v1.2.1

ByteCrawl docs

Focused crawling for AI agents: point it at a site and get back the pages most relevant to a topic (Shark-Search, OPIC), plus everyday scraping — static HTML, dynamic JS, hidden APIs, sessions and LLM-ready Markdown.

Four ways in, listed below in order of how much you have to install. The first three — agent skill, MCP and the HTTP API — need nothing on your machine and no key. Install the Python library when you want no caps, JS rendering, or ByteCrawl in your own code.

Agent skill

If you are an AI agent — or you are pointing one here — skip these docs and read the skill instead. It is a single Markdown file that picks the right path for the job (hosted API, MCP, Python library, or browser) and gives working calls for each.

Read and follow https://bytecrawl.vercel.app/agent-onboarding/SKILL.md

Paste that line into Claude Code, Cursor, or any agent — or read the skill yourself. No install and no key on any path. It also documents how to pick a crawl strategy and how to tune Shark-Search — see Focused crawlers below.

That line is for using it now — the agent reads the file, does what you asked, and installs nothing. To have it permanently, ask for that instead:

Set up bytecrawl for me: add the MCP server and install the skill.
The steps are at https://bytecrawl.vercel.app/agent-onboarding/SKILL.md

The difference is not the wording, it is who is giving the instruction. An agent will not modify your machine because a web page told it to — nor should it — but it will when you ask.

Install it once — then `/bytecrawl`

Pasting the line above works for one turn. Installed as a skill it is there in every session, and the agent reaches for it on its own:

mkdir -p ~/.claude/skills/bytecrawl && \
  curl -so ~/.claude/skills/bytecrawl/SKILL.md \
  https://bytecrawl.vercel.app/agent-onboarding/SKILL.md

Same file this page serves, so there is nothing to keep in sync. Run it bare and it asks what you want; give it a task and it just does it.

MCP server

Expose ByteCrawl to any MCP-capable agent (Claude Code, Claude Desktop, Cursor). The agent gets six tools — the same six the playground offers: focused_crawl, compare_strategies, fetch_markdown, extract, list_links and fetch_json_api.

Hosted — nothing to install

claude mcp add --scope user --transport http bytecrawl https://bytecrawl.vercel.app/mcp

Same six tools, no Python on your machine. Static HTML only, rate-limited per IP, crawls capped at 10 pages and strategy comparisons at 6 per strategy, and it refuses non-public URLs (SSRF guard). For heavy use or JS-rendered sites, run it locally.

Local — only if you need JS pages or bigger crawls

# install and register with Claude Code
pipx install bytecrawl[mcp]
claude mcp add --scope user bytecrawl -- bytecrawl-mcp

Or in any MCP client config:

{"mcpServers": {"bytecrawl": {"url": "https://bytecrawl.vercel.app/mcp"}}}

This one costs an install, so reach for it only when the hosted server cannot do the job. pipx rather than pip because it is a command-line app, and a system Python refuses a plain pip install with externally-managed-environment. Needs Python 3.10+; the hosted one does not, being just HTTP.

Then just ask

You don't call the tools yourself. With the server connected you ask in plain language and the agent picks one:

Use bytecrawl to find everything on python.org about the packaging ecosystem

Read https://example.com/pricing with bytecrawl and give me the plans as a table

Which crawl strategy does best on wikipedia.org for "san francisco"? Compare them

Saying the name is worth the two syllables: most agents ship their own single-page fetcher and will reach for that by default. Naming bytecrawl is what gets you a focused crawl instead of one page read in isolation.

HTTP API

The simplest way to try ByteCrawl — one hosted endpoint. Everything but url is optional.

# a page → clean Markdown
curl "https://bytecrawl.vercel.app/api?url=quotes.toscrape.com"

# focused crawl by topic
curl "https://bytecrawl.vercel.app/api?url=en.wikipedia.org/wiki/Silicon_Valley&method=crawl&query=san+francisco"

# scrape a CSS selector
curl "https://bytecrawl.vercel.app/api?url=books.toscrape.com&method=extract&select=h3+a::attr(title)"

Param	Description
`url`	Page or site to fetch (required).
`method`	`markdown` (default) · `text` · `html` · `links` · `json` · `extract` · `crawl`
`query`	Topic to rank pages by (with `method=crawl`).
`select`	A CSS selector to scrape (with `method=extract`).
`strategy`	`shark` (default) · `opic` · `bfs`

Static-only and SSRF-guarded; crawls are capped at 10 pages. For JS rendering or heavier use, run the library or MCP server locally.

Install the library

Everything above runs without installing anything. Install when you want no page caps, JS rendering, or ByteCrawl inside your own Python code.

# slim core (requests + beautifulsoup4 + lxml)
pip install bytecrawl

# optional extras
pip install bytecrawl[llm]       # Markdown for LLMs
pip install bytecrawl[browser]   # Playwright (then: playwright install chromium)
pip install bytecrawl[mcp]       # local MCP server

No extra fails silently: if one is missing, the ImportError names the exact command to run.

Quickstart

Create a Scraper, fetch a page, extract typed records.

from bytecrawl import Scraper

bot = Scraper()

page = bot.static("https://books.toscrape.com")
books = page.extract("article.product_pod", {
    "title": "h3 a::attr(title)",
    "price": "p.price_color::text",
})
print(books[0])  # {'title': 'A Light in the Attic', 'price': '£51.77'}

Scraper

The entry point. Pick a strategy explicitly, or let fetch() decide.

Method	Returns	What it does
`Scraper(user_agent, delay, timeout)`	Scraper	Create a client. `delay` adds a pause between requests (rate-limit).
`.static(url)`	Page	Strategy 1 · plain HTML via requests + BeautifulSoup.
`.browser(url, wait, scroll)`	Page	Strategy 2 · real Chromium for JS-rendered pages. `wait` = CSS selector to await.
`.api(url, params)`	Page	Strategy 3 · GET a JSON API; result in `page.json()`.
`.fetch(url, strategy)`	Page	Auto: tries static, falls back to browser if the page looks empty (JS).
`.crawl(start, item, fields, next_page, pages)`	list[dict]	Strategy 4 · follow the `next_page` link across pages, extract records.
`.session()`	Session	Strategy 5 · an authenticated, cookie-persistent session.

crawl example

items = bot.crawl(
    "https://quotes.toscrape.com/",
    item="div.quote",
    fields={"frase": "span.text::text", "tags[]": "a.tag::text"},
    next_page="li.next a::attr(href)",
    pages=3,
)

Page

What every fetch returns. Holds the HTML (or JSON), timing, and extraction helpers.

Member	Returns	What it does
`.css(sel)`	str \| None	First match of a selector.
`.css_all(sel)`	list[str]	All matches.
`.extract(item, fields)`	list[dict]	A list of records. `fields` maps name → selector.
`.links(raw=False)`	list[str]	Every link as an absolute URL, deduplicated, without `#fragments`, `mailto:` or asset files. `raw=True` returns the untouched `href` values.
`.json()`	Any	The parsed JSON (when fetched via `.api()`).
`.markdown(main_only=True)`	str	Clean Markdown for LLMs. `main_only` keeps just the main content.
`.tokens(of=None)`	int	Rough token estimate (~4 chars/token).
`.elapsed` · `.method` · `.status`	—	Timing in seconds, strategy used, HTTP status.

page = bot.static(url)
md = page.markdown()
print(page.tokens(), "->", page.tokens(of=md))  # 2755 -> 423

Session

An authenticated session. Cookies persist across requests; chainable.

Method	What it does
`.login(url, data, csrf_field=None)`	POST credentials. If `csrf_field` is set, reads the hidden token from the form first.
`.bearer(token)`	Set an `Authorization: Bearer` header for token APIs.
`.fetch(url)`	GET a protected page using the live session.

s = bot.session().login(
    "https://quotes.toscrape.com/login",
    data={"username": "user", "password": "pass"},
    csrf_field="csrf_token",
)
home = s.fetch("https://quotes.toscrape.com")

Focused crawlers

Three strategies share one loop — pop a URL, fetch it, score its links, push them back — so the same page budget across strategies is a fair comparison. They differ only in how the frontier is ordered.

Class	Orders the frontier by	Use when
`SharkSearch(query, ...)`	Topic similarity, inherited by child links with decay.	You know the topic. Best default.
`OPIC(query="", ...)`	Importance — "cash" flowing along links.	You want the hubs, with no topic in mind.
`BFS(query="", ...)`	Distance from the seed.	You want even coverage, or a baseline.

from bytecrawl import BFS, OPIC, SharkSearch, pagerank

r = SharkSearch(query="san francisco", delay=0.5).crawl(url, max_pages=20)
r.top(10)             # highest-relevance pages
r.relevant(0.1)       # everything above a threshold
r.stats               # requests, errors, elapsed
pagerank(r.graph)     # offline importance, to compare against OPIC

Picking one. Every surface takes the same three names — shark (default), opic, bfs. On the HTTP API: &method=crawl&query=TOPIC&strategy=opic. On MCP: focused_crawl(url, query, strategy="opic"). In Python, instantiate the class. An unknown name is rejected, not silently defaulted — the API returns 400 and the MCP tool raises ValueError.

SharkSearch requires query: without a topic it has nothing to rank. BFS and OPIC accept it too, but only to score the pages in the result — it does not change their traversal order.

What the choice buys you. From en.wikipedia.org/wiki/Silicon_Valley, query san francisco, 20 pages each: Shark returns 20 pages above 0.1 relevance against 4 for OPIC and 1 for BFS, and its best page scores 0.7774 against BFS's 0.1068 — 7.3x. Reproduce it with result.relevant(0.1). A lower threshold will not close the gap: 20 pages leaves thousands of URLs still queued, so coverage never converges the way it does on a small site.

Tuning Shark-Search. Two parameters from the paper, both exposed: delta (0.5) is how fast a branch with no signal fades — the decay is δⁿ with depth, so lower abandons dead branches sooner. gamma (0.8) is how much a link's score comes from its parent versus its own anchor text: score = γ·inherited + (1−γ)·local. Lower it when anchor text on the target site is descriptive.

Selector syntax

Selectors are plain CSS plus two pseudo-suffixes, and a list marker on field names.

Form	Result
`"p.price"` or `"p.price::text"`	Text content of the element.
`"a::attr(href)"`	The value of an attribute.
`"tags[]": "a.tag::text"`	Field name ending in `[]` collects a list of values.

Choosing a strategy

Go from cheapest to heaviest. The right order saves time and avoids bans:

API? Check DevTools → Network for a JSON endpoint. Cleanest and fastest → .api()
Static HTML? If "View source" already shows the data → .static()
JS-only? Empty source but data on screen → .browser()
Many pages, known pagination? → .crawl() with next_page=
A whole site, ranked by topic? → a focused crawler
Behind login? → .session().login()

Try it live in the Methods playground.