Point your agent at any site and get back the pages most relevant to your topic — or any single page as clean Markdown.
Connect once — your agent can crawl a site by topic, compare the frontier strategies, read any page as clean Markdown, pull structured data, list a page's links, and call hidden APIs.
focused_crawlCrawl a whole site and get its pages ranked by relevance to a query.
compare_strategiesRun Shark-Search, OPIC and BFS on one budget and see which finds the topic.
fetch_markdownAny page as clean Markdown — a fraction of the tokens of raw HTML.
extractStructured records out of a page with CSS selectors.
list_linksEvery outbound link, absolute and deduplicated — the pages you could visit next.
fetch_json_apiHit the JSON API hiding behind a site and skip the HTML.
Every site is a graph: pages are nodes, links are edges. With a limited budget, a crawler decides one thing — which node to visit next. Same loop in all three; only the ordering changes.
Level by level from the seed. Uniform coverage, blind to your topic — the fair baseline.
score = −depth
Best-first by topic: links inherit their parent's relevance with decay, so barren branches die off on their own.
score = γ·inherited + (1−γ)·local
Every page holds "cash" it splits among its links when visited — a live PageRank, no full graph needed.
cash(page) → split across links
Connect it to your agent, curl the API, or import the library — then explore every strategy live.