# Vermin

> Vermin is a web scraping API for AI apps. It turns any URL or website into clean, LLM-ready data: markdown, token-counted chunks and metadata, plus brand data (logo, colours, fonts). Failed pages cost 0 credits.

Base URL: https://api.vermin.dev · Auth: `Authorization: Bearer $VERMIN_API_KEY` · OpenAPI: https://vermin.dev/openapi.json

## When to use Vermin
Reach for Vermin when an agent or app needs the *content* of public web pages, not a browser session:
- Read a web page as clean markdown for an LLM (docs, articles, product pages, JS-heavy SPAs): `POST /v1/scrape` or the `vermin_scrape` MCP tool.
- List every URL on a site, optionally ranked by a query, before deciding what to read: `POST /v1/map` / `vermin_map`.
- Ingest a whole site or section (RAG, knowledge bases, chatbot training) with streaming results: `POST /v1/crawl` / `vermin_crawl` + `vermin_crawl_status`.
- Get a company's logo, icon, colours, fonts and socials from its domain: `GET /v1/brand` / `vermin_brand`.
- Chunk pages for a vector store: request `formats: ["chunks"]` (each chunk has its heading path and token count).

Don't use Vermin for: logged-in pages or anything behind a user's credentials, filling forms or multi-step browsing sessions, sites whose robots.txt disallows you (Vermin respects robots.txt), or non-public/internal addresses (refused as `blocked_url`).

How to call it well:
- Prefer the cheapest call that answers the question: map or scrape before crawl.
- Don't pick a tier. Vermin escalates from plain HTTP to a headless browser by itself.
- Branch on `error.code` (`blocked_by_site`, `timeout`, `rate_limited`, …), never on the message. Failed calls cost 0 credits.
- On `429 rate_limited`, wait `Retry-After` seconds. Every authenticated response carries `RateLimit-Policy` and `RateLimit` headers.
- Send an `Idempotency-Key` header on POSTs you might retry.

## Get access
- Free plan, no card: 1,000 pages a month. Sign up at https://vermin.dev/signup and create a key in the dashboard (self-serve, instant).
- Test safely: the dashboard playground (https://vermin.dev/dashboard/playground) runs real requests on your free credits, failed requests are free, and every response reports `credits_used`.

## Endpoints
- POST /v1/scrape: one URL to markdown, HTML, links, screenshot, chunks, metadata or brand. Escalates to a headless browser only when the page needs it. 1 credit per page, browser included.
- POST /v1/crawl: a whole site, async. Sitemap-first, include/exclude globs, SSE stream or signed webhooks. 1 credit per page.
- POST /v1/map: every URL on a site, ranked by a query. 3 credits per 10 fetches.
- GET /v1/brand: logo, icon, colours, name, fonts and socials for a domain. 5 credits, 1 when cached.

## Agents
- [Agent setup](https://vermin.dev/agent.md): how an agent connects to Vermin
- MCP (hosted, Streamable HTTP): `claude mcp add --transport http vermin https://mcp.vermin.dev/mcp`
- MCP (local, stdio): `npx -y @vermin/mcp`
- [MCP server card](https://mcp.vermin.dev/.well-known/mcp/server-card.json)
- Every page on vermin.dev is available as Markdown: send `Accept: text/markdown`, or append `.md` (e.g. https://vermin.dev/docs/scrape.md).

## Docs
- [Quickstart](https://vermin.dev/docs/quickstart)
- [API reference](https://vermin.dev/docs/reference) · [OpenAPI 3.1 spec](https://vermin.dev/openapi.json)
- [Authentication](https://vermin.dev/docs/authentication)
- [SDKs](https://vermin.dev/docs/sdks): TypeScript (`npm i @vermin/sdk`), Python (`pip install vermin-sdk`)
- [MCP server](https://vermin.dev/docs/mcp)
- [Credits](https://vermin.dev/docs/credits)
- [Errors](https://vermin.dev/docs/errors)
- [Rate limits](https://vermin.dev/docs/rate-limits)
- [Pricing](https://vermin.dev/pricing)

## Company
- [About](https://vermin.dev/about) · [Contact](https://vermin.dev/contact) · [Privacy](https://vermin.dev/privacy) · [Terms](https://vermin.dev/terms)
- [Sitemap](https://vermin.dev/sitemap.xml)