Scrape

Scrape fetches a single URL and returns the formats you ask for. With the default tier: "auto" it tries a plain fetch, runs a quality check, and only escalates to a headless browser when the page actually needs one. You pay for the tier that worked: 1 credit (rendering and screenshots included), +1 when you use browser actions.

Example#

curl -X POST "https://api.vermin.dev/v1/scrape" \
  -H "Authorization: Bearer $VERMIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/blog/post",
    "formats": ["markdown", "chunks", "screenshot"],
    "chunkSize": 800
  }'

Request#

Formats#

Actions#

Need to click a cookie banner, scroll a feed or type into a search box first? Pass up to 20 actions; they run in order in a real browser before the page is captured.

{
  "url": "https://example.com/feed",
  "actions": [
    { "type": "click", "selector": "#accept-cookies" },
    { "type": "scroll" },
    { "type": "wait", "ms": 1500 }
  ]
}

Screenshot size and element capture#

Screenshots use a 1280 × 800 desktop window by default. Set screenshotOptions to choose the window, the pixel density and whether to emulate a phone, or to capture just one element. Page actions run in the same window first, so you can open a menu or switch a tab before the capture.

{
  "url": "https://vermin.dev",
  "formats": ["screenshot"],
  "screenshotOptions": {
    "viewport": { "width": 390, "height": 844, "deviceScaleFactor": 3, "mobile": true },
    "selector": "header"
  }
}

screenshotOptions needs the screenshot or screenshotFull format. If the capture fails, you still get the page and its other formats, without a screenshot, and quality.escalations says why: screenshot:element_not_found, screenshot:invalid_selector, screenshot:element_not_visible or screenshot:timeout.

Site markdown#

Many docs sites (Next.js, Docker, GitHub Docs, Microsoft Learn, Cloudflare and others) publish their own markdown for each page. Set siteMarkdown: true and we ask for it (Accept: text/markdown) and return it as-is when it passes the quality check, with method: "markdown". It's about twice as fast as extracting the HTML, and sites that don't have it still send HTML in the same request, so it never costs extra credits.

It's off by default because the site's version usually carries less metadata: title and description come from its front matter or first heading, and the language and canonical URL are often missing. On our benchmark its content also scored slightly below our own extraction. It's ignored with the html, rawHtml and links formats, which need the page itself.

Caching#

Public pages are cached for two days by default and a cache hit costs 1 credit, served from the nearest data center in milliseconds (cached: true). Set maxAge (ms) to decide how fresh is fresh enough, or maxAge: 0 to always scrape live. Requests with custom headers, actions or screenshot formats are never cached or served from cache, and neither are pages the site marks no-store or private.

Response#

{
  "success": true,
  "data": {
    "url": "https://example.com/blog/post",
    "finalUrl": "https://example.com/blog/post",
    "markdown": "# Shipping at the edge\n\n...",
    "chunks": [
      { "index": 0, "text": "# Shipping at the edge ...", "headingPath": ["Shipping at the edge"], "tokens": 781 }
    ],
    "screenshot": "https://api.vermin.dev/artifacts/screenshots/7f3c….png?sig=…",
    "metadata": { "title": "Shipping at the edge", "language": "en", "statusCode": 200 },
    "tier": "render",
    "cached": false,
    "quality": {
      "score": 0.93,
      "issues": ["js_required"],
      "escalations": ["fetch→render: js_required"],
      "reason": "The fetched HTML was an empty app shell; the rendered page had the article."
    }
  },
  "credits_used": 1,
  "latency_ms": 1240,
  "request_id": "req_01J9V3P0ZC"
}

The quality report#

quality is Vermin's answer to "why is my markdown empty?":

• score: 0–1 confidence that the content is complete and is the page's real main content.
• issues: what we noticed. thin_content, js_required, bot_wall, cookie_wall, login_wall, soft_404, truncated, non_html.
• escalations: what the pipeline did about it, e.g. fetch→render: js_required.
• reason: the same story in words.

robots.txt#

Scrape checks the site's robots.txt (for verminbot, then *) before fetching, and refuses disallowed URLs with blocked_url at 0 credits. The verdict is cached per origin for an hour. If robots.txt can't be reached in time, the scrape goes ahead; a 5xx or 429 from robots.txt counts as "disallow everything". Pass ignoreRobots: true only when you have permission to fetch the page.

Errors#

Common ones: blocked_url (private/local address or robots.txt), blocked_by_site, timeout, capacity_exceeded, and not_configured if you ask for a tier this deployment doesn't have. All free. See Errors.