Webhooks & streaming

Crawls can run for minutes. Rather than polling, let Vermin tell you what's happening: either stream events over SSE while you're connected, or receive webhooks at your own endpoint.

SSE is covered on the Crawl page. The rest of this page is about webhooks.

Subscribing#

Two ways, and they combine:

• Per crawl: add webhook to POST /v1/crawl. Signed with your organisation's webhook signing secret (Dashboard → Webhooks).
• Dashboard endpoints: register a URL in Dashboard → Webhooks to receive events for every crawl (and usage alerts), each signed with that endpoint's own whsec_… secret.

{
  "url": "https://docs.example.com",
  "limit": 500,
  "webhook": {
    "url": "https://you.dev/hooks/vermin",
    "events": ["page", "completed", "failed"],
    "metadata": { "tenant": "acme" }
  }
}

events defaults to all of them: started, page, completed, failed, cancelled. metadata is echoed back in every delivery. A crawl cancelled (or failed) before it starts still sends crawl.cancelled (or crawl.failed), without a crawl.started first.

Payload#

{
  "id": "evt_01J9V9F6TV",
  "type": "crawl.page",
  "createdAt": "2026-09-28T09:41:12Z",
  "data": {
    "id": "crawl_01J9V4A7",
    "url": "https://docs.example.com/guides/intro",
    "document": { "url": "https://docs.example.com/guides/intro", "markdown": "# Intro\n…", "tier": "fetch" }
  },
  "metadata": { "tenant": "acme" }
}

For crawl.started, crawl.completed, crawl.failed and crawl.cancelled, data is the crawl status (status, total, completed, failed, credits_used, and error when it failed). Headers: Vermin-Signature, Vermin-Event (e.g. crawl.page) and Vermin-Delivery-Attempt.

Verifying signatures#

Every delivery carries:

Vermin-Signature: t=1790000000,v1=e19cbcc55e778366b144a95f57151266a21ffbf9da41a16e8654d511fe1afbec

v1 is the lowercase hex HMAC-SHA256 of "<t>.<raw request body>", keyed with your signing secret. To verify:

1. Read the raw body bytes. Don't parse and re-serialise JSON first: whitespace matters.
2. Split the header on ,; take t and every v1 value (during secret rotation there can be several).
3. Compute HMAC-SHA256(secret, t + "." + rawBody) as hex and compare it to each v1 in constant time. Accept if any match.
4. Reject if |now - t| > 300 seconds, to stop replays.

import { createHmac, timingSafeEqual } from "node:crypto";

export function verifyVermin(rawBody: string, header: string, secret: string, toleranceS = 300): boolean {
  const parts = header.split(",").map((p) => p.trim().split("="));
  const t = parts.find(([k]) => k === "t")?.[1];
  const sigs = parts.filter(([k]) => k === "v1").map(([, v]) => v);
  if (!t || !sigs.length || Math.abs(Date.now() / 1000 - Number(t)) > toleranceS) return false;
  const expected = Buffer.from(createHmac("sha256", secret).update(`${t}.${rawBody}`).digest("hex"));
  return sigs.some((s) => s.length === expected.length && timingSafeEqual(Buffer.from(s), expected));
}

The SDKs are gaining built-in verifyWebhook helpers (PHP's Webhook::verify is shown above); shared test vectors live alongside the spec so every implementation agrees byte-for-byte.

Delivery and retries#

• Respond with any 2xx within 10 seconds. Do heavy work asynchronously: enqueue, then acknowledge.
• Anything else, or a timeout, is retried with exponential backoff: 10s, 20s, 40s… up to 1 hour between attempts, 8 attempts in total.
• Respond 410 Gone to stop retries for that delivery.
• Events that never get through are kept as dead letters in Dashboard → Webhooks, where you can inspect them.
• Deliveries can arrive out of order or more than once. De-duplicate on the event id (identical across retries) and order on createdAt.