Batches

Hand us a list of URLs and stop holding a connection open. Every one is read on the same terms as a single scrape, and you either poll or we call you.

Submit the list

POST /v1/batches

curl -X POST https://api.scrapa.app/v1/batches \
  -H "authorization: Bearer sc_live_your_key" \
  -H "content-type: application/json" \
  -d '{
    "urls": ["https://example.com/a", "https://example.com/b"],
    "formats": ["markdown"]
  }'

The answer

{
  "data": {
    "id": "b7c1e0a4-5f2d-4c8b-9e13-2a6d4f8c1b90",
    "status": "queued",
    "total": 2,
    "completed": 0,
    "failed": 0,
    "skipped": 0,
    "pool": "residential",
    "cost": 0,
    "expiresAt": "2026-09-07T09:12:00.000Z"
  }
}

HTTP/1.1 202 Accepted
x-request-cost: 0

You get an id back, not pages. That is what 202 means here. Every URL is checked before anything is stored, so a bad one in a list of five thousand is a 400 you can fix now rather than a surprise forty minutes in.

Read it back

GET /v1/batches/{id}

curl "https://api.scrapa.app/v1/batches/b7c1e0a4-5f2d-4c8b-9e13-2a6d4f8c1b90" \
  -H "authorization: Bearer sc_live_your_key"

# …and once it has finished, the pages themselves:
curl "https://api.scrapa.app/v1/batches/b7c1e0a4…/?items=true&from=0" \
  -H "authorization: Bearer sc_live_your_key"

Polling a batch is free and tells you where it got to. Ask for items when you actually want the pages: they come back up to two hundred at a time, and since positions are dense you page with from plus however many you just got.

completed counts every URL that has reached an end, so it is what you compare against total. failed and skipped are both inside it, and they are not the same thing.

Or let us call you

webhook

{
  "urls": ["https://example.com/a", "https://example.com/b"],
  "webhook": {
    "url": "https://your-app.example.com/hooks/scrapa",
    "events": ["batch.completed"]
  }
}

Pass a webhook and we POST when the batch is done. The signing secret comes back once, on the create. We mint it rather than take one, so you never put a credential on the wire to us, and losing it means creating another batch. Ask for batch.item.completed as well when a long batch should be usable before it ends.

Verifying a delivery

import { createHmac, timingSafeEqual } from "node:crypto";

// A delivery signed longer ago than this is refused, so nobody who
// copies one on its way to you can send it to you again later.
const MAX_AGE_SECONDS = 300;

export function verify(secret: string, rawBody: string, header: string): boolean {
  const t = /(?:^|,)t=(\d+)(?:,|$)/.exec(header)?.[1];
  const v1 = /(?:^|,)v1=([0-9a-f]+)(?:,|$)/.exec(header)?.[1];
  if (!t || !v1) return false;
  if (Math.abs(Date.now() / 1000 - Number(t)) > MAX_AGE_SECONDS) return false;
  const expected = createHmac("sha256", secret).update(`${t}.${rawBody}`).digest();
  const given = Buffer.from(v1, "hex");
  return given.length === expected.length && timingSafeEqual(given, expected);
}

The scrapa-signature header is t=<unix>,v1=<hmac>: HMAC-SHA256 over the timestamp, a dot, and the raw body exactly as it arrived. Compare in constant time, and refuse a delivery signed more than five minutes ago, as the sample does, so nobody can send you an old one again.

We deliver to ports 80 and 443 only, and we do not follow redirects. A redirect is a second host nobody checked.

What it costs

Nothing when you submit, and nothing is reserved. Each URL then costs exactly what the same call would have cost on its own, and the pages the web refused cost nothing here too, unless your own browser.actions script ran in them first. cost on the batch is what it has spent so far, and it grows while it runs. Quote it first if you want the ceiling.

Because nothing is held up front, a batch can outlive the credits that started it. The URLs we never reached are skipped, not failed: that is not a verdict about the URL, and you can submit exactly those positions again. The same happens to a batch still running 24 hours after you submitted it: its results can no longer be read, so we stop instead of charging for pages nobody will see.

In TypeScript

Submit, wait, read

import { Scrapa } from "@scrapa/sdk";

const scrapa = new Scrapa(process.env.SCRAPA_API_KEY!);

const batch = await scrapa.batches.create({
  urls: pages.map((p) => p.url),
  formats: ["markdown"],
});

// Poll until every URL has a verdict, then read them in pages.
let status = await scrapa.batches.get({ id: batch.id });
while (status.completed < status.total) {
  await new Promise((r) => setTimeout(r, 5_000));
  status = await scrapa.batches.get({ id: batch.id });
}

for (let from = 0; from < status.total; ) {
  const page = await scrapa.batches.get({ id: batch.id, items: true, from });
  for (const item of page.items ?? []) {
    if (item.status === "done") save(item.url, item.result!.markdown);
    else if (item.status === "failed") retryLater(item.url, item.error!.code);
  }
  from += page.items?.length ?? status.total;
}

Nothing here is batch-specific except the waiting: an item's result is the same object scrape() hands back, and an item's error is the same code, so whatever you already wrote for one page works for five thousand.