Engines

fetch

A plain HTTP request through the proxy. It is the whole answer for a server-rendered page, an API response, an RSS feed or a documentation site. That is most of the web, still.

Send a form or JSON

{
  "url": "https://example.com/search",
  "method": "POST",
  "headers": { "content-type": "application/x-www-form-urlencoded" },
  "body": "q=paving"
}

Some search pages only answer a POST. Set method to "POST", put what the form sends in body, and give its type in headers: the form type above, or application/json for JSON. Like Chrome, it sends an Origin header with the site's own address. If the site's form sits on another page, put that page in a referer header and the Origin follows it. You get the results page back, read like any other.

A POST is sent once. Sending it again could repeat what it did, so it is never retried from another address or tried another way, and it always uses the fetch engine. The body can be up to 64 KB. If the site redirects, a 301, 302 or 303 opens the next page with GET and no body. A 307 or 308 sends the body again only if the protocol, host and port stay the same. Anywhere else, even http to https or www to the bare domain, it stops with UPSTREAM_BLOCKED and names the new address, so you can send the POST there yourself.

Download the original file

import { writeFileSync } from "node:fs";
import { Scrapa } from "@scrapa/sdk";

const scrapa = new Scrapa(process.env.SCRAPA_API_KEY!);
const { raw } = await scrapa.scrape({ url: "https://example.com/report.pdf", formats: ["raw"] });
if (raw) writeFileSync("report.pdf", Buffer.from(raw.base64, "base64"));

Ask for raw and you get the file exactly as the site sent it: raw.base64 holds the bytes and raw.contentType says what they are. One more line saves it to disk.

Files up to 5 MB come back whole. A larger one answers FILE_TOO_LARGE at no charge, never half a file. It costs what a fetch costs: 1 credit, plus bandwidth past the first megabyte. The pdf format is a different thing: the browser printing a page.

Read a PDF or a Word file

import { Scrapa } from "@scrapa/sdk";

const scrapa = new Scrapa(process.env.SCRAPA_API_KEY!);
const page = await scrapa.scrape({ url: "https://example.com/report.pdf" });
console.log(page.markdown); // "# Annual report\n\n## Results\n\nRevenue grew in every region."
console.log(page.metadata.pages); // 12

Point url at a PDF or a Word (.docx) file and it comes back as text, like any page. Big text becomes a heading and the gaps between lines split the paragraphs, so markdown, text, links and extract all work on it. metadata.pages says how many pages the PDF has.

We read files up to 25 MB, and from a PDF the first 200 pages and the first 5 million characters. A longer PDF sets metadata.truncated to true. It costs what the engine that read it costs: 1 credit with fetch, 5 with the browser, plus bandwidth past the first megabyte. We read one file for 15 seconds at most, and stop sooner if timeoutMs runs out. Then the call answers DOCUMENT_TIMEOUT at no charge. A Word file that holds more than 32 MB once unzipped answers FILE_TOO_LARGE, and a file link that answers a short web page instead of the file (often a check that you are human) answers UPSTREAM_BLOCKED. Neither costs anything. A dead link is the exception: its 404 or 410 page is the site's answer, so it comes back with that status and is charged like any page. A scanned PDF holds pictures of text, not text, so it comes back with little or none.

browser

A real Chrome (Google's own build, not a headless shell) loads the page, runs its scripts and waits for the network to settle. It is what a single-page app, a lazy-loaded list or a cookie wall needs, and it is the only engine that can produce a screenshot or a PDF.

{
  "url": "https://app.example.com/dashboard",
  "engine": "browser",
  "formats": ["markdown", "screenshot"]
}

auto, the default

Fetch first. If the body comes back a JavaScript shell (no readable text, a mount point, a plea to enable scripts), the same URL is re-read in the browser and you are charged the browser row. If the fetch answered properly, you paid for a fetch.

You are never charged for both engines. A page that answered on the first rung is billed once; an escalation that fails hands the credits back and the fetch's page stands, with metadata.needsJs saying why it looked thin.

Screenshots and PDFs

Both come back base64-encoded in the answer, and both force the browser engine whatever engine says. Asking for a screenshot and a PDF together costs no more than asking for one. It is the same page load.

Telling the browser what to wait for

{
  "url": "https://app.example.com/prices",
  "engine": "browser",
  "browser": {
    "waitForSelector": ".price",
    "viewport": { "width": 390, "height": 844 }
  }
}

A single-page app draws its content after the page loads. browser.waitForSelector waits for the element you actually need, which beats guessing at a delay. If it never shows up, you get NAV_TIMEOUT at no charge, not the half-built page. browser.waitMs is the guess, for the scripts no selector describes. Both come out of the timeout rather than adding to it.

browser.viewport sizes the window the page renders in. Set it when a site serves a different layout by width. browser.fullPage screenshots the whole scrollable page instead of just the window.

Click and type before the page is read

{
  "url": "https://news.example.com/article/42",
  "browser": {
    "actions": [
      { "type": "click", "selector": "#accept-cookies", "all": true },
      { "type": "scrape" }
    ]
  }
}

Some pages hide what you want behind a cookie banner, a button or a search box. browser.actions is a list of steps a real Chrome runs one after another, in the order you list them, before the page is read. A step can wait, click, type, press a key, scroll, take a screenshot, keep the HTML or run your own script. Here the first step accepts the cookies and the second keeps the page's HTML, which comes back in actions.scrapes. all: true clicks every match and is not an error when there is none, because many sites show the banner only to some visitors. The formats you ask for read the page as the last step left it.

Steps always use the browser engine, at the browser price of 5 credits, with nothing added per step. You can send up to 50 steps, and all the waits together have to fit inside timeoutMs. If a step does not work, like a click on a selector that matches nothing, the call answers ACTION_FAILED, names the step and why, and costs nothing. A script is the exception. Once one starts, the call is charged even if that script throws, a later step fails, the page crashes or times out, the site blocks it, the script leads to an address we refuse, your own proxy says no, or one of your includeTags or excludeTags selectors is too slow for the page it built, because the script may already have read the page. If you cancel a call with a script after we have sent it to the browser, you still pay the browser price, because we cannot tell whether the script already ran. If the failure is ours (a 5xx), you still get the credits back.

When a site says no

{ "url": "https://shop.example.com/item/42", "retry": 2 }

A blocked page costs nothing unless your own script ran in it first (see above), so both ways to answer one are free to try. retry asks again from a different address, up to twice. The exit rotates every try, which is what gets past a block on the address rather than on you.

{ "url": "https://shop.example.com/item/42", "engine": "fetch", "escalate": false }

escalate: false does the opposite: one try, then the site's verdict. Reach for it when a run has to stay cheap and predictable, because with auto escalating is what charges the browser row instead of the fetch one. It names its engine, since auto is the escalation.

How long a page gets

{ "url": "https://slow.example.com", "timeoutMs": 45000 }

The default budget is generous enough for a slow render and bounded so nothing hangs. Set timeoutMs per call when you know better. A page that never arrives answers NAV_TIMEOUT, which costs nothing, unless your own browser.actions script ran in the page first.

Read the same page again for free

{ "url": "https://shop.example.com/item/42", "maxAge": 600000 }

Add maxAge and a page your account read recently can come back from our copy instead of from the site. It is the oldest copy you accept, in milliseconds, up to 3600000 (one hour); the example takes one up to 10 minutes old. A copy costs 0 credits, and the answer says cached: true and, in cachedAt, when the page was read. With no copy that recent, we read the page now, charge the usual price and keep that copy for up to an hour. We may drop a copy sooner, to make room or on a deploy, and then the next call reads the page again at the usual price.

A copy is yours alone: no other account ever gets it. It has to match the URL, the country and the engine. A batch keeps its own copies, apart from single calls. auto takes either kind of copy, fetch only a plain one and browser only a rendered one. extract still costs 5 credits, because the model reads the copy again. A POST, or a call with headers, your own proxy, browser options, a screenshot, a PDF, raw or escalate: false, always reads the page now. A blocked page is never kept, so the next call asks the site again, and neither is a page that still needs JavaScript (metadata.needsJs).