Map

One call

curl -X POST https://api.scrapa.app/v1/map \
  -H "authorization: Bearer sc_live_your_key" \
  -H "content-type: application/json" \
  -d '{"url": "https://example.com"}'

{
  "data": {
    "urls": [
      { "url": "https://example.com", "lastmod": "2026-10-01" },
      { "url": "https://example.com/pricing", "lastmod": "2026-09-12" },
      { "url": "https://example.com/blog/hello" }
    ],
    "source": "both",
    "truncated": false,
    "pool": "residential",
    "bytes": 48211
  },
  "meta": { "cost": 2, "creditsRemaining": 985, "requestId": "0f3c…" }
}

Send the site's address. You get back its URLs, each one once, with the date the sitemap says the page last changed when it gives one.

Where the URLs come from

First the site's robots.txt, which names its sitemaps. Then the three places sites usually keep one: /sitemap.xml, /sitemap_index.xml and /sitemap.txt. A sitemap that lists other sitemaps is followed, and a gzipped one is unzipped. At the same time we read the page you named and keep its links.

sitemap picks the sources. include, the default, reads both. only reads the sitemaps alone. skip reads only the page's links, for a site with no sitemap.

Only the URLs you want

{
  "url": "https://example.com",
  "search": "blog",
  "includeSubdomains": true,
  "limit": 500
}

Only the host you named comes back, with or without www. Set includeSubdomains to keep blog.example.com as well. search keeps the URLs that contain a word, upper or lower case alike. limit caps the list: 5,000 unless you say, 30,000 at most.

One spelling per page

Sites list the same page several ways. We drop the part after the #, the utm_ tracking parameters and the trailing slash, and sort what is left of the query, so each page comes back once.

truncated: true means the site may list more than you got. Either limit cut the list, or we could not read the robots.txt or a sitemap to the end. That happens when the site refused it, it was too big, it came after the first 25 sitemaps, the map ran out of time, or our own proxy failed. When the fault is ours, you get the URLs found so far and the map costs nothing.

From a URL to a page

import { Scrapa } from "@scrapa/sdk";

const scrapa = new Scrapa(process.env.SCRAPA_API_KEY!);

const { urls } = await scrapa.map({ url: "https://example.com", search: "blog" });
const pages = await Promise.all(
  urls.slice(0, 3).map(({ url }) => scrapa.scrape({ url })),
);

A map hands you addresses, not content. Read the pages you need with scrape, or send the whole list to a batch.

What it costs

2 credits per map, whatever it finds. Bandwidth through the managed pool is metered past the first megabyte, with every file the map read counted together.

A site that lists nothing is charged, because it answered. A site that refused every file costs nothing, and so does anything we broke. To see the price without making the call, use POST /v1/map/quote.

The shell spelling

curl "https://api.scrapa.app/v1/map?url=https://example.com&search=blog&sitemap=only" \
  -H "authorization: Bearer sc_live_your_key"

The same operation as a GET, for when a JSON body is in the way. The proxy block flattens into query parameters. The SDK method is map() and the MCP tool is map_site.