/v1/sitemap endpoint

Full URL list from sitemap.xml. Falls back to shallow crawl if no sitemap exists.

Language:
POST

/v1/sitemap

5 credits

Attempts to fetch and parse sitemap.xml (and sitemap index files). If no sitemap is found, performs a shallow crawl of the homepage to discover linked pages. Useful for bulk-scraping or building crawl queues.

Request body

fieldtypereqdescription
urlstringyesRoot URL of the site. Must include protocol (https://).
options.maxPagesnumbernoMax URLs to return. Default: 500.
options.timeoutnumbernoTimeout in ms. Default: 30000. Max: 60000.

Example

curl -X POST https://contextforai.ashutoshswamy.in/v1/sitemap \
  -H "Authorization: Bearer cfai_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Response

200 OK
{
  "url": "https://example.com",
  "source": "sitemap.xml",
  "count": 48,
  "pages": [
    { "url": "https://example.com/", "lastmod": "2026-07-01", "priority": 1.0 },
    { "url": "https://example.com/about", "lastmod": "2026-06-15", "priority": 0.8 }
  ],
  "extractedAt": "2026-07-01T10:00:00Z"
}
← Full API reference: authentication, errors, rate limits, billing