How to extract all URLs from a website's sitemap (and find broken links)

Q: Can it find broken links in my sitemap?

Yes. With checkStatus every URL is requested once and you get the status code, final status after redirects and the redirect chain. Filter for 4xx/5xx to find pages that should not be in the sitemap.

Updated 2026-10-10 · Tool used: Sitemap Scraper - All URLs from XML Sitemaps + Status Check on Apify · Price: $0.20 per 1,000 URLs ($0.50 per 1,000 status checks); sites without a sitemap are free

Sitemaps are the fastest way to get a complete list of a website's pages: no crawling, no guessing. But big sites split them into sitemap indexes, gzip files and image or news sitemaps, and online sitemap viewers stop after a few hundred URLs.

This guide shows how to turn the sitemaps of one or hundreds of websites into a flat table (URL, lastmod, change frequency, priority, images, hreflang alternates), filter it by URL pattern or date, optionally check the HTTP status of every URL to find broken and redirected pages, and get only new URLs on a schedule.

Open Sitemap Scraper on Apify →

Steps

Add startUrls: domains, sitemap URLs or robots.txt URLs. Sitemaps are discovered from robots.txt and common paths.
Optionally filter with includeUrlPatterns (e.g. /blog/) and lastModifiedWithinDays.
Turn on checkStatus for a broken link and redirect audit, or Only new URLs to monitor a competitor's new pages.

Example input

{
  "startUrls": [
    "https://apify.com",
    "https://crawlee.dev"
  ],
  "maxUrlsPerSite": 300,
  "includeUrlPatterns": [],
  "checkStatus": true
}

Example output (one item, shortened)

{
  "site": "crawlee.dev",
  "url": "https://crawlee.dev/python/blog",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "sitemapUrl": "https://crawlee.dev/python/sitemap.xml",
  "imageCount": 0,
  "images": [],
  "videos": [],
  "alternates": [],
  "news": null,
  "statusCode": 200,
  "finalStatusCode": 200,
  "finalUrl": "https://crawlee.dev/python/blog",
  "redirectChain": [],
  "responseTimeMs": 581,
  "contentType": "text/html",
  "checkError": null,
  "scrapedAt": "2026-10-09T00:50:05.814Z"
}

Every run's dataset can be downloaded as JSON, CSV, Excel, XML or HTML from the Apify Console, or via https://api.apify.com/v2/datasets/<datasetId>/items?format=csv.

Run it from code

Python

from apify_client import ApifyClient  # pip install apify-client

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run_input = {
  "startUrls": [
    "https://apify.com",
    "https://crawlee.dev"
  ],
  "maxUrlsPerSite": 300,
  "includeUrlPatterns": [],
  "checkStatus": true
}
run = client.actor("tinyrex/sitemap-scraper").call(run_input=run_input)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

JavaScript / Node.js

import { ApifyClient } from 'apify-client'; // npm i apify-client

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('tinyrex/sitemap-scraper').call({
  "startUrls": [
    "https://apify.com",
    "https://crawlee.dev"
  ],
  "maxUrlsPerSite": 300,
  "includeUrlPatterns": [],
  "checkStatus": true
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

curl (no SDK)

# Runs the actor and returns the results directly (CSV here; use format=json for JSON)
curl -X POST "https://api.apify.com/v2/acts/tinyrex~sitemap-scraper/run-sync-get-dataset-items?format=csv" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://apify.com", "https://crawlee.dev"], "maxUrlsPerSite": 300, "includeUrlPatterns": [], "checkStatus": true}' > results.csv

Get your API token in Apify Console → Settings → API & Integrations. The free Apify plan includes monthly credit that is enough to try this.

Use it from an AI agent (MCP)

The Apify MCP server exposes this actor as a tool for Claude, Cursor, VS Code, n8n and other MCP clients. Add this to your client's MCP config (or use the OAuth flow at mcp.apify.com instead of a token):

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=tinyrex/sitemap-scraper",
      "headers": {
        "Authorization": "Bearer <YOUR_APIFY_TOKEN>"
      }
    }
  }
}

Then ask in plain language, e.g. "Extract urls from sitemap for this list and give me a table".

Pricing

Pay per result: $0.20 per 1,000 URLs ($0.50 per 1,000 status checks); sites without a sitemap are free. You can cap the cost of every run in the run options.

FAQ

How do I get all URLs of a website?

If the site has a sitemap (most do), reading it is the fastest and most complete way. Enter the domain; the actor finds the sitemaps through robots.txt and the usual paths and follows sitemap indexes.

Can it find broken links in my sitemap?

Yes. With checkStatus every URL is requested once and you get the status code, final status after redirects and the redirect chain. Filter for 4xx/5xx to find pages that should not be in the sitemap.

What if a site has no sitemap?

It is reported in the SITES record and costs nothing. Use a crawler for sites without sitemaps.

Try it on Apify →