POST /find_sitemap_urls • Credits: 1 credit per call
Find Sitemap URLs reads the sitemap files of one domain and returns the page URLs that match your keywords. The response holds the sitemap file list, the total URL count, the filtered URL count, and the filtered URLs. Find Sitemap URLs costs 1 credit per call, and the price does not change with the number of URLs.
What does Find Sitemap URLs do?
A sitemap is an XML file that lists the pages of a website. Find Sitemap URLs finds those files on a domain, reads every page URL inside them, and keeps the URLs that match a keyword list. A single call answers a question such as "which pricing pages does this domain publish?". Find Sitemap URLs runs on RichAPI infrastructure, and it calls no third-party data provider. Find Sitemap URLs returns URLs only. To read the content of a URL, send that URL to the Web Scraper endpoint.
Which endpoint does Find Sitemap URLs use?
Method:
POSTURL:
https://api.richapi.ai/api/v1/find_sitemap_urlsAuthentication:
x-api-keyheaderResponse: Synchronous JSON
Credits: 1 credit per call
How to send a Find Sitemap URLs request
Copy your API key from the RichAPI dashboard.
Set the
domainparameter to the website domain. A scheme is optional.Set the
keywordsparameter to a comma-separated keyword list. To return every URL, omitkeywords.Send a
POSTrequest tohttps://api.richapi.ai/api/v1/find_sitemap_urls.Send the API key in the
x-api-keyheader.Read the
urlsarray and thefiltered_urlscount from the response.
CAUTION: Send one call for each domain. RichAPI charges 1 credit for every 200 response, and a domain with no sitemap also costs 1 credit.
cURL
curl -X POST https://api.richapi.ai/api/v1/find_sitemap_urls
-H "x-api-key: YOUR_API_KEY"
-H "Content-Type: application/json"
-d '{
"domain": "stripe.com",
"keywords": "pricing,billing",
"exact_match": false
}'
Python
import requests
response = requests.post(
"https://api.richapi.ai/api/v1/find_sitemap_urls",
headers={"x-api-key": "YOUR_API_KEY"},
json={
"domain": "stripe.com",
"keywords": "pricing,billing",
"exact_match": False,
},
timeout=60,
)
response.raise_for_status()
print(response.json())
Node.js
const response = await fetch("https://api.richapi.ai/api/v1/find_sitemap_urls", {
method: "POST",
headers: {
"x-api-key": "YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
domain: "stripe.com",
keywords: "pricing,billing",
exact_match: false,
}),
});
const data = await response.json();
console.log(data);
Which parameters does Find Sitemap URLs accept?
The table holds the fields of the HTTP request body. Send these names to https://api.richapi.ai/api/v1/find_sitemap_urls. The dashboard, the MCP tool find_sitemap_urls, and the n8n operation find_sitemap_urls use the same three names, so no name mapping applies.
Parameter | Type | Required | Default | Description |
| string | Yes | None | The domain to read. |
| string | No |
| A comma-separated keyword list. An empty value returns every URL. |
| boolean | No |
| Set to |
Find Sitemap URLs applies the defaults in the table. Any other field returns status 400 with the message Unknown input field(s): <name>.
How does the keyword filter work?
Find Sitemap URLs splits the keywords value on commas and converts each keyword to lower case. The filter runs against the URL path, not against the host name and not against the query string.
| Rule |
|
| The URL path holds the keyword as a substring. |
|
| The last path segment equals the keyword. |
|
One keyword match is enough for a URL to pass the filter. If keywords is empty, Find Sitemap URLs applies no filter, and filtered_urls equals total_urls.
How does Find Sitemap URLs discover a sitemap?
Find Sitemap URLs requests
/robots.txton the domain.Find Sitemap URLs collects every
Sitemap:line in that file. A relative path resolves against the domain root.If
/robots.txtnames no sitemap, Find Sitemap URLs requests/sitemap.xml,/sitemap_index.xmland/sitemap-index.xml.Find Sitemap URLs parses each XML file. A
<sitemap><loc>entry names a nested sitemap file, and Find Sitemap URLs follows that file up to 3 levels deep.A
<url><loc>entry is a page URL. Find Sitemap URLs collects it and counts it intotal_urls.Find Sitemap URLs stops at 50,000 page URLs.
Find Sitemap URLs skips a sitemap file that fails to load. Find Sitemap URLs also skips a sitemap file with malformed XML. Neither case fails the request.
Find Sitemap URLs blocks a private, loopback, link-local or reserved address. A request for such a domain returns 400.
What does Find Sitemap URLs return?
The next block is an example. It shows the response for the request body in the code samples above.
{
"domain": "https://stripe.com",
"sitemap_urls": [
"https://stripe.com/sitemap/partition-0.xml",
"https://stripe.com/sitemap/sitemap.xml"
],
"total_urls": 1842,
"filtered_urls": 4,
"urls": [
"https://stripe.com/pricing",
"https://stripe.com/pricing/local-payment-methods",
"https://stripe.com/billing",
"https://stripe.com/billing/pricing-models"
]
}
Which fields does the Find Sitemap URLs response hold?
Field | Type | Description |
| string | The normalized domain root, with the scheme and the host name. |
| array of strings | The sitemap file URLs that Find Sitemap URLs read or tried to read. |
| integer | The number of page URLs found across all sitemap files, before the keyword filter. |
| integer | The number of page URLs that match the keyword filter. Equals |
| array of strings | The page URLs that match the keyword filter. 50,000 URLs at most. |
How much does Find Sitemap URLs cost?
Find Sitemap URLs costs 1 credit per call. The charge is flat. A response with 1 URL and a response with 40,000 URLs both cost 1 credit. 1 credit costs $0.02 on the 1,000 and 5,000 credit packs, and less on larger packs. The tables use $0.02. RichAPI charges the credit on a 2xx response only.
Calls | Credits | Cost |
1 | 1 | $0.02 |
10 | 10 | $0.20 |
100 | 100 | $2.00 |
1,000 | 1,000 | $20.00 |
10,000 | 10,000 | $200.00 |
Find Sitemap URLs or Sitemap Extractor: which endpoint is cheaper?
Find Sitemap URLs costs 1 credit per call. Sitemap Extractor (web_sitemap) costs 1 credit for every URL in the response. The two prices diverge fast on a large sitemap.
Sitemap size | Find Sitemap URLs credits | Find Sitemap URLs cost | Sitemap Extractor credits | Sitemap Extractor cost |
10 URLs | 1 | $0.02 | 10 | $0.20 |
100 URLs | 1 | $0.02 | 100 | $2.00 |
500 URLs | 1 | $0.02 | 500 | $10.00 |
5,000 URLs | 1 | $0.02 | 5,000 | $100.00 |
40,000 URLs | 1 | $0.02 | 40,000 | $800.00 |
A 500-URL sitemap costs 1 credit through Find Sitemap URLs and 500 credits through Sitemap Extractor. That is $0.02 against $10.00 for the same website.
CAUTION: Set the limit parameter on every Sitemap Extractor request. Sitemap Extractor bills on the URL count, and an uncapped request against a large sitemap costs hundreds of credits.
What is the difference between Find Sitemap URLs and Sitemap Extractor?
The two endpoints read the same kind of file and return different things. Find Sitemap URLs returns a keyword-filtered URL list plus the sitemap file list. Sitemap Extractor returns the full URL list and a count.
Item | Find Sitemap URLs | Sitemap Extractor |
Path |
|
|
Website parameter |
|
|
Keyword filter | Yes, with | No |
Sitemap file list in the response | Yes, in | No |
Total URL count before the filter | Yes, in | No |
Result cap | 50,000 URLs, fixed |
|
Crawl fallback for a website with no sitemap | No | Yes, up to 100 URLs |
Cached response | No | Yes, with |
Price | 1 credit per call | 1 credit per URL returned |
Use Find Sitemap URLs to find the pages that match a keyword. Use Find Sitemap URLs to count the pages of a domain. Use Sitemap Extractor to take delivery of every URL. Use Sitemap Extractor to cap the result count with limit. Use Sitemap Extractor for a website that publishes no sitemap.
What happens on a domain with no sitemap?
Find Sitemap URLs runs no crawl fallback. If /robots.txt names no sitemap and the three default sitemap paths return no valid XML, the response is a 200 with empty results:
{
"domain": "https://example.com",
"sitemap_urls": [
"https://example.com/sitemap-index.xml",
"https://example.com/sitemap.xml",
"https://example.com/sitemap_index.xml"
],
"total_urls": 0,
"filtered_urls": 0,
"urls": []
}
The sitemap_urls array lists the paths that Find Sitemap URLs tried. A 200 response with 0 URLs costs 1 credit. For a website with no sitemap, send the request to Sitemap Extractor. Sitemap Extractor crawls the website and returns up to 100 URLs.
Which errors can Find Sitemap URLs return?
Status | Meaning | What to do |
| The | Send a JSON object with a public |
| The API key is missing, wrong or revoked. | Send a valid key in the |
| The organization has too few credits left, or the team reached its monthly spend limit. | Buy credits in the RichAPI dashboard. |
| This API key has no access to this endpoint ( | Call |
| The endpoint name is wrong. | Send the request to |
| The request used a method other than | Send the request with |
| The request rate is over the limit. | Wait, then send the request again. For a higher limit, contact support. |
| The API failed. | Send the request again. If the failure repeats, contact support. |
Every error response holds an error field with the reason. RichAPI charges credits on a 2xx response only. A 400, 401, 402, 403, 404, 405, 429 or 500 response costs 0 credits.
Frequently asked questions
How much does Find Sitemap URLs cost?
Find Sitemap URLs costs 1 credit per call. 1 credit costs $0.02 on the 1,000 and 5,000 credit packs. The charge is flat, so a call that returns 40,000 URLs costs the same 1 credit as a call that returns 1 URL.
Which is cheaper, Find Sitemap URLs or Sitemap Extractor?
Find Sitemap URLs is cheaper on every sitemap larger than one URL. A 500-URL sitemap costs 1 credit ($0.02) through Find Sitemap URLs. The same sitemap costs 500 credits ($10.00) through Sitemap Extractor, because Sitemap Extractor charges 1 credit for each URL.
Does Find Sitemap URLs read robots.txt?
Yes. Find Sitemap URLs requests /robots.txt first and collects every Sitemap: line. If /robots.txt names no sitemap, Find Sitemap URLs falls back to /sitemap.xml, /sitemap_index.xml and /sitemap-index.xml.
Does Find Sitemap URLs follow a sitemap index?
Yes. A sitemap index is a sitemap file that points at other sitemap files. Find Sitemap URLs follows a nested index up to 3 levels deep and lists every file it read in the sitemap_urls array.
Does a domain with no sitemap cost credits?
Yes. The call returns a 200 response with an empty urls array and a total_urls of 0, and RichAPI charges 1 credit. Find Sitemap URLs runs no crawl fallback, so a website with no sitemap returns no URLs.
How many URLs can Find Sitemap URLs return?
Find Sitemap URLs returns 50,000 URLs at most, and that cap is fixed. The total_urls field reports the number of page URLs found before the keyword filter ran.
Does Find Sitemap URLs return the content of each page?
No. Find Sitemap URLs returns URLs only. To read the content of a URL, send that URL to the Web Scraper endpoint. Web Scraper returns the HTML, the markdown, the links and the page title.