POST /web_sitemap • Credits: 1 credit per URL returned

Sitemap Extractor returns the page URLs in the sitemap of one website. The response holds a data.urls array and a data.total_count field. Sitemap Extractor charges 1 credit for each URL that it returns, so a response with 500 URLs costs 500 credits.

What does Sitemap Extractor do?

A sitemap is a file that lists the pages of a website. Most websites publish that file at /sitemap.xml. Sitemap Extractor reads the file and returns the page URLs as a JSON array. Sitemap Extractor first requests /sitemap.xml on the domain. If that file is absent, Sitemap Extractor reads the Sitemap: line in /robots.txt. If both files are absent, Sitemap Extractor discovers the URLs with a crawl of the website. Sitemap Extractor is the URL discovery endpoint of the RichAPI web scraper family.

Which endpoint does Sitemap Extractor use?

Method:POST
URL:https://api.richapi.ai/api/v1/web_sitemap
Authentication:x-api-key header
Response: Synchronous JSON
Credits: 1 credit per URL returned

How to send a Sitemap Extractor request

Copy your API key from the RichAPI dashboard.
Set the url parameter to the website root URL.
Set the limit parameter to the highest number of URLs that you accept.
Send a POST request to https://api.richapi.ai/api/v1/web_sitemap.
Send the API key in the x-api-key header.
Read the data.urls array from the response.

CAUTION: Sitemap Extractor charges 1 credit for each URL in the response. A response with 500 URLs costs 500 credits. 500 credits cost $10.00. Set limit on every request.

cURL

curl -X POST https://api.richapi.ai/api/v1/web_sitemap   
  -H "x-api-key: YOUR_API_KEY"   
  -H "Content-Type: application/json"   
  -d '{
    "url": "https://stripe.com",
    "limit": 100,
    "cache": true
  }'

Python

import requests

response = requests.post(
    "https://api.richapi.ai/api/v1/web_sitemap",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={
        "url": "https://stripe.com",
        "limit": 100,
        "cache": True,
    },
    timeout=60,
)
response.raise_for_status()
print(response.json())

Node.js

const response = await fetch("https://api.richapi.ai/api/v1/web_sitemap", {
  method: "POST",
  headers: {
    "x-api-key": "YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://stripe.com",
    limit: 100,
    cache: true,
  }),
});

const data = await response.json();
console.log(data);

Which parameters does Sitemap Extractor accept?

The table holds the three fields of the request body. RichAPI accepts these three names and no other name.

Parameter	Type	Required	Default	Description
`url`	string	Yes	None	The website root URL or a direct sitemap URL.
`limit`	integer	Yes	None	The maximum number of URLs to return, from `1` to `10000`.
`cache`	boolean	No	`false`	Set to `true` to accept a cached sitemap.

CAUTION: RichAPI rejects any other field name with status 400 and the message Unknown input field(s). Send limit, and never send max_urls.

What does Sitemap Extractor return?

The next block is an example. It shows the response for a request with limit set to 5.

{
  "data": {
    "urls": [
      "https://stripe.com/",
      "https://stripe.com/pricing",
      "https://stripe.com/payments",
      "https://stripe.com/billing",
      "https://stripe.com/customers"
    ],
    "total_count": 5,
    "source": "xml"
  },
  "meta": {
    "cache_hit": false,
    "execution_time_ms": 640,
    "quality_warning": null
  }
}

Which fields does the Sitemap Extractor response hold?

Field	Type	Description
`data.urls`	array of strings	The URLs that Sitemap Extractor discovered. RichAPI charges 1 credit for each entry.
`data.total_count`	integer	The number of URLs found.
`data.source`	string	Where the URLs came from: `xml`, `xml_index`, `robots`, `robots_index`, `map` or `crawl`.
`meta.cache_hit`	boolean	`true` when the response came from the cache.
`meta.execution_time_ms`	integer	The run time of the call in milliseconds.
`meta.quality_warning`	string or null	A warning when no URL was found. Otherwise `null`.

How does the limit parameter control the cost?

Sitemap Extractor bills on the number of entries in data.urls, not on the call. The limit parameter caps the number of URLs in the response, so limit caps the credit charge. A large ecommerce sitemap holds tens of thousands of URLs. One request with limit set to 10000 against that sitemap can cost 10,000 credits.

RichAPI holds limit credits at the start of the request and releases the unused part after the response. If the balance is under that hold, the request returns 402 and costs no credit.

`limit`	Maximum credits	Maximum cost
10	10	$0.20
50	50	$1.00
100	100	$2.00
500	500	$10.00
1,000	1,000	$20.00

CAUTION:limit caps the charge. It does not lower it. If the sitemap holds fewer URLs than limit, RichAPI charges the smaller number.

How much does Sitemap Extractor cost?

Sitemap Extractor costs 1 credit for each URL in the response. RichAPI counts the entries in data.urls. 1 credit costs $0.02 on the 1,000 and 5,000 credit packs. Larger packs cost less per credit. A response with 250 URLs costs 250 credits, and 250 credits cost $5.00.

URLs returned	Credits	Cost
1	1	$0.02
25	25	$0.50
100	100	$2.00
250	250	$5.00
500	500	$10.00
1,000	1,000	$20.00
5,000	5,000	$100.00

The count is per response, not per call. Ten requests that each return 100 URLs cost 1,000 credits. 1,000 credits cost $20.00.

How long does Sitemap Extractor cache a sitemap?

Sitemap Extractor caches a sitemap for 24 hours. The cache key holds the website URL and the limit value, so a new limit value reads the live website again. The cache default is false. To read a cached sitemap, set cache to true.

CAUTION: A cached response costs the same credits as a live response. RichAPI bills on the entries in data.urls in both cases.

What happens on a website with no sitemap?

Sitemap Extractor tries four sources in order. It requests /sitemap.xml first. It reads the Sitemap: line in /robots.txt second. It maps the website links third. It crawls the website last. A crawl returns 100 URLs at most. A higher limit value does not raise that limit.

If every source returns nothing, the response holds an empty data.urls array and a data.total_count of 0. That response costs 0 credits.

Which errors can Sitemap Extractor return?

Status	Meaning	What to do
`400`	The request body failed validation, or `limit` is below `1` (message `limit must be at least 1.`).	Correct the parameter that the message names. Send `url`, `limit` and `cache` only.
`401`	The API key is missing, wrong or revoked.	Send a valid key in the `x-api-key` header.
`402`	The balance is under the hold of `limit` credits, or the team reached its monthly spend limit.	Buy credits in the RichAPI dashboard, or lower `limit`.
`403`	`Endpoint not available on your plan`: this API key has no access to this endpoint.	Call `GET /api/v1/my-endpoints` to list the endpoints this key can use. For access, write to [email protected].
`404`	The endpoint name is wrong.	Send the request to `/web_sitemap`.
`405`	The request used a method other than `POST`.	Send the request again with the `POST` method.
`422`	The service could not process a value, for example a `limit` above `10000`, a URL that does not start with `http://` or `https://`, or a private network address.	Send `limit` from `1` to `10000` and a full public HTTP or HTTPS URL.
`429`	Too many requests.	Read the `Retry-After` header. Wait that many seconds, then send the request again.
`500`	The API failed.	Send the request again. If the failure repeats, write to [email protected].
`502`	The upstream service failed.	Wait a few seconds, then send the request again.
`504`	The website took too long to answer.	Lower `limit`, then send the request again.

RichAPI charges credits on a 2xx response only. A 400, 401, 402, 403, 404, 405, 422, 429, 500, 502 or 504 response costs 0 credits.

Frequently asked questions

What is a sitemap?

A sitemap is a file that lists the pages of a website, and search engines read it. Most websites publish the file at /sitemap.xml or name it in /robots.txt.

How do I control the credit cost of a Sitemap Extractor request?

Set the limit parameter on every request. Sitemap Extractor charges 1 credit for each URL in the response. A limit value of 100 caps the charge at 100 credits, that is $2.00.

Why does my request return "Unknown input field(s)"?

The request body holds a field name that RichAPI does not accept. Sitemap Extractor accepts url, limit and cache only, so replace max_urls with limit.

Does a website with no sitemap cost credits?

If Sitemap Extractor finds no URL, the response holds an empty data.urls array, and the call costs 0 credits. If the crawl fallback finds URLs, each URL costs 1 credit.

Does a cached sitemap cost credits?

Yes. A cached response costs 1 credit for each URL in data.urls, and that is the same price as a live response. To read the live website, set cache to false.

What is the difference between Sitemap Extractor and Find Sitemap URLs?

Sitemap Extractor returns the full URL list for 1 credit per URL. Find Sitemap URLs filters the sitemap by keyword and costs 1 credit per call.

Does Sitemap Extractor return the content of each page?

No. Sitemap Extractor returns URLs only. To read the content of a URL, send that URL to the Web Scraper endpoint.

Sitemap Extractor API