POST /web_scrape • Credits: 1 credit per call

Web Scraper returns the full content of one public web page as HTML and markdown. The response holds a data object with the page HTML, the markdown, the title and the links. Each call costs 1 credit.

What does Web Scraper do?

A web page holds the content that your workflow needs, but the raw page is hard to parse. Web Scraper solves that problem. The endpoint loads the page in a real browser with anti-bot protection. It returns the finished HTML and a clean markdown version of the same page. One call covers one URL. Web Scraper is the general-purpose endpoint of the RichAPI web scraper family.

Which endpoint does Web Scraper use?

Method:POST
URL:https://api.richapi.ai/api/v1/web_scrape
Authentication:x-api-key header
Response: Synchronous JSON
Credits: 1 credit per call

How to send a Web Scraper request

Copy your API key from the RichAPI dashboard.
Set the url parameter to the full page URL.
Set the formats parameter to the output formats that you need.
Send a POST request to https://api.richapi.ai/api/v1/web_scrape.
Send the API key in the x-api-key header.
Read data.markdown or data.html from the response.

CAUTION: Each Web Scraper call costs 1 credit. A cached result also costs 1 credit.

cURL

curl -X POST https://api.richapi.ai/api/v1/web_scrape   
  -H "x-api-key: YOUR_API_KEY"   
  -H "Content-Type: application/json"   
  -d '{
    "url": "https://example.com",
    "formats": ["markdown"],
    "cache": true
  }'

Python

import requests

response = requests.post(
    "https://api.richapi.ai/api/v1/web_scrape",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={
        "url": "https://example.com",
        "formats": ["markdown"],
        "cache": True,
    },
    timeout=60,
)
response.raise_for_status()
print(response.json())

Node.js

const response = await fetch("https://api.richapi.ai/api/v1/web_scrape", {
  method: "POST",
  headers: {
    "x-api-key": "YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://example.com",
    formats: ["markdown"],
    cache: true,
  }),
});

const data = await response.json();
console.log(data);

Which parameters does Web Scraper accept?

Parameter	Type	Required	Default	Description
`url`	string	Yes	None	The full HTTP or HTTPS URL of the page to scrape.
`formats`	array of strings	No	`["html", "markdown"]`	The output formats to include in the response.
`cache`	boolean	No	`false`	Set to `true` to return a cached page instead of a live scrape.

If you omit cache, the RichAPI gateway sends cache: false, and Web Scraper scrapes the live page.

Which output formats can you request?

The formats parameter takes an array of strings. Each value adds one kind of content to the response.

Value	What the value adds
`html`	The page HTML after the browser runs the JavaScript, in `data.html`.
`markdown`	The page content converted to markdown, in `data.markdown`.
`links`	The links found on the page, in `data.links`.

Web Scraper ignores any other value. The response holds no raw HTML field and no screenshot. If formats is empty or absent, Web Scraper returns html and markdown.

What does Web Scraper return?

Web Scraper returns a data object and a meta object. Here is an example response for https://stripe.com/about.

{
  "data": {
    "html": "<html><head><title>About Stripe</title></head><body><h1>About Stripe</h1></body></html>",
    "markdown": "# About Stripe\n\nStripe is a financial infrastructure platform for businesses.",
    "links": [
      "https://stripe.com/pricing",
      "https://stripe.com/jobs",
      "https://stripe.com/newsroom"
    ],
    "title": "About Stripe"
  },
  "meta": {
    "cache_hit": false,
    "execution_time_ms": 2841,
    "quality_warning": null
  }
}

Which fields does the Web Scraper response hold?

Field	Type	Description
`data.html`	string	The full page HTML. If `formats` omits `html`, the field is empty.
`data.markdown`	string	The page content converted to markdown. If `formats` omits `markdown`, the field is empty.
`data.links`	array of strings	The links found on the page. If `formats` omits `links`, the array is empty.
`data.title`	string	The page title.
`meta.cache_hit`	boolean	The cache state. A value of `true` marks a page served from the cache.
`meta.execution_time_ms`	integer	The run time of the call in milliseconds.
`meta.quality_warning`	string or null	The reason the page returned no content. The field is `null` after a successful scrape.

Does Web Scraper render JavaScript?

Yes. Web Scraper opens each page in a real browser. The data.html field and the data.markdown field hold the content after the JavaScript runs. A single-page application built with React, Vue or Angular returns real content, not an empty page shell.

What happens on a page that blocks scrapers?

Web Scraper applies anti-bot protection on every call. Some pages still block the request. If the scrape returns nothing, Web Scraper returns status 200 with an empty data.html field and an empty data.markdown field. The meta.quality_warning field then names the reason.

`meta.quality_warning`	What the value means
`null`	Web Scraper read the page.
`Page returned no relevant data`	Web Scraper reached the page, but the page held no HTML and no markdown.
`Scraping failed - no data could be retrieved for this URL`	Web Scraper did not reach the page at all.

CAUTION: A blocked page still costs 1 credit. RichAPI charges the credit on every 200 response. Make sure that meta.quality_warning is null before you store the result.

How long does Web Scraper cache a page?

Web Scraper holds each scraped page in a cache for 4 hours. The cache is off by default. To read a page from the cache, set cache to true. A second call for the same URL and the same formats inside that window then returns the cached page, and meta.cache_hit is true. A cached call and a live call both cost 1 credit.

When do you use Web Scraper instead of a specialized endpoint?

When you want the page content itself, use Web Scraper. When you want one extracted signal, use a specialized endpoint. A specialized endpoint returns structured fields and saves you the parse step.

Endpoint	What the endpoint returns	Credits
`web_scrape`	The full page as HTML, markdown, links and title.	1 per call
`web_meta_tags`	OpenGraph, Twitter Card and SEO meta tags, plus the title, description, images and favicon.	1 per call
`web_json_ld`	The JSON-LD structured data blocks, such as products, articles and organizations.	1 per call
`web_emails`	Email addresses found across several pages of one website.	2 per call
`web_social_links`	Social media profile links, such as Facebook, X, LinkedIn, Instagram, YouTube and TikTok.	1 per call
`web_tech_stack`	The detected technologies, such as the CMS, analytics, CDN, frameworks and payment tools.	1 per call
`web_pixels`	The tracking pixels, such as Google Tag Manager, GA4, Facebook Pixel, LinkedIn Insight and TikTok Pixel.	1 per call
`web_sitemap`	Every URL in the sitemap of one website.	1 per URL returned
`website_intelligence`	One combined report: meta tags, tech stack, pixels, social links, emails, SSL and headers.	5 per call

When is website_intelligence cheaper?

Website Intelligence bundles five signals: meta tags, tech stack, pixels, social links and emails. Separately those five cost 6 credits, because Website Email Extractor costs 2 credits and the other four cost 1 credit each. One website_intelligence call costs 5 credits, so five signals is the break-even point.

Signals you need for one domain	Single-purpose cost	`website_intelligence` cost	Cheaper choice
3 (meta tags, pixels, social links)	1 + 1 + 1 = 3 credits	5 credits	The single-purpose endpoints
4 (add tech stack)	1 + 1 + 1 + 1 = 4 credits	5 credits	The single-purpose endpoints
5 (add emails)	1 + 1 + 1 + 1 + 2 = 6 credits	5 credits	`website_intelligence`

For four signals or fewer, the single-purpose endpoints cost less.

How much does Web Scraper cost?

Web Scraper costs 1 credit per call. 1 credit costs $0.02 on the 1,000 and 5,000 credit packs, and less on larger packs. The table uses $0.02. The credit count does not change with the page size or with the number of formats that you request.

Calls	Credits	Cost
1	1	$0.02
100	100	$2.00
1,000	1,000	$20.00
10,000	10,000	$200.00

Which errors can Web Scraper return?

Status	Meaning	What to do
`400`	The request body failed validation.	Correct the parameter that the message names. Send a full HTTP or HTTPS URL.
`401`	The API key is missing, wrong or revoked.	Send a valid key in the `x-api-key` header.
`402`	The organization has no credits left, or the team reached its monthly spend limit.	Buy credits in the RichAPI dashboard.
`403`	`Endpoint not available on your plan`: this API key has no access to this endpoint.	Call `GET /api/v1/my-endpoints` to list the endpoints this key can use. For access, write to [email protected].
`404`	The endpoint name is wrong.	Send the request to `/web_scrape`.
`405`	The request used a method other than `POST`.	Send the request again with the `POST` method.
`422`	The service could not process a value, for example a URL that does not start with `http://` or `https://`, or a private network address.	Send a full public HTTP or HTTPS URL.
`429`	The request rate is over the limit.	Wait, then send the request again. For a higher limit, write to [email protected].
`500`	The API failed.	Send the request again. If the failure repeats, write to [email protected].
`502`	The upstream service failed.	Wait a few seconds, then send the request again.
`504`	The target page took too long.	Send the request again.

RichAPI charges no credit for a 400, 401, 402, 403, 404, 405, 422, 429, 500, 502 or 504 response.

Frequently asked questions

Does Web Scraper follow links to other pages?

No. One Web Scraper call scrapes one URL. To collect every URL on a website, send a Sitemap Extractor request first. Then send one Web Scraper call per URL.

Can Web Scraper read a page behind a login?

No. Web Scraper accepts no login credentials and no cookies. It reads only a page that a public visitor can open.

Does a failed scrape cost a credit?

A blocked page returns status 200 with empty content, and that call costs 1 credit. A 4xx or 5xx error costs no credit.

What is the difference between Web Scraper and Website Intelligence?

Web Scraper returns the raw content of one page for 1 credit. Website Intelligence returns a combined report with meta tags, tech stack, pixels, social links, emails, SSL and headers for 5 credits.

Can Web Scraper reach an internal or private address?

No. Web Scraper rejects a URL that points to a private network address or to an internal host. Send a public HTTP or HTTPS URL.

How do I get markdown only?

Set the formats parameter to ["markdown"]. The response then holds data.markdown, and the data.html field stays empty.

Web Scraper API