About this endpoint
What it does
Scrapes the URL provided in the url query parameter and returns the original URL, a UTC timestamp for when the scrape ran, and a scraped_data object containing extracted page content and metadata.
Query Parameter(s)
| Attribute | Type | Mandatory | Description |
|---|---|---|---|
| url | String | Yes | The URL to scrape. Must be a valid URI. |
Response
Returns a JSON object with url as a string, timestamp as a date-time string, and scraped_data as an object. The scraped_data object may include extracted links, metadata, content representations, language detection, and keyword/index data.
| Attribute | Type | Mandatory | Description |
|---|---|---|---|
| url | String | Yes | The URL that was scraped, echoed back from the request. Must be a valid URI. |
| timestamp | String | Yes | UTC timestamp of when the scrape was performed. Format: date-time. |
| scraped_data | Object | Yes | Extracted page data. Includes the page title, link data, metadata, content representations, detected language, and keyword index where available. |
| scraped_data.links | Object | No | Links found on the page, split into internal and external. |
| scraped_data.links.external | String Array | No | External links found on the page. Each item is a URI. |
| scraped_data.links.internal | String Array | No | Internal links found on the page. Each item is a URI. |
| scraped_data.title | String | No | The page's <title> content. |
| scraped_data.headers | Object | No | Document-level metadata pulled from the <head>. |
| scraped_data.headers.charset | String | No | Character set declared by the page. Nullable. |
| scraped_data.headers.viewport | String Array | No | Viewport meta values. |
| scraped_data.headers.canonical | String | No | Canonical URL. Must be a valid URI. Nullable. |
| scraped_data.headers.csrfToken | String | No | CSRF token value. Nullable. |
| scraped_data.headers.contentType | String | No | Content type value. Nullable. |
| scraped_data.meta_tags | Object | No | Standard <meta> tag values. |
| scraped_data.meta_tags.image | String | No | Image meta value. Nullable. |
| scraped_data.meta_tags.author | String | No | Author meta value. Nullable. |
| scraped_data.meta_tags.keywords | String Array | No | Keyword values from meta tags. |
| scraped_data.meta_tags.description | String | No | Meta description value. Nullable. |
| scraped_data.open_graph | Object | No | Open Graph (og:*) meta tag values, keyed by tag name. |
| scraped_data.content_html | String | No | The page's main content rendered as HTML. |
| scraped_data.twitter_card | Object | No | Twitter/X Card (twitter:*) meta tag values, keyed by tag name. |
| scraped_data.content_lists | Object Array | No | Lists extracted from the page. |
| scraped_data.content_lists[].type | String | No | List type, such as ul or ol. |
| scraped_data.content_lists[].items | String Array | No | Items contained in the list. |
| scraped_data.content_markdown | String | No | The page's main content rendered as Markdown. |
| scraped_data.detected_language | String | No | Detected ISO 639-1 language code for the page. |
| scraped_data.content_structured | Object Array | No | Ordered list of structural content blocks extracted from the page. |
| scraped_data.content_structured[].tag | String | No | HTML tag name of the block. |
| scraped_data.content_structured[].content | String | No | Text content of the block. |
| scraped_data.content_keywords_index | String Array | No | Extracted keyword and phrase index for the page content. |