Extract web content with Superhighway's /scrape endpoint

Superhighway's /scrape endpoint takes a URL and returns clean markdown text — no browser, no JS rendering, no HTML parsing. The common use case: feed any webpage straight into your LLM context with a single API call.

Quick start (Python)

import os, requests

API_KEY = os.environ["SUPERHIGHWAY_API_KEY"]

resp = requests.get(
    "https://superhighway.walls.sh/scrape",
    params={"url": "https://example.com/article"},
    headers={"Authorization": f"Bearer {API_KEY}"},
    timeout=30,
)
resp.raise_for_status()
data = resp.json()

print(data["title"])     # page title
print(data["markdown"])  # clean markdown — ready for LLM context
print(data["url"])       # canonical URL

Quick start (JavaScript)

const resp = await fetch(
  "https://superhighway.walls.sh/scrape?url=" + encodeURIComponent("https://example.com/article"),
  { headers: { Authorization: `Bearer ${process.env.SUPERHIGHWAY_API_KEY}` } }
);
const { title, markdown, url } = await resp.json();

LangChain integration — custom document loader

Wrap /scrape in a function that returns a LangChain Document, ready to drop into any chain:

from langchain_core.documents import Document
import requests, os

def load_url(url: str) -> Document:
    r = requests.get(
        "https://superhighway.walls.sh/scrape",
        params={"url": url},
        headers={"Authorization": f"Bearer {os.environ['SUPERHIGHWAY_API_KEY']}"},
        timeout=30,
    )
    r.raise_for_status()
    d = r.json()
    return Document(page_content=d["markdown"], metadata={"title": d["title"], "source": d["url"]})

# Use in a chain
doc = load_url("https://docs.example.com/page")
# feed doc.page_content to your LLM

MCP tool usage (Claude, Cursor, Windsurf)

Run the Superhighway MCP server so your agent can scrape pages natively:

npx -y superhighway-mcp

The MCP server exposes a scrape tool — agents pass a URL and call it directly, no HTTP plumbing required.

x402 autonomous payment (no API key needed)

Skip signup entirely: an agent with a wallet can pay per call over x402. The first request returns 402 with payment terms; the agent pays $0.002 USDC on Base and retries with an X-PAYMENT header:

import requests

# First call gets 402 with payment terms
resp = requests.get("https://superhighway.walls.sh/scrape", params={"url": "https://example.com"})
# resp.status_code == 402 -> agent pays USDC on Base, retries with X-PAYMENT header

See the x402 pay-per-call guide for the full autonomous payment flow.

Response structure

{
  "url": "https://example.com/article",
  "title": "Page Title",
  "markdown": "# Page Title\n\nMarkdown content of the page...",
  "text": "Page Title Plain text content of the page...",
  "length": 1234
}

markdown is the LLM-ready content; text is the same content with formatting stripped; length is the markdown character count.

When to use /scrape vs /search vs /research

Get your API key at /pricing (free tier: 1,000 calls/month). For more integration options see the full guide list.