Content crawler

Point it at a URL and get readable text back, not HTML. Use it to read a competitor's content, gather source material before writing, or pull a site's own copy into a model.

POST/v1/pages/crawl

Crawl

Starts at the URL you give it, follows links up to depth, and returns every page it reads as text. Cookie banners and navigation are stripped; headings and paragraph breaks are kept, because that structure is what makes the text usable by a model.

Costs 2.28 credits per page requested. 5 pages is 11.4 credits, 20 pages is 45.6. You are charged for the pages you ask for, not the pages that come back, so ask for what you need.

Crawling respects robots.txt. A page that blocks crawlers is skipped rather than fetched, so a run can come back with fewer pages than you asked for: pages_requested and count are both in the response so you can see when that happened.

The crawl stays on the site you point it at. Each page gets 20 seconds before it is abandoned, and the whole run returns at most 150,000 characters of text, which is roughly twenty average articles.

Required attributes

Name
url
Type
string
Description
Page to start from. example.com/blog and https://example.com/blog both work. Private and internal addresses are rejected.

Optional attributes

Name
pages
Type
integer
Description
Pages to crawl, 1 to 20. Default 5. Capped by your plan.
Name
depth
Type
integer
Description
How many links deep to follow, 0 to 2. Default 1. 0 reads only the starting page, which is the cheapest way to pull one article.

Response fields

Name
pages[].text
Type
string
Description
The page as text, capped at 20,000 characters per page.
Name
pages[].truncated
Type
boolean
Description
Whether a cap was hit, so you know the page had more to say. True for a page over 20,000 characters, and for the page where the run's 150,000 character budget ran out.
Name
pages_requested
Type
integer
Description
What you asked for and were charged for. Compare it against count to see how many pages the site actually let us read.
Name
pages[].word_count
Type
integer
Description
Words in the returned text. Useful for judging depth against your own pages before you write.
Name
word_count
Type
integer
Description
Total across every page in the run.

Request

POST

/v1/pages/crawl

curl -X POST https://suvjexogmfpvxjcktvuk.supabase.co/functions/v1/api/v1/pages/crawl \
  -H "Authorization: Bearer $PENGU_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/blog", "pages": 5, "depth": 1}'

Response

{
  "data": {
    "start_url": "https://example.com/blog",
    "crawled_at": "2026-09-20T09:30:18.907Z",
    "pages_requested": 5,
    "count": 5,
    "word_count": 6142,
    "pages": [
      {
        "url": "https://example.com/blog/how-we-cut-churn",
        "title": "How we cut churn by 40%",
        "description": "The three changes that moved the number, and the two that did not.",
        "word_count": 1480,
        "truncated": false,
        "text": "# How we cut churn by 40%\n\nWe started with ..."
      }
    ]
  },
  "meta": {
    "credits_charged": 8,
    "credits_remaining": 2792,
    "request_id": "0e3b6d91-4c2a-4f88-9d17-5a6b2e8c1740"
  }
}