Website URL Extractor

What the website URL extractor does

Give it a starting URL and it crawls the site the way a search engine would: fetch the page, collect the internal links, fetch those, and keep going up to the limit you set. The result is the list of URLs the site exposes through links, which is not always the list the owner thinks it has.

What the list tells you

• Pages you forgot exist: old landing pages, test pages and duplicate paths that still get crawled.
• How deep the site goes. Anything more than three clicks from the home page struggles to rank; the crawl depth column shows it.
• Which sections carry the most internal links. That is where the site is telling Google its priorities are.
• On a competitor site, the full inventory of pages they publish, which is the starting point for a content gap analysis.

Good practice

• Start from the canonical host (with or without www, whichever redirects to the other).
• Cap the crawl on large sites. A few hundred URLs is enough to see the structure.
• Respect robots.txt on sites you do not own. The tool does; if a section is blocked, it stays out of the list.