Website crawler
Crawl a whole site from its home page, or check a URL list pasted in or imported from TXT, CSV or TSV.
100% free and open source
OctopusOn SEO Crawler
A free SEO spider and website audit tool that crawls like a search engine: JavaScript rendering, broken links, redirect chains, duplicate content, hreflang and an explainable SEO score, with no URL limit.
Windows, macOS & Linux · SEO and GEO · Updated continuously · No account, no limits
Everything a website crawler needs, free
One SEO spider for site audits, JavaScript SEO, migrations and monitoring: 100+ checks in 21 categories, each with an explanation and a fix.
JavaScript SEO
Render pages with your installed Chrome or Edge and compare raw and rendered titles, canonicals, directives, words and links.
Broken link checker
Find 404s, server errors and broken internal links, with the pages and anchor texts that point to them.
Redirect chain checker
Follow every redirect hop, spot chains and loops, and see which links still point at redirected URLs.
Duplicate content checker
Exact duplicates on titles, descriptions, H1s and page text, plus near-duplicate pages found with SimHash.
Hreflang checker
Validate return links, self-references, x-default and language codes in HTML, HTTP headers and PDFs.
Schema validator
Test JSON-LD structured data for Article, Product, FAQ, Breadcrumb, Organization and LocalBusiness rich results.
XML sitemap generator
Build an XML sitemap from a crawl, split into a ZIP at 50,000 URLs or 50 MB.
A free SEO spider that crawls like Googlebot
Crawl as Googlebot Smartphone, Googlebot Desktop, Bingbot, Chrome or your own user agent, at a polite speed, with robots.txt read the way Google reads it.
- Spider mode or list mode, with URL, depth and time limits and include and exclude patterns
- 1–64 parallel requests, speed profiles and a requests-per-second cap
- robots.txt: respect it, ignore it but report, or ignore it; X-Robots-Tag per crawler
- XML sitemaps, sitemap indexes and gzip files, with their redirect chains
- Live progress, pause and resume, several crawls at once and saved crawls to reopen
JavaScript rendering for JavaScript SEO
Single-page apps and client-side frameworks are crawled the way search engines render them, in a fresh, isolated browser context for every URL.
- Uses the Chrome, Chromium or Edge already on your computer
- Desktop, mobile or custom viewport
- Raw vs rendered: title, description, canonical, robots directives, word count and links
- Flags a noindex removed by JavaScript and re-crawls failed renders
Site audit with an explainable SEO score
A 0–100 SEO score built from 100+ checks. Every deduction leads to the URLs behind it, with an explanation and a fix.
- Response codes, redirects and security: HTTPS, HSTS and mixed content
- Titles, meta descriptions, headings, canonicals, directives and pagination
- Images, structured data, Open Graph and Twitter cards, mobile viewport
- Prioritised fixes, grouped into nine systems
Ready for AI search and GEO
Check that AI crawlers can reach your content, next to the classic SEO checks.
- AI readiness: AI bot access in robots.txt and llms.txt
- Structured data that rich results and AI answers rely on
- Pair it with Log Analyser to see which AI bots really visit
Crawl comparison, exports and Search Console data
Follow a site from one crawl to the next, hand results to clients and find quick wins in Google data.
- Compare two crawls on 30 SEO fields and see new, fixed and remaining findings
- Filtered exports to Excel, CSV and JSON from every tab
- Choose the columns of every tab
- SEO Opportunities: low-CTR pages, near wins and traffic losses from Search Console and GA4 data
OctopusOn SEO Crawler vs Screaming Frog SEO Spider
The main differences at a glance.
| Feature | OctopusOn SEO Crawler | Screaming Frog SEO Spider |
|---|---|---|
| Price | Free, $0 | Free up to 500 URLs; £199 / $279 per user per year |
| URLs per crawl | No limit | 500 free; unlimited with a licence |
| Source code | Open source | Proprietary |
| JavaScript rendering | Included | Paid version |
| Save and reopen crawls | Included | Paid version |
| Windows, macOS and Linux | Yes | Yes |
Screaming Frog details from screamingfrog.co.uk, checked in October 2026.
SEO Crawler questions
Is OctopusOn SEO Crawler really free?
Yes. It is 100% free and open source (MIT or Apache-2.0). There is no paid version, no licence key and no URL limit; the only limits are your computer's memory and disk.
Is it a Screaming Frog SEO Spider alternative?
Yes. It covers the core of a technical SEO crawl for free: spider and list modes, JavaScript rendering, response codes, redirects, canonicals, hreflang, duplicates, structured data, XML sitemaps and exports. A few Screaming Frog features, such as custom extraction and PageSpeed Insights, are not available yet; the full comparison lists them.
Can it render JavaScript?
Yes. It uses the Chrome, Chromium or Edge installed on your computer and compares the raw HTML with the rendered page.
How many URLs can it crawl?
There is no built-in limit: a URL limit of 0 means unlimited. Very large sites are bound only by your computer's memory and disk.
Does my crawl data leave my computer?
No. Crawls run and are stored on your computer. Nothing is uploaded and no account is needed.
Which systems does it run on?
Windows, macOS (Apple Silicon and Intel) and Linux (AppImage, deb and rpm). The octopuson-crawler command-line tool is available for scripts and servers.
Command line
Every feature also works from the terminal, for scripts and scheduled jobs.
octopuson-crawler crawl https://example.com --render javascript
octopuson-crawler export latest --tab internal --out internal.xlsxDesktop app
Windows
- Installer · x64OctopusOn.SEO.Crawler_0.1.0_x64-setup.exe · 12 MB
62a4c7e708b02d50… - MSI package · x64OctopusOn.SEO.Crawler_0.1.0_x64_en-US.msi · 17.1 MB
73233a3752dddea5…
If SmartScreen warns about an unknown publisher, click “More info” → “Run anyway”.
macOS
Linux
- AppImage · x64OctopusOn.SEO.Crawler_0.1.0_amd64.AppImage · 94 MB
487bde2555f38331… - Debian / Ubuntu package · x64OctopusOn.SEO.Crawler_0.1.0_amd64.deb · 21.4 MB
92f1e6dbe3fe4b8b… - Fedora / RHEL package · x64OctopusOn.SEO.Crawler-0.1.0-1.x86_64.rpm · 21.4 MB
51b108736bae23a2…
AppImage: make it executable (chmod +x) and run it — or install the .deb or .rpm package.
Command line
| File | Size | SHA-256 |
|---|---|---|
| octopuson-crawler-crawler-v0.1.0-aarch64-apple-darwin.tar.gz | 14.7 MB | 26ad53920be4b297afb9… |
| octopuson-crawler-crawler-v0.1.0-x86_64-apple-darwin.tar.gz | 15.9 MB | 473dd14900a48f9b3a69… |
| octopuson-crawler-crawler-v0.1.0-x86_64-pc-windows-msvc.zip | 12.8 MB | 82b577000c44159de1b4… |
| octopuson-crawler-crawler-v0.1.0-x86_64-unknown-linux-gnu.tar.gz | 17.4 MB | 690e1c55f8eb5364ab32… |
Run your first free site audit
Download OctopusOn SEO Crawler, enter a URL and get an explainable SEO score in minutes.
Download free