Verified bots, not user agents
110+ crawlers identified and checked against the operators' official IP lists and reverse DNS. Fake Googlebots are flagged, not counted.
New: robots.txt compliance and AI referral reports
Your server logs know what Google and AI bots really do
A free SEO log file analyser. OctopusOn Log Analyser turns raw access logs into crawl insights: which pages Googlebot and AI crawlers fetch, where they hit errors, which bots are fake and where your crawl budget goes.
100% free and open source · Windows, macOS & Linux · Your logs stay on your computer
Reads the logs you already have — format detected automatically
- Nginx
- Apache
- IIS
- CloudFront
- AWS ALB
- HAProxy
- Traefik
- Caddy
- Cloudflare
- Google Cloud
- JSON
- CSV
From raw log lines to SEO decisions
Built by SEOs for agencies that analyse dozens of sites every month.
AI visibility
See what GPTBot, ClaudeBot, PerplexityBot and ChatGPT-User fetch — and which visits ChatGPT, Perplexity or Gemini send you.
robots.txt compliance
Which bots request disallowed URLs, which ones your robots.txt blocks entirely — evaluated exactly like Google does.
Crawl budget
Crawl frequency per URL and directory, days since the last crawl, parameter URLs, redirects and static files eating your budget.
Orphan & uncrawled pages
Match crawler exports and XML sitemaps against logs; extra columns are joined like a VLOOKUP.
39 reports + query language
Ready-made reports and a simple form for your own filters. Save, share and export to Excel, CSV or JSON.
Countries & networks
Country and network (ASN) of every client IP reveal scraper clouds and where fake bots come from. Looked up locally.
Crawl alerts
Sudden Googlebot drops, AI crawl spikes and server-error bursts are highlighted on the overview.
AI search is here
Know when AI assistants read — and recommend — your pages
Training crawlers, AI search indexers and real-time assistant fetches are separated, so you see which content feeds models, which gets cited, and the human visits AI assistants send back.
- Separate categories for AI training, AI search and AI assistants
- Errors served to AI bots — every failed fetch is a lost citation
- Visits referred by ChatGPT, Perplexity, Gemini, Copilot and Claude
- Bandwidth and estimated CO₂ per crawler
Three steps to your first insight
- 1
Drop your logs
Plain, gzip, zstd, bzip2 or zip — files or whole folders. Re-imports are safe.
- 2
We verify the bots
Every bot hit becomes verified, spoofed or unverified using official data.
- 3
Act on the reports
Fix errors bots hit, stop crawl waste, unblock what matters and track AI visibility.
110+crawlers recognised
12+log formats
39ready-made reports
750Klines per second
Automate it
A command line for every report
The octopuson-log CLI shares the data with the desktop app. Script monthly reports, schedule imports on your servers and export straight to Excel.
- ~750,000 lines per second
- Runs 100% on your computer
$ octopuson-log import acme.com /var/log/nginx/ Format combined (detected) Hits 4,812,330 stored · 0 duplicates Throughput 768,104 lines/s $ octopuson-log query acme.com --template googlebot-errors 'date:last7d' url status hits last_seen /old-category/shoes 404 1,204 2026-10-12 /api/cart 500 311 2026-10-12 $ octopuson-log robots acme.com bot group blocked disallowed verified gptbot gptbot yes 24,320 24,320 googlebot * 12 0
Bot identification and verification
User agents are trivial to fake. Every bot hit is checked, so reports count real crawlers only.
- 110+ crawlers in categories: search, AI training, AI search, AI assistants, ads, SEO tools, social, monitoring
- Official IP lists of Google, Bing, OpenAI, Anthropic, Perplexity, Apple and more, plus forward-confirmed reverse DNS
- Verdict per hit: verified, spoofed or unverified — re-check any time
- Your own bots: custom user-agent patterns with IP ranges, IP list URLs or reverse DNS
AI crawlers and AI search
Understand how AI platforms use your content — and what they send back.
- Daily activity of every AI crawler and the URLs it fetched
- Errors served to AI bots and assistants
- Human visits referred by AI assistants, with landing pages
- Bandwidth and estimated emissions per bot
robots.txt compliance
Check every logged bot request against your live robots.txt or any version of it.
- Google-style evaluation: most specific user-agent group, longest match, Allow wins ties
- Disallowed requests per bot, from verified and from fake IPs
- Bots blocked site-wide, such as AI training crawlers
- Export every violation with the deciding rule
Crawl budget and technical health
Find where crawlers waste time and what they cannot reach.
- URL summary: last status, last crawl, days since last crawl, size, speed and hits per bot
- Response codes per URL with 1xx–5xx columns and inconsistent responses
- Redirects, 4xx/5xx, slow and large URLs, parameter URLs, static files
- Alerts for crawl drops, spikes and server-error bursts
Uncrawled and orphan pages
Combine your crawl data with what bots actually did.
- Crawler CSV exports, plain lists and XML sitemaps (with sitemap indexes)
- Listed but never crawled, crawled but not listed (orphans)
- All other columns of your export joined to the log data
- Period comparison per URL, directory or bot — before and after a migration
Your data, your way
Fast, private and scriptable.
- Formats detected automatically; custom Nginx and Apache formats supported
- Import settings per site: bots only, include/exclude URLs, strip parameters
- Query language with saved queries; Excel, CSV and JSON exports
- Share a whole site as one archive; command-line tool for automation
OctopusOn Log Analyser vs Screaming Frog Log File Analyser
The main differences at a glance.
| Feature | OctopusOn Log Analyser | Screaming Frog Log File Analyser |
|---|---|---|
| Price | Free, $0 | Free with limits; £99 per year |
| Log lines | No limit | 1,000 free; unlimited with a licence |
| Projects | No limit | 1 free; unlimited with a licence |
| Source code | Open source | Proprietary |
| Search engine bot verification | Official IP lists and reverse DNS | Yes |
| Windows, macOS and Linux | Yes | Yes |
Screaming Frog details from screamingfrog.co.uk, checked in October 2026.
Frequently asked questions
Is OctopusOn Log Analyser really free?
Yes. The desktop app and the CLI are free and open source, with no limit on log lines or sites.
Do my logs leave my computer?
No. Everything is stored and analysed locally. The only downloads are public crawler IP lists and, if you enable it, the IP geolocation database.
Which log formats are supported?
Nginx and Apache (including custom formats), IIS, CloudFront, AWS load balancers, HAProxy, Traefik, Caddy, Cloudflare, Google Cloud, JSON and CSV logs. The format is detected automatically.
How large can my logs be?
Hundreds of millions of lines. Logs are imported at roughly 750,000 lines per second and stored compressed at about 8% of their original size.
Which languages does the app speak?
The desktop app is available in English and Turkish; reports and exports work the same in both.
Command line
Every feature also works from the terminal, for scripts and scheduled jobs.
octopuson-log import acme.com /var/log/nginx/
octopuson-log query acme.com --template url-summary -o urls.xlsxDesktop app
Windows
- Installer · x64OctopusOn.Log.Analyser_0.2.0_x64-setup.exe · 12.5 MB
359971af71843543… - MSI package · x64OctopusOn.Log.Analyser_0.2.0_x64_en-US.msi · 18.1 MB
12a0f338ca14e9ab…
If SmartScreen warns about an unknown publisher, click “More info” → “Run anyway”.
macOS
Linux
- AppImage · x64OctopusOn.Log.Analyser_0.2.0_amd64.AppImage · 94.5 MB
74b966200224068e… - Debian / Ubuntu package · x64OctopusOn.Log.Analyser_0.2.0_amd64.deb · 22.1 MB
cf20bbde96dfda22… - Fedora / RHEL package · x64OctopusOn.Log.Analyser-0.2.0-1.x86_64.rpm · 22.1 MB
54221ce58ae7b6a9…
AppImage: make it executable (chmod +x) and run it — or install the .deb or .rpm package.
Command line
| File | Size | SHA-256 |
|---|---|---|
| octopuson-log-log-v0.2.0-aarch64-apple-darwin.tar.gz | 16.1 MB | bd585fa5fdba057caefe… |
| octopuson-log-log-v0.2.0-x86_64-apple-darwin.tar.gz | 17.7 MB | 77f839e15a0e091903e0… |
| octopuson-log-log-v0.2.0-x86_64-pc-windows-msvc.zip | 14.6 MB | b832c325e8f2915fa441… |
| octopuson-log-log-v0.2.0-x86_64-unknown-linux-gnu.tar.gz | 19.2 MB | 715e530f7d8001105040… |
See your site through Googlebot's eyes
Download the app, drop in a log file and get your first report in minutes.
Download free