Crawl Budget From Log Files
Finds what crawlers actually spend time on, which is rarely what you want them on.
Free — no card needed
Create a free account
to open Crawl Budget From Log Files — and the other 364 prompts across 21 categories.
Already have an account?
Running it, start to finish
- Establish whether crawl budget is genuinely a constraint for your site size.
- Compare where crawl goes against what you want crawled.
- Block the waste and fix the linking before touching sitemaps.
What you get back
The output this produces, every time.
- Says plainly whether crawl budget is even a concern, since it is heavily over-diagnosed.
- Compares actual crawl distribution against what you said matters.
- Distinguishes robots blocking from noindex, which are confused and do different things.
Getting better results
Where this usually goes wrong, and how to avoid it.
- Check site size first. For a few thousand pages this is almost never the constraint, and the time is better spent elsewhere.
- Verify the crawler is real. Logs are full of bots claiming to be search engines, and including them distorts every conclusion.
- Fix linking before sitemaps. A sitemap is the weakest of the available signals and the one most often reached for first.
More SEO prompts
All SEO →- SEO
Keyword Research by Intent
Groups queries by what the searcher wants rather than by volume, and says which you can realistically win.
ClaudeChatGPT001Open → - SEO
Topic Cluster Architecture
Designs a pillar-and-cluster structure with a build order, flagging the pages that would cannibalise each other before you write them.
ClaudeChatGPT000Open → - SEO
Page Speed Triage
Fixes what affects the metric on the pages that rank, rather than chasing a site-wide score.
ClaudeChatGPT000Open →
Written for The AI University. Every prompt in this library is original work — authored, tested and revised here, not collected from elsewhere. 365 of them, free with an account.