How crawl budget works – my latest SEL column
Ian Lurie Jan 6 2011
My latest Search Engine Land column just went live. It’s about Crawl Budget and how it may work. Having re-read it, though, I want to make sure I add even more qualification to what I wrote:
- I talk about crawl budget a lot, but the most important part of this article, really, is ‘making the most of your crawl budget’. That’s what Vanessa Fox refers to as ‘crawl efficiency’. If you have a huge crawl budget but 90% of it is sucked up by duplicate content and such, then you won’t see much benefit.
- Search engines apply PageRank to individual pages, not to entire sites. However, I definitely see evidence that they look at some version of average PageRank, or domain authority, or whatever you want to call it, site-wide. That appears to impact crawl budget. There may not be a causal link – it may simply be that sites with lots of pages that all have higher PageRank get higher crawl budgets as a side effect of deep links to individual site pages.
This is a very deep, geeky topic – I hope I don’t ruffle any feathers too badly, but I also hope that a lot of people smarter than I will pick up the discussion, so we can all learn something.
Chairman & Principal Consultant
Ian Lurie is Chairman and Principal Consultant of Portent Inc., an Internet marketing agency that has provided Internet marketing, including PPC, SEO, social and analytics services, since 1995. Read More