How crawl budget works – my latest SEL column

Ian Lurie

My latest Search Engine Land column just went live. It’s about Crawl Budget and how it may work. Having re-read it, though, I want to make sure I add even more qualification to what I wrote:

  1. I talk about crawl budget a lot, but the most important part of this article, really, is ‘making the most of your crawl budget’. That’s what Vanessa Fox refers to as ‘crawl efficiency’. If you have a huge crawl budget but 90% of it is sucked up by duplicate content and such, then you won’t see much benefit.
  2. Search engines apply PageRank to individual pages, not to entire sites. However, I definitely see evidence that they look at some version of average PageRank, or domain authority, or whatever you want to call it, site-wide. That appears to impact crawl budget. There may not be a causal link – it may simply be that sites with lots of pages that all have higher PageRank get higher crawl budgets as a side effect of deep links to individual site pages.

This is a very deep, geeky topic – I hope I don’t ruffle any feathers too badly, but I also hope that a lot of people smarter than I will pick up the discussion, so we can all learn something.

Ian Lurie
CEO & Founder

Ian Lurie is CEO and founder of Portent and the EVP of Marketing Services at Clearlink. He's been a digital marketer since the days of AOL and Compuserve (25 years, if you're counting). He's recorded training for, writes regularly for the Portent Blog and has been published on AllThingsD, Smashing Magazine, and TechCrunch. Ian speaks at conferences around the world, including SearchLove, MozCon, Seattle Interactive Conference and ad:Tech. He has published several books about business and marketing: One Trick Ponies Get Shot, available on Kindle, The Web Marketing All-In-One Desk Reference for Dummies, and Conversation Marketing. Follow him on Twitter at portentint, and on LinkedIn at

Start call to action

See how Portent can help you own your piece of the web.

End call to action
Close search overlay