If you were given the opportunity to witness a Ferrari being made, given an inside look into making one of the most amazing machines on earth. And then after you got this inside look at how this incredible work of craftsmanship was made, and you then were able to speak at a conference for car mechanics and Ferrari enthusiasts, what would you talk about? You could talk about the hand stitched italian leather, the near perfectly tuned v-12 engine that produces 800+ horsepower, or the aerodynamic design that beautifully mixes engineering and style. There are a lot of factors that make a Ferrari, well a Ferrari. And when you could talk about any of them, would you choose to talk about the sourcing of materials?
I recently attended BrightonSEO, where we heard from an ex-Sr. Google Engineer named Fili Wiese, who, while at Google, worked on the documentation for manual action penalties. Which means, Fili could have talked about anything related to SEO, but he chose to talk about a subject that, in my opinion, is carried over and will be carried over as a top ranking factor for all of Google's existence: The ability for Google’s bots to crawl, fetch and index content from a site efficiently.
At first glance, that sounds like the least exciting topic he could have picked. It's like passing over the V-12 to talk about where the aluminum comes from. But that's exactly the point. Ferrari can stitch the finest leather and tune the engine to perfection, and none of it matters if the materials going into the car are flawed. In SEO, your content, your links and your on-page work are the leather and the engine. Crawling and indexing are the materials. If Google can't reliably reach your pages, read them and store them, nothing else you've done gets a chance to count.
Googlebot believes what your server tells it
Fili's session was centered on HTTP responses, the status of a page your server sends back every time Googlebot requests a URL. Most of us only look at these when something breaks: a 404 here, a redirect there. His point was that Googlebot reads every one of them and adjusts its behavior based on what status is returned. Your server is constantly telling Google which pages exist, which have moved, which haven't changed and whether the site can handle more requests. When those messages are wrong, Google acts on the wrong information.
A few common examples:
- A 200 OK on a page that says "not found" or shows an empty category. Google has to work out on its own that the page is a soft 404, and it spends crawl activity on URLs that should have told it to move on.
- Redirect chains, where one URL redirects to another that redirects again. Every extra hop is another request before Googlebot reaches the content, and chains tend to build up quietly after migrations and URL changes.
- 5xx errors and 429 responses. Googlebot reads these as a sign the server is struggling and slows down to avoid overloading the site. That protects your server, but it also means new and updated pages get picked up later. If the errors persist on a URL, it can eventually fall out of the index.
- An unreachable robots.txt file. If Google can't fetch it because of a server error, it may hold off on crawling the site and eventually de-index your entire site, rather than risk crawling something you meant to block.
- No support for conditional requests. When a server returns 304 Not Modified for pages that haven't changed, Googlebot can skip downloading them again and spend that effort on pages that have.
Crawl efficiency can give you the competitive edge
Google will not spend unlimited time and resources crawling your site and making sure it catches ever page. So it is important to take advantage of every crawl request received from Google and make sure that the Googlebot is seeing only the most important content on the site. Every ranking depends on Google having a current, accurate copy of your page. A product page recrawled two weeks after a price change, a new article that sits undiscovered for days, or a page that keeps dropping out because of intermittent server errors will all lose to competitors whose sites are easier to crawl. That's why I'd argue crawl efficiency has been a top ranking factor from the beginning and will stay one. The algorithms on top keep changing. The need to fetch and index the page first does not, and the same is true for the AI crawlers now feeding AEO strategies.
Where to look on your own site
Fili's advice for finding these problems came down to two sources: Google Search Console and your server logs.
In Search Console, the Crawl Stats report (under Settings) breaks Googlebot's requests down by response code, file type and purpose, and flags host problems like robots.txt fetch failures and server connectivity errors. A growing share of 5xx or redirect responses, or a sudden drop in total crawl requests, is worth digging into. The Page Indexing report then shows which URLs Google crawled but chose not to index, and why.
Server logs show what Googlebot actually requested, not what you assume it requested. Running them through a tool like Screaming Frog's Log File Analyser shows which sections get crawled most, how often key pages are revisited, and where Googlebot is spending requests on parameter URLs, redirects or error pages. Comparing that log data against a regular site crawl is usually where the surprises turn up: important pages Googlebot rarely visits, and low-value URLs it hits constantly.
Back to the Ferrari
Nobody buys a Ferrari for the quality of its raw materials, but every part of the car depends on them. I often say that you could have the best content in the world, but if Google can’t see it,that content will never be shown in search results, and the chances of the intended user finding it are very low. Your site works the same way. Content, links and brand get the attention, and they should. Underneath all of it is the question Fili spent his whole session on: when Googlebot asks your server for a page, does it get a clear, honest and fast answer? Get that right, and everything else you build has a chance to perform. Get it wrong, and you're fitting hand-stitched leather onto a car with a cracked frame.
