What is “mobile-first” indexing?
Google said they would predominantly use the mobile version of the content for indexing and ranking. Historically, they were using the desktop version when evaluating the relevance of the page and defining its rankings. In other words, whatever meta data, content, structured data, and overall set-up you have on mobile is what Google will crawl and rank. If both versions are equivalent, great. However, what about websites that use a separate (m.) website that is not linked to the desktop version?
Before we got them on board, one of our clients had a separate (m.domain) website i.e. each desktop page had an equivalent mobile version on an (m.) subdomain. Although this is not ideal, if properly linked and following Google’s best practices, it shouldn’t have been a problem for mobile-first. One of the first things Google recommends is ensuring both versions are linked with the correct rel=canonical and rel=alternate link elements so they establish the main URL and don’t see them as duplicates. Unfortunately, something our client had not implemented yet. They thought Google would kindly wait until they had made the change to flip the mobile switch.
What actually happened was a 57% drop in the number of valid pages (from 775 to 330) in Google Search Console (GSC) in November 2018. Currently, the valid pages have dropped even further to 221. As there was no notification in Google Search Console and we didn’t yet have access to their (m.) account, we started investigating what had happened overnight and why there was a sudden drop.
The below is a screenshot from their desktop GSC account, showing the drop in “valid” pages in the new GSC.
/sitemap.xml submitted to Google:
After further investigation of the “Excluded” section, we also identified 419 pages have been recently crawled but excluded from Google’s index. We have then selected example URLs from the list and used the new function “Inspect URL”, which was not giving us any further explanation why this might have happened. The pages were simply marked as “Crawled – currently not indexed”.
We couldn’t understand why Google would exclude the pages, the mobile linking has been wrong since the website was launched in 2017. Then our technical team had an idea, what about checking Google’s cache for the page, specifically checking the source code and the parsing of the URL. And there it was, the proof we needed!
The cached source code clearly showed:
1. Even when the desktop URL is requested, Google's mobile bot cached the mobile - (m.) version of the page
2. The page was crawled after 1st Nov when we suspected Google switched the client to a mobile-first indexing and we saw the drop in valid pages
3. The canonical tag is self-referencing, not pointing to the desktop equivalent, which is not following Google's best practices.
And there it was, the realisation that Google had crawled the page with the mobile bot, had seen the desktop and mobile versions and decided to cache the mobile page, while dropping the desktop equivalent from the index. But how, they hadn’t received a notification yet? Have Google run out of patience and just switched the website to mobile-first?
A few week later, our suspicions have been confirmed and the same URL returned the following results in “Inspect element” directly in GSC. “Crawled as Googlebot smartphone”, the mobile-first indexing has just landed, and the website wasn’t ready. There was still no notification in GSC!
Fixing their mDot site
The client had a long way to go to ensure optimal optimisation, however, as a first step, we recommended fixing the linking between both versions to avoid any more pages dropping from the index. More specifically, we said:
- On the desktop page, add a special link rel=”alternate” tag pointing to the corresponding mobile URL. This helps Googlebot discover the location of your site’s mobile pages.
- On the mobile page, add a link rel=”canonical” tag pointing to the corresponding desktop URL.
- Avoid using relative URLs on the mobile website as this may still point to the mobile website. Use absolute URLs.
- Google might not tell you when the mobile-first indexing has been switched for your website. Regularly check for any anomalies with your indexed and excluded pages.
- Be extra vigilant if you have a separate (m.) website and follow Google’s best practices how to best prepare your website. At a minimum, make sure your desktop and mobile pages are correctly linked.
- Sometimes GSC might experience a report delay. Use alternative methods as checking the cache to see what Google has crawled and when.
- If the traffic and index coverage have dropped, involve a specialist to perform a full audit and prepare a plan how to equip the website for the mobile-first.
The mobile-first indexing is here to stay, and Google has taken several years to carefully test and evolve their indexing to keep up with the ever-changing mobile world. If you want to maintain a healthy SEO strategy and outpace fellow competitors, you need to get there, and you need to act fast!