I fairly often get approached from website owners or business owners who are having trouble with how Google is indexing or not indexing their website(s). Most often a new website has just been launch and now they are noticing that Google isn’t seeing or displaying the site’s pages and/or posts as they had hoped. When I am asked to help get to the bottom of what is causing the indexing issues in Google I usually follow the same process for troubleshooting issues putting myself in the shoes of the Google crawlers that browse your website and determine how it should be displayed on the search engine results page.
10 Step Process for Troubleshooting Google Indexing and SEO Problems
Below are 10 different methods I use to get to the bottom of problems with how a website is getting indexed in Google. I asked my friend Betsy if I could use her new WordPress site LegosInMyLouis.com as the sample site for troubleshooting. I asked to use her site particularly because it was a common example of some fundamental WordPress issues I resolve all the time. Most of the Google indexing issues I discover on her site are common for a new WordPress installation so I know they will be helpful to many others reading this post.
I mostly freestyle the screencast videos so please pardon my lack of production preparation.
1. Search ‘site:yourwebsite.com’ in Google
I always start here by first evaluating how Google is currently indexing the website. Doing a site:yourwebsite.com search only replacing with your website URL will show you how Google is currently prioritizing all the pages on your website. The results on the first page should be your most common pages such as homepage, about page, contact page, blog, ect. If you see other pages that are not familiar or seem weird on the first page of Google search results after doing the site: search then that’s a sign Google is not correctly indexing your site.
In the video below I take a look at Betsy’s site as well as my personal site to help illustrate how to perform this check.
Note: Many of the issues I pointed out in the above video are easily fixed by applying the settings I review below in the WordPress Settings and SEO Plugin step.
2. Use WayBack Machine archive.org to Understand a URL’s History
This is a tool I use when I do not know the history of a domain name or URL and am asked to troubleshooting SEO issues with a site. The free tool takes snapshots of sites on the web over time so you can see what a site looked like years back. This helps to determine if the Google crawlers have any sort of preconception of the site currently displayed at the URL.
This is helpful especially when a domain name with history or premium domain was used to launch a new site. Google also keeps track of a domain’s history so in order for us to troubleshoot any issues the Google crawlers are having with a site we should also make ourselves aware of the history of the domain name on the site.
Below I use the example site and my personal site to show you how to tool works.
3. Run a Website Speed Test
Running a website speed test isn’t always necessary to resolve website indexing issues but is an important step for successful SEO as a whole. I’ll usually use a site speed test if I am troubleshooting Google indexing issues on a certain page or post for example. Site speed Google has mentioned is a ranking factor therefor something that might contribute to indexing issues. Since resolving indexing issues in Google is a process that takes time for the Google crawlers to return it makes it worth it to also evaluate site speed at this time as well in order to make necessary improvements efficiently.
Two Free Tools for Website Speed Testing
tools.pingdom.com Website Speed Test – I use this speed test because it has been a trusted authority in the site speed testing space for years. The results of the test are often more accurate and useful than PageSpeed Insights mentioned below. But ideally, you’d use both tools and attempt to address the bigger issues found by both tools.
Google Developers PageSpeed Insights – This is a very strict tool and it won’t hesitate to give most sites a red very bad results upon test. The issues this tool points out are meant to be used as guidelines rather than requirements. You want to be in the green on this tool but I’ve seen plenty of well-performing and properly indexed website who get a red rating on this tool so take it with a grain of salt is my advice here. Don’t let it keep you up at night.
Below is the video I illustrate these two tests on our sample domain.
4. Review the Homepage Source Code
Or in the case of a single page/post indexing issue look at the source code of that specific page or post. This is a very powerful step for me as I have been developing website for many years and can easily identify any issues in the code that may be preventing Google from properly indexing a website. For those unlike me who have less or no experience with HTML you should still attempt this step and see if you can gain any insight based on the tips I give.
To help those who have less knowledge of source code I want to narrow your focus the head section of the Source code. This is usually where most of the SEO related items are applied to the page therefor the most critical part of the source code to look at when troubleshooting indexing issues.
In the video below you can see how to view the source code and how I evaluate it checking for common problems that cause indexing issues with Google.
5. Firebug Console for Firefox Browser
As a front-end web developer and designer the Firebug tool for Firefox Browser is by far the most useful tool in my toolbox. Turns out it’s also very handy for getting to the bottom of website indexing in Google or performance issues. It’s handy because although you can view the page source as I showed above it’s also important to see how the page gets served from the server. The Source Code is the end result of what you see in the browser and then I use Firebug to dig deeper into what happens when the Google crawler gets served the page content. You get insights into any errors or items in the header, footer or page content that when loaded from the server may not be working correctly or as they should be.
Once again our sample site for this post illustrates how this is useful for solving indexing issues in the video below.
6. Homepage/Site Design Review
More often than not websites are designed with SEO as an afterthought and people come to me after launch wondering why their site is not indexing properly. These folks often want to point blame at their design or web development company for building them a site that doesn’t work right in search engines. The design or development company usually responds by telling them to hire an SEO expert or get on their SEO maintenance plan which sounds like a never-ending expense to most business owners that they don’t always understand.
Reality is that sometimes sites are designed where a certain component or section of the site is not SEO friendly for some reason and causes indexing problems in Google. Happens more often than not actually because most Web Developers are not trained in SEO or they were not paid for that part maybe. This is my personal sweet spot for troubleshooting as for most people these types of issues are very easy to overlook and difficult to identify when present. As a Web Designer and Developer, I have a ton of experience in getting to the root of these types of issues and/or avoiding them in the first place.
Below I will review the homepage of our sample site and talk over some of what I notice about the design.
7. Google Search Console (formerly Google Webmaster Tools)
Having trouble getting your website to index properly in Google? Why not let Google tell you why that is? Here is your solution: Google Search Console. It amazes me how many people approach me to help with SEO or help them fix their indexing problems yet they have not even set up Google Search Console. Although some of the other methods I describe here are often more effective for me as a whole this tool is like getting the answers to the test a lot of the time.
Like Google PageSpeed Insights, what you find here should also be taken with a grain of salt but then again it is coming directly from the source. Google crawlers are much smarter than anyone gives them credit and although issues with indexing are pointed out in GSC(Google Search Console) often times Google will adjust accordingly to take any issues into account. Meaning just because it shows indexing issues doesn’t mean you are negatively being affected by them or that by fixing them it will solve all your problems. Just to manage some expectations there.
This tool will also often tell you if the domain name or URL has any manual penalties in place by Google that might be causing any indexing issues.
Fetch as Google is a great tool within Google Search Console although in the video below I do not use it because I want to go and fix the known issues I have already identified in my process first then fetch as Google.
Our sample website owner Betsy I asked for her to add me to the Google Search Console accounts and admitted she had not even signed up or set it up yet. In the video below I take a look through her GSC account and it’s interesting to see all that isn’t identified in relation to the issues I’ve already discovered with her site in this post.
8. WordPress Settings and WordPress SEO Plugin Configuration
There are some fundamental WordPress settings and plugins I recommend in order to make sure WordPress does exactly what we want as far as on-page SEO goes.
WordPress SEO by Yoast – Watch the very last video at the end of this post to see how I configure the plugin the address the issues I’ve identified while troubleshooting our sample site in this post.
Permalinks – You can also see in the last video of this post how Permalinks should be configured.
Watch the last video of this post to see how I correctly setup these settings.
9. Google Mobile-Friendly Test
As of 4/21/2105 it’s become increasingly important to ensure your website is mobile-friendly. If you have issues with indexing your website on mobile then you should take into consideration resolving the issues identified by the tool.
10. Additional Google Indexing Troubleshooting Resources
While checking some other resources in the making of this post I kept a log for you to get lost in below. Don’t get caught up here playing with every tool and resource and just try to stick to the basics of this post. I’ll add to this list as I find additional resources.
8 Reasons Why Your Site Might Not Get Indexed – MOZ
Is Your Website Being Indexed Properly by Google? – Search Engine Journal
Solving Website Indexing Problems – Search Engine Land
Firebug for Firefox
SEOquake for Firefox
W3C Markup Validation Service
Google Search Console
How I Fix the Identified Google Indexing Issues
Over the course of this blog post, I have identified over 10 different areas with SEO/crawler issues on our sample site legosinmylouis.com that need to be addressed. I held off from fixing the issues while I was troubleshooting in the videos also contained in this post so that I could run through it all together at the end of this post in a video which is below.
Here are a few question’s I’ve gotten via e-mail that I thought I would share here for those looking for similar answers:
I don’t understand what Google means when it asks me in the beginning to “. Download this HTML verification file. [googleXXXXXXXXXXX] ?