I’ve published some content before on how to fix issues with how Google is crawling and indexing your site. From those older posts, I get people who reach out to me and ask for me to help them solve the particular indexing problems they are experiencing with Google. The following case study is an example of one of those e-mails I received and decided I would help at no charge if I could share the results of my troubleshooting with my audience on my blog and they agreed.
Below I’ll share the correspondences and troubleshooting that took place eventually leading to a solution for this company to fix their indexing issues and start focusing back on their content marketing.
To keep this post simple I am merely going to post the e-mail thread below with some correspondence omitted that is unrelated or confidential.
Here is the e-mail that initially showed up in my e-mail box:
I hope this email finds you well. First, let me say Im a big fan of your website. Secondly I found your post about https://raleighleslie.com/google-not-indexing-all-blog-post-pages/ very informational and easy to read.
Thank you for taking the time and posting this. About 3 months ago we launched a new website with the help of a 3rd party company. Since then Iv been studying SEO and noticed that google is only indexing a few of our webpages and not all of them.
When I reached out to the company they said it was “googles fault”. Even when I search for exact phases from my blog they are not showing up. I was wondering if you could look into it for me and to see if it is how the website is designed or if it is in fact “googles fault”.
Our website company has since tried to put me in contact with their “seo specialist” or to sign up for their “maintenance plan” however I think there is something bigger at hand here and think it’s the way the website maybe designed. At the end of the day I may hire a firm to help us with SEO but for now I am simply just trying to have all of my pages indexed which should be standard and easy…
I greatly appreciate any insight or input you can share with me. Thank you!
to which I responded:
Hey Steve –
Thanks for reaching out. So glad to hear when people get value from my blog!
I don’t have time until later this week to look into this further for you but I was curious and took a quick look around.
Looks like a really nice website that company built for you. Well designed and developed.
As far as SEO for that site, they’ve got you setup on a great foundation but they are right in recommending you work with an SEO specialist to achieve your specific goals. I also commend you though for self-educating yourself enough thus far to know what you are after it seems.
I’d be happy to pop in and take a look at the back end of that site if you want to send me WordPress logins. I’d also be willing to use it as a case study for a blog post on my site, making it anonymous if you want, so that I could help you through the indexing issue you’re having. Particularly I want to look at the Yoast WordPress SEO settings and theme settings.
On quick look: I noticed that specifically your blog/news post pages are what are not indexing. I looked and didn’t see the specific nofollow or noindex code that might cause that so I didn’t immediately find a switch in the code that would resolve your issues.
Have you setup Google Webmaster Tools? Have you submitted your sitemap there?
Are those blog posts unique content are are the copied or duplicate somewhere else on the web? You or someone else writes them all?
Your company does not maintain any social media accounts? If you posted your blog/news content there that would be a helpful signal for the Google crawlers to come by as well…
What specific keyword terms are you most concerned with? Seems like you’d just like your blog/news titles to show up in Google which should be attainable…
I think you’re right though about not needing an SEO maintenance plan but instead just to resolve the immediate indexing issues you’re experiencing. Then if you keep up the blogging and keep doing a better job there and with the rest of your content marketing that should be enough as long as you have the little things tweaked right to start.
I hear ya… hard to investing in creating more content when the posts are not even indexing in Google…
Let me know how that sounds to you.
Cheers – Raleigh
Steve was very appreciative and responded with this…
That would be awesome and would like that as a case study!
My wordpress is: […]
I do have a webmaster tools and submitted our site map on there. I have recently disabled the yoast wordpress and tried the google sitemap plug in. ( still no luck )
These blog posts are all unique and I wrote them all with the help of a copy editor.
We do in fact have social media, linkedin, facebook and google+ that I post on and link back to website.
I’m not so concerned with keywords… I’m more concerned with adding content that isn’t showing up on google. These blog posts are typically 500 words with all unique information.
Please let me know if there is anything additional I can provide you. I would love your insight to what could be the issue! Thank you very much and I look forward to speaking further and hopefully one day reading a blog about this.
After taking a look around the site a bit deeper I came back to Steve with this:
Perfect that worked. Everything in there looks pretty good.
Would you be able to give me a quick timeline of what you’ve done since the site launch 3 months ago briefly?
I did notice one thing that could be part of the issues:
When the google crawler hits your site and starts crawling the only links to your blog posts exisit on this page:
But you are using a Ajax post loading plugin that generates the content dynamically. So when I look at the source code there is no actual links to the recent posts re:
So even though you are posting links to the posts on social media and are submitting sitemap to Google Webmaster Tools the crawlers may still be having trouble crawling your blog roll if you use that ajax plugin…
The fancy loading is cool but could be a factor in the indexing issue you are having. I recommend ditching that plugin to eliminate it as a possible cause for the problem..
Let me know what you think. I gotta jump on some other work today so I’ll get back to this later.
Steve came back to me with this helpful information:
So we launched the site 3 months. Besides regularly posting to the “blog/news” section I also added meta tags and keywords for the yoast SEO plugin. All of our posts are also posted on linked in, fbook and google+.
I submitted site map to google and did google fetch about a week after the launch.
About 30 days ago I noticed that sections of the website were not showing up on google so I resubmitted to google.
2 weeks disabled the yoast plugin and tried the google site map plugin.
Since then I have made 1 or 2 more blog posts.
Other than that I didn’t do anything else. I’m not smart enough to be dangerous when it comes to website design and am not familiar with the ajax plugin. I would be willing to disable it but I guess I would like to see what that makes the website looks like first? Or keep the look of it and use a different plugin?
Our market is a niche and our competition does not focus on their website. I think we could broaden our reach and land more clients if I was able to leverage our blog posts increase our ranking for various keywords. I think without google indexing our blog posts the blog is pretty much useless.
I greatly appreciate you checking this out and please let me know if there is anything additional I can provide! Thank you
My response to Steve:
Thanks for the breakdown that helps a ton.
Honestly you are doing a great job thus far doing everything you can. Things take time especially with Google most people like me will tell you any changes you make could take up to 3 months to have an effect.
I could not encourage you enough with pushing forward with your content marketing strategy as you’ve described. But you’re right you definitely needs to fix the indexing issues.
I recommend you discontinue the use of that Ajax loader plugin to display your blog posts. They also offer an SEO add-on for the plugin that might help but deactivating the plugin and having the post load normally would be the best bet toward better indexing sooner. The add-on is $25 bucks and might be worth it to try right away.
You might need a developer to complete that work as it might not be as simple as deactivating the plugin or adding the add-on as the development company might have coded it into the theme.
Thanks – Raleigh
Steve then reported back to me with:
Awesome! Thanks for the information. Let me touch base with our company president and make him aware of the current situation. We may go back to our website designer and give them an opportunity to fix the issue.
I also want to touch base with him about our marketing budget and share with him the importance of hiring an SEO specialist who can really help us out. I will share with him your hourly rates and see if we can move forward on doing some work together.
Thank you again for your insights!!
Hey thanks for reaching out Steve. Glad I could help.
Let me know if you have any questions in the meantime but I honestly think the root of the issue is with the Ajax plugin and would love the storybook ending to the case study blog post to be true haha.
Thanks – Raleigh
After a little time I followed up with Steve to see what had happened since I rooted out the issue:
Just took a quick look at the source code for your blog and it appears you were able to fix the issue and the post teasers are now showing up in the source code. That should help the rest of your blog show up in Google over the next few weeks I’d suspect.
Did you just install that Ajax SEO add-on for that plugin or can I ask what other work was performed?
Thanks hope all is well.
He sent this response back to me:
Good Afternoon Raleigh,
As my website design company was poking around trying to figure out the issue here is what they said…
“I updated the Ajax Load More plugin, and added the “Preload” option, which preloads a specific amount of posts before sending any ajax requests. This allows google to see those preloaded posts. I tested on Google Web Master Tools and had a positive result.
So right now you have 20 posts preloading. If you have more than 20 posts in WP (not the case right now) the other posts will be loaded with the click of the button at the bottom of the page that says “Old Posts”.
I did the same thing for Project posts.
This should encourage google bots to index these posts. However, this doesn’t explain why google would not index other pages such as the About page. That’s a regular page without any ajax. So, I fetched and submitted each page to the google index.
You can search google for site:watchdogpm.com to see the results.
You had also asked me about Archived Projects. Those pages actually exist. Archived Projects are just projects that were marked as archived. WordPress automatically creates the pages URLs. Two solutions:
1. Add a redirect, so that if someone somehow lands there (there are no links to get to them) then they will be redirected to the main projects page.
2. Add a message such as “this projected has been archived. click here to to go the main projects page”.
The “Ajax Load More” plugin we use, has an SEO AddOn, which we recently purchased. I will install it once I have done enough testing with it. “
Not sure what else they did but it looks like they are now showing up on google which makes me happy.
On a separate note how many hours would you expect to do a SEO audit and to fix any issues we may have? I am trying to rank higher for a couple of words in a niche market… Much appreciated!
And from there I considered the issue resolved on my end and responded with this last e-mail before publishing this post:
Ah yes I saw that option to preloaded some posts too and thought that might be a good workaround solution that might do the trick.
The only problem with that is the button to ‘load more posts’ is not in the HTML still as are the post teasers now. So Google still might have trouble crawling past those pre-loaded posts if you know what I mean as it will never see the load more posts.
The SEO add-on plugin fixes another common issue not as directly related to the issue you were having but I would definitely recommend they install it once they feel comfortable with it for you.
I’m working a productizing a service right now which is exactly what you’ve asked for and I get all the time. An SEO audit product/service where I point out the issues and solutions with a little focus defined by the clients. I’d be interested in hearing from you what would be the most value for that in terms of how I packaged that service if you have any input.
Thanks for the insight. Hope to publish that post soon.
If you would like to know the process I use from troubleshooting Google Indexing issues just follow that link to another blog post I published.
Please post comments or questions about the context of this blog in the comments section below.