Why is Google NOT Indexing ALL My Blog Post Pages?

Recently I came across a question about a common problem regarding blog post indexing in Google. The business had about 70 blog posts that they thought were each on their own page through their HubSpot blog. Yet – Their Google Webmaster Tools account only shows Google indexing 27 pages. Here is how they and you can get your blog posts indexed on Google with a few best practices.

Find out if Your Blog Posts are Indexing

In order to do this, it is wise to start by getting an idea of how Google crawlers are viewing your website and blog. You can do this using what is a called a ‘search operator’ which is basically a command you can type in Google that will serve only certain results. In this case, you want to use the ‘site:’ search operator to see what pages Google is indexing on your domain.

Google Site Search Operator
click to enlarge

For example:

Try a search in Google like this:
site:yourdomain.com

You can see in the screenshot here or if you tried it for yourself how Google “sees” your website. In the particular case mentioned at the start of this post when I did a search like this for them the search engine results page was not pretty. Bunch of blog category, tag, and author archive pages showed up on the first page under the homepage. No bueno. Out of the box by default popular websites platforms like HubSpot and WordPress have this inherent issue which is easily resolved if you know the best practices and proper use of the no-index meta tag. So in the example business, I am using their indexed archive pages were being given more priority by Google than his individual blog post pages that were on the 9th or 10th page of Google results when I ran that search operator command to see. What was going on is that those tag and author archive pages, when indexed, are very rich in content and google is crawling those more often and more completely. It’s also giving them precedence in the SERPs.

What do those results look like compared to Google Webmaster Tools?

Limited data but you can see not as visual or insightful as the search command.
Limited data but you can see not as visual or insightful as the search command.

If you instead go into Google Webmaster Tools and look you can get similar data but not in the visual way you get it with the search command referenced above. Never-the-less you should do both if you are troubleshooting or monitoring your website’s indexing status.

You can understand how the search command above paints a much better picture than Google Webmaster Tools in terms of how Google is prioritizing which pages it indexes on your website and blog.

Don’t get me wrong GWT can be extremely helpful in diagnosing crawler or indexing issues with Google. You should use both tools and make sure that all actionable items are addressed so that your website and blog pages are all index as you’d ideally want them.

Why Google Hates Your Blog Post Pages

Think about it this way: Maybe you only have 1 tag and or 1 author that you put on every post. When Google indexes that tag/author archive page it’s going to have the exact same content on it as the author’s archives page or the blog roll.

  1. Blog Category Archive Pages are Indexing – Consider setting those to no-index.
  2. ‘Tag’ and ‘Author’ Archive Pages are Indexing – These are meant to be used for user navigation not so much for people searching Google. What is happening is that tag and author archive pages, when indexed, are very rich in content and Google is crawling those more often and more completely. It’s also giving them precedence in the SERPs.
  3. Duplicate Content – The above two items can result in duplicate content on your own site. Duplicate content is a topic of its own and worth mentioning here but not going into the details of that now.

Recommended WordPress Yoast SEO Settings for Blog posts Indexing on Google:

I create a 4-minute video showing me setting the default setting for WordPress SEO by Yoast to avoid these issues in WordPress:

Note: This video is not the solution for everyone but meant as general guidelines for avoiding the types of issues mentioned in the rest of this post. Just remember that your individual situation may be unique and consulting with an SEO expert is the best way to go.

How to get Google to Rank Your Blog Posts

So how does Google know which to rank or crawl more often and which does it give less priority too since both pages contain the same content? Here are some tips on how to get the correct pages ranked and crawled more often.

  1. Google Webmaster Tools Sitemaps – You should always submit a sitemap to Google Webmaster Tools.

    Larry Levenson of Sigma Web Marketing considers this  a  “great way to see what Google is “seeing”, correct 404 errors, and more. And gosh, the price is right (free)!”


    Meaning
    : Submit a sitemap and use the above-mentioned tools to correct errors.

  2. Social Signals for Blog Posts – Believe it or not Google does rely heavily on social signals for indexing and ranking content. This was a mystery for a bit but I think the overall consensus now is that social signals are a big part of helping Google crawlers do their job better and quickly index content.
    Meaning: How many likes, tweets, and +1’s do your posts have?
  3. Internal Linking – Do you have contextual internal links on the indexed posts and pages of the site that link to the non-indexed blog posts? Internal links help Google crawlers crawl your entire site and crawl linked posts and pages more often.
    Meaning: Do you have procedures to utilize internal linking properly on your content?
  4. Category or Tag Archive Pages Indexing – Sometimes those will take precedence over the individual blog post pages. Google has a tendency to prefer to rank the category page higher than individual posts contained within that category.
    Meaning: It is most often wise to set no-index, follow on category and tag archive pages as described in the video above.

Additional Resources for Hubspot

If you are using Hubspot, here is a resource to you make necessary corrections to your settings;

Additional Resource

The post above post will walk you through how, when using Hubspot, you might want to remove certain web pages from the SERPs (search engine results pages), and how to go about doing it. It does not dive specifically into how to set archive pages to no-index but that is something HubSpot’s great support team can more than likely help you with.

The Bottom Line

Following these steps will help you ensure that your blog categories, tags and author archive pages are not taking authority over your individual posts.  Avoiding a conflict for Google when trying to determine which pages to rank and crawl more often. Following the above steps will allow your blog post pages to be index better and rank higher while avoiding duplicate content issues on your own site.

I explained this to the example business I mentioned at the start of this post and here is his response:

Raleigh, Wow. thanks very much for your insights. Also, breaking down the concepts into a metaphor I can understand. I’ll have to review this with my Website guy, and if we have any other questions as we journey through this, I’ll get back to you. Again, thanks for sharing your expertise. I owe you a coffee sometime ; )

Since he sent that he has told me that he checked everything and found there is “lots to do” so it seems he is on the right path to Google indexing each one of his blog post pages with the priority it deserves.

Want to read more? Check out my other Google Indexing post: How to Troubleshoot Google Indexing Issues

Need more help?

Please feel free to seek SEO expert help over at Codeable

Hire WordPress Experts
WordPress Self-Audit Master Checklist Tool
21 replies
    • Raleigh
      Raleigh says:

      Hey Nick – Thanks, comments like that are exactly why I love what I do. Glad you found the tips in this article helpful for your blogs indexing in the SERPs! Just a few simple tricks that can go a long way for improving how Google views your site.

      Reply
  1. Mark
    Mark says:

    I’m noticing that since October my site is being crawled the same but indexing is less. have you got any ideas why that would happen? I have noticed my homepage cache is out of date by over a month which isn’t usual. Any ideas?

    Reply
  2. Charles B
    Charles B says:

    Thanks for this Raleigh. I couldn’t understand why Google wasn’t indexing my newer blog posts (older oned okay, though). Hopefully these Yoast settings will do this trick.

    Reply
      • Raleigh
        Raleigh says:

        Hey Charles – I also hope the settings work for you… I quickly looked at wordful and it seems you have some indexing archive issues that you’ve got corrected with the plugin now. Good news – I think you are getting to the root of your problem :)

        I’ve seen Google crawlers re-crawl in a matter of days yet sometimes it takes months or possibly what seems like never. If you:

        1. generate and submit a new sitemap to google webmaster tools
        2. ‘Fetch as Google’ and ‘Submit to index’ in google webmaster tools

        That might help speed things up a bit. -Raleigh

        Reply
        • viki
          viki says:

          I have recently created a blog on blogger but my problem it Google is not indexing them. I created and submitted a sitemap on webmaster. Shared post on more than dozens of website. Where as my another 5-6 day older blog on different websites like wordpress, weebly is indexed within 3 days. Should I need to change any setting in robots.txt or somewhere else. Is I am missing something in blogger? Please help!!

          Reply
          • Raleigh Leslie
            Raleigh Leslie says:

            Viki – Do you have a link to your site or is it: http://topfreeclassifiedsiteslist.blogspot.in/ ?

            I’d be patient with it as sometime it takes more time for Google to index posts. Also it’s unclear to me if you are talking about the whole site or just some or all of the posts on it.

            Also overall the site above seems very spammy to me. It’s all about link building the old way and directly violates Google’s terms of service with advice given on the site dealing with targeting sites for link equity.

            Viki I’m sorry I’m unable to help you out here more but if I was you I’d take a giant step back and and completely reconsider your entire approach. Thanks for the comment. -Raleigh

    • Raleigh Leslie
      Raleigh Leslie says:

      Looks like your site is fairly new. Sometimes Google crawlers don’t come and crawl your site everyday and I suspect in a little time these new posts of yours will get indexed.
      Few things to help:
      1. Make sure you put internal links from old articles to new articles.
      2. Fix your social sharing button icons and get more social shares
      3. Submit a sitemap to Google Webmaster Tools and check for any HTML Issues there
      4. Put authoritative outbound links in your posts linking to authorities on the subject of the article.
      5. Promote the article with seek and assist strategies where you find someone asking about tips for purchasing a laptop for instance.
      6. Quote and cite authorities in your article that will likely link back to the article from their social accounts because they were featured.

      Hope that helps. Have patience and stay focus on the next next article don’t get caught up making sure you see each one index. Cheers.

      Reply
    • Raleigh Leslie
      Raleigh Leslie says:

      In this post I actually advise people to set their tags and category pages(archives) to noindex because much of the time it prevents duplicate content especially on thin sites with not many posts in a certain category or tag. Many people mistake category and tag usage for SEO optimization which they are actually meant instead for allowing users to sort content and are not meant for showing up in google necessarily. When they are indexed you get problems like your category and tag pages out ranking your actual content pages and posts which is problematic.

      Without looking into your specific scenario I am unable to offer much help. Is your site pretty new or thin on content in various categories or tags? Did you try search the source code on those category and tag pages for the noindex code to insure it wasn’t sneaking in there if you are trying to get them indexed?

      Reply
  3. Vikesh Sharma
    Vikesh Sharma says:

    Hi Raleigh.. I have a problem with the indexing. Google suddenly stop indexing most of my new posts.. Although i used fetch as google option every time i published any post. Please suggest something. please reply me in the email.

    Reply
    • Raleigh Leslie
      Raleigh Leslie says:

      Hi Vikesh – I’d take a look and respond here but I don’t see your website URL? Also you should not need to ‘fetch as google’ each time you publish a post that’s overkill.

      Reply
  4. Marcelo
    Marcelo says:

    Thanks for the post Raleigh! I´ve tried changing the sitemap, sending the posts URLs via search console, altering the configurations on wordpress and nothing was working. Then I used the practices you showed here and all our posts were indexed in minutes!

    Reply
  5. David
    David says:

    If using the Yoast plug-in, will the old indexed tag/category pages be removed automatically after making changes in the Yoast settings, or is there more manual work that needs to be done? Taking this a bit further, I’m assuming it’s not helpful to have a category and/or tags sitemap in the search console?

    Also, I noticed a discrepancy with a page in my search console – The pages sitemap for my website shows my newer pages as being recognized and counted. However, if I input the url into the search bar in the search console, it shows some as being NOT indexed (with no reason given). Is this a contradiction I should not be concerned with, or should I investigate why the page(s) may not be indexed?

    Thanks for the help.

    Reply
    • Raleigh Leslie
      Raleigh Leslie says:

      Hey David, thanks for the question and your patience in my reply.

      To your first question, once you set the no-index setting in yoast for categories and tags, there is no more work to be done. It will take some time, 1-3+ months, for Google to re-crawl and de-index the tag/category pages if it does at all. Also yes, no need to have those cat/tag pages in the sitemap either.

      Your second question is a bit unclear. But you can have pages in your sitemap that Google decides not to index as findable via the search engine… or it can just take more time for it to recognize the pages. To get the page indexed in Google just make sure you have internal links pointing to it and maybe even some external links from other sites pointing to the page or page that links to the page you want indexed.

      That’s about all the general advice I can really give without digging in more… if you need more help I suggest asking for expert help on Codeable Thanks and good luck with it.

      Reply

Trackbacks & Pingbacks

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply