Google indexed pages – does Google know about your money pages?

Have you checked to see how many pages from your blog are in Google’s index? I think it is reasonable to assume that every page you have published is indeed indexed by Google and that each of those pages should be found when you search for a unique phrase from that page.

But the chances of that being the case are actually quite slim.

And while this might seem like a massive problem, when you dig deeper, it might not actually be that much of an issue.

I am doing a lot of SEO work this month. To track the progress and success of this work I first had to make a note of the current stats for the areas I want to improve. This includes, amongst other things, my Alexa ranking, daily traffic, inbound links and some data from my Google Webmaster Tools account.

One of the factors I recorded from my Google Webmaster Tools account is the number of pages from this site that are indexed and cached by Google. Three days ago figure was 133. Yesterday it was down to 129 and today the figure has dropped by another 1.

The total is now 128.

To find out how many of your pages have been indexed, log in to your Google Webmaster Tools account, click on Site Configuration > Sitemaps and read the information on that page.

This is assuming you have a GWT account and you have installed a sitemap. If not, register for your account here and get the XML Sitemap Generator plugin here.

Establishing which pages are in Google’s index

Manually establishing which pages are indexed is not a simple task. I had a look around for a free tool that might do this, but I couldn’t find anything. I eventually bought Andy Black’s Index Checker, which, for $7.00, is an absolute steal. It runs through all the pages on your server (which can be imported from an XML sitemap) and tells you if they have been cached by Google.

I was surprised, and dissapointed, to find out that 22 pages have not been indexed. And some of those were pages I hoped would earn me a commission through an affiliate sale.

This is the full list created by Index Checker

From the above list I can see a couple of problem URLs which can easily be fixed, a couple of pages I wouldn’t expect to be indexed eg search results and thanks, a few out of date pages (mostly related to Google products, such as Wave) and a couple of false positives, but I would have expected Google to have indexed and cached the vast majority of them. Especially the Privacy Policy and Terms page, which have been online since day one.

As far as I can ascertain there are three possible issues here:

1) Duplicate content

I changed domains a few months ago. I set up 301 redirects (through cPanel) from the old domain for each page. Having dug deeper it looks like there are a two that have not worked properly. I have set these up again.

2) Affiliate links

A few of the pages I expect to have been indexed have affiliate links within the body text. The links point to products on Amazon and are not cloaked in any way. This could be a problem.

3) Stray characters in the filename

I can’t be sure if the stray characters are the cause of the problem, but it does seem likely. This – %e2%80%93 – was caused by a hyphen and this – %e2%80%99 – was caused by an apostrophe.

How to get the pages into Google’s index

Now that I have used Andy’s tool and established which pages are not in Google’s index, I have to do my best to get them in, or back in if they’ve been removed. A method I tried a couple of days ago, when I realized that there were so many missing pages, was to ping my archives page using Pingomatic. This hasn’t done much good yet, but it doesn’t mean it hasn’t worked. It may take a few days for the data in GWT to update.

Instead of waiting for this to happen, I will give Google a little nudge and include (below) links to the unindexed pages I would like Google to know about. I don’t have to include links to the Privacy or Terms page as those links are in the footer (although they haven’t worked yet!).

This has been a very interesting exercise and I will closely watch the progress of page indexation in the future. If you have never monitored the amount of pages from your blog Google has in its index, I think it is worthwhile spending a little time finding out. You could be in for a shock.

Have you experienced similar problems with your your blog? If so, I would be very interested to know about your situation and what you did to resolve the problems you experienced.

Related posts: