This post is part of the weekly Q&A section. Just use the contact form if you want to submit a question.
Rick Regan asks:
I have a two-part question about how Google indexes sites:
1) Is Google supposed to index an entire site at once or does it do so incrementally? My blog (on my own domain) appears to be getting indexed incrementally, to the point where it’s taken a month to index all 30+ of my pages. Most of those pages were present before I manually submitted my URL to Google. I’m wondering if I’m doing something wrong or not doing something I should be doing. Does this have anything to do with my blog being new, or having no external links pointing to it?
2) Does Google eventually drop noindex and 404 pages? I have archive pages that got indexed but I have since added to them the “noindex” tag. I also deleted an empty category that Google now gets a 404 on. Will Google eventually remove those pages as it re-crawls my site?
I will answer to each question individually.
1) Most of the times Google will index a new website gradually, yes. That is at least what I have observed with most of my websites. The speed at which Google will index all your internal pages will depend on different factors though.
If you get some very trusted and relevant backlinks, and on top of that you also have a very efficient internal link structure, all your internal pages will get indexed fast. If, on the other hand, you have very few backlinks and a poor link structure, it might take a while before you get to see all your pages indexed.
Keep in mind that using the manual URL submission to Google will have a small impact upon the speed and breadth of your indexation. In fact many people recommend that if you want to get a site indexed fast you should NOT use that feature, and rather focus on getting some trusted backlinks to your site.
2) Yes, eventually Google will fix those issues. New sites don’t get crawled very often, so that is certainly the reason. As soon as Google finds out about the “noindex” tag, for example, it will remove those pages from their index.
That is probably the best way to check. If your site is new Google does not visit it so often.
Get some good quality links and that should help. If you keep adding content Google will return more often.
I want Google to index my blog more frequently. My site has been indexed and I post new content every day. However the last time Google visited the site was 5 days ago.
I use the site:URL command to check in Google if my new content has been indexed. Not sure if there is another method.
@JohnMenskil I know how to do that. I was not the one looking for that information anyway. I use Google Webmaster tools for that.
@Edward I’m guessing you already figured this out, but there are a few ways to learn how many of your pages are indexed by Google. I’ll mention a couple:
1) In Google’s search page type in “site:www.domain.com” The results will show you how many pages have been indexed by Google.
2) Use websites such as Alexa and Web Grader to find out how many times Googles has indexed your website’s pages.
If you use wordpress, the headers on the pages can be used as meta descriptions too.
That is called your meta description. You can change that within your website controls. If you don’t google will just take part of your page. You have about 150 characters and its also a good idea to use your keyword in there.
how do I get google to update the cached information on my website? I am talking about the description of my website found in the search results of google.
Only Google can answer that question for you Paul G. They index pages that are of interest to them. Depends on Keywords you use.
Some great ideas here.
Googlebot visits one of my sites everyday but does not cache it very often. It seems that even though the bot is crawling the site they are not indexing the pages. Is this only done on the days that they cache the site?
Is this correct?
If anyone can help I would appreciate it.
Indexing can take a while as Google has billions of web pages to crawl. But submitting an XML sitemap does help as I have seen them download it in a few minutes after submitting it to them.
good thoughts but I still have problem with Indexing my my website on google 🙁
It can take several days or weeks for Google to index your site. Have you submitted it to Google using an XML sitemap also?
Thanks.
I’m having the most frustrating time trying to get my site indexed! GRRrrrrr!
I have a 30 page website built on x site pro 2 full of unique content, i have at least 200 backlinks, I have my articles and videos popping up on the front page of google from sites like hub pages, squidoo, scribd, youtube, metacafe, dailymotion, veoh, go articles, article city, digg, reddit, stumbleupon…. and many more; however its been 3 days since i finished my site and published it to the web but its still not indexed! Grrrrrr!
I know its not indexed because when i use google page rank tracker it says its not indexed!! I have always used this tool and I know it works, so why so long this time google???
Does anyone have any idea why its taking so long??
You can check for yourself, pregnancy-central.com (make sure you include the hyphen!)
Always better to not put it under maintenance for a while as you never know when the bots will arrive. Use a web directory submission software to get more backlinks.
My website markmystep.com got listed on the Google fairly quickly. But when I started developing the website and put it under maintenance, it somehow got lost and is not showing up on Google search.
I have put it online now and still it is not showing up on search results. As per your suggestions, I am now working of creating some backlinks. Hope this works out well.
Thanks for your post. If you could provide any other suggestion, which would speed up the process, that would be helpful.
Another way is to try to submit to as many web directories as possible.
I have set up several different sites and is often an annoyance to get them listed up quickly I agree with everyone else though the fastest way is backlinks usually asking a couple of friends to link to me in there blogs and such gets your page up a lot quicker especially if they are a regular blogger.
I’ve found submit xml sitemaps helps but I experimented with one of my domains and the backlinks had a much quicker effect.
The idea of going with tools other than direct Google submission seems to make a lot of sense.
We are just starting out and will look into backlinking and possibly creating sitemaps for our sites.
GVMarketing
How do I find a list of my indexed URLs by Google?
Thanks.
Backlinks means high traffic sites having a higher page rank than you linking to your site. Simply means they have your URL somewhere on their site pointing to you.
Yes, Make sure you dont have major errors in your code, validate your html, link building will also help.
What does everyone mean by backlinks?
I’m a newbie so give me some leeway
Hey,
That’s a nice piece of info….
Is there a specific time frame that Google follows for refreshing the Google’s search positioning… If yes, please let me know about it.
Thanks, guys, there’s some really helpful stuff here. I will try it out and report back what works for me.
I also submit sitemaps to Google, Yahoo and Live.I also use Google Webmaster Tools. I have some anecdotal evidence that when you have errors on your site like duplicate titles and meta-descriptions that this seems to have an adverse affect on the pages that are indexed.
since last one year i build this web site,and also verified by webmaster,after this i can’t understand after the expiry one year my web is not working properly,being an affiliate with click bank,having nickname and with other sites,my web site makes not a single dollar,what is the short falls in my site i could not under stand.please help to make this site in full functioned
My site is already indexed by google. When I serach using site:mysiteurl, its ok, but when I google link:mysiteurl it does not show eventhough I have done alot of backlinking. Why?
in this month i lost half of my google index and that was suck, i don’t know what happen …
is google filtering the site content? and deleting the duplicate index?
Thanks alot, this article is definately worth taking note of.
Socially bookmarked to bring awareness 🙂
Karl
yes dear this is true .
Good backlinks from google trusted sites always will get your site indexed more quickly. We have had performed many experiements to find out what gives the best approach.
Good backlinks from google trusted sites always will get your site indexed more quickly. We have had performed many experiements to find out what gives the best approach.
(1) manual site submission – can be slow, google will typically send the bot round within day to check that the site has indexable content, and returning a 200.Then over a period of weeks gradually more pages are crawled and indexed.
(2) Links on High PR / High volume sites ( we believe that google puts more priorty on traffic that is flowing to the site rather than page rank). Typically you get index inclusion after a couple of days.
(3) Now here is the 24 carret nugget, Site registration period – pay a bit extra and get your domain registered for 10 years and you will get almost immediate indexing. Google have it in their patent – just think about it – with yearly domains around $3 for bulk registrations, google sandbox yearlings but mark 10 year regies as more likely to web sites with conviction behind them rather than spam / squeeze sites. We registered WebTrafficAgents.com and on the second day were at page 4 of 27Million results.
Now Google put the new site in the “Sandbox”. Hence it takes much time to index. Better to use the XML sitemap for gradual indexing.
Putting a sitemap in Google’s webmaster tools also speeds up inclusion. Well worth the (short) time it takes to set up there.
YES,backlinks are very important to a new blog.
Also you will be indexed quickly by adding blog to some social bookmarks.
I agree with the trusted backlinks approach. It’s probably the fastest way to get your domain indexed. Aged domains are also good if you can get them.
Daniel,
i have a new blog and hope that google will index the domain soon before i put more post. Yes, i am working on link building. Thanks for your articles. Useful for me.
Good 2 know you
Aiko
Very Good Post.
I have a number of websites that I have created and had indexed quite quickly.
I use a few different methods to do this. Submit content to sites that Google loves like ezinearticles and digg.
I also submit sitemaps to Google, Yahoo and Live.
I also use Google Webmaster Tools. I have some anecdotal evidence that when you have errors on your site like duplicate titles and meta-descriptions that this seems to have an adverse affect on the pages that are indexed.
You can also use Google Webmaster Tools to see how often Google crawls your site.
On a somewhat related note, I’m really confounded about the disparity of the search results between Google, Yahoo, and Live (MSN). I submit sitemaps to each of these and they are crawling my site, but the return can be radically different.
For example, if I use the image search feature of each to search for “site:www.2dolphins.com logo” I get:
Live – 112 results
Google – 72 results
Yahoo – 22 results
And the funny thing is that there’s very little overlap – not many of the results from Yahoo also show up in the Live image search results.
Yes, #2 is usually just a matter of time before it all gets cleaned up. You just have to be patient because there’s nothing you can do to speed up the process. Consider it a lesson learned I guess!
if we have used manual URL submission, what should we do in order to be better indexed by Google?
@Cristian: Webmaster tools says that tool is for “urgent” removal (mine was not urgent – just wanted to remove duplicate content). They recommend noindex or robots.txt otherwise.
@Calvin: I have a similar problem. The number of URLs Webmaster tools says it’s indexed from my sitemap doesn’t match the URLs returned in a “site: ” search for my site. I don’t know why.
Wouldn’t a site map help? If you loaded a sitemap xml from day one, wouldn’t that help Google easily find all your pages?
Daniel,
A number of my sites seem to have dropped off Google. Even when I use “site:mysubdomain.mydomain.com”, Google says it can’t find anything.
But when I look at Google Webmaster Tools, it said my sites are in the index.
Any idea what’s going on?
Excellent article…
Just DIGG it…..
I’m glad I stopped by today. I’ve always been curious about the “noindex” tag. I’ve been meaning to research this, but I’ve procrastinated this forever. Maybe I knew I would stop by Daily Blog Tips and find the answer. Thanks!
Rick: For question #2, Google Webmaster Tools now allows you to submit URLs for removal. I submitted a set and had them all removed within 4-5 days.
I think the best way to let google index your sites quickly is submit your web pages to some social bookmark sites, like digg, mixx, reddit, and then you can get a lot of traffic.
@Matej, yeah social bookmarks that pass link juice are a good option to get indexed.
Daniel, thanks for answering my question.
In the time that has passed since I asked the question Google did eventually index all my pages (well, all but one for some reason) . And yes, the noindexes and 404s eventually got removed as you said they would.
I have a few backlinks now and I see from my stats that Google crawls my site more frequently now. Most of my posts seem to get indexed within an hour of posting.
Usually my sites get indexed in a couple of hours. I usually submit my site on Digg with the exact keywords. It helps me to get traffic ( google-digg-my site ) and I get indexed really fast.