This post is part of the weekly Q&A section. Just use the contact form if you want to submit a question.
Rick Regan asks:
I have a two-part question about how Google indexes sites:
1) Is Google supposed to index an entire site at once or does it do so incrementally? My blog (on my own domain) appears to be getting indexed incrementally, to the point where it’s taken a month to index all 30+ of my pages. Most of those pages were present before I manually submitted my URL to Google. I’m wondering if I’m doing something wrong or not doing something I should be doing. Does this have anything to do with my blog being new, or having no external links pointing to it?
2) Does Google eventually drop noindex and 404 pages? I have archive pages that got indexed but I have since added to them the “noindex” tag. I also deleted an empty category that Google now gets a 404 on. Will Google eventually remove those pages as it re-crawls my site?
I will answer to each question individually.
1) Most of the times Google will index a new website gradually, yes. That is at least what I have observed with most of my websites. The speed at which Google will index all your internal pages will depend on different factors though.
If you get some very trusted and relevant backlinks, and on top of that you also have a very efficient internal link structure, all your internal pages will get indexed fast. If, on the other hand, you have very few backlinks and a poor link structure, it might take a while before you get to see all your pages indexed.
Keep in mind that using the manual URL submission to Google will have a small impact upon the speed and breadth of your indexation. In fact many people recommend that if you want to get a site indexed fast you should NOT use that feature, and rather focus on getting some trusted backlinks to your site.
2) Yes, eventually Google will fix those issues. New sites don’t get crawled very often, so that is certainly the reason. As soon as Google finds out about the “noindex” tag, for example, it will remove those pages from their index.
Ok thank you Ed…I will wait for Google to update the snapshots then. I just think it has something to do on how the webpage was designed or coded but according to you it was Google’s behavior on taking snapshots only sparingly..:D
It takes time for Google to do updates on an individual basis. It looks for links more often than it takes snapshots. So be patient and it will happen.
hey guys i really found this one disturbing. Google had crawled my site again just yesterday and took the snapshot on my current page. It managed to retain the cached version of the page that was indexed. But the problem is the image shown in the search results still the (Blank) one or the (“Preview Coming Soon”) that i stated on my previous post. I really dont know why they cant show me the correct image of the page in the search results. Can anyone explain this to me?…i need your patience guys…:(
Who is SEO Quake? What queries do you run once a day?
Thanks.
I usually use SEO Quake because when you do a site command of that website you can sort by PR. I also set my Google to 100 links per page. I also use livepr.info but you can only run queries once an hour 🙁
Hope this helps!
Charles
You can set the visit frequency of the Google Bot in your WM tools.
Yeah, i already submitted my site to Google using (webmaster tools). By the way do we have a tool that will monitor the Googlebot’s activity (e.g crawl/index schedules for our site) aside from WM tools?..thank you
Have you submitted your site to Google?
Google still did not revisit my site up to now. They were mentioning some techniques above which i already do to no avail. Is there really an effective way to let google revisit our site and re-indexed it?…sorry if this question was already answered on previous comments but i want to be more precise and brief this time…:(
You will have to wait for Google to re-visit the site in this case.
The space is an extra character that is interpreted differently by browsers. I would use the one without the space to avoid confusion.
Im also using the one without space, though im getting “0” results in it. Does this mean that google didn’t found my links at all or had my links became invalid?…when i perform (link:www.mysite.com) on google, ive got “0” links found..
Wow good to hear that Ed 🙂 but the problem is google did not visited my site again until now. And also what is the difference between typing (link:mysite.com w/o space recommended by google) which does gave me “0” results and typing (link: mysite.com w/ space) which shows some of the links that ive already made. I have made a couple of backlinks already, does this means that the first command (w/o space) w/c is suggested by google is incorrect?
They should eventually. Google stores a page cache until it changes.
Thank you Ed for your question, you kinda remind me of my big mistake here. My site was indeed not ready yet when google managed to indexed it. When google tried to crawl my page it was still under maintenance, so that’s maybe the reason why the cached version always looked like what it is “Preview Coming Soon”..(I found this by performing cache:mysite.com). But my site was already go on live now, should the snapshots will change to the actual page?
Was the page ready when indexed by Google?
When my i check my indexed pages from google, it will return a snapshots showing “Preview coming soon”. Clicking the link will return the correct page but i wonder why google was able to display the correct snapshot of my page that was indexed and instead it only display a blank picture with text “Preview Coming soon”. Can we do something about this?
You can check this for yourself:
site:portconnectintl.com
There is no way you can control the Google Bot.
I feel our site are in google sandbox, I want to know how to get out of the effect quick, Is there tips for deal with this trouble?
I noticed weird thing. I think it depends on which google site I am querying currently. When I do site:mysite from different location like Australia, UK, India google gave me different result.
Hmm pretty interesting…..
That might happen when Google re-visits the site. There is no way of knowing what the bot might index.
I noticed google has indexed 22 of my pages but now again it came back to 4 pages. Do you have any idea why this will happen??
Thanks in advance.
It really depends on what Google would like to index. That is out of your control.
I just switched my domain on a pre-existing site over last week. I have about 500 pages that need indexing. Currently only 14 pages have been indexed, and already submitted a site map.
I’m expecting all my current pages to be updated within 2 months. I really hope I’m wrong on this estimation. But so far it seems like it’s been adding 2-4 pages a day.
…………
PS: sorry if this is duplicate post…not sure if it submitted the 1st time.
Have you tried Facebook? Not sure if it works as well as Digg.
I would like for Google to index my site more frequently. My site is already indexed and I post new several times a week. However the last time Google visited the site was 13 days ago…any suggestions on how to increase frequency? I have posted links to twitter, digg, wordpress, FB etc with no luck.
Digg is a nice way to get indexed, but you should also look at other Web 2.0 sites, such as Twitter, Hubpages, etc.
Digg has been around for a few years, so it’s not as valued as it once was.
Posting my sites to Digg always seems to short-circuit this process. I’m a little worried about the sandbox issue one poster alluded to. Should I be worried about publishing or indexing my sites too soon?
You will have to wait till Google’s next visit. You cannot tell them anything about how to do their job. They will index any or all pages on your site depending on the appropriate content/keywords they deem fit to use. A title can help with the indexing process, but it is not written in stone.
Thank you.
Hi Ed,
Thanks for the reply. Now it seems my site is indexing. But the thing is I changed some settings little bit like the way title appears and something. Google indexed my page according to my previous setting which was blog name – title.
IS there a way to tell Google re-index all my pages according to new page structure?
Many Thanks.
There a lot of things Google looks at to determine if any of your pages warrants indexing by their engine. What site are you referring to?
Good Luck.
It really depends on what and how Google decides to index.
I just build up my website and I am wondering how to increase page rank. Even if I searched using keyboards that I tagged, Google doesn’t show up my page for next 10 pages. 🙁
Any suggestion besides wait and see.
Hi,initially my website posts was indexed very quickly within hours,but now it takes many days for my post to be indexed.This all happens after i got pagerank 2
Hai Daniel,
Nice info. Digg always become my choice for fast google index!
It takes time. But Google does find everyone eventually.
Hi Juanita, don’t give up. If you already register yourself to Google webmaster tools, and google analytics, it will need a while. I also almost gave up, because i havent found any datas of my blog, but tonight, Google finally found me 🙂
Hi. Il ike your advice. I have my first blog up and running. But don’t know when google will index it. I hope it will be soon. Please visit and leave some comments and advice. Thanks
They normally do index it pretty quick. Only if there is a problem, it might take longer.
One Question i want to ask you that when i post a new post in my blog and when i search my blog with blog name i shows only few old and some time aonly formal pages it never shows latest post on search engine can u suggest me what the matter i am as begnnier wnt some tips from u r side,
Many times I observed that Google indexes pages almost instantly.
I am exploring what processes are involved in fastest google indexing
Where is your site hosted? What is the name? It can take anywhere from a day to a week or even a month for Google to index your site.
I have my website hosted a week back. i have added the google code to my website. But when i search the google i cant find my website. I have added keywords also on my site. What else do i need to do to top it up on Google. Its a corporate website hence cant host it on other websites or cross link it. Please help.
@Dave having it for sale should not matter with it being indexed. They are two different issues.
I called my hosting service and everything looked correct. I have several websites hosted with them and they all have indexed with no problems.
I am now using adwords for securing traffic. I have back links from high traffic sites and regular websites as well. We will have to wait it out and see what happens.
I still think it has something to do with having it for sale and information on the website listing it for sale.
@Dave that seems to be the problem. I would call upi the hosting service to dig into this more.
I do not pay to have a website indexed. I have had many websites indexed over the years, but this one will not index. I paid more than a normal reg. fee for the name and it has some history going back 6 or 7 years. This is one of three that will not index on Google.
I did receive a report that the Nameservers for the geoname.com that may be poisoning DNS caches. Replies from these nameservers contain bogus Referrals to other nameservers for top-level Zones. If the poisoning succeeds, future queries for names in the Zone Poisoned would go to one of the Referral nameservers instead of its truly authoritative server.
I do not know if this is the problem or what it all means.
As far as a site map, Google has to look at it first and I have a tracker and google analytics and they have not been on the site yet.
What were for sale? You do not need to pay anything to get indexed. Have you submitted a sitemap to Google yet?
For the first time, I am have a hard time getting indexed on Google.
I usually get indexed within just a few days. Both KalamazooRealEstate.com and KalamazooHomesForSale.com will not index.
They both were for sale and I did pay more than the standard rate.
Is this one of the reasons why they will not index?
They do that as there might be something in the URL or the page it connects to of interest to them and worth indexing.
Does anyone else have a problem, that Google tend to index irrelevant pages like tags and categories (if we are talking about blogs) instead of actual pages?
@Tom, getting quality backlinks like @Andrew mentioned is a good way too. It makes you more popular and so you might get more Google visits.
@Tom when you build your XML Sitemap, change the frequency of pages you update daily to “daily”. This way Google knows of it and will come to it on a daily basis. No guarantees though.