Printable View
Another search engine question.
I see Google is downloading our XML sitemap every day.
Google reports it has indexed 2,198 of the 3,533 pages in the sitemap.
We're adding dozens to 100+ pages a day to our site.
So, they're grabbing our sitemap everyday, but it takes awhile to get the URLs actually in their index.
Would anybody care to guess how long it typically takes to get a page that has been downloaded (in the sitemap) in to the index?
Thank you SEO gods!
It doesn't have so much to do with Google downloading the sitemap. It's the spiders crawling your site. Each visit they may only crawl a few pages. They don't crawl the entire site at once to avoid slowing up your site and sucking up bandwidth.
It depends on the site. Some sites are only visited once a month. Sites with a higher PR or larger number of pages will get visited every few days. Super high authority sites (ESPN, USAToday, CNN.com, etc.) will get crawled every few hours typically.
Thanks as always Mike.
Ok, I went back in to Webmaster Tools, and found this.
Pages crawled per day
1,074 High
376 Average
84 Low
It looks like the average for July was around 500 pages per day crawled. Hmm, but I don't know what they're crawling, might not be the new pages I guess.
So, Google uses three steps?
1) Download sitemap,
2) crawl site,
3) add to index
It looks like there is a time gap between crawling and updating the index. Sound right to you?
Ok, I understand, patience is my only option. Will reserve judgment on our new content for awhile.
You can speed up the indexing process by building backlinks to those new pages or by bookmarking them at sites like Digg, Reddit, etc.
I can see how backlinks might speed up crawling, but given that Google already has the URLs, would backlinks speed up indexing?Quote:
Originally Posted by MikeF421
It's not a big deal really, I was just curious to learn a bit more about the process. You know how it is, you're curious if something you're doing is working, and there's no way to know until your pages get indexed.
I guess the solution is...
Back to work. :)
The theory is that if the spiders see other places pointing to a site, they'll think, "Hey this might be something we want to check out."
I do it all the time to speed up indexing of my sites, but I've never worked on a site with 8 billion pages like yours, or a site that was adding so many new pages so often.