Robots.txt Unreachable - Sitemaps

rashbrook

Not sure if anyone is still reading this thread - but on the chance that someone is -

I looked at webmaster tools again today and found that I have the little yellow triangle with the exclamation point on my sitemap again - and that I'm up to 27,000 crawl errors.

Under the sitemap section I found the following verbatim: "When we tested a sample of the URLs from your Sitemap, we found that the site's robots.txt file was blocking access to some of the URLs. If you don't intend to block some of the URLs contained in the Sitemap, please use our robots.txt analysis tool to verify that the URLs you submitted in your Sitemap are accessible by Googlebot. All accessible URLs will still be submitted." " http://www.ashbrook-photography.com/keyword/bisonProblem detected on: Feb 1, 2011"

I haven't changed a thing on my site. Per that, I would conclude that this is a smugmug issue.

OffTopic

Twoofy wrote: »

Hello,

I am pretty sure that the redirects are actually related to old keywords that were at some point deleted or changed, but not removed from the sitemaps.

Mine are not old keywords that were deleted or changed. My error list contains almost every keyword on my site. For example:

http://www.loricareyphoto.com/keyword/california


"California" is a keyword for the majority of images on my site.

More generic examples:

/historic
/empty
/dawn
/musician
/night
/north america
/orange county
/road
/lava
/alpenglow


"desert' isn't blocked, but 'desert wildflowers', 'desert aster'. 'desert bluebells', 'desert canterbury bells', 'desert five spot', 'desert foxes', 'desert globemallow', 'desert gold', 'desert mallow', 'desert pincushion', 'desert star', 'desert sunflower' and 'deserted' all show as restricted.

Looking at my Crawl Stats, while it's not at Zero like it was with the December problem, it is barely in single digits. At the end of January I had an average of 3,000 pages crawled per day and then it dropped like a rock on about January 31 and is barely a blip now.

jachang

My site map not working any more
(Sorry--I just realized I put this in the wrong forum. It's supposed to be under Support.)

I looked at my Google Analytics account last week, and saw that there were several errors on the site map. I resubmitted everything, but I still have the big red X's beside sitemap galleries and sitemap images.

I checked for crawl errors, and there were over 31,000 errors for "url restricted by robots.txt".

What's going on?

I haven't done anything on my site, no changes in anything, so I don't know why this happened. Can anyone help?

Thanks,

Jean

jachang

I'm getting the same problem. I have over 31,000 crawl errors, saying "URL restricted by robots.txt." I couldn't go through all 31,000 of them, but the first few thousand are all from January 30 and 31. When I look to see the crawl stats, they have dropped down to zero as of January 31.

What's happening?

I have red X's on my sitemaps, and it says:

General HTTP error: 404 not found
We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit. I resubmitted, and get the same errors.

Please let me know how I can fix this. Thanks.

Jean

Andy

Thread closed please do ask questions here
http://www.dgrin.com/showthread.php?p=1554783#post1554783