Googlebot cannot access the robots.txt.

mishenkamishenka Banned Posts: 470 Major grins
edited May 27, 2013 in SmugMug Support
I just received an email form Google informing me that their crawler encountered 5 errors while attempting to access your robots.txt in the last 24 hours. Here is the text of the email:

"Over the last 24 hours, Googlebot encountered 5 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 50.0%. "

Since I have no control over the robot.txt file I have to ask you, SmugMug Support, to clarify this unfortunate incident.

Thank you,
mike

michaelshapirophotography_problem01-XL.jpg

Comments

  • rainforest1155rainforest1155 Registered Users Posts: 4,566 Major grins
    edited December 9, 2012
    Mike,
    That's a temporary glitch that likely occurred during our last maintenance. You can ignore the message from Google as you can see that Google didn't encounter any more issues on the following day.

    These messages from Google are pretty much only useful if you host your own site to make you aware of possible issues. With SmugMug, we take care of your site and likely by the time Google emails you about anything, the issue has already been resolved.
    Now, if you'd be getting constant errors for a week or so, that might be something to look into.
    Sebastian
    SmugMug Support Hero
  • mishenkamishenka Banned Posts: 470 Major grins
    edited December 9, 2012
    Thank you.
  • EyenventEyenvent Registered Users Posts: 21 Big grins
    edited May 24, 2013
    Same here, only for three days in a row now. I fetch as google and it comes back with that error, cannot crawl. Should I worry? headscratch.gif
    http://eyenvent.com
    http://www.danielweddings.com
  • mbonocorembonocore Registered Users Posts: 2,299 Major grins
    edited May 24, 2013
    Eyenvent wrote: »
    Same here, only for three days in a row now. I fetch as google and it comes back with that error, cannot crawl. Should I worry? headscratch.gif
    http://eyenvent.com

    I would not worry about these, per Sebastian's response above.

    Michael
    www.mb-photography-sf.com
    Facebook
    Google+
    Twitter
    Photo Blog
  • FergusonFerguson Registered Users Posts: 1,345 Major grins
    edited May 25, 2013
    mbonocore wrote: »
    I would not worry about these, per Sebastian's response above.

    Michael

    I don't believe it. I just checked and for me it's been going on for 6 days, reaching 100% failure (last three days over 75% failure), and with server errors every day since late march (per Google).

    I have another thread, where I was told they are making a tradeoff -- they don't have capacity for both Google and users, so they are cutting off Google on purpose. Sounds a bit different from this answer?
    http://captivephotons.com
  • mbonocorembonocore Registered Users Posts: 2,299 Major grins
    edited May 27, 2013
    Eyenvent and Michael,

    Are you still receiving the same errors, and if so, could you please See if you can drill deeper for more information on these specific errors?

    Thank you.

    Michael
    www.mb-photography-sf.com
    Facebook
    Google+
    Twitter
    Photo Blog
Sign In or Register to comment.