Webmaster Tools: Increase in server errors

Darter02

I'm seeing a large spike in errors. About 1700 of them as of yesterday. 504 errors.

Google detected a significant increase in the number of URLs we could not access. It is likely that your server had an internal error or was busy when attempting to process these requests.

tomoscott

I got the same message - 200 errors in one day!

KQuinlan

Same here.. usually have 1 or 2 server errors a day from google bot... but today google detected 1336 server errors on my site while attempting to crawl.

richW

Could one or all of you post back one of the error messages you're receiving?

ChancyRat

I got it just now as well:

Increase in server errors

Google detected a significant increase in the number of URLs we could not access. It is likely that your server had an internal error or was busy when attempting to process these requests.
Recommended action

• Check the Crawl Errors page in Webmaster Tools.
• Check your scripts and script permissions.
• Examine the log files on the server for your site for scripts or pages that might be crashing.
• Consider addressing the load on your server.

ChancyRat

Link takes me to:

https://support.google.com/webmasters/answer/2409437?hl=en

ChancyRat

Sorry, don't know what to look for so posting stuff I find:

504 Error

Googlebot couldn't access the contents of this URL because the server had an internal error when trying to process the request. These errors tend to be with the server itself, not with the request.

ChancyRat

And the server errors went from 3 per day, to 35 on 6/26, to 1,950 so far today (6/27/14).

richW

richW wrote: »

Could one or all of you post back one of the error messages you're receiving?

We changed CDN providers and Google is crawling us a lot more aggressively. Over time it should calm down.
I just received the same email from Google. I ran the 'Fetch as Google' and received the same errors. After about 15 minutes I ran the 'Fetch as Google' again and received the Green Checkmark.
I went from HTTP/1.1 504 Gateway Time-out to HTTP/1.1 200 OK

I've contacted our Operations Team and they'll be looking into the Google crawl errors too.

ChancyRat

richW wrote: »

We changed CDN providers and Google is crawling us a lot more aggressively. Over time it should calm down.
I just received the same email from Google. I ran the 'Fetch as Google' and received the same errors. After about 15 minutes I ran the 'Fetch as Google' again and received the Green Checkmark.
I went from HTTP/1.1 504 Gateway Time-out to HTTP/1.1 200 OK

I've contacted our Operations Team and they'll be looking into the Google crawl errors too.

Rich, thanks for - er, whatever you said. lol, this is what I heard:

"Blah blah science jargon blah blah calm fetch green blah will check."

Are we supposed to do anything?

LPC

Me too -

Google detected a significant increase in the number of URLs we could not access. It is likely that your server had an internal error or was busy when attempting to process these requests.

Recommended action
Check the Crawl Errors page in Webmaster Tools.
Check your scripts and script permissions.
Examine the log files on the server for your site for scripts or pages that might be crashing.
Consider addressing the load on your server.
Learn more in our Help Center.

AperturePlus

I received the same mail - 200 errors.

When I click on one of the errors, I get "Googlebot couldn't access the contents of this URL because the server had an internal error when trying to process the request. These errors tend to be with the server itself, not with the request."

RobLoud

richW wrote: »

Could one or all of you post back one of the error messages you're receiving?

I have had the same notification..... My errors have jumped from around 25 (changed folder/gallery descriptions) to nearly 500 within the last week.

Google detected a significant increase in the number of URLs we could not access. It is likely that your server had an internal error or was busy when attempting to process these requests.

Recommended action
Check the Crawl Errors page in Webmaster Tools.
Check your scripts and script permissions.
Examine the log files on the server for your site for scripts or pages that might be crashing.
Consider addressing the load on your server.
Learn more in our Help Center.

essex1970

same here:

- 788 server errors
- 88 not found
- fetching is either redirected or unreachable
- /sitemap-galleryimages.xml got warning, some URLS are unreachable.

pabloconrad

Good to know Smug is working on this.

I had over 600. Usually have none. Every link I checked works and verified via Google.

I just marked all as fixed.

pabloconrad

On a good note:

Found a typo in one of my URLs and fixed it.

bwg

Like richW said, Google is hitting us really hard right now after changing CDN providers. We throttle the traffic to bots so they don't take down the rest of the site. This will resolve itself over time, there is nothing you need to do/worry about.

AperturePlus

Thanks bwg - good to know.

What does confuse me is that some are really strange errors, like: http://www.apertureplus.com/keyword/_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;1;havelock/

I am just trying to understand Webmaster Tools - where do these searches come from?

rainforest1155

That looks more like a link formatting issue in Google. If it's indeed the link Google attempted to access, we don't have a way to tell where they got it from.

AperturePlus

Thanks Sebastian

Many, if not most of the errors, are in the form as per my above post, including many '1's... like

http://www.apertureplus.com/keyword/_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;_;1;2;agra/

Do we just ignore them?

shandrew

Google's bot/crawler problems were resolved on 6/30 8 PM Pacific. However, due to how google webmaster tools lags in how it reports traffic (especially when it hits very large traffic levels, as it did here), it will continue to report high error levels for several days until it has completed processing the data from prior to 6/30 8 PM Pacific--for some reason it records the data under the date it was processed rather than the date when the events occurred.

Unless you are very interested in google's bot/crawler activity itself, I recommend turning off Google Webmaster Tools' email notifications, as they simply aren't terribly useful--they are not designed well for large scale sites like SmugMug's. If you want free and simple external site monitoring, you can try a service like pingdom or siteuptime.

Darter02

Six days after your post (07/10) I'm still getting almost 6,000 errors/day. How long should this take again?

thenickdude

Just got this warning from Google Webmaster Tools:

http://photos.chickensmoothie.com/: Googlebot can't access your site

Over the last 24 hours, Googlebot encountered 58 errors while attempting to retrieve DNS information for your site. The overall error rate for DNS queries for your site is 81.7%.

Our other subdomains show no DNS errors, so the error seems to be due to SmugMug's DNS.

mbonocore

Darter02 wrote: »

Six days after your post (07/10) I'm still getting almost 6,000 errors/day. How long should this take again?

Hard to say...I wish we knew all the details of how Google does what they do, but of course, it's not really public knowledge. It will calm down over time, but remember, this doesn't have any effect on actual viewers. They should not be experiencing anything different.

elizab

Same problem here. Still showing ca 1500 errors per day, two and a half weeks after the problem first started. While it doesn't affect people wanting to access our sites presumably it does affect google bots crawling our sites and therefore our search rankings?



www.madaboutgreys.com

RobLoud

Not sure if you have already done this and it may be a stupid question, but have you marked all of your errors as fixed?
I initially had a high errors and even after the fix they peaked even more. I marked all of my errors as fixed and within a couple of days my errors had then gone back down to zero and I have had no issues since.



Robin

denisegoldberg

RobLoud wrote: »

Not sure if you have already done this and it may be a stupid question, but have you marked all of your errors as fixed?
I initially had a high errors and even after the fix they peaked even more. I marked all of my errors as fixed and within a couple of days my errors had then gone back down to zero and I have had no issues since.

Thanks for that tip Robin - I've now done the same and will check the results in a few days.

--- Denise

RobLoud

denisegoldberg wrote: »

Thanks for that tip Robin - I've now done the same and will check the results in a few days.

--- Denise

I hope it works for you and others. Please post back in a few days to let us know whether it works or not.

Robin

AperturePlus

It seems to be happening again, to me at least.

I have just received a notification from Google of an increase in server errors.

Anyone else?
Clive

Darter02

RobLoud wrote: »

Not sure if you have already done this and it may be a stupid question, but have you marked all of your errors as fixed?

Robin

OK, I tried that. It took a few days of marking things fixed to get to zero. They only show the first 1000 errors, so you have to wait to see the next 1000. It looked fine until today. I'm upwards of 5K errors again. It looks like the same keyword URL errors as before.

chipj

Yes, I'm seeing the same. In fact this time I have more than twice the number of reported errors than the June 27th incident. They all appear to 504 errors.

AperturePlus wrote: »

It seems to be happening again, to me at least.

I have just received a notification from Google of an increase in server errors.

Anyone else?
Clive