Google Email

PhotoDavid78 · March 31, 2013

I got an email from google this morning that said:

http://www.davidweissphotos.com/: Increase in server errors

Google detected a significant increase in the number of URLs we could not access. It is likely that your server had an internal error or was busy when attempting to process these requests.

Can someone please let me know what this is all about? They also provided a link with tools to see what is going on but I don't want to click the link in case it isn't a real email from google.

Thanks

rainforest1155 · March 31, 2013

Hi David,

You don't have to click the link in the email. You can just log in to your Google Webmasters to view details.

That said, in most cases the things reported in these emails shouldn't concern you at all. It's most likely for some *past* temporary failed attempts by Google to access certain pages. By the time you receive the email, Google usually has retried to access those pages and got the info it needed.
The emails are only relevant to people that host their own web site and may use that infor provided by Google for doing troubleshooting on their server. That's not something you have to deal with if you use SmugMug as we take care of keeping the systems running.

Of course, if there's an issue with a specific page that's reoccurring and still happens, feel free to post details.

timnosenzo · March 31, 2013

I got the same email, and my Google webmaster tool account is showing a lot of crawl errors.

timnosenzo · March 31, 2013

3,865 503 errors to be exact.

rainforest1155 · March 31, 2013

Are there any errors from today? If the errors are from the past and are not occurring repeatedly for the same URL over the course of a couple of days leading up to today, it's most likely water under the bridge and not interesting for SmugMug users.

Any older errors are really only interesting if you were to host your own site and server.

timnosenzo · March 31, 2013

Looks like they're from 3/28

Screen Shot 2013-03-31 at 11.37.21 AM.jpg

Djm3006 · March 31, 2013

PhotoDavid78 wrote: »

I got an email from google this morning that said:

http://www.davidweissphotos.com/: Increase in server errors

Google detected a significant increase in the number of URLs we could not access. It is likely that your server had an internal error or was busy when attempting to process these requests.

Can someone please let me know what this is all about? They also provided a link with tools to see what is going on but I don't want to click the link in case it isn't a real email from google.

Thanks

I also got the same

denisegoldberg · March 31, 2013

I just received the same email from google.

Mar 30, 2013

Google detected a significant increase in the number of URLs we could not access. It is likely that your server had an internal error or was busy when attempting to process these requests.

Was there an overall smug problem yesterday?

--- Denise

timnosenzo · March 31, 2013

I just got another one. It's up to over 5,000 errors.

zacHer0 · March 31, 2013

Here is a cut/paste from one of our engineers. Sorry, this is all the info I have. We should be able to get more information for you tomorrow if you have questions.

For about one week, Google's web crawling robot has been sending
SmugMug customers over 3x the normal levels of traffic. SmugMug
restricts the level of bot traffic to prioritize actual usage, and
when bots go over a certain level of traffic we respond with an HTTP
code "503" which advises Google's robot to come back later. This
tracks Google's recommendations for responses to overaggressive bot
traffic. The downside of this is that Google's webmaster tools
needlessly alerts users that their site is responding with "come back
later". This would be a concern if it were happening with human
traffic, but with bot traffic it's not a problem.

Djm3006 · March 31, 2013

zacHer0 wrote: »

Here is a cut/paste from one of our engineers. Sorry, this is all the info I have. We should be able to get more information for you tomorrow if you have questions.

For about one week, Google's web crawling robot has been sending
SmugMug customers over 3x the normal levels of traffic. SmugMug
restricts the level of bot traffic to prioritize actual usage, and
when bots go over a certain level of traffic we respond with an HTTP
code "503" which advises Google's robot to come back later. This
tracks Google's recommendations for responses to overaggressive bot
traffic. The downside of this is that Google's webmaster tools
needlessly alerts users that their site is responding with "come back
later". This would be a concern if it were happening with human
traffic, but with bot traffic it's not a problem.

Thanks for that

Darter02 · April 1, 2013

I've had over 3,000 for the last two days.

My question is, do these errors affect Google placement in search results?

AperturePlus · April 1, 2013

I have received the same. Yesterday, I cleared about a thousand and today it is refelcting a further 145.

pgman · April 1, 2013

Wrong

zacHer0 wrote: »

For about one week, Google's web crawling robot has been sending
SmugMug customers over 3x the normal levels of traffic. SmugMug
restricts the level of bot traffic to prioritize actual usage, and
when bots go over a certain level of traffic we respond with an HTTP
code "503" which advises Google's robot to come back later. This
tracks Google's recommendations for responses to overaggressive bot
traffic. The downside of this is that Google's webmaster tools
needlessly alerts users that their site is responding with "come back
later". This would be a concern if it were happening with human
traffic, but with bot traffic it's not a problem.

The problem with return a 503 to Google is that 503 is defined as:

Resource unavailable

As far as Google is concerned, the web site is down. Try another time. Each time that Google gets this message it will wait longer until it decides not to come anymore.

A much better answer would be for smugmug to return a 509 which means:

Bandwidth Limit Exceeded

Then the website gets less penalized.

SM problems is that it cannot say to Google to crawl less often, only the registered website owner can set that, so...

jinspin · April 1, 2013

At first I saw my google page ranking dropped significantly over 3 days recently from like usual page 4-5 to 7-8 and page 7-8 to 11-12 for different keywords, So I was wondering what was wrong. And then I saw from webmaster i had over 30,000 server URL errors around same time. I even did manual Fetch of certain URL's and sometimes it would fail. I thought it was just me but from these posts this seems system wide. So stuff like this does affect your google search ranking. I am hoping they resolve this because this hurts.

pgman wrote: »
The problem with return a 503 to Google is that 503 is defined as:
Resource unavailable
As far as Google is concerned, the web site is down. Try another time. Each time that Google gets this message it will wait longer until it decides not to come anymore.

A much better answer would be for smugmug to return a 509 which means:
Bandwidth Limit Exceeded
Then the website gets less penalized.

SM problems is that it cannot say to Google to crawl less often, only the registered website owner can set that, so...

zacHer0 · April 1, 2013

I'm told that Google has just fixed the issue. Please let us know if you continue to see problems.

PhotoDavid78 · April 2, 2013

Thanks Zac

Darter02 · April 3, 2013

zacHer0 wrote: »

I'm told that Google has just fixed the issue. Please let us know if you continue to see problems.

I had an increase in server errors.

zacHer0 · April 3, 2013

Darter02 wrote: »

I had an increase in server errors.

Do you still have an increase in server errors?

Darter02 · April 3, 2013

zacHer0 wrote: »

Do you still have an increase in server errors?

I just took that screencap, so yes.

zacHer0 · April 3, 2013

Darter02 wrote: »

I just took that screencap, so yes.

Hmm, this might be a separate issue. The screenshot link below would indicate that Google fixed the problem on the 1st.

https://www.evernote.com/shard/s74/sh/765e6636-8ed8-4695-8d4e-72ba355fcd70/1e12482e222abf6a03e23c6fcb913eae

Is anyone else still seeing the increase in server errors?

jinspin · April 3, 2013

zacHer0 wrote: »

Hmm, this might be a separate issue. The screenshot link below would indicate that Google fixed the problem on the 1st.

https://www.evernote.com/shard/s74/sh/765e6636-8ed8-4695-8d4e-72ba355fcd70/1e12482e222abf6a03e23c6fcb913eae

Is anyone else still seeing the increase in server errors?

I also checked and I had server URL errors though this is less than 30000 or 25000 errors i got before.

21,259 URL errors
response code 503
detected on 4/1/13
it gave me list of top 1000 URL errors. I had cleared this list before when I got errors before.
I did a google fetch of my homepage and all links from my homepage.
I have been getting server errors a day or two after I do fetch

zacHer0 · April 3, 2013

There must be a backlog that Google is working through so it might take a little while for things to normalize.

chaselton · April 3, 2013

Glad I found this page. I just got the email, but according to the stats the detected errors stop on the 1st

LiveAwake · April 3, 2013

chaselton wrote: »

Glad I found this page. I just got the email, but according to the stats the detected errors stop on the 1st

Yes, I am still getting errors. I'm getting more like 400-500/day, not thousands like some, but my graph shows these errors steadily increasing through the most recent tally of 489 yesterday (4/2).

timnosenzo · April 4, 2013

Still getting them as of yesterday.

Screen Shot 2013-04-04 at 2.28.19 PM.jpg

Darter02 · April 4, 2013

zacHer0 wrote: »

There must be a backlog that Google is working through so it might take a little while for things to normalize.

"Backlog" eh? I don't think that word means what you think it means.

Still over 3K today... All 503s.

Most of the last days seemed to be links for

/somekeyword

If that helps solve this issue.

jamesl · April 5, 2013

Add me to the list of those seeing the 503 errors. Started on 3/30 with 2500 and on 4/4 still around 2600. It seems like this is still an issue.

Darter02 · April 5, 2013

Today I have 3,121 errors.

zacHer0 · April 5, 2013

You all should be showing up in Google searches just as usual, not to worry

.

Our engineer that looked into this says it's nothing to worry about. Google is just not reporting things as when they occurred. He believes that because of the huge blow-up on their end that they have a ton of backlog to process. The stuff i showed on the graph of googlebot traffic was only the traffic we accepted; the actual amount they tried was probably several times greater.

The alerts from Google webmaster tools simply aren't useful right now due to their bugginess from last week.

If you are technical, you can verify that you site works for googlebot by running:

curl -L -I -A "Googlebot/2.1 (+http://www.google.com/bot.html)"
http://www.ronlutzii-photographer.com/

Edit: Just realized the screenshot of the graph he gave me is broken. I asked him for an updated screenshot and will share it when I have it.

Edit: updated screenshot - https://www.evernote.com/shard/s74/sh/9e9080e6-9a77-450c-bc93-dd257ea38735/b122d3f72054d2397ebbe29fff376ed5/res/5fd382e9-f1df-41d9-87e6-0b9892570ed9/skitch.png

jamesl · April 5, 2013

zacHer0 wrote: »

You all should be showing up in Google searches just as usual, not to worry .

I understand that my existing crawled images are all working, however the errors seem to indicate that anything that has changed since the 30th is now not being indexed.

I'll check back in a few days to see if the backlog on their end has cleared.

Thanks,

James

Google Email

Comments