I'm getting Webmaster URL "server errors"

ChancyRat

I just got a Google alert about a significant number of URL errors.
Most of them appear to be keyword-related, and they're all dated 10/18/15.
This is the 854 in the "server error" tab (image).

The 108 in the "other" tab are also from 10/18/15, and are also related to keywords.

When I click any of the URLs, I'm taken to the keyword results page for that keyword, as if there is no problem with the URL.

(The "not found" tab, those are predominantly from my renaming a Folder URL. :cry)

Not sure what to make of this?

ChancyRat

The reason I'm asking about this is that I made no changes to how keywords are organized or coded on my site. So this sudden set of URL errors, out of the blue, doesn't this speak to something behind the curtain with Smugmug?

owilliams

Hi Gwen,

Thanks for getting a hold of us. I've had a look at your site (www.joinrats.com/) and I'm not having any trouble with it on my end. Everything with your domain seems to be acting just fine. Do you have any more information about the trouble you are seeing there? Is this a google webmaster alert? If you could reply back to us by email (help@smugmug.com) we'll be happy to get you connected over with one of our domain experts here at SmugMug to help get a better idea of what is going on.

Best,
-Oliver
SM Support

ChancyRat

Whoa the numbers of errors are climbing
Oliver, thanks, but some - thing - is wrong.
Now at 5,772, from 800-something, in 3 days. :cry

Any other SM sites with this kind of problem?

Appear to be all keyword urls.

Desktop:
"Errors that occurred when your site was crawled by google bot".

Other:
"Google was unable to crawl this URL due to an undetermined issue."

One example of the details from one of the problem URLs:
Last crawled: 10/17/15First detected: 10/17/15
Googlebot couldn't access the contents of this URL because the server had an internal error when trying to process the request. These errors tend to be with the server itself, not with the request. More info.

# If you're reading this, you belong at a job you love: http://www.smugmug.com/jobs/  # See http://help.smugmug.com/customer/portal/emails/new if you'd like to apply to be whitelisted for crawling SmugMug  User-agent: Adsbot-Google Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: Alexabot Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: archive.org_bot Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: baiduspider Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: bingbot Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: BingPreview Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: ccbot Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: CloudflarePrefetch Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: facebookexternalhit Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: FriendFeedBot Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: FunnelBack Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: Google Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: Google-Site-Verification Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: Google-Sitemaps Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: Googlebot Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: Googlebot-Image Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: Googlebot-Mobile Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: Googlebot-News Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: Googlebot-Video Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: gsa-crawler Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: houzzbot Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: ia_archiver Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: Mediapartners-Google Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: msnbot Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: Pingdom Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: rogerbot Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: ScoutJet Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: Slurp Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: Spinn3r Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: Teoma Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: Twitterbot Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: Yandex Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: Yeti Disallow: /date Disallow: /popular/ Disallow: /admin/ Allow: /api/developer Allow: /api/doc Allow: /api/v2/doc Disallow: /api/ Disallow: /cart/ Disallow: /hack/ Disallow: /gallery/ Disallow: /search/ Disallow: /services/api/json/ Disallow: /services/api/php/ Disallow: /services/api/rest/ Disallow: /services/api/xmlrpc/ Disallow: /test/ Disallow: /go/   User-agent: * Disallow: /  Sitemap: http://www.joinrats.com/sitemap-index.xml

ChancyRat

Yes, google webmaster sent me the email alert and the images are from within webmaster.

leftquark

My site appears to be fine. Are the URL's it give you actual valid URL's? Is it only keyword URL's? Can you send us a few examples?

ChancyRat

I downloaded the csv file, but it's too large to attach here. Aaron I think someone should look at it. I can't get it to you via DG, right?

The numbers are in the thousands.
Today: 3,900 desktop, 4491 other
Smartphone 5529

I see about 200 regular URLs, and most of the rest are keywords.
No matter which URL I click to check, they all go to a valid page.
I'll send a file of them to the help email.

ChancyRat

I sent a file to Help and asked them to forward to you. Thanks, Aaron. SSooooooo confused.