GoDaddy SEO Robot Issue

Tom Potter · February 1, 2014

Hey All,

I have my domain name with GoDaddy. The following, in quotes, is an error I get, when I check on the health of my SEO:

"You have a Robots.txt file, but it's blocking one or more major search engines (Bing, Yahoo, or Google)."

This is the recommended solution:

Determine what pages of your site can be crawled and by whom. Then, create the appropriate robots.txt file and upload it to the root of your hosted site. If you do not have FTP access to your hosting account, specify your crawling preferences, create the corresponding Meta Tag, and paste it in the <head></head> section of your site's HTML.

This preference allows all site pages to be crawled by all search engines.

I am not tech savy with this stuff. Can anyone help?

Thank Very Much

Ferguson · February 10, 2014

I host my domain on GoDaddy, never noticed any kind of health check there, can you provide a pointer?

Robots.txt is built into Smugmug, and it does prevent certain areas from being scanned, but generally they are normally appropriate areas.

I.e. what they are telling you to do, should already be done for you, by Smugmug. It may be helpful if there's more details behind it as to what precisely it is complaining about. And one way may be to point everyone to where you get that error, so (at least for those with GoDaddy accounts) can check our own and see?

EntropicTendencies · February 12, 2014

Complaints about robots.txt are common if you feed the tool your base domain without the www (e.g. tompotterphotography.com). Try using www.tompotterphotography.com.

Barrie
Smugmug Support Hero

Tom Potter · February 18, 2014

EntropicTendencies wrote: »

Complaints about robots.txt are common if you feed the tool your base domain without the www (e.g. tompotterphotography.com). Try using www.tompotterphotography.com.

Barrie
Smugmug Support Hero

Thanks Ferguson & Barrie for your help!
Barrie - exactly where is it that you are suggesting I make the "www" adjustment?

Thanks!

DanCarl97 · February 18, 2014

If you go to webmaster tools, then site settings from the menu that shows when you hover over your URL on the right (also on the image below), you'll be presented with the screen below.

Then, you can select between www. and not www..

ChancyRat · February 18, 2014

I just tried to select the www option and it gave me this message:
"Part of the process of setting a preferred domain is to verify that you own http://mysitename.com/. Please verify http://mysitename.com/."
Then it reverted me back to the "don't set a preferred domain".
So I'm confused, should I or shouldn't I select the www option?
And then why would it ask me to verify the address without the www?
But the revert me back to no choice?

DanCarl97 · February 18, 2014

You should select the www. option.

Strange that it should ask you to verify again. Maybe try that and hope it works. That's all I got.

ChancyRat · February 18, 2014

Are you saying I should unverify and then verify? I have to unverify before i can verify. Not sure unverifying is the right thing to do?

EntropicTendencies · February 18, 2014

You should add both yourdomain.whatever and www.yourdomain.whatever.

Both need to be verified (though if I remember correctly they will use the same meta tag).

Then you will be able to set the preferred domain as the www variant. Add your site map only on the www variant.

Barrie
Smugmug Support Hero

ChancyRat · February 18, 2014

EntropicTendencies wrote: »

You should add both yourdomain.whatever and www.yourdomain.whatever.

Both need to be verified (though if I remember correctly they will use the same meta tag).

Then you will be able to set the preferred domain as the www variant. Add your site map only on the www variant.

Barrie
Smugmug Support Hero

Thank you Barrie, I stumbled through webmaster and managed to achieve what you suggested.

GoDaddy SEO Robot Issue

Comments