robots.txt file

justawebbie

How can I get the robots.txt file opened so our Google search appliance can index our smugmug site?

http://photomgr.smugmug.com

I sent an email to SmugMug support but have not heard back from them at all.

Andy

Hi, I answered this question from our help desk days ago... hold on a sec and I'll find it.
Yup, here it is fro December 9th:



Can you please check your spam folders at work? Thanks!

The answer is, http://photomgr.smugmug.com/robots.txt

justawebbie

I found that file but how do i go about getting it changed to allow our google gsa search crawl our smugmug site? Right now it does not allow it to crawl it at all.

Andy

justawebbie wrote: »

I found that file but how do i go about getting it changed to allow our google gsa search crawl our smugmug site? Right now it does not allow it to crawl it at all.

I'm really not sure - I do know that you cannot modify the robots.txt file, but that Google sure can. Not sure why your Google appliance can't though

justawebbie

I wonder if anyone else might know why the GSA can not index the site. They tell me it is the robots.txt blocking them from doing it. I will get more details from our OIT department after the holidays and post here.

Thank you though Andy.

justawebbie

Our OIT department sent this to me. I think we need to have the robots.txt allow our Google search appliance to search for Google.

Google itself may be allowed, but perhaps not our GSA (search.alaska.edu) Do you know how to get this changed, I am a pro user?

rainforest1155

Your service seems to use a separate user agent that we don't allow at this point. You can't manually add any services to the robots.txt file, I'm sorry.
Maybe you could edit the user agent used for your GSA service? This page seems to suggest that changing the name is possible:
http://code.google.com/apis/searchappliance/documentation/50/help_gsa/crawl_headers.html

May try using a useragent that's listed in our robots.txt file, like the Googlebot ?

If you'd like to suggest supporting of another search useragent, you could add your suggestion here:
http://feedback.smugmug.com/forums/17723-smugmug

justawebbie

Thank you for help. I have posted into Smugmugs feedback forum. I let my OIT department know about the useragent. I hope we can resolve this issue.

Andy

We've updated our robots.txt to allow for this, thanks for being so passionate about it!

The 48th Ronin

So is there any way we can edit our robots.txt file? I want to use MajesticSEO to run reports and need to add:

User-Agent: MJ12bot
Disallow:

Thanks. Greetings form Canada's Far East.

Brian Carey
http://www.briancareyphotography.com/

The 48th Ronin

Just reread this post, looks like we can't at this time. :-(

paullantz

robots.txt file for google adsense crawler
I received a message from google adsense.
Wondering if robots.txt file can be modified as suggested to them if this has not been done already.
thanks very much,

Paul Lantz

Large number of failed ad crawls
76

failed crawls
last week
We noticed that the AdSense ad crawler is having some issues accessing your site on smugmug.com. This issue is caused by a misconfigured robots.txt file, which is blocking the ad crawler from viewing certain sections of your site.

Last week, we detected 76 failed crawl requests. Because of this, your AdSense ads are less targeted and are generating less revenue.

View crawler errors
To fix this, you’ll need to edit your robots.txt file to allow our AdSense crawler by adding these two lines to the very top:

User-agent: Mediapartners-Google
Disallow:

This will allow the AdSense ad crawler to access pages on your site that already have AdSense code on them. As a result, you and your users will benefit from more targeted, relevant ads.

It's important to note that making this change will not impact your Google search or SEO rankings. Adding these two lines to your robots.txt file will simply deliver better, more relevant ads to pages with AdSense code already on them. Pages that don’t have AdSense ad code will not be affected.

For more in-depth information, make sure that you read our blog post about this on the Inside AdSense blog.

rainforest1155

paullantz wrote: »

I received a message from google adsense.
Wondering if robots.txt file can be modified as suggested to them if this has not been done already.
thanks very much,

Paul Lantz

Large number of failed ad crawls
76

What specific crawls failed and on which date? Chances are that the crawler tried to access parts of the site it is not supposed to or perhaps those links were no longer valid?

If you don't want to post the links in the public forum, email our HelpDesk with details.

paullantz

urls adsense crawler could not follow
http://paullantz.smugmug.com/search/?searchWords=baby&searchType=InUser&NickName=paullantz&x=0&y=0
http://paullantz.smugmug.com/search/?searchWords=Fort+Albany&searchType=InAlbum&AlbumID=25782496&x=0&y=0
http://paullantz.smugmug.com/search/?searchWordsShort=October&searchType=InAlbum&AlbumID=25782496&x=0&y=0
http://paullantz.smugmug.com/search/?searchWordsShort=moosonee&searchType=InAlbum&AlbumID=861351&x=0&y=0


very much wish I understood this better

Paul

jfriend

paullantz wrote: »

http://paullantz.smugmug.com/search/?searchWords=baby&searchType=InUser&NickName=paullantz&x=0&y=0
http://paullantz.smugmug.com/search/?searchWords=Fort+Albany&searchType=InAlbum&AlbumID=25782496&x=0&y=0
http://paullantz.smugmug.com/search/?searchWordsShort=October&searchType=InAlbum&AlbumID=25782496&x=0&y=0
http://paullantz.smugmug.com/search/?searchWordsShort=moosonee&searchType=InAlbum&AlbumID=861351&x=0&y=0


very much wish I understood this better

Paul

Those are not URLs that should be crawled by a search engine (those are programmatic search results, not an actual web page on your site) so the robots.txt file is probably doing the right thing.

rainforest1155

That's correct, /search is one of the folders that isn't available to crawlers. For a full listing, you can take a look at the robots.txt file on your site:
nickname.smugmug.com/robots.txt
(replace nickname with your own site nickname)