Most important: Google Adsense's crawler has been locked out by the new robots.txt

www.paolosmeraldi.com

It seems that with the new robots.txt file Smugmug has locked out the Google Adsense crawler.

Here is the new file: as you can see the User agent Mediapartners-Google that is, the one used by Google Adsense, has been left out.

Please allow this crawler as soon as possible as people using Adsense like me (http://photos.paolosmeraldi.com) are now publishing only public service ads (meaning we do not get any money).

Please note, this isn't the googlebot, have a look here, the two are different

https://www.google.com/adsense/support/bin/answer.py?answer=10532

Thanks



# If you're reading this, you belong at a job you love: http://www.smugmug.com/jobs/
# See http://www.smugmug.com/help/emailreal if you'd like to apply to be whitelisted for crawling this site

User-agent: baiduspider
Disallow: /admin/
Disallow: /cart/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /search/

User-agent: Googlebot
Disallow: /admin/
Disallow: /cart/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /search/

User-agent: msnbot
Disallow: /admin/
Disallow: /cart/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /search/

User-agent: naverbot
Disallow: /admin/
Disallow: /cart/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /search/

User-agent: ScoutJet
Disallow: /admin/
Disallow: /cart/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /search/

User-agent: Slurp
Disallow: /admin/
Disallow: /cart/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /search/

User-agent: twiceler
Disallow: /admin/
Disallow: /cart/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /search/

User-agent: Yandex
Disallow: /admin/
Disallow: /cart/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /search/


User-agent: *
Disallow: /

Andy

We'll check into this - thanks for bringing it up!

www.paolosmeraldi.com

Thanks Andy; it seems quite simple to me.

I have chosen Smugmug because it was possible to integrate adsense into it, and now I wish that I can continue using it as it is a very good service!

Sheaf

This should be fixed. Sorry!

www.paolosmeraldi.com

Yes, I have checked right now.

My website is starting to show ads again.

This happened right at the moment while my visits were increasing!

Well it happens, thanks.

Paolo