Options

Most important: Google Adsense's crawler has been locked out by the new robots.txt

www.paolosmeraldi.comwww.paolosmeraldi.com Registered Users Posts: 28 Big grins
edited August 7, 2010 in Bug Reporting
It seems that with the new robots.txt file Smugmug has locked out the Google Adsense crawler.

Here is the new file: as you can see the User agent Mediapartners-Google that is, the one used by Google Adsense, has been left out.

Please allow this crawler as soon as possible as people using Adsense like me (http://photos.paolosmeraldi.com) are now publishing only public service ads (meaning we do not get any money).

Please note, this isn't the googlebot, have a look here, the two are different

https://www.google.com/adsense/support/bin/answer.py?answer=10532

Thanks



# If you're reading this, you belong at a job you love: http://www.smugmug.com/jobs/
# See http://www.smugmug.com/help/emailreal if you'd like to apply to be whitelisted for crawling this site

User-agent: baiduspider
Disallow: /admin/
Disallow: /cart/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /search/

User-agent: Googlebot
Disallow: /admin/
Disallow: /cart/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /search/

User-agent: msnbot
Disallow: /admin/
Disallow: /cart/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /search/

User-agent: naverbot
Disallow: /admin/
Disallow: /cart/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /search/

User-agent: ScoutJet
Disallow: /admin/
Disallow: /cart/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /search/

User-agent: Slurp
Disallow: /admin/
Disallow: /cart/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /search/

User-agent: twiceler
Disallow: /admin/
Disallow: /cart/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /search/

User-agent: Yandex
Disallow: /admin/
Disallow: /cart/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /search/


User-agent: *
Disallow: /

Comments

Sign In or Register to comment.