Robot.txt

JiggajaggsJiggajaggs Registered Users Posts: 8 Beginner grinner
edited October 15, 2012 in SmugMug Support
Hi
My wife's site disappeared from local searches on google -
I checked with webmaster tools and see the following URLs are blocked - is this normal ?

It says.......

# If you're reading this, you belong at a job you love: http://www.smugmug.com/jobs/
# See http://www.smugmug.com/help/emailreal if you'd like to apply to be whitelisted for crawling this site

User-agent: Adsbot-Google
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/

User-agent: archive.org_bot
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/

User-agent: baiduspider
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/

User-agent: facebookexternalhit
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/

User-agent: FunnelBack
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/

User-agent: Googlebot
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/

User-agent: Googlebot-Image
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/

User-agent: Googlebot-Mobile
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/

User-agent: Google-Sitemaps
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/

User-agent: gsa-crawler
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/

User-agent: ia_archiver
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/

User-agent: Mediapartners-Google
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/

User-agent: msnbot
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/

User-agent: naverbot
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/

User-agent: ScoutJet
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/

User-agent: Slurp
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/

User-agent: Spinn3r
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/

User-agent: twiceler
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/

User-agent: Twitterbot
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/

User-agent: Yandex
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/


User-agent: *
Disallow: /


Sitemap: http://www.xclusivphotos.co.uk/sitemap-index.xml


Thats it !

Thanks for your assistance :thumb
Dave

Comments

  • rainforest1155rainforest1155 Registered Users Posts: 4,566 Major grins
    edited October 15, 2012
    Yes, that's normal as we have parts of the site search engines should not index. So for all search engines that are allowed, you'll see "Disallow" for the folders they should not index - the rest of your wife's site they can index just fine.
    There's also an up to date sitemap for the site which specifically links search engines that support sitemaps to the most important section of the site.

    Furthermore, the site is indexed by Google already. Just search for:
    site:xclusivphotos.co.uk
    and you'll see all the pages Google currently has indexed. I see more than a thousand results for that search.

    To get more search engine coverage, you'll have to add more information to your site, like adding captions and keywords for individual photos is a very important step. For more suggestions, take a look at the SEO page and follow the suggestions there.
    Sebastian
    SmugMug Support Hero
  • JiggajaggsJiggajaggs Registered Users Posts: 8 Beginner grinner
    edited October 15, 2012
    Thanks for the reply - puts our minds at rest and I guess we (Royal We ! ) have some work to do indexing images now. Thanks again.
Sign In or Register to comment.