Robot.txt
Jiggajaggs
Registered Users Posts: 8 Beginner grinner
Hi
My wife's site disappeared from local searches on google -
I checked with webmaster tools and see the following URLs are blocked - is this normal ?
It says.......
# If you're reading this, you belong at a job you love: http://www.smugmug.com/jobs/
# See http://www.smugmug.com/help/emailreal if you'd like to apply to be whitelisted for crawling this site
User-agent: Adsbot-Google
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: archive.org_bot
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: baiduspider
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: facebookexternalhit
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: FunnelBack
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: Googlebot
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: Googlebot-Image
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: Googlebot-Mobile
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: Google-Sitemaps
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: gsa-crawler
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: ia_archiver
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: Mediapartners-Google
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: msnbot
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: naverbot
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: ScoutJet
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: Slurp
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: Spinn3r
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: twiceler
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: Twitterbot
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: Yandex
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: *
Disallow: /
Sitemap: http://www.xclusivphotos.co.uk/sitemap-index.xml
Thats it !
Thanks for your assistance :thumb
Dave
My wife's site disappeared from local searches on google -
I checked with webmaster tools and see the following URLs are blocked - is this normal ?
It says.......
# If you're reading this, you belong at a job you love: http://www.smugmug.com/jobs/
# See http://www.smugmug.com/help/emailreal if you'd like to apply to be whitelisted for crawling this site
User-agent: Adsbot-Google
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: archive.org_bot
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: baiduspider
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: facebookexternalhit
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: FunnelBack
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: Googlebot
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: Googlebot-Image
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: Googlebot-Mobile
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: Google-Sitemaps
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: gsa-crawler
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: ia_archiver
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: Mediapartners-Google
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: msnbot
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: naverbot
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: ScoutJet
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: Slurp
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: Spinn3r
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: twiceler
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: Twitterbot
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: Yandex
Disallow: /admin/
Disallow: /cart/
Disallow: /date/
Disallow: /hack/
Disallow: /popular/
Disallow: /search/
Disallow: /test/
Disallow: /VIP/
Disallow: /vip/
Disallow: /go/
User-agent: *
Disallow: /
Sitemap: http://www.xclusivphotos.co.uk/sitemap-index.xml
Thats it !
Thanks for your assistance :thumb
Dave
0
Comments
There's also an up to date sitemap for the site which specifically links search engines that support sitemaps to the most important section of the site.
Furthermore, the site is indexed by Google already. Just search for:
site:xclusivphotos.co.uk
and you'll see all the pages Google currently has indexed. I see more than a thousand results for that search.
To get more search engine coverage, you'll have to add more information to your site, like adding captions and keywords for individual photos is a very important step. For more suggestions, take a look at the SEO page and follow the suggestions there.
SmugMug Support Hero