would like to be able to crawl my site

ablichter

With any spider I used so far I only was able to crawl the first level of my/any site (new design) even when using one of the user agents (bingbot, googlebot) allowed according to robots.txt.
For different reason (e.g. checking for broken links, etc. pp) I would like to be able to crawl at least my own site...

ablichter

ablichter wrote: »

With any spider I used so far I only was able to crawl the first level of my/any site (new design) even when using one of the user agents (bingbot, googlebot) allowed according to robots.txt.
For different reason (e.g. checking for broken links, etc. pp) I would like to be able to crawl at least my own site...

Ping.

ablichter

ping

thenickdude

That should already be working if you're faking the Googlebot user agent (required to trigger SmugMug to render crawlable HTML instead of JavaScript). Which spider software are you using?

ablichter

Lamah wrote: »

That should already be working if you're faking the Googlebot user agent (required to trigger SmugMug to render crawlable HTML instead of JavaScript). Which spider software are you using?

You are right, thanks. I use an online crawler -> http://www.auditmypc.com/xml-sitemap.asp

thenickdude

In the settings for the crawler, set the user-agent to "Googlebot". That's enough to trigger SmugMug to serve plain HTML for you, and also matches the robots.txt restrictions.

ablichter

Lamah wrote: »

In the settings for the crawler, set the user-agent to "Googlebot". That's enough to trigger SmugMug to serve plain HTML for you, and also matches the robots.txt restrictions.

Yepp, did already and worked. Thanks