Robot txt???

sbphoto0124

I'm currently trying to optimize my site and keep being told that I need to update this file. Apparently some major search engines are being blocked. Any idea how I would update this to make sure I am appearing as I should be?

phaserbeam

A robots.txt might be helpful if you host a site on your own, but for smugmug i recommend to check the settings here if you do not want to let google find your photos.

thenickdude

SmugMug's robots.txt only blocks certain behind-the-scenes folders that aren't useful for search engines anyway. You don't need to worry about it. You can see a list of excluded folders here:

http://www.sherlockphotography.org/robots.txt

sbphoto0124

Thanks for the info I called my domain host and inquired about this and they said they can't do anything on their end. I have my account settings set to be found on Google so not quite sure what the issue is. Hopefully the stuff I've done so far will help my site's visibility somehow. Otherwise it's back to square one.

Thank you all for the replies

richpepp

I have my account settings set to be found on Google so not quite sure what the issue is

Getting found by Google is painful but does actually work. We have a gallery we put up in July and if I search google for a bit of text from one of the photo captions such as 'A girl at a North Korean orphanage eats rice with food supplements' then I can see that the image is in Google web search (but not image search). So robots.txt isn't blocking Google as far as I can see. The trick to getting found is always to have good captions, keywords and gallery descriptions as well as good inbound links to your site from other places so Google thinks other people like your site as well. At an absolute minimum you should have a link to your site in your signature in this forum

It takes a bit of time for stuff to appear but it does appear in the end. Inbound links are important though.

As Lamah says, don't worry about robots.txt - it's only blocking some behind the scenes folders that provide a second way of getting to your photos. In any case we can't control it

Google image search is another matter though

sbphoto0124

Thank you Rich but according to my domain provider one of the search engines is being blocked but I have no way of knowing which.

thenickdude

Is this just coming from a stupid tool that looks at your robots.txt file and checks "for problems"? All it's telling you is that some search engines are blocked from some parts of your site, which is absolutely true. For example, a range of search engines including Google are blocked from the directory /admin/, which should never be showing up in search results. But there are no blocks for any search engine for the parts of your site that are supposed to be public, like your galleries.

richpepp

sbphoto0124 wrote: »

Thank you Rich but according to my domain provider one of the search engines is being blocked but I have no way of knowing which.

It's quite easy to see really as the robots.txt file is just in plain text. Just add robots.txt to the end of your URL in the address bar of any browser and you should see it (e.g. photos.miseast.org/robots.txt for me). What you will see is a bunch of what are called 'user agents' which is what search engines use to identify themselves along with a list of areas of the site that they aren't allowed to go to. The areas listed are all areas that you don't really want searched. None of your regular galleries or folders will be listed there. All of the major search engines will be in that file but only with a list of restricted areas, not being blocked altogether

Rich

(edit - go on, put a link to your site in your signature. That will help you appear in google unless you already have lots of inbound links from elsewhere)

sbphoto0124

Thanks Lamah and Rich! Yes I looked at the files and sure enough as you stated the stuff that's private is blocked which is fine! I've never heard of such a thing (new to this whole SEO thing) so I'm coming across a lot of jargon that I'm not familiar with. Seems that everything is working fine and I've done what I can for now to increase visibility online.

I appreciate all the feedback!

richpepp

If you are new to the SEO thing then the single most important thing to do is to build links to your site from other places. Google uses these (amongst other things) to work out how important your site is and hence how high up the list to put it when it returns search results. The best way to find incoming links is to sign up for Google webmaster tools but you can also get a quick idea by typing 'link:. However you have used .smugmug.com rather than www..com which would have been much better. Although they both end up at the same place you want to always use your full site name where possible so you don't waste any links.

sbphoto0124

I have to look into that Rich but thanks for the kudos and help. I definitely need all the help I can get. Most of my business is word of mouth and trying o develop a bigger client base has proven harder online than expected. !

richpepp

sbphoto0124 wrote: »

I have to look into that Rich but thanks for the kudos and help. I definitely need all the help I can get. Most of my business is word of mouth and trying o develop a bigger client base has proven harder online than expected. !

Online is indeed hard. I think it was a good 6 months before our site started showing up properly in Google but it does actually happen. Key is good captions and keywords so Google knows what the photos are about. You've also got your address at the bottom of every page which is brilliant as it allows google to localise it's search and it's more likely to put you near the top if it's someone near you doing the search. Remember that there are lots of other people wanting to do the same as you so to get higher up in Google you have to work that little bit harder. Keep the blog going

Great photos btw

Rich

btw you can speed up getting online visitors by using Google AdWords but it tends not to be cheap so you really have to work at converting visitors into paying customers.

sbphoto0124

I did Adwords and stopped about a month ago it was costing me an arm and a leg between bing, fb AND google. I decided to pause it until I could optimize because I wasn't seeing any good return on the expense at the time.

Yes I just added the address last night for that main reason (I've been doing a lot of reading on things I can do to optimize). I know I'm not the only one especially in my area where there's 20 photos per block