Options

Robots.txt Unreachable - Sitemaps

135

Comments

  • Options
    denisegoldbergdenisegoldberg Administrators Posts: 14,246 moderator
    edited January 2, 2011
    I see /sitemap-base.xml.gz and /sitemap-index.xml for my site, both with green check marks. Before this change I believe there was also an entry for images. I have uploaded new photos and applied keywords since this change was pushed to the live site.

    Can you confirm what should be listed under Sitemap on the https://www.google.com/webmasters/tools/sitemap-list?.... page?

    --- Denise
  • Options
    denisegoldbergdenisegoldberg Administrators Posts: 14,246 moderator
    edited January 3, 2011
    There still appears to be something wrong here - things seem to be moving in the wrong direction.

    /sitemap-index.xml is showing with a red x again. Yesterday I resubmitted, and the status flipped to a green check mark. Today the red x is back.

    And - the URLs submitted and URLs in web index seem to have stayed at the same values for days in spite of the fact that I have added new galleries, new photos, and keyworded photos.

    Seems to imply that there is still a problem somewhere. Can you help?

    --- Denise
  • Options
    psenior1psenior1 Registered Users Posts: 125 Major grins
    edited January 3, 2011
    There still appears to be something wrong here - things seem to be moving in the wrong direction.

    /sitemap-index.xml is showing with a red x again. Yesterday I resubmitted, and the status flipped to a green check mark. Today the red x is back.

    And - the URLs submitted and URLs in web index seem to have stayed at the same values for days in spite of the fact that I have added new galleries, new photos, and keyworded photos.

    Seems to imply that there is still a problem somewhere. Can you help?

    --- Denise

    I had the same today, I resubmited the index and its OK now. Worth a try if you haven't already
    website - http://www.snrmac.com
    facebook - my facebook page please LIKE me!
  • Options
    denisegoldbergdenisegoldberg Administrators Posts: 14,246 moderator
    edited January 3, 2011
    psenior1 wrote: »
    I had the same today, I resubmited the index and its OK now. Worth a try if you haven't already
    I resubmitted it yesterday, was fine after the resubmission - but it switched back to a red x this morning.

    And honestly I don't believe we should need to resubmit every day (or even every week). Something still feels wrong here.

    --- Denise
  • Options
    AndyAndy Registered Users Posts: 50,016 Major grins
    edited January 3, 2011
    And honestly I don't believe we should need to resubmit every day (or even every week).

    You don't. Let Twoofy reply, he will. Thanks.
  • Options
    psenior1psenior1 Registered Users Posts: 125 Major grins
    edited January 4, 2011
    still no change in the contents of the sitemap index, yet the lastmod date was only yesterday. If the contents of the sitemap-base has changed, should the galleries and image sitemaps have been generated by now?
    <sitemapindex xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/siteindex.xsd">
    &#8722;
    <sitemap>
    <loc>http://www.snrmac.com/sitemap-base.xml.gz</loc>
    <lastmod>2011-01-03T20:00:23Z</lastmod>
    </sitemap>
    </sitemapindex>
    
    website - http://www.snrmac.com
    facebook - my facebook page please LIKE me!
  • Options
    denisegoldbergdenisegoldberg Administrators Posts: 14,246 moderator
    edited January 4, 2011
    I resubmitted it yesterday, was fine after the resubmission - but it switched back to a red x this morning.
    Interesting. I didn't resubmit again yesterday but this morning the red x is gone, back to a green check. Given that things are switching between available and not available and given that I have uploaded photos since the last update of the numbers I still find it interesting that the numbers shown are not changing at all. And by the way they are much much lower than they were before the change.

    --- Denise
  • Options
    psenior1psenior1 Registered Users Posts: 125 Major grins
    edited January 6, 2011
    Andy wrote: »
    You don't. Let Twoofy reply, he will. Thanks.

    thats good, but when!?
    website - http://www.snrmac.com
    facebook - my facebook page please LIKE me!
  • Options
    TwoofyTwoofy Registered Users Posts: 171 Major grins
    edited January 8, 2011
    Interesting. I didn't resubmit again yesterday but this morning the red x is gone, back to a green check. Given that things are switching between available and not available and given that I have uploaded photos since the last update of the numbers I still find it interesting that the numbers shown are not changing at all. And by the way they are much much lower than they were before the change.

    --- Denise

    Hi Denise,

    It looks like your sitemaps have been re-generated now. I validated the XML and all the urls too. Usually when I the sitemaps in this status, its just a matter of waiting for Google to get the webmaster stats updated. But, I can safely say that Google is fetching and processing your sitemaps at this time. We just need Google to finish doing their side of things.

    - Greg
  • Options
    TwoofyTwoofy Registered Users Posts: 171 Major grins
    edited January 8, 2011
    psenior1 wrote: »
    still no change in the contents of the sitemap index, yet the lastmod date was only yesterday. If the contents of the sitemap-base has changed, should the galleries and image sitemaps have been generated by now?
    <sitemapindex xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/siteindex.xsd">
    &#8722;
    <sitemap>
    <loc>http://www.snrmac.com/sitemap-base.xml.gz</loc>
    <lastmod>2011-01-03T20:00:23Z</lastmod>
    </sitemap>
    </sitemapindex>
    

    I am surprised by this too. I'll have an answer for you on Monday.

    - Greg
  • Options
    denisegoldbergdenisegoldberg Administrators Posts: 14,246 moderator
    edited January 8, 2011
    Twoofy wrote: »
    It looks like your sitemaps have been re-generated now. I validated the XML and all the urls too. Usually when I the sitemaps in this status, its just a matter of waiting for Google to get the webmaster stats updated. But, I can safely say that Google is fetching and processing your sitemaps at this time. We just need Google to finish doing their side of things.
    Thanks for checking Greg.

    Things do look better, but I am still seeing a red X for /sitemap-images.xml.gz. Is that a google catch-up type of thing, or is there still something odd going on?

    Screen grab below is from just a minute ago.

    I just resubmitted that entry as a sitemap, so I guess I'll see tomorrow if anything changes.

    --- Denise
  • Options
    TwoofyTwoofy Registered Users Posts: 171 Major grins
    edited January 8, 2011
    Thanks for checking Greg.

    Things do look better, but I am still seeing a red X for /sitemap-images.xml.gz. Is that a google catch-up type of thing, or is there still something odd going on?

    Screen grab below is from just a minute ago.

    I just resubmitted that entry as a sitemap, so I guess I'll see tomorrow if anything changes.

    --- Denise

    I am going to guess its in the same ballpark as "Google Catchup", but can you give me a screenshot of what you see when you click the one with the red X? It should show us the error when you do that.

    - Greg
  • Options
    denisegoldbergdenisegoldberg Administrators Posts: 14,246 moderator
    edited January 8, 2011
    Twoofy wrote: »
    I am going to guess its in the same ballpark as "Google Catchup", but can you give me a screenshot of what you see when you click the one with the red X? It should show us the error when you do that.
    It showed a 404 error before I resubmitted. I am still seeing the red x on the upper level entry but now when I drill down there is no longer a red x showing.

    I'll let you know what happens tomorrow - if it's a catch up thing then I'll assume the red x on the upper level should be gone tomorrow. Fingers crossed...

    --- Denise
  • Options
    TwoofyTwoofy Registered Users Posts: 171 Major grins
    edited January 8, 2011
    If it showing a 404 error, it may take more time for that to disappear. But, I think its working okay - because it says there are 8,943 URLs in that sitemap file. Obviously if it was not able to fetch it (which is what the 404 would be) it would not know how many URLs were in it. I'm thinking that the 404 error is probably just an old error and will flush out of their system eventually (though probably more then a day, but I will keep my fingers crossed too).

    - Greg
  • Options
    New England PhotographyNew England Photography Registered Users Posts: 5 Beginner grinner
    edited January 8, 2011
    I cant even get a sitemap generated from Smugmug. I wish I could upload one.
  • Options
    TwoofyTwoofy Registered Users Posts: 171 Major grins
    edited January 8, 2011
    I cant even get a sitemap generated from Smugmug. I wish I could upload one.

    I do not understand what you mean. The sitemaps are generated automatically. Can you elaborate?

    - Greg
  • Options
    psenior1psenior1 Registered Users Posts: 125 Major grins
    edited January 8, 2011
    Twoofy wrote: »
    I am surprised by this too. I'll have an answer for you on Monday.

    - Greg


    OK - thanks.
    website - http://www.snrmac.com
    facebook - my facebook page please LIKE me!
  • Options
    denisegoldbergdenisegoldberg Administrators Posts: 14,246 moderator
    edited January 9, 2011
    Twoofy wrote: »
    ....because it says there are 8,943 URLs in that sitemap file. Obviously if it was not able to fetch it (which is what the 404 would be) it would not know how many URLs were in it. I'm thinking that the 404 error is probably just an old error and will flush out of their system eventually (though probably more then a day, but I will keep my fingers crossed too).
    Greg -
    Showing all green today, although it looks like it will possibly take quite a while to show anything in the index. Yes, I know, that's Google, not smug!

    Right now the report shows what is probably a reasonable number of URLs submitted but not that many in web index. And all of those in the web index are in /sitemap-base.xml.gz. No entries in the images or galleries sitemaps. Does that sound normal to you?

    Thanks!

    --- Denise
  • Options
    TwoofyTwoofy Registered Users Posts: 171 Major grins
    edited January 9, 2011
    Greg -
    Showing all green today, although it looks like it will possibly take quite a while to show anything in the index. Yes, I know, that's Google, not smug!

    Right now the report shows what is probably a reasonable number of URLs submitted but not that many in web index. And all of those in the web index are in /sitemap-base.xml.gz. No entries in the images or galleries sitemaps. Does that sound normal to you?

    Thanks!

    --- Denise

    Yeah, that sounds okay for now. I should point out that even though Webmaster Tools says there is nothing indexed - there probably are. The only real way to know for sure is to do one of the various google searches (site:..., etc) to see what is actually in their index. Most of my sites show a very small number (according to Webmaster Tools), but have an enormous number of pages that are actually in the index. I wish Webmaster Tools was more precise on this... :(

    - Greg
  • Options
    TwoofyTwoofy Registered Users Posts: 171 Major grins
    edited January 10, 2011
    psenior1 wrote: »
    OK - thanks.

    Hello,

    Should be good to go now. Here is your current sitemap-index:

    [HTML]<?xml version="1.0" encoding="utf-8"?>
    <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9&quot; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance&quot; xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/siteindex.xsd"&gt;
    <sitemap>
    <loc>http://www.snrmac.com/sitemap-base.xml.gz</loc&gt;
    <lastmod>2011-01-10T08:52:13Z</lastmod>
    </sitemap>
    <sitemap>
    <loc>http://www.snrmac.com/sitemap-galleries.xml.gz</loc&gt;
    <lastmod>2011-01-10T08:55:00Z</lastmod>
    </sitemap>
    <sitemap>
    <loc>http://www.snrmac.com/sitemap-images.xml.gz</loc&gt;
    <lastmod>2011-01-10T09:01:53Z</lastmod>
    </sitemap>
    </sitemapindex>
    [/HTML]
    - Greg
  • Options
    psenior1psenior1 Registered Users Posts: 125 Major grins
    edited January 11, 2011
    great, thanks. I've now got the three - base, images and galleries all with a tick.

    thanks for your help!
    website - http://www.snrmac.com
    facebook - my facebook page please LIKE me!
  • Options
    cmancman Registered Users Posts: 75 Big grins
    edited January 11, 2011
    I am very sorry, but I would ask the distinguished audience to see, how this is done by Google on Blogger.com (the example blog of Denise Goldberg):

    blog - http://denisegoldberg.blogspot.com/ ;

    file robots.txt - http://denisegoldberg.blogspot.com/robots.txt ;

    sitemap - http://denisegoldberg.blogspot.com/feeds/posts/default?orderby=updated (just feed).

    P.S. And there is no need to reinvent the wheel. fitness4.gif
  • Options
    denisegoldbergdenisegoldberg Administrators Posts: 14,246 moderator
    edited January 11, 2011
    cman wrote: »
    I am very sorry, but I would ask the distinguished audience to see, how this is done by Google on Blogger.com (the example blog of Denise Goldberg):
    I'm a bit confused here - all you are showing is the feed to my blog. We are talking about sitemaps on our smug sites here, not just a feed. Yes, there is a feed on my blog, and there are feeds on my smug site too - those feeds have been there for a long, long time. Feeds are not a site map.

    What are you trying to say here?

    --- Denise
  • Options
    denisegoldbergdenisegoldberg Administrators Posts: 14,246 moderator
    edited January 11, 2011
    Twoofy wrote: »
    Yeah, that sounds okay for now. I should point out that even though Webmaster Tools says there is nothing indexed - there probably are.
    The numbers are continuing to improve - looks like a waiting game for now. I have done site searches in google, and you're right, there are more pages shown there then in Webmaster tools.

    Thanks again.

    --- Denise
  • Options
    cmancman Registered Users Posts: 75 Big grins
    edited January 11, 2011
    ...... Feeds are not a site map.

    What are you trying to say here?

    --- Denise

    Really? sunglasses2.gif

    Please look at the last line of Your file robots.txt:

    sitemap = feed
  • Options
    denisegoldbergdenisegoldberg Administrators Posts: 14,246 moderator
    edited January 11, 2011
    cman wrote: »
    Please look at the last line of Your file robots.txt:
    sitemap = feed
    All that shows is the last 25 posts to my blog. I suppose that could be viewed as helpful, but honestly, you can see the last 25 posts by just reading my blog, or by browsing the contents of the side bar. The most useful aspect of the feed is to allow people to subscribe so they don't need to look for new entries, the new entries come to them.

    In my opinion a blog is very different from a set of galleries. I want my photos and galleries to be found - and not necessarily in last entered / modified sequence. A feed doesn't do that. I want people to be able to find sets of blog posts easily as well, and that's why I sometimes assign labels to my posts.

    What do you think a site map should be? I would argue that (just) a feed is not good enough, yet that's what you seem to be proposing.

    --- Denise
  • Options
    cmancman Registered Users Posts: 75 Big grins
    edited January 11, 2011
    Denise Goldberg, thanks for answer.

    Please compare:

    1.
    - http://www.google.com/images?hl=en&q=site:denisegoldberg.com&um=1&ie=UTF-8&source=og&sa=N&tab=wi (12 results)
    - http://www.google.com/search?hl=en&ie=ISO-8859-1&q=site%3Adenisegoldberg.com (About 18,200 results)

    indexed images / indexed pages = 0,0007

    and

    2.
    http://www.google.com/images?hl=en&q=site:denisegoldberg.blogspot.com/&um=1&ie=UTF-8&source=og&sa=N&tab=wi (About 139 results)
    http://www.google.com/search?hl=en&ie=ISO-8859-1&q=site%3Adenisegoldberg.blogspot.com%2F (About 283 results)

    indexed images / indexed pages = 0.49

    Sitemap is not an end in itself, but only one of the means to improve the indexing of sites. wink2.gif
  • Options
    cmancman Registered Users Posts: 75 Big grins
    edited January 11, 2011
    Denise Goldberg, in conclusion, I would like to draw your attention that this is done directly by Google (!) (on Blogger.com). bye%20bye.gif
  • Options
    rashbrookrashbrook Registered Users Posts: 92 Big grins
    edited January 20, 2011
    I've returned. I've been moved into a new position in my day job and haven't had enough time to monitor this as closely for a couple weeks.

    Here it is - until today everything looked fine - when I looked. Today I looked and saw that my sitemap is being submitted fine, but that there are zero (0) urls in my web index. ???
  • Options
    rashbrookrashbrook Registered Users Posts: 92 Big grins
    edited January 30, 2011
    A positive note - today I searched while not logged in to google and saw that after 2.5 months, if I search against my business name, my site comes up 4th in the search results. Much better.

    Still have a green check and a good robots.txt crawl as well.
This discussion has been closed.