checking hyper links

synaturesynature redwoodtwigPosts: 189Registered Users Major grins
edited March 11, 2016 in SmugMug Support
I want to run a link checker against my reorganized site. The W3 schools one tells me that robots.txt prevents it from checking:

Line: 57 http://cdn.smugmug.com/include/js/smugpage/core-config-c6ae44ff9abf61a296800acb209cd62e.js
Status: (N/A) Forbidden by robots.txt
The link was not checked due to robots exclusion rules. Check the link manually.

Line: 75 http://www.redwoodtwig.com/Studios
Status: (N/A) Forbidden by robots.txt
The link was not checked due to robots exclusion rules. Check the link manually.

The first one is fine, I don't need that checked. But the second one is a main area of my site. Does this mean google is not going to look into Studios and catalog what's there?

I tried a couple of other link checkers and they are telling me the same thing. Setting aside my major concern about SEO, is there a link checker that will ignore robots.txt and crawl through and give me a report so I can make sure my reorganization is done.
Brandon Smith
http://redwoodtwig.com
Sony A900 with 1.4 85mm prime; A77, and couple other cameras floating around that see less usage. Lightroom, usually the latest version.

Comments

  • Lille UlvenLille Ulven Lille Ulven ScandinaviaPosts: 518Registered Users Major grins
    edited March 6, 2016
    Hi synature

    Just to roll out simple made mistakes: when you open the organizer and look at the settings of your Studios Folder/Gallery do you have in the section "Security & Sharing" the setting for "Web Searchable" set to "site setting" / "yes"? If not that might be what is preventing that Folder/Gallery from being marked as "searchable" in your robots.txt. (Double checking might just be worth it - has been worth it for me at least a couple of times before :D)

    Good luck

    Lille Ulven
    http://www.lilleulven.com - The Photos of my travels
    http://blog.lilleulven.com - The Stories of my travels
  • synaturesynature redwoodtwig Posts: 189Registered Users Major grins
    edited March 6, 2016
    Hi synature

    Just to roll out simple made mistakes: when you open the organizer and look at the settings of your Studios Folder/Gallery do you have in the section "Security & Sharing" the setting for "Web Searchable" set to "site setting" / "yes"? If not that might be what is preventing that Folder/Gallery from being marked as "searchable" in your robots.txt. (Double checking might just be worth it - has been worth it for me at least a couple of times before :D)

    Good luck

    Lille Ulven

    I checked again to make sure, and it is site setting / yes and over in the site settings I have the discover me things turned on.

    This link checker has the option to ignore robots.txt, so I was able to find the half dozen or so broken ones. http://website-link-checker.online-domain-tools.com/ The firefox tool also worked, though only one page at a time.

    I am still concerned that google is not crawling into my public pages like a good SE is supposed to.
    Brandon Smith
    http://redwoodtwig.com
    Sony A900 with 1.4 85mm prime; A77, and couple other cameras floating around that see less usage. Lightroom, usually the latest version.
  • ChancyRatChancyRat Soooo Beginner Grinner Oakland, CAPosts: 2,130Registered Users Major grins
    edited March 11, 2016
    I have a page with tons of reference links, so I was dedicated to finding a working link-checker. Good to go with Check My Links, a Chrome Extension:
    https://chrome.google.com/webstore/detail/check-my-links/ojkcdipcgfaekbeaelaapakgnjflfglf?hl=en-GB

    It color-codes all the links on the page for Valid, Warnings, and Invalid.
    Even checks my navbar links. :)
  • synaturesynature redwoodtwig Posts: 189Registered Users Major grins
    edited March 11, 2016
    ChancyRat wrote: »
    I have a page with tons of reference links, so I was dedicated to finding a working link-checker. Good to go with Check My Links, a Chrome Extension:
    https://chrome.google.com/webstore/detail/check-my-links/ojkcdipcgfaekbeaelaapakgnjflfglf?hl=en-GB

    It color-codes all the links on the page for Valid, Warnings, and Invalid.
    Even checks my navbar links. :)

    Yes, that one works fine, as does a similar one in firefox. But only one page at a time.
    Brandon Smith
    http://redwoodtwig.com
    Sony A900 with 1.4 85mm prime; A77, and couple other cameras floating around that see less usage. Lightroom, usually the latest version.
  • ChancyRatChancyRat Soooo Beginner Grinner Oakland, CAPosts: 2,130Registered Users Major grins
    edited March 11, 2016
    synature wrote: »
    Yes, that one works fine, as does a similar one in firefox. But only one page at a time.

    Ahh, very true, yes.
  • ChancyRatChancyRat Soooo Beginner Grinner Oakland, CAPosts: 2,130Registered Users Major grins

    I'm> @ChancyRat said:

    I have a page with tons of reference links, so I was dedicated to finding a working link-checker. Good to go with Check My Links, a Chrome Extension:
    https://chrome.google.com/webstore/detail/check-my-links/ojkcdipcgfaekbeaelaapakgnjflfglf?hl=en-GB

    It color-codes all the links on the page for Valid, Warnings, and Invalid.
    Even checks my navbar links.

    I'm curious what others do regarding finding broken links on pages on your site?
    I used this tool back in 2016 and found it immensely useful. However, when i went to re-add it to Chrome just now, a pop-up message told me this:

    I don't actually understand what this message is saying. I mean, "websites you visit" - means the Smugmug page where my links are? Or the external URLS that the links go to?
    Read and change? Change would mean---?
    Would this be normal? Not excessive? If you think it is abnormal and excessive, can you recommend a better tool? Thanks.

  • JenuineJenuine Big grins Posts: 88Registered Users SmugMug Employee

    @ChancyRat the best thing at this juncture would be to reach out to the Support Heroes with the details of the concern, [email protected] They will take a look and advise.

    Jen
    SmugMug Support Hero
Sign In or Register to comment.