checking hyper links
synature
Registered Users Posts: 191 Major grins
I want to run a link checker against my reorganized site. The W3 schools one tells me that robots.txt prevents it from checking:
Line: 57 http://cdn.smugmug.com/include/js/smugpage/core-config-c6ae44ff9abf61a296800acb209cd62e.js
Status: (N/A) Forbidden by robots.txt
The link was not checked due to robots exclusion rules. Check the link manually.
Line: 75 http://www.redwoodtwig.com/Studios
Status: (N/A) Forbidden by robots.txt
The link was not checked due to robots exclusion rules. Check the link manually.
The first one is fine, I don't need that checked. But the second one is a main area of my site. Does this mean google is not going to look into Studios and catalog what's there?
I tried a couple of other link checkers and they are telling me the same thing. Setting aside my major concern about SEO, is there a link checker that will ignore robots.txt and crawl through and give me a report so I can make sure my reorganization is done.
Line: 57 http://cdn.smugmug.com/include/js/smugpage/core-config-c6ae44ff9abf61a296800acb209cd62e.js
Status: (N/A) Forbidden by robots.txt
The link was not checked due to robots exclusion rules. Check the link manually.
Line: 75 http://www.redwoodtwig.com/Studios
Status: (N/A) Forbidden by robots.txt
The link was not checked due to robots exclusion rules. Check the link manually.
The first one is fine, I don't need that checked. But the second one is a main area of my site. Does this mean google is not going to look into Studios and catalog what's there?
I tried a couple of other link checkers and they are telling me the same thing. Setting aside my major concern about SEO, is there a link checker that will ignore robots.txt and crawl through and give me a report so I can make sure my reorganization is done.
Brandon Smith
http://redwoodtwig.com
Sony A7r4 with a selection of Rokinon Cine primes that I'm really enjoying learning how to use.
http://redwoodtwig.com
Sony A7r4 with a selection of Rokinon Cine primes that I'm really enjoying learning how to use.
0
Comments
Just to roll out simple made mistakes: when you open the organizer and look at the settings of your Studios Folder/Gallery do you have in the section "Security & Sharing" the setting for "Web Searchable" set to "site setting" / "yes"? If not that might be what is preventing that Folder/Gallery from being marked as "searchable" in your robots.txt. (Double checking might just be worth it - has been worth it for me at least a couple of times before
Good luck
Lille Ulven
I checked again to make sure, and it is site setting / yes and over in the site settings I have the discover me things turned on.
This link checker has the option to ignore robots.txt, so I was able to find the half dozen or so broken ones. http://website-link-checker.online-domain-tools.com/ The firefox tool also worked, though only one page at a time.
I am still concerned that google is not crawling into my public pages like a good SE is supposed to.
http://redwoodtwig.com
Sony A7r4 with a selection of Rokinon Cine primes that I'm really enjoying learning how to use.
https://chrome.google.com/webstore/detail/check-my-links/ojkcdipcgfaekbeaelaapakgnjflfglf?hl=en-GB
It color-codes all the links on the page for Valid, Warnings, and Invalid.
Even checks my navbar links.
Yes, that one works fine, as does a similar one in firefox. But only one page at a time.
http://redwoodtwig.com
Sony A7r4 with a selection of Rokinon Cine primes that I'm really enjoying learning how to use.
Ahh, very true, yes.
I'm> @ChancyRat said:
I'm curious what others do regarding finding broken links on pages on your site?
I used this tool back in 2016 and found it immensely useful. However, when i went to re-add it to Chrome just now, a pop-up message told me this:
I don't actually understand what this message is saying. I mean, "websites you visit" - means the Smugmug page where my links are? Or the external URLS that the links go to?
Read and change? Change would mean---?
Would this be normal? Not excessive? If you think it is abnormal and excessive, can you recommend a better tool? Thanks.
@ChancyRat the best thing at this juncture would be to reach out to the Support Heroes with the details of the concern, help@smugmug.com. They will take a look and advise.
SmugMug Support Hero