Why is Facebook crawling through SmugMug?

W.W. Webster

StatCounter has reported Facebook crawling through my SmugMug site on a number of occasions over recent weeks, most recently in the past few minutes.



What possible interest can (or should) they have? I'm perfectly happy for Google, Bing, Yahoo and numerous other search engines to index my images - which they do - but Facebook? They don't provide a search function, and they have no rights to my images.

Does SmugMug facilitate or encourage Facebook in this? I don't trust those b*st*rds! :huh

Wendee

Hi DubDub,

Sorry about this, but I'm not really sure. Do you know if anyone has linked to your site on Facebook? If images were posted or linked that could be what's showing up. Like this one:

http://www.facebook.com/WellingtonPhotographicSociety/posts/213828098689791

I'd suggest trying to contact Facebook to see if they have more insight about why they'd be showing up as crawling your pages. That's not really helpful of me though, sorry!

W.W. Webster

Wendee wrote: »

Sorry about this, but I'm not really sure. Do you know if anyone has linked to your site on Facebook? If images were posted or linked that could be what's showing up.

Thanks, but no reason to be sorry, Wendee!

I don't see why if I, or anyone else, for that matter, links to a SmugMug image on Facebook, that Facebook needs to come nosing around my site! They've got the link, what else are they looking for?

I seldom see Google or Bing spooking around, but I imagine that's because SmugMug pushes keywords, etc to them. Some Asian search sites come looking, which is fine, but Facebook has no need for this data, as I see it.

jfriend

W.W. Webster wrote: »

StatCounter has reported Facebook crawling through my SmugMug site on a number of occasions over recent weeks, most recently in the past few minutes.



What possible interest can (or should) they have? I'm perfectly happy for Google, Bing, Yahoo and numerous other search engines to index my images - which they do - but Facebook? They don't provide a search function, and they have no rights to my images.

Does SmugMug facilitate or encourage Facebook in this? I don't trust those b*st*rds!

If you or someone else tries to post link to a page from your site on FB, FB will crawl the linked page to find out what's on it to try to make a title for the post, see if there are any images on that page that can be used a thumbnail, etc... If it's this, I don't think there's anything nefarious going on here - they're just try to make a decent user experience for people that try to post a link to your page.

W.W. Webster

jfriend wrote: »

If it's this, I don't think there's anything nefarious going on here - they're just try to make a decent user experience for people that try to post a link to your page.

No - it's not 'this'!

jfriend

W.W. Webster wrote: »

No - it's not 'this'!

How do you know it's not this?

W.W. Webster

jfriend wrote: »

How do you know it's not this?

Because the timing of Facebook's incursions is quite random and very occasional. They do not coincide with the timing of any links I create to my SmugMug images. Furthermore, Statcounter does not identify (i.e. know about?) the specific links they access.

But more to the point, this suggests SmugMug facilitates Facebook access, or can they crawl around unassisted? Surely this can't happen without SmugMug allowing it?

jfriend

W.W. Webster wrote: »

Because the timing of Facebook's incursions is quite random and very occasional. They do not coincide with the timing of any links I create to my SmugMug images. Furthermore, Statcounter does not identify (i.e. know about?) the specific links having been accessed.

But more to the point, does SmugMug facilitate this or can Facebook gain access unassisted? Surely they can't crawl around without SmugMug allowing it?

All public galleries can be crawled by anything. They're on the web for people or machines to look at whenever they want. That's how search engines find your public stuff. This should not be a surprise to you or be anything that you are concerned about. If it's on the internet and public, then it is available for viewing to all, man or machine.

Legitimate crawlers are supposed to follow the rules that Smugmug puts in the robots.txt file, but that is voluntary - there is no absolute enforcement.

Facebook could also be looking at your page when somebody else on the internet posts a link in their FB page to your site. If it's a crawler looking at your page, then statcounter would probably not see it because crawlers don't cause javascript to get executed which statcounter generally needs.

Andy

W.W. Webster wrote: »

Because the timing of Facebook's incursions is quite random and very occasional. They do not coincide with the timing of any links I create to my SmugMug images. Furthermore, Statcounter does not identify (i.e. know about?) the specific links they access.

But more to the point, this suggests SmugMug facilitates Facebook access, or can they crawl around unassisted? Surely this can't happen without SmugMug allowing it?

We don't do anything special for Facebook Like John says, they, like other sites can crawl your site (think: google, bing, yahoo)

W.W. Webster

Thanks John.

jfriend wrote: »

All public galleries can be crawled by anything. They're on the web for people or machines to look at whenever they want. That's how search engines find your public stuff.

Of course, but my point is that Facebook is not a 'search engine'!

jfriend wrote: »

Facebook could also be looking at your page when somebody else on the internet posts a link in their FB page to your site.

That's possible, I suppose, but again - why?

jfriend wrote: »

If it's a crawler looking at your page, then statcounter would probably not see it because crawlers don't cause javascript to get executed which statcounter generally needs.

I understand, as whenever (rarely) Google, Yahoo, or one of the Chinese or Korean search engines, etc come looking, StatCounter never reports specific page URLs being accessed, which is entirely understandable (and acceptable).

What I don't understand (nor appreciate) is that Facebook, which is not a search engine and which shows no regard for photographers' copyright in its terms of service, should be crawling about in SmugMug users' sites - even if (as you suggested previously), "they're just try(ing) to make a decent user experience for people that try to post a link to (my) page".

We have an expression in this country for claims and assertions as dubious as this - "yeah right"!

W.W. Webster

Andy wrote: »

We don't do anything special for Facebook

Thanks Andy - that's clear, and long may it continue!

jfriend

W.W. Webster wrote: »

Thanks John.
Of course, but my point is that Facebook is not a 'search engine'!

That's possible, I suppose, but again - why?

I understand, as whenever (rarely) Google, Yahoo, or one of the Chinese or Korean search engines, etc come looking, StatCounter never reports specific page URLs being accessed, which is entirely understandable (and acceptable).

What I don't understand (nor appreciate) is that Facebook, which is not a search engine and which shows no regard for photographers' copyright in its terms of service, should be crawling about in SmugMug users' sites - even if (as you suggested previously), "they're just try(ing) to make a decent user experience for people that try to post a link to (my) page".

We have an expression in this country for claims and assertions as dubious as this - "yeah right"!

I explained in my previous post the situation where I know that facebook looks at your page to suggest titles and find images associated with a link to your site that somebody else is putting on a facebook page. That seems a perfectly legitimate use to me, no different than all the other services that go looking for an image they can make a thumbnail out of to represent a link. If you want more info than we have here, then you probably need to go ask Facebook. I don't understand why you're so angry about it here and being so unappreciative of those trying to answer your questions.

W.W. Webster

jfriend wrote: »

I don't understand why you're so angry about it here and being so unappreciative of those trying to answer your questions.

I'm neither! I'm sorry you got that impression.

I'm just trying to understand what's going on and, as always, you've been very helpful!

Thanks!

jcdill

W.W. Webster wrote: »

Of course, but my point is that Facebook is not a 'search engine'!

Are you sure about that?

https://www.google.com/search?num=100&hl=en&q="does+facebook+crawl"

In this podcast, around the 35 minute mark, they talk about how/why Facebook crawls a site. They says that if your site has a FB connect or like button, FB crawls your site once every 24 hours.

They "say" the search results come from Bing, but I bet they rank higher the sites that have FB connect or like buttons, over non-FB connected sites.

jc