automigration from Flickr to Smugmug?

dmd

I'm an increasingly unhappy Flickr user who'd like to move to Smugmug. I'm mostly unhappy because of (a) the ugly Flickr interface and (b) the HORRIFICALLY bad resizing algorithm Flickr uses.

The trouble is I've got more than 3500 photos on Flickr, with hundreds of tags. I've put many hundreds of hours into tagging, titling, and sorting my photos into sets. In order for me to switch to Smugmug, it would be absolutely essential that some automated tool exist that would copy all my photos from Flickr to Smugmug with tags, titles, and sets intact.

Both services support titles, descriptions, tags, and sets, and have API calls to read and write that information, so it doesn't seem like this would be very difficult really - the main challenge would be in detecting and recovering from errors.

Does such a thing exist already? Or is anyone planning on writing one? If it were the beginning of the summer, and not the end, I'd write it myself, but I don't have the time now as the second year of grad school starts for me in a few days...

Andy

dmd wrote:

I'm an increasingly unhappy Flickr user who'd like to move to Smugmug. I'm mostly unhappy because of (a) the ugly Flickr interface and (b) the HORRIFICALLY bad resizing algorithm Flickr uses.

The trouble is I've got more than 3500 photos on Flickr, with hundreds of tags. I've put many hundreds of hours into tagging, titling, and sorting my photos into sets. In order for me to switch to Smugmug, it would be absolutely essential that some automated tool exist that would copy all my photos from Flickr to Smugmug with tags, titles, and sets intact.

Does such a thing exist? Is anyone planning on writing one? If it were the beginning of the summer, and not the end, I'd write it myself, but I don't have the time now as the second year of grad school starts for me in a few days...

hiya dmd - that's such a great idea. i hope one of our smugmug hackers takes some serious thought to this...

Nikolai

S*E to the rescue
Star*Explorer would upload your 3,500 pictures like nothing. You would probably have to re-tag them again, but other than that - there is no limit to what you can do!:):
And if you have limited upload bandwidth - check out S*E's mass upload service...

HTH

Andy

Nikolai wrote:

Star*Explorer would upload your 3,500 pictures like nothing. You would probably have to re-tag them again, but other than that - there is no limit to what you can do!:):
And if you have limited upload bandwidth - check out S*E's mass upload service...

HTH

does s*e take them off of flickr?

luke_church

andy wrote:

does s*e take them off of flickr?

I will devote up to 2 hours of time to this this afternoon. I might be able to have something done that takes them off Flickr into folders.

I don't know about you, but re-tagging 3500 photos would be something of a show-stopper for me.

luke_church

dmd wrote:

Does such a thing exist already? Or is anyone planning on writing one? If it were the beginning of the summer, and not the end, I'd write it myself, but I don't have the time now as the second year of grad school starts for me in a few days...

Just to check you do run Windows don't you? The only way I can even try to develop something fast enough is against .NET. It'll be a very fast hack, but might just do the trick (esp. in combination with S*E)

Luke

tibu

A good reason to use IPTC
Hi

Just a short comment, this is one very good reason to use IPTC for captions and keywords. I've been using them for quite a while since smugmug supports them and I'm pretty sure flicker supports them too

In that case migrating should be a breeze, just reupload

Carlos

dmd

Let's see if I can respond to everything:

Retagging everything would definitely be a showstopper. The whole point of this is to preserve the metadata - tags, titles, descriptions, sets. Moving the photos themselves, THAT is easy. I could do that right now, with no effort. It's the metadata that's what's important to me, as I've spent hundreds if not thousands of hours of my life getting it right.

I don't run Windows on my own machine, but I do have access to one at work, so that's all right.

IPTC - I'm not sure what you'd propose using IPTC for in this context... generally tag/title/description/set information gets added to my photos days if not weeks after the photos are uploaded to Flickr. Once the photos are on Flickr, the IPTC data is not modified - that is to say, if you download the photo, the IPTC data does not contain up to date caption etc. information. IPTC is only used for initial population of these fields, which is useless for me, as I make extensive use of Flickr's batch organization tools (which I assume Smugmug has equivalents of?).

Nikolai

No, but..

andy wrote:

does s*e take them off of flickr?

... I was assuming that the man actually keeps his originals locally...

And as to the tagging.. well, I'm just copying the files over the Net, metadata is a separate part:-) This task looks essentially like porting data from one database to another, both databases being serving similar purposes but having different structure...

luke_church

OK, so 35/120 minutes in and I have a way of listing all of the meta-data from a Flikr user account.

DMD -> Please can you send me your Flikr username aso that I can do testing on it? If you'd prefer to do it privately PM on DGrin or email both work. The problem is that otherwise I'm kind of stuck with my limited 'test free' account, which I can't download much from because of the bandwidth restrictions.

dmd wrote:

Retagging everything would definitely be a showstopper. The whole point of this is to preserve the metadata - tags, titles, descriptions, sets. Moving the photos themselves, THAT is easy. I could do that right now, with no effort. It's the metadata that's what's important to me, as I've spent hundreds if not thousands of hours of my life getting it right.

Right, the issue here is how we associate the meta-data with the photos? Do you want me to trawl your photos from Flikr or has the resizing algorithm perminantly trashed them? If yes, we need to look at way of mapping the meta-data onto the data on your HD so that we can re-run the uploads to Smugmug...

(PS, if your hesitant about joing Smugmug if the meta-data transfer could be fixed, now is the time to say so, I'm kind of busy these days... :-))

>I don't run Windows on my own machine, but I do have access to one at work, so that's all right.

OK, it must be running at least .NET 1.1, if that isn't installed, you'll need Admin rights to install it, but don't worry, it's fairly painless. I've never seen a computer its damaged yet, and I have installed it on a lot of them!

Looking forwards to hearing from you

Cheers,

Luke

luke_church

Nikolai wrote:

...
And as to the tagging.. well, I'm just copying the files over the Net, metadata is a separate part:-) This task looks essentially like porting data from one database to another, both databases being serving similar purposes but having different structure...

indeed.

Speaking of which, I think there might be a slight issue...

Flickr as much as I can see only offers one level of categorisation, the 'Set'. Smugmug offers 3, Category -> Sub Category -> Album.

My inital thoughts are Set = Album, create a single Category, Sub Category and put all of the Albums in it.

Guidance anyone?

I'm switching to other work until I get a response from DMD on Meta-data file alignment and categories... (Have you changed your local file names after the files were uploaded? If not it should be possible to trivially auto-align them)

Luke

dmd

My Flickr username is 3e. You don't need my password; everything necessary for this process can be done through the API with read permissions.

If you're wondering how to associate metadata with photos, I guess you haven't started to look at the Flickr API documentation yet, as I can't imagine how you'd get the metadata without having it be associated with photos.

All the original photos are on Flickr - the computer the application runs on will only be a conduit. The photos are no longer on my own computer.

tibu

More in IPTC
Hi

In this context IPTC is useless.

The comment is that it is better to have all that info (title, description, keywords, location, etc) originally in the file as IPTC.

What I do is that right after storing the pictures in my PC I use some IPTC software (normally IMatch) to set those fields. Then when they are uploaded to Smugmug/Flickr/Other host that supports IPTC, they are automatically tagged and captioned.

Even better if whatever service somehow disappear you still have all your data safely stored in your local backup (assuming you have one of course)

I don't know if flickr has some exceptionally good tagging tools but I bet you can find similar capabilities on commonly available software.

Regards
Carlos

luke_church

dmd wrote:

My Flickr username is 3e.

Thank you.

You don't need my password; everything necessary for this process can be done through the API with read permissions.

Indeed, I got that bit thanks :-)

If you're wondering how to associate metadata with photos, I guess you haven't started to look at the Flickr API documentation yet, as I can't imagine how you'd get the metadata without having it be associated with photos.

Hmm... Strangely, I have been looking at the API documentaiton, I find it kind of helps when writing software. :

I was simply assuming in the same manner that Nikolai did that you would want to re-upload from locally cached originals, but as that isn't the case it certainly makes life easier for me, I'll just have to pull them down of Flickr.

No worries, I'll just scrape them.

Any comments as to how you would like them arranged in Categories/Albums?

Many thanks

Luke

dmd

luke_church wrote:

but as that isn't the case it certainly makes life easier for me, I'll just have to pull them down of Flickr.

Any comments as to how you would like them arranged in Categories/Albums?

Yeah, as the originals are all safe and sound on Flickr, just use those.

As for organization - Flickr, as you know, doesn't have subalbums (which is yet another reason I want to move!). For the time being, I'd just want the Flickr structure mirrored - that is, just generate all my sets as albums; once the Flickr structure is precisely mirrored on Smugmug, I can start moving stuff around how I want it.

By the way - are there plans in the works at Smugmug to richen the API? It seems woefully inadequate.

For instance:
adding new tags to an image on Flickr:

  flickr.photos.addTag(photo_id,tags)

adding new tags to an image on Smugmug:

  imageinfo = smugmug.images.getInfo(ImageID)
  keywords = newkeyword + imageinfo['Keywords']
  smugmug.images.changeSettings(ImageID,{,,keywords})

bmcent1

Luke_Church, Hi, I'm new here, on a trial membership to Smug Mug and alread a "Pro" on Flickr. I'm pretty sure I will become a member but I'm still mulling it.

Can you tell me if anything ever became of this thread? I find I need almost the exact same tool!

There are lots of things I really like about SmugMug, and I plan to keep more of my pics here than on Flickr. However, I am crazy about Flickr's "Organizr!" It just works really well for me.

I have heard of software such as Imatch, but I'd prefer to add keywords to my photos on Flickr after I upload because I often think of new ones and see associations as I upload more, organize, and browse what's there.

I need a tool to pull down my original uploads from Flickr (or find the images on my local drive), grab all the tags from Flickr, and then insert them into the IPTC keywords back on my local copy. Then I can upload the local copy to SmugMug with all the tags included and all will be in sync.

I like the idea of mapping Sets to Albums and sorting out categories and subs later.

Any chance of this happing soon? I'm a sys admin, not a great coder at apps over a few pages long, but I have some experience with PERL, PHP, and C... if there is anything I can do to help I'd give it a try!

Andy

bmcent1 wrote:

Luke_Church, Hi, I'm new here, on a trial membership to Smug Mug and alread a "Pro" on Flickr. I'm pretty sure I will become a member but I'm still mulling it.

Can you tell me if anything ever became of this thread? I find I need almost the exact same tool!

There are lots of things I really like about SmugMug, and I plan to keep more of my pics here than on Flickr. However, I am crazy about Flickr's "Organizr!" It just works really well for me.

I have heard of software such as Imatch, but I'd prefer to add keywords to my photos on Flickr after I upload because I often think of new ones and see associations as I upload more, organize, and browse what's there.

I need a tool to pull down my original uploads from Flickr (or find the images on my local drive), grab all the tags from Flickr, and then insert them into the IPTC keywords back on my local copy. Then I can upload the local copy to SmugMug with all the tags included and all will be in sync.

I like the idea of mapping Sets to Albums and sorting out categories and subs later.

Any chance of this happing soon? I'm a sys admin, not a great coder at apps over a few pages long, but I have some experience with PERL, PHP, and C... if there is anything I can do to help I'd give it a try!

I'll ping Luke for you

luke_church

bmcent1 wrote:

Luke_Church, Hi, I'm new here, on a trial membership to Smug Mug and alread a "Pro" on Flickr. I'm pretty sure I will become a member but I'm still mulling it.

Can you tell me if anything ever became of this thread? I find I need almost the exact same tool!

So the current status is that I have successfully migrated 1 user as a concept demonstration, and pretty much every spare hour I have is being put into building the alpha prototype for end users.

There are lots of things I really like about SmugMug, and I plan to keep more of my pics here than on Flickr. However, I am crazy about Flickr's "Organizr!" It just works really well for me.

Cool.

Any chance of this happing soon?

Yeah, lots of chance :

I'm a tad reluctant to commit to time frames, but something will have to go really badly wrong for the alpha build to not be out by the end of this month. If things go really well it might be more like the middle of next week....

if there is anything I can do to help I'd give it a try!

Fancy doing some early testing on the prototype?

It'll be running on a .NET 2.0 platform, so you'll need to be running Windows 2000 or XP. (I don't remember whether ME can do .NET 2.0, Win 98 can't)

Would this be a problem?

Hope this helps....

Luke

bmcent1

luke_church wrote:

Yeah, lots of chance :

I'm a tad reluctant to commit to time frames, but something will have to go really badly wrong for the alpha build to not be out by the end of this month. If things go really well it might be more like the middle of next week....

Wow! Awesome! Now I guess I have to make my SmugMug membership permanent... free trial expires tomorrow ... A tool like this / integration like this will make it very worth it!

luke_church wrote:

Fancy doing some early testing on the prototype?

It'll be running on a .NET 2.0 platform, so you'll need to be running Windows 2000 or XP. (I don't remember whether ME can do .NET 2.0, Win 98 can't)

Would this be a problem?

Hope this helps....

Luke

Sure!! I'd love to alpha test. ... so long as you're pretty sure it won't ruin my Win XP or local copies of my photos. No prob if it runs amok on the SM side, I'm just getting underway there anyway. I can put .net on my PC when your prototype is ready.

Don't want to cause feature creap, but I have a related idea:

How difficult do you think it would be to run a MD5SUM or SHA message digest on the image part of a JPEG file? I would love to be able to store this hash in some EXIF field, possibly even the title/name field, as a way to quickly find duplicates. If compressed differently, two images won't match, but at least the exact same image with different tags would match (if you can hash strictly the image data.) I never trust file names to be unique or match. Maybe a wishlist feature or possibly another utility altogether, but I thought it could be very handy.

Thanks again and let me know when you have something ready to test!

bmcent1

luke_church wrote:

I'm a tad reluctant to commit to time frames

Understandable! I hope things are going okay! Any better idea on time frame? Need any testing yet?

luke_church

bmcent1 wrote:

Understandable! I hope things are going okay! Any better idea on time frame? Need any testing yet?

Sorry, this is going to sound pathetic, but I'll tell it to you straight.

Current target date for alpha release is sometime around the 15th of March, hopefully a little sooner, maybe a few days later.

Reasons: (The bad news)

-> I'm currently in the middle of the software project from hell (and its nice associated 18 hour days)

-> I have serious RSI, which is currently in the middle of a bad patch, so every spare hour I can find I spend trying to rest my hands.

But now, the good news:

-> I have now released a product based on the same codebase as SmugFlickr, and its surviving in the field, even though it was only given very light testing. So it looks like the codebase is reasonably sound. SmugFlickr's test cases are a whole lot more aggressive, so it should be fairly sound software :

-> Workarounds for the remaining Smugmug bugs have been implemented and are passing their daily automated tests every time :

-> My client delivery on the project from hell is early March. Anything not done by then, won't be done. So once that's gone, I can switch back to finishing other stuff I should be doing now, then once that's done I'm all SmugFlickr... :

-> The hosting architecture for deliverying SmugFlickr and SmugTools to a serious user base and providing high-uptime servers for the updates it will need to pull down is all running and looking pretty solid. :

Have faith, it's coming.

Hopefully that didn't sound too wingy, I'm really sorry I couldn't give you a better answer.

Luke

luke_church

How difficult do you think it would be to run a MD5SUM or SHA message digest on the image part of a JPEG file? I would love to be able to store this hash in some EXIF field, possibly even the title/name field, as a way to quickly find duplicates. If compressed differently, two images won't match, but at least the exact same image with different tags would match (if you can hash strictly the image data.) I never trust file names to be unique or match. Maybe a wishlist feature or possibly another utility altogether, but I thought it could be very handy.

I smell version 2, this is a related feature that someone else has been asking for. :

Hope you're enjoying Smugmug in the meantime.

Luke

bmcent1

luke_church wrote:

Hope you're enjoying Smugmug in the meantime.

Oh yeah! Definately.

Thanks for the update. SmugFlickr should be great!

luke_church

Alpha test variant is now out in the wild. :

Please see: http://www.dgrin.com/showthread.php?p=289716

Sorry for the delay

Luke

Andy

luke_church wrote:

Alpha test variant is now out in the wild. :

Please see: http://www.dgrin.com/showthread.php?p=289716

Sorry for the delay

Luke

bmcent1

Awesome! Thanks for the heads up. I'll check it out soon and leave any productive feedback I can in the main thread.