AlbumFetcher - Download multiple galleries w/ embedded captions and keywords
wellman
Registered Users Posts: 961 Major grins
*Admin Edit*
Hey everyone unfortunately AlbumFetcher is no longer being supported and I've removed the link fro the Wiki page. Try out RapidFetcher instead, it works with Macs, Win or Linux! (Adobe Air app).
You can find RapidFetcher along with other downloaders here: http://wiki.smugmug.net/display/SmugMug/Hacks+and+Apps
*/Admin Edit*
===
Note from Greg... AlbumFetcher still works! No development since 2009, but many folks still use it successfully.
Current version: v0.3.6 (July 8, 2009) - Zipped Application | Exe File (if you're afraid of zips) | Source
I'm pleased to announce the release of AlbumFetcher! AlbumFetcher is a little pet project of mine which I've decided to make freely available to the SmugMug community. I've received amazing amounts of help (and ideas) through these forums, and this my way of saying thanks. It's a simple little app that serves two main functions:
1. Download one or many SmugMug galleries at once
2. Embed captions and keywords into your downloaded images' IPTC headers
AlbumFetcher should run on Win2000, WinXP and above. It has been reported to run under Wine on Linux and through Parallels on a Mac. It's a single file, so it's easily portable and doesn't need to be installed. AlbumFetcher is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This DGrin thread will be the project's home page. Feedback is extremely appreciated, and if there are bugs and/or other issues (as there always are), please let me know here. This is a hobby, so I can't hope to live up to SmugMug's standard of customer service, but I will try to help you out if you're having an issue with the software. One word of caution - beware of the total number of bytes you download. You could easily run up against your monthly bandwith limit.
Again, this is yours to use forever for free.
Enjoy!
-Greg
How Does it Work?
UPDATE, Nov 14 2006:
I've incorporated some of Pat Kane's requests and some improvements of my own into v0.2. New features include:
-Greg
UPDATE, Nov 16 2006:
Pat Kane pointed out a bug in the folder selector, which is now fixed. Current version is now 0.2.1. Thanks, Pat!
UPDATE, Jan 11 2008:
I'm pleased to announce the release of AlbumFetcher v0.3. There are several improvements in this release. Read on for details.
UPDATE, Jan 12 2008:
Scott Walter pointed out a bug which caused the app to crash when downloading GIFs, which is now fixed. Current version is now 0.3.1. Thanks, Scott!
UPDATE, Jan 29 2008:
A few people were having similar issues. Basically AlbumFetcher was bombing out if it tried to download an image which was missing (for whatever rare reason) on SmugMug. This is now handled more gracefully in new Version 0.3.2 (link above) - AlbumFetcher will stop the download and report the error (instead of crashing).
UPDATE, Mar 3 2008:
AlbumFetcher is now compatible with SmugMug's security changes (use of keys).
UPDATE, Dec 15 2008:
It's been a pleasure over the past two years to see AlbumFetcher become a useful little download utility for members of the SmugMug family. The users of AlbumFetcher deserve a product that will continue to evolve and keep up with the latest SmugMug features (hidden photos, SmugVault, etc). To use the catchphrase circulating at work, I no longer have the personal bandwidth to keep AlbumFetcher current.
This thread is a call for help for someone to take the AlbumFetcher reins. The code is simple Microsoft Visual C++, using MFC. If there's anyone interested in taking the project over as-is or porting it to something cross-platform, let me know.
Thanks, and thanks again to those who've had kind words and future ideas for AlbumFetcher.
UPDATE, June 17 2009:
Version 0.3.4 is out. This fixes some issues uncovered by some API changes. AF has to be smarter about how it reads responses from SmugMug now. Also, a fix suggested by t0mt0m related to photos from iPhones has been incorporated. Thanks for your patience on this one; the issue was hard to find.
UPDATE, July 7 2009:
Version 0.3.5 is out. Thanks to derf for finding a bug which caused AlbumFetcher to die when the filename recorded by SmugMug contains invalid characters for a directory structure. Now fixed.
UPDATE, July 8 2009:
I forgot to update one of the version number lines of text in v0.3.5. To keep things straight, I updated everything to v0.3.6, which is now available via the links above.
UPDATE, March 23 2010:
Thanks to SamirD, AlbumFetcher has a more permanent home. Drop.io is great, but it's 1000 limit was making it a pain to keep the files served up consistently. Thanks for the space, SamirD! Links updated above...
UPDATE, Aug 15 2011:
AlbumFetcher is still alive and kicking! Moved download links to public Dropbox space. Links updated above...
UPDATE, Aug 1 2013:
AlbumFetcher is currently dead, given SmugMug's new awesomeness and new API. It might be easily resurrected by someone who can do some C++ updating. Source code is still available.
Hey everyone unfortunately AlbumFetcher is no longer being supported and I've removed the link fro the Wiki page. Try out RapidFetcher instead, it works with Macs, Win or Linux! (Adobe Air app).
You can find RapidFetcher along with other downloaders here: http://wiki.smugmug.net/display/SmugMug/Hacks+and+Apps
*/Admin Edit*
===
Note from Greg... AlbumFetcher still works! No development since 2009, but many folks still use it successfully.
Current version: v0.3.6 (July 8, 2009) - Zipped Application | Exe File (if you're afraid of zips) | Source
I'm pleased to announce the release of AlbumFetcher! AlbumFetcher is a little pet project of mine which I've decided to make freely available to the SmugMug community. I've received amazing amounts of help (and ideas) through these forums, and this my way of saying thanks. It's a simple little app that serves two main functions:
1. Download one or many SmugMug galleries at once
2. Embed captions and keywords into your downloaded images' IPTC headers
AlbumFetcher should run on Win2000, WinXP and above. It has been reported to run under Wine on Linux and through Parallels on a Mac. It's a single file, so it's easily portable and doesn't need to be installed. AlbumFetcher is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This DGrin thread will be the project's home page. Feedback is extremely appreciated, and if there are bugs and/or other issues (as there always are), please let me know here. This is a hobby, so I can't hope to live up to SmugMug's standard of customer service, but I will try to help you out if you're having an issue with the software. One word of caution - beware of the total number of bytes you download. You could easily run up against your monthly bandwith limit.
Again, this is yours to use forever for free.
Enjoy!
-Greg
How Does it Work?
- Download AlbumFetcher (zip file) with the link above.
- Use WinZip or your favorite zip program to extract AlbumFetcher.exe (the program) to the folder of your choice (maybe the desktop).
- Double-click the AlbumFetcher program icon.
- If you're an account owner: In the top-right corner, enter your SmugMug ID and password. If you're a guest: Switch the login type selector to Guest. Then enter the SmugMug ID of the account you'd like to work with. You'll only have access to listed public galleries. (SmugMug ID means either your email address which is registered with SmugMug or the yourname part of yourname.smugmug.com. Your ID and password go directly to SmugMug; there's nobody listening in between.)
- Click List Albums. A listing of all your albums will appear.
- In the Download Options box (on the right), pick a folder to save albums to. Also pick the image sizes and other options as you wish.
- Click the list to select albums for download. Use Ctrl-clicking or Shift-clicking to select more than one.
- Click Fetch Albums. AlbumFetcher will now download your requested albums at the size you specified.
- Have a tasty beverage.
UPDATE, Nov 14 2006:
I've incorporated some of Pat Kane's requests and some improvements of my own into v0.2. New features include:
- Datetime stamp for each line in log file
- Files with same original name are now handled successfully (an underscore is prepended to the second file's name).
- Users can optionally skip over previously downloaded images (helpful when a download fails and you want to start over, but not re-download all the successful images).
- Category/subcategory info is shown as a field in the list control.
- A thumbnail of the image currently being downloaded is displayed.
- Total time statistics are recorded when a job is completed successfully.
- MD5 hashes are now compared. Files that fail the comparison are left alone, but a note is made in the log file.
-Greg
UPDATE, Nov 16 2006:
Pat Kane pointed out a bug in the folder selector, which is now fixed. Current version is now 0.2.1. Thanks, Pat!
UPDATE, Jan 11 2008:
I'm pleased to announce the release of AlbumFetcher v0.3. There are several improvements in this release. Read on for details.
- Multiple login options - Users may now login to SmugMug as account owner, guest, or guest with site password. If you're uncomfortable with other people being able to bulk-download your images, set your external linking option (in gallery customization) to "No."
- Album-level password support - Guests may download password-protected galleries (with the proper password of course).
- Selectable image size for download - Choose which size you want! AlbumFetcher will find the next largest available if logged in as guest and requested size is not available.
- Video support - Choose which size you want! AlbumFetcher will find the next largest available if logged in as guest and requested size is not available. Support is also included for legacy mpeg-1 videos.
- As inspired by SmugMug, AlbumFetcher is stretchy!
- No more runtime errors - (Fingers crossed.) AlbumFetcher has essentially been completely rewritten. I found a few scenarios that caused the app to crash during my development process, and they've been eliminated. I've successfully downloaded thousands of images while testing. Let me know if you still have runtime problems.
UPDATE, Jan 12 2008:
Scott Walter pointed out a bug which caused the app to crash when downloading GIFs, which is now fixed. Current version is now 0.3.1. Thanks, Scott!
UPDATE, Jan 29 2008:
A few people were having similar issues. Basically AlbumFetcher was bombing out if it tried to download an image which was missing (for whatever rare reason) on SmugMug. This is now handled more gracefully in new Version 0.3.2 (link above) - AlbumFetcher will stop the download and report the error (instead of crashing).
UPDATE, Mar 3 2008:
AlbumFetcher is now compatible with SmugMug's security changes (use of keys).
UPDATE, Dec 15 2008:
It's been a pleasure over the past two years to see AlbumFetcher become a useful little download utility for members of the SmugMug family. The users of AlbumFetcher deserve a product that will continue to evolve and keep up with the latest SmugMug features (hidden photos, SmugVault, etc). To use the catchphrase circulating at work, I no longer have the personal bandwidth to keep AlbumFetcher current.
This thread is a call for help for someone to take the AlbumFetcher reins. The code is simple Microsoft Visual C++, using MFC. If there's anyone interested in taking the project over as-is or porting it to something cross-platform, let me know.
Thanks, and thanks again to those who've had kind words and future ideas for AlbumFetcher.
UPDATE, June 17 2009:
Version 0.3.4 is out. This fixes some issues uncovered by some API changes. AF has to be smarter about how it reads responses from SmugMug now. Also, a fix suggested by t0mt0m related to photos from iPhones has been incorporated. Thanks for your patience on this one; the issue was hard to find.
UPDATE, July 7 2009:
Version 0.3.5 is out. Thanks to derf for finding a bug which caused AlbumFetcher to die when the filename recorded by SmugMug contains invalid characters for a directory structure. Now fixed.
UPDATE, July 8 2009:
I forgot to update one of the version number lines of text in v0.3.5. To keep things straight, I updated everything to v0.3.6, which is now available via the links above.
UPDATE, March 23 2010:
Thanks to SamirD, AlbumFetcher has a more permanent home. Drop.io is great, but it's 1000 limit was making it a pain to keep the files served up consistently. Thanks for the space, SamirD! Links updated above...
UPDATE, Aug 15 2011:
AlbumFetcher is still alive and kicking! Moved download links to public Dropbox space. Links updated above...
UPDATE, Aug 1 2013:
AlbumFetcher is currently dead, given SmugMug's new awesomeness and new API. It might be easily resurrected by someone who can do some C++ updating. Source code is still available.
0
Comments
Fast and easy.
Thanks....
Really nice job - it works wonderful!
Smug since 2006
SmugMug Help
PhotoscapeDesign
Sounds pretty sweet mate
I haven't had a chance to try it out yet but I will when I get a chance.
Thanks for contributing.
David
SmugMug API Developer
My Photos
I just tried the application and downloaded one gallery. Total download was about 55-MBytes and it only took a few minutes.
I'm pleased to say that the program was easy to use and it worked w/out problems.
Here are a few feature requests to consider for a later version:
1. Add a date/time stamp to the log file in front of each recorded action, e.g.,
2006-11-12 18:44:22 Totaling bytes for album Test.
2006-11-12 18:44:34 Downloading album description for Test.
2006-11-12 18:44:40 Downloading IMG_1117.JPG from Test (1 of 25 total).
2. Once a download is complete, present some statistics in the Status window, e.g., Successfully downloaded 25 images, 55-Mbytes total in 3 minutes 14 seconds.
3. Show a thumbnail of the image currently being downloaded.
4. Allow a pick list for the e-mail address box (handy for users with more than one account)
5. Allow the list of Galleries (Albums) to be sorted.
6. Retain the Category > Subcategory > Album structure from the user's site and allow that to be shown where Album Title is currently shown.
These are just suggestions. Even without any of them being implemented, this is still a very nice utility. Thanks again for sharing.
Regards,
Pat
1. What happens to an original image's IPTC data (assuming it had data when it was uploaded) if the "Embed IPTC data" box is checked? Is the original data appended, overwritten, other?
2. Are you using an existing, well-established library to write the IPTC data to the originals?
Thanks for the kind words! I'm glad you like it (or in devbobo's case, the idea of it ).
Pat, thanks for the great suggestions; I'll get some of those worked into v0.2. Let me see if I can answer some of your questions...
Any IPTC data that is pre-existing in the image is preserved, with the exception of caption and keyword, which are overwritten with the SmugMug values.
For example, say you captioned an image in Bridge or Lightroom and put some geographical data in an IPTC field, too. Now you upload to SmugMug. SM should read the caption out for you and place it in SM's database. Now say you edit the pre-populated caption to something else. AlbumFetcher (if the IPTC box is checked) will overwrite the original's embedded caption with the one from SmugMug, but that geographical data will be untouched. Make sense?
I'm using the Exiv2 library (with some edits to get it to compile under MS VC++ 2005). It's GPL'ed, which is why I GPL'ed my project, too.
Swim for Them | WellmanHouse.net | AlbumFetcher | SmugShowBuilder
Version 0.2 of AlbumFetcher is out. Details in the first post...
Swim for Them | WellmanHouse.net | AlbumFetcher | SmugShowBuilder
Here's a minor bug -- I selected Cancel in the Download to Folder Select dialogue box and the previously selected folder name was replaced with a back slash. On Cancel, I would have expected the folder name to be preserved.
IPTC explanation received and understood. I figured that this would be the behaviour. I add quite a bit of metadata to my images and even though smugmug doesn't currently use it all (e.g., headline, city, state, etc.), I didn't see any reason for it not to be preserved on download. Fortunately, you confirmed that it would be.
Thanks for the pointer to the Exiv2 library. Their web site says the right types of things I was looking for, e.g., "a sophisticated write algorithm avoids corrupting the MakerNote."
Since you brought it up here's another suggestion for a future release: fill in the EXIF GPSInfo tags on download with any latitude/longitude assignment made within smugmug. I'm not sure if this information even available via the smugmug API, but it would be handy for those that don't geocode their images prior to upload. This isn't something I'm requesting for myself (I geocode prior to upload), but it seems like it would be a nice feature for those that use smugmug's tools. I'm sure this would be low on the priority list of things to do, but I figured I'd throw it out there anyway!
Check your paypal for a few drinks on me. You've earned them.
Oops. Thanks for the heads-up. I'll take care of that one.
If you want to verify this for yourself, Google a free app called Exifer. It will show you (and allow you to edit) all EXIF and IPTC tags. I did some testing to make sure other IPTC fields were left alone, but I wouldn't mind some secondary confirmation.
That's a fantastic idea. That will also go into the list for v0.3. My goal here was to make a "true" Smugmug backup tool that would save all my caption/keyword work, since I do most of mine on the site. The GPSInfo is a further extension of the same principle, so I will be happy to work that in.
Sweet! Thank you! Thanks also for your suggestions.
Swim for Them | WellmanHouse.net | AlbumFetcher | SmugShowBuilder
I still have Exifer on the hard drive. I used to use it to rename my photos to a date/time stamp, but that job is now taken care of by Downloader Pro.
It will probably be a couple of weeks before I get a chance, but I'd be glad to run some images through the process to see if any metadata gets clipped.
I took a look at the API documentation, and I don't see this as being available, yet. Once Smugmug makes it available, I can use the Exiv2 library to get this implemented. I'll make an official feature request.
Swim for Them | WellmanHouse.net | AlbumFetcher | SmugShowBuilder
Has anyone seem the same tool for Flickr? Have you worked with the Flickr API to do the same thing? Specifically I am interested in using Flickr as a geotagging interface because it works so well. Here is a relevant link...
http://www.flickr.com/forums/ideas/28952/
Thanks! Glad you liked it.
I'm not exactly sure what might be going on here, but I have some ideas. It's possible that whatever image managemant program you're using (before you upload to SmugMug) is sticking the image number in as a keyword (I've seen this before), and that the same thing is happening with titles showing up in keywords.
Nothing too fancy is going on with AlbumFetcher. It just overwrites the original image's caption and keyword IPTC fields with the strings provided by SM's API. If you'd like to check out what data is embedded in an image before AlbumFetcher does its work, Exifer is a good little free tool.
If there really does seem to be something squirrelly going on with AlbumFectcher, you can send me a link to an original image that will work as an example (you can PM me the link if you like). That being said, I'm on holiday and away from my development tools for a few days.
I haven't ever used Flickr, and honestly don't have plans to in the near future. It's a good idea, though. Hopefully SM will beef up their geotagging interface soon.
Thanks again for the kind words!
-Greg
Swim for Them | WellmanHouse.net | AlbumFetcher | SmugShowBuilder
Yes and it's a smugmug behaviour. smugmug tries to make keywords from file names. Sometimes it makes sense to do so, e.g., "Jim and Steve.jpg", and other times it does not, e.g., "Picture 022.jpg".
You probably don't see 22 as a keyword on your smugmug page as smugmug is programmed to ignore it; however, if you go to that image on smugmug and click on Edit next to your Keywords, you'll see 22 as one of the entries. Yes, it is there even if it doesn't show up.
Because it is there, AlbumFetcher adds it as a keyword on download.
Thanks for the great explanation!
Swim for Them | WellmanHouse.net | AlbumFetcher | SmugShowBuilder
I needed to grab several images the other day. This would have been a great tool to have had. At least, I'll have it next time.
Any chance that future development might include multiple individual images within a gallery/album?
Thank you so much for your efforts.
My galleries: http://ghealy.smugmug.com
My Blog: http://photosbyhealy.blogspot.com
It would take a little work, but that's a good idea. It will probably be a good while before I get back around to adding new features, but I'll put this on the list.
Swim for Them | WellmanHouse.net | AlbumFetcher | SmugShowBuilder
This is so great. Thanks for doing this!
Portfolio • Workshops • Facebook • Twitter
You're quite welcome. It's been nice to "give back" a little, after all the great help I've received from this forum.
Swim for Them | WellmanHouse.net | AlbumFetcher | SmugShowBuilder
www.ivarborst.nl & smugmug
-Greg
Swim for Them | WellmanHouse.net | AlbumFetcher | SmugShowBuilder
Actually, yes, but with a caveat. I would only do it for *specific* galleries/folders and only with a specific group of people.
I would do it for personal wedding (snapshots), parties, family gathering, etc. I normally have a p&s with me at all times, and I have captured some very blackmail worthy shots since and everything wants copies of all of them.
2) I think letting other people download your images is fine if you've made them public... If you've password protected them, or perhaps even locked the originals away then this needs to remain protected / locked away (I don't have a problem with people downloading the Medium / Large images instead of course)... But I don't see why anyone should have a problem with people downloading whole galleries...
If people can do it already (albeit manually), allowing them to do it via this tool is a great idea...
3) I too do a lot of my sorting and keywording on SmugMug... It really helps later when you're trying to track down stuff... So the feature request (I think it's a big one... Are you sure you're ready?):
Downloads based on keywords! Letting people choose to do this within a category / sub-category would be excellent option too of course...
The reason I thought of this is because I'm busy making a physical photobook as a present, and I'd love to be able to download all my "best" tagged images within certain galleries / categories / sub-categories... It would save me oodles of time! You get the picture?
Cheers!
85mm f/1.8, 17-50 + 28-75m f/2.8 lenses
iMac 24" 2.8 GHz Intel Core 2 Extreme, 4Gb RAM, OSX 10.5.7
http://encosion.com/ | http://encosion.smugmug.com/
Other than that problem this tool will come in handy in the future. Thanks for writing it and making it available.
I Reject Your Reality And Substitute My Own - Adam Savage
The keyword thing is an interesting idea. I'd have to delve back into the code to be sure, but I don't think it would be that bad to implement. I'm grabbing all the metadata before the photos download, so it would just be a matter of scanning for matches and paring down the list.
This will almost certainly be in the next version, but I can't say when that will come.
Swim for Them | WellmanHouse.net | AlbumFetcher | SmugShowBuilder
I am not saying that I disagree with you, but I know lot of photogs especially in the Pro Forum that might start foaming at the mouth over this feature.
SmugMug API Developer
My Photos
Thanks for your kind words. What kind of file names were giving you problems? I'd like to get that fixed if possible.
-Greg
Swim for Them | WellmanHouse.net | AlbumFetcher | SmugShowBuilder
This seems like a bit of a pandora's box to me. I think it would be very useful for some, but the potential for abuse seems very high. Imagine someone with dishonorable intentions getting ahold of this thing and starting some sort of DOS attack on SmugMug by bulk downloading bajillions of images.
If this were to work, I think what I'd really need would be a new option for the users in their gallery customizations - something like "allow bulk download." It would default to FALSE, and need to be available via the API. Of course, that implies a system-wide change to Smugmug just for my little app, which is unlikely, I would think. However, giving people explicit control over whether they want to allow this would be the ONLY way it could be pulled off successfully, IMO.
Thoughts?
Swim for Them | WellmanHouse.net | AlbumFetcher | SmugShowBuilder
In the spirit of the Internet, personally I feel that if it's "public", then it shouldn't matter how the person gets hold of it...
On the "DOS" issue: If someone's going to go through the trouble to do that (a botnet targeting random SmugMug sites) there's nothing currently stopping them from doing so... Whether you create this feature in your app or not, they'll still do it - they certainly won't be lacking the skills!
85mm f/1.8, 17-50 + 28-75m f/2.8 lenses
iMac 24" 2.8 GHz Intel Core 2 Extreme, 4Gb RAM, OSX 10.5.7
http://encosion.com/ | http://encosion.smugmug.com/
Well, I'm not sure how or when it happened but it seems that one time while uploading my images instead of the image file name alone the whole file path was uploaded. Another strange file name occured while using DxO Optics Pro. The photoshop plugin creates a file that looks like this: "[DefaultOutput] 7093_Ren Faire 0.jpg" That not only tripped up AlbumFetcher but also froze my galleries up pretty good. Hope that helps.
I Reject Your Reality And Substitute My Own - Adam Savage