AlbumFetcher - Download multiple galleries w/ embedded captions and keywords

wellman

poppopjohn wrote:

I need to download a number of smugmug photos that I've gathered up by a keyword search. Is there anyway to download photos displayed on smugmug via keyword search? I sent an email to smugmug asking if there's a way to create an album from photos resulting from a keyword search and they said no.

I've had this request before, and it would be very useful, I'll admit. I'll start mulling how to do it...

wellman

OmahaMama wrote:

I put a little something into your PayPal.

Thanks!

OmahaMama wrote:

And any of you reading this who PANICKED like I did when it didn't work, you should send a little something his way, too. Hint hint.

And double-thanks!

wellman

wellman wrote:

As best I can test, AlbumFetcher is back in working order (updated to v0.3.4). Thanks for your patience, folks.

You can find the updated links at the top of the thread.

After releasing v0.3.4, I've gotten a lot of good suggestions/requests for future improvements. If you have any more, let me know...

• Keyword-based downloading (an oldie but goodie)
• Fetch photos based on gallery or sharegroup URL
• Auto-restart after abort

derf

Still getting runtime errors, when in a folder with many photos, I tried to delete the files it was having a problem with but then it just errors on the next file. Awesome software, but this kink is driving me nuts. The software is great on folders that do not have this issue. Any ideas on changes I can make.

wellman

derf wrote:

Still getting runtime errors, when in a folder with many photos, I tried to delete the files it was having a problem with but then it just errors on the next file. Awesome software, but this kink is driving me nuts. The software is great on folders that do not have this issue. Any ideas on changes I can make.

If you like, you can send me a link to a gallery you're having problems with, assuming it's public and non-passworded. (PM if you like, rather then replying in-thread.) Perhaps I can debug.

I'm headed out into the wilderness with the family for a few days, so it will be a while before I can examine.

derf

http://derf.smugmug.com/gallery/832225_W9sEk#37169210_t7kzZ
and
http://derf.smugmug.com/gallery/830976_yJVdr#37169223_PccZA

I will pm my user and pass :ivar

wellman

New version 0.3.5
New version 0.3.5 is available, fixing a bug uncovered by derf. Original post updated.

agallia

wellman wrote:

New version 0.3.5 is available, fixing a bug uncovered by derf. Original post updated.

Greg, where is 0.3.5 available?

Andy

agallia wrote:

Greg, where is 0.3.5 available?

wellman wrote:

Original post updated

http://www.dgrin.com/showpost.php?p=422013&postcount=1

agallia

Andy wrote:

http://www.dgrin.com/showpost.php?p=422013&postcount=1

Downloaded and ran. Still says 0.3.4 at top of window?

wellman

agallia wrote:

Downloaded and ran. Still says 0.3.4 at top of window?

Whoops. Goof on my part not updating the text in the title bar. Ugh. I'll fix tomorrow and post back. Sorry 'bout that...

harish_agawane

How Can I fetch album with waterMark
Hi Greg,

How can I download album with watermark as I want to upload those photos auther sites like flicker, Facebook, webs.

Waiting for your reply

Thanks & Regards,
Harish S. Agawane
http://www.harish-agawane.com/

wellman wrote:

**Mac User? Go here.**

I'm pleased to announce the release of AlbumFetcher! AlbumFetcher is a little pet project of mine which I've decided to make freely available to the SmugMug community. I've received amazing amounts of help (and ideas) through these forums, and this my way of saying thanks. It's a simple little app that serves two main functions:

1. Download one or many SmugMug galleries at once
2. Embed captions and keywords into your downloaded images' IPTC headers

AlbumFetcher should run on Win2000, WinXP and above. It has been reported to run under Wine on Linux and through Parallels on a Mac. It's a single file, so it's easily portable and doesn't need to be installed. AlbumFetcher is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This DGrin thread will be the project's home page. Feedback is extremely appreciated, and if there are bugs and/or other issues (as there always are), please let me know here. This is a hobby, so I can't hope to live up to SmugMug's standard of customer service, but I will try to help you out if you're having an issue with the software. One word of caution - beware of the total number of bytes you download. You could easily run up against your monthly bandwith limit.

Again, this is yours to use forever for free. However, if you're a fan of my little app and feel so inclined, you can send me the electronic equivalent of the adult beverage of your choice here.

Enjoy!
-Greg


Current version: v0.3.5 (July 7, 2009) - Application | Source


How Does it Work?

1. Download AlbumFetcher (zip file) with the link above.
2. Use WinZip or your favorite zip program to extract AlbumFetcher.exe (the program) to the folder of your choice (maybe the desktop).
3. Double-click the AlbumFetcher program icon.
4. If you're an account owner: In the top-right corner, enter your SmugMug ID and password. If you're a guest: Switch the login type selector to Guest. Then enter the SmugMug ID of the account you'd like to work with. You'll only have access to listed public galleries. (SmugMug ID means either your email address which is registered with SmugMug or the yourname part of yourname.smugmug.com. Your ID and password go directly to SmugMug; there's nobody listening in between.)
5. Click List Albums. A listing of all your albums will appear.
6. In the Download Options box (on the right), pick a folder to save albums to. Also pick the image sizes and other options as you wish.
7. Click the list to select albums for download. Use Ctrl-clicking or Shift-clicking to select more than one.
8. Click Fetch Albums. AlbumFetcher will now download your requested albums at the size you specified.
9. Have a tasty beverage.

UPDATE, Nov 14 2006:
I've incorporated some of Pat Kane's requests and some improvements of my own into v0.2. New features include:

1. Datetime stamp for each line in log file
2. Files with same original name are now handled successfully (an underscore is prepended to the second file's name).
3. Users can optionally skip over previously downloaded images (helpful when a download fails and you want to start over, but not re-download all the successful images).
4. Category/subcategory info is shown as a field in the list control.
5. A thumbnail of the image currently being downloaded is displayed.
6. Total time statistics are recorded when a job is completed successfully.
7. MD5 hashes are now compared. Files that fail the comparison are left alone, but a note is made in the log file.

Again, enjoy!
-Greg


UPDATE, Nov 16 2006:
Pat Kane pointed out a bug in the folder selector, which is now fixed. Current version is now 0.2.1. Thanks, Pat!


UPDATE, Jan 11 2008:
I'm pleased to announce the release of AlbumFetcher v0.3. There are several improvements in this release. Read on for details.

1. Multiple login options - Users may now login to SmugMug as account owner, guest, or guest with site password. If you're uncomfortable with other people being able to bulk-download your images, set your external linking option (in gallery customization) to "No."
2. Album-level password support - Guests may download password-protected galleries (with the proper password of course).
3. Selectable image size for download - Choose which size you want! AlbumFetcher will find the next largest available if logged in as guest and requested size is not available.
4. Video support - Choose which size you want! AlbumFetcher will find the next largest available if logged in as guest and requested size is not available. Support is also included for legacy mpeg-1 videos.
5. As inspired by SmugMug, AlbumFetcher is stretchy!
6. No more runtime errors - (Fingers crossed.) AlbumFetcher has essentially been completely rewritten. I found a few scenarios that caused the app to crash during my development process, and they've been eliminated. I've successfully downloaded thousands of images while testing. Let me know if you still have runtime problems.

UPDATE, Jan 12 2008:
Scott Walter pointed out a bug which caused the app to crash when downloading GIFs, which is now fixed. Current version is now 0.3.1. Thanks, Scott!


UPDATE, Jan 29 2008:
A few people were having similar issues. Basically AlbumFetcher was bombing out if it tried to download an image which was missing (for whatever rare reason) on SmugMug. This is now handled more gracefully in new Version 0.3.2 (link above) - AlbumFetcher will stop the download and report the error (instead of crashing).


UPDATE, Mar 3 2008:
AlbumFetcher is now compatible with SmugMug's security changes (use of keys).


UPDATE, Dec 15 2008:
It's been a pleasure over the past two years to see AlbumFetcher become a useful little download utility for members of the SmugMug family. The users of AlbumFetcher deserve a product that will continue to evolve and keep up with the latest SmugMug features (hidden photos, SmugVault, etc). To use the catchphrase circulating at work, I no longer have the personal bandwidth to keep AlbumFetcher current.

This thread is a call for help for someone to take the AlbumFetcher reins. The code is simple Microsoft Visual C++, using MFC. If there's anyone interested in taking the project over as-is or porting it to something cross-platform, let me know.

Thanks, and thanks again to those who've had kind words and future ideas for AlbumFetcher.


UPDATE, June 17 2009:
Version 0.3.4 is out. This fixes some issues uncovered by some API changes. AF has to be smarter about how it reads responses from SmugMug now. Also, a fix suggested by t0mt0m related to photos from iPhones has been incorporated. Thanks for your patience on this one; the issue was hard to find.


UPDATE, July 7 2009:
Version 0.3.5 is out. Thanks to derf for finding a bug which caused AlbumFetcher to die when the filename recorded by SmugMug contains invalid characters for a directory structure. Now fixed.

wellman

wellman wrote:

Whoops. Goof on my part not updating the text in the title bar. Ugh. I'll fix tomorrow and post back. Sorry 'bout that...

Moved to version 0.3.6 with all the version numbers matching each other. Links at top of thread.

wellman

harish_agawane wrote:

Hi Greg,

How can I download album with watermark as I want to upload those photos auther sites like flicker, Facebook, webs.

Waiting for your reply

Thanks & Regards,
Harish S. Agawane
http://www.harish-agawane.com/

To download shots with your watermark, log in to your site through AlbumFetcher as a guest. Assuming you've applied watermarks to a particular gallery, any non-original-sized image you download will have the watermark. (Originals do not get watermarked by SmugMug.)

agallia

wellman wrote:

Moved to version 0.3.6 with all the version numbers matching each other. Links at top of thread.

Excellent, Greg! Used 0.3.6 as 'owner' on both desktop and laptop with no problems. Downloaded quickly including album title .txt file. Thanks for your hard work! Much appreciated.

If there is a wish list, add option to change download folder so we don't have to do it each time AlbumFetcher is used.

Allen

agallia wrote:

Excellent, Greg! Used 0.3.6 as 'owner' on both desktop and laptop with no problems. Downloaded quickly including album title .txt file. Thanks for your hard work! Much appreciated.

If there is a wish list, add option to change download folder so we don't have to do it each time AlbumFetcher is used.

Isn't the download folder user browser dependent?


Greg, one thing that I think confuses some is how to download, took me awhile to figure out, kept looking for zip link.
Can you add a line under this in post #1
Current version: v0.3.6 (July 8, 2009) - Application | Source
Clicking Application or Source will open another page with the zip file download button.

agallia

Allen wrote:

Isn't the download folder user browser dependent?


Greg, one thing that I think confuses some is how to download, took me awhile to figure out, kept looking for zip link.
Can you add a line under this in post #1
Current version: v0.3.6 (July 8, 2009) - Application | Source
Clicking Application or Source will open another page with the zip file download button.

I was referring to downloading albums, not downloading the AlbumFetcher .exe program. Default for downloading albums to my computer is always C:/AlbumFetcher unless I change it each time.

wellman

agallia wrote:

I was referring to downloading albums, not downloading the AlbumFetcher .exe program. Default for downloading albums to my computer is always C:/AlbumFetcher unless I change it each time.

I'll file that idea as a possibility. AlbumFetcher is designed to be light and portable. However, I suppose I could allow the option to save some user options to a settings file, which would be used if it exists. I'll mull it over...

wellman

Allen wrote:

Isn't the download folder user browser dependent?


Greg, one thing that I think confuses some is how to download, took me awhile to figure out, kept looking for zip link.
Can you add a line under this in post #1
Current version: v0.3.6 (July 8, 2009) - Application | Source
Clicking Application or Source will open another page with the zip file download button.

OK, done.

Jimbo

File Names
Great job on the downloader! I have a question. If I login in as owner and download a gallery, the original filename is used. If I login as Guest, a random number (assigned by SmugMug) is used. I was wondering if there is a way to have this downloader use the original file name when loging in as a guest to a password protected gallery. This way if friends that have downloaded a gallery can refer to the filename and I will know what image they are refering to. Is this possible? Thanks, Jimbo

sirensofsong

Thanks for this great tool. It will really help with posting watermarked samples of my smugmug images to flickr.

wellman

Jimbo wrote:

Great job on the downloader! I have a question. If I login in as owner and download a gallery, the original filename is used. If I login as Guest, a random number (assigned by SmugMug) is used. I was wondering if there is a way to have this downloader use the original file name when loging in as a guest to a password protected gallery. This way if friends that have downloaded a gallery can refer to the filename and I will know what image they are refering to. Is this possible? Thanks, Jimbo

As far as I know, this isn't possible with the current SmugMug API. Only account owners get access to the original filenames.

You could of course give out your password, but that would be, um, dangerous.

Sorry I can't be of help...

wellman

sirensofsong wrote:

Thanks for this great tool. It will really help with posting watermarked samples of my smugmug images to flickr.

Glad you find it useful.

auburnheritagephoto

First of all, great little program. I am very much enjoying it, I typically use Smugmug to backup my files and recently had a hard drive crash on me, so I am having to use your program to grab batches of files. I was wondering if there is a way that you can add an auto-retry function to the program. I keep getting a glitch here or there which stops the download. I am pretty sure that it is my internet that is glitching, but I was just thinking that a check box which auto-retry's would be nice, for overnight stuff. Thanks so much for the program, it really is great.

wellman

Latest capture on feature requests...

• Keyword-based downloading (an oldie but goodie)
• Fetch photos based on gallery or sharegroup URL
• Auto-restart after abort
• Save settings (local download folder, etc) in sidecar file

Jimbo

Large Number of files and Unviewed Galleries

wellman wrote:

As far as I know, this isn't possible with the current SmugMug API. Only account owners get access to the original filenames.

You could of course give out your password, but that would be, um, dangerous.

Sorry I can't be of help...

Thanks for the reply. I, and some other users have noticed a few other things. It seams that on galleries with a large number of pictures, an error will be encountered somewhere during the download. Have you seen this and is this something that can be fixed?

Second, I have also noticed that some galleries show "0" files when I know that there are photos in that gallery. No photos will be downloaded in this situation. If I go in and view every page in that gallery and wait until all thunbnails are seen, AlbumFetcher will now see the total number of pictures in that gallery and the download will work. Any ideas?
Thanks again for all you do with this application. I know it takes a lot of time and effort and is greatly appriciated!

wellman

Jimbo wrote:

Thanks for the reply. I, and some other users have noticed a few other things. It seams that on galleries with a large number of pictures, an error will be encountered somewhere during the download. Have you seen this and is this something that can be fixed?

Does this occur on the same image in that large gallery each time it's re-tried from scratch? If so, I might be able to figure out what property of that particular image is causing the failure.

Jimbo wrote:

Second, I have also noticed that some galleries show "0" files when I know that there are photos in that gallery. No photos will be downloaded in this situation. If I go in and view every page in that gallery and wait until all thunbnails are seen, AlbumFetcher will now see the total number of pictures in that gallery and the download will work. Any ideas?

I'm guessing here, but this sounds like it might have something to do with SmugMug keeping images that haven't been viewed in a while over on Amazon S3. Might be an API issue.

Jimbo wrote:

Thanks again for all you do with this application. I know it takes a lot of time and effort and is greatly appriciated!

Thanks!

Jimbo

Greg,
I have been doing some testing to see if I can find some more information for this issue that I am seeing. Following are some observations:

It seems that the majority of the errors that we are seeing are with the Status message of “Error in fetching image data. Download aborted”. This seems to be on albums that have more images and/or larger total gallery size. In this situation, the download never starts. AlbumFetcher appears to be timing out while trying to gather the image data from SmugMug. One thing that I noticed is that when I removed the password on 2 of the galleries that failed to start the download, and tried to download again, the download started. I aborted since these albums have a fairly large amount of files in them and would take some time. Adding the password back in and restarting AlbumFetcher, I get the error message. If you would like to see the error, you can login as a guest with the SmugMug ID of “flashlitephotos”. The test gallery is AlbumFetcherTest with a password of “nature”, all lower case, and has 80 images. I’ve tried this multiple times on this gallery to make sure that I am entering the password correctly. Same results and is consistent. Let me know if you have any specific questions that may help debug.

I am still trying to gather more data on downloads that stop during the download process. Not sure if it is the same image or not at this time. I will keep you posted.

Thanks,
Jim

wellman wrote:

Does this occur on the same image in that large gallery each time it's re-tried from scratch? If so, I might be able to figure out what property of that particular image is causing the failure.



I'm guessing here, but this sounds like it might have something to do with SmugMug keeping images that haven't been viewed in a while over on Amazon S3. Might be an API issue.




Thanks!

wellman

Jimbo wrote:

Greg,
I have been doing some testing to see if I can find some more information for this issue that I am seeing. Following are some observations:

It seems that the majority of the errors that we are seeing are with the Status message of “Error in fetching image data. Download aborted”. This seems to be on albums that have more images and/or larger total gallery size. In this situation, the download never starts. AlbumFetcher appears to be timing out while trying to gather the image data from SmugMug. One thing that I noticed is that when I removed the password on 2 of the galleries that failed to start the download, and tried to download again, the download started. I aborted since these albums have a fairly large amount of files in them and would take some time. Adding the password back in and restarting AlbumFetcher, I get the error message. If you would like to see the error, you can login as a guest with the SmugMug ID of “flashlitephotos”. The test gallery is AlbumFetcherTest with a password of “nature”, all lower case, and has 80 images. I’ve tried this multiple times on this gallery to make sure that I am entering the password correctly. Same results and is consistent. Let me know if you have any specific questions that may help debug.

I am still trying to gather more data on downloads that stop during the download process. Not sure if it is the same image or not at this time. I will keep you posted.

Thanks,
Jim

Thanks for the info, Jim. I was trying to download a gallery of 400ish images last night and saw the same thing. I'll update my to-do list...

wellman

Latest capture on feature requests...

• Bug: AF has problems with big galleries. The download never starts. Eliminating passwords helps in some cases.
• Keyword-based downloading (an oldie but goodie)
• Fetch photos based on gallery or sharegroup URL
• Auto-restart after abort
• Save settings (local download folder, etc) in sidecar file