App to make full backup of all your photos and videos

tommyblue

As a SmugMug user I wanted to keep a local backup of all my photos and videos. As you probably know the only way to do it is to open each gallery and download it, this is all but user-friendly.
I'm a programmer so I decided to make my own software to do it. It can run periodically and will download new elements, skipping exising ones. It uses your API access tokens to access the account, loops over all galleries and downloads each photo and video to a folder you choose.
It requires some programming skills to be downloaded and configured, but then you'll have a full backup of your account.
It's completely open source, feel free to use it, change it, share it, etc.
Can be downloaded from github, project: tommyblue/smugmug-backup (sorry, I can't post link as I'm a new user)
Feedbacks are welcome

SamirD

Thank you! What are the system requirements for this? I see on your blog someone got it working for Windows, implying that it doesn't work in windows normally?

tommyblue

@SamirD any system that can build a GoLang executable is supported. I work with both Mac and Linux, but I think it works with Windows too

begbirumerah

@tommyblue Thank you for the program.
I have successfully run your program on my Windows 10.
The gallery names are preserved on my E: as folder names.
However there is only 1 file in each folder
Looks like all images are dumped as a single binary file (not *.jpg as I uploaded) and that single file size is much less than total MB of all images in that gallery.
I wasn't sure how to use "file_names" so I used all the keys you listed.

The program and its files are in E:\Smugbak\
Here's my config.toml:
~~~~~~~~~~~~~~~~
[authentication]
api_key = "S2CVD...<hidden>"
api_secret = "Gh4hz...<hidden>"
user_token = "xDg9b9q...<hidden>"
user_secret = "fFBbb...<hidden>"

[store]
destination = "/SmugBak/TheBak"
file_names = "FileName ImageKey ArchivedMD5 UploadKey"
use_metadata_times = true
force_metadata_times = true
~~~~~~~~~~~~~~~~

Here's the last few lines of the message as appeared on my E:\ prompt
~~~~~~~~~~~~~~~~
time="2020-10-09T09:50:33+08:00" level=info msg="Saved \\SmugBak\\TheBak\\August/FileName ImageKey ArchivedMD5 UploadKey"
time="2020-10-09T09:50:33+08:00" level=info msg="Getting https://photos.smugmug.com/photos/i-cZm9tJ2/0/D/i-cZm9tJ2-D.jpg"
time="2020-10-09T09:50:35+08:00" level=info msg="Saved \\SmugBak\\TheBak\\August/FileName ImageKey ArchivedMD5 UploadKey"
time="2020-10-09T09:50:35+08:00" level=info msg="Backup completed."

E:\SmugBak>

begbirumerah

Hi tommyblue,
so I removed file_names altogether from my config.toml.
Ran smug-mug-backup.exe successfully.
Now I can see the jpg files listed in the respective folders.
Next question, how to backup Title, Caption, and Keywords for each iof the jpg files?

thank you in advance,
Beg

GreatSave9

A+ work!! Thank you! It as very easy to setup and run.
However, is there a way to choose a specific folder or gallery? Because if I download my entire site, I imagine I would need a 3TB or 4TB drive ready to DL.
If I could avoid some galleries or folders that I know I wouldn't need, that would help as I know I want like 70% of my site backed up and 30% I don't care for.

But definitely A+ work and it seems great.

GreatSave9

Just FYI, the plan is to use a 4TB hard drive to DL my site.
Then at the same time, I have that drive setup to sync to Amazon Photos through their desktop app.
So as one app is downloading, the other is uploading. So I have a backup of my SM site.

WHY!? Someone may ask.... well, I already have Amazon PRIME so I get unlimited photos with them. (Including RAW) and the biggest factor, Amazon has Facial Recognition and its the easiest and fastest way to do that with so many images of friends/family. Their interface is abit clunky and annoying, but again, its a backup, and It should work!

Thank you again for your app! Hope this info helps anyone else.

timblaktu

@begbirumerah I made the same mistake you did here, and I'm a 25yr experienced software developer, so don't feel too bad ;-). The `file_names` parameter is a go lang string template, which means you must include the dot `.` before those annotations for the correct substitutions to be made at runtime. What you did was to tell the program to name EVERY file literally "FileName ImageKey ArchivedMD5 UploadKey". Most certainly not what you intended. When you deleted the file_names line in your config, you forced the app to use the default, which is `"{{ .FileName }}". I created a PR on tommyblue's project to clarify this detail a bit.

@GreatSave9 what mechanism are you using to upload your local smugmug_backup archive to Amazon Photos? You imply that the upload to Amazon is concurrent with the smugmug_backup download process, so I assume you have something that's monitoring the local smugmug_backup archive and uploading to AMZ at intervals..

Everyone, I'd be curious to get your ideas here on this thread on desired features in this app. I've raised this feature request with my particular needs:

https://github.com/tommyblue/smugmug-backup/issues/21

...and I'd be curious if others have the same needs, and/or wanted to present other needs. I'm not sure if @tommyblue will have the time or the desire to work on these, so for now, I'm just wanting to collect "user stories" about your needs WRT making and managing SmugMug site backups.

dvdmon

Hoping someone can help. I tried installing this app but I'm stuck at the compiling stage. I'm on a Windows 10 machine and copied files from make-4.2.1-without-guile-w32-bin.zip into my C:\ProgramFiles\Git\mingw64\ based on a stackoverflow article. I get the following error when I try make build:

tommyblue

Hello everybody, just a quick heads up to inform anybody interested in backing up their account that the software has evolved over the years. It's now at version 1.5 and it's able to perform an initial backup of ~200GB in 1 hour or so and following, incremental, backups in few minutes. You can download ready-to-launch executables for Windows, Mac and Linux from the release page