smuggo v0.1 released

movestill · January 14, 2016

All, I just released the initial version of smuggo. It's designed for command line uploading to SmugMug. This is my main use case:

As the final step in my CaptureOne workflow, I set the Open With application to be smuggo (wrapped in an Automator script). Thus, each processed image is automatically sent to a staging gallery on SmugMug when it completes.

It can also be used for mass uploading.

I provided executables for Windows, OS X, and Linux (64 bit).

https://github.com/movestill/smuggo/releases

Tim

animefans · August 7, 2016

movestill wrote: »

All, I just released the initial version of smuggo. It's designed for command line uploading to SmugMug. This is my main use case:

As the final step in my CaptureOne workflow, I set the Open With application to be smuggo (wrapped in an Automator script). Thus, each processed image is automatically sent to a staging gallery on SmugMug when it completes.

It can also be used for mass uploading.

I provided executables for Windows, OS X, and Linux (64 bit).

https://github.com/movestill/smuggo/releases

Tim

I apologize for digging this thread out, but I have to thank the OP for sharing his code
I have been meaning to learn smugmug api so I can do command line upload, but never find the time, nor know enough technical to tackle it on my own
With this tool, I am able to do the upload with relative ease

The only issue I come across with the tool is searching existing album (smuggo albums).

It is giving me the following error

[kkfong@devcent core]$ ./smuggo albums
getUser response: 401 Unauthorized
No Uri object found in getUser response.
[kkfong@devcent core]$

But I am able to get the album id through smugmug's live API page, so at least I have workaround

Once I learn the go language, and smugmug's REST API, maybe I can fix this bug

Still, a big thanks to movestill for sharing his work!

K.K.

movestill · September 20, 2016

animefans wrote: »
The only issue I come across with the tool is searching existing album (smuggo albums).

It is giving me the following error
[kkfong@devcent core]$ ./smuggo albums
getUser response: 401 Unauthorized
No Uri object found in getUser response.
[kkfong@devcent core]$
But I am able to get the album id through smugmug's live API page, so at least I have workaround

Once I learn the go language, and smugmug's REST API, maybe I can fix this bug

Still, a big thanks to movestill for sharing his work!

K.K.

Hi K.K., I just noticed your post. I think you may not have authorized smuggo properly, if you're getting "401 Unauthorized". You may need to revoke authorization for smuggo via SmugMug's Account Settings page and then reauthorize it. I think I had to do something like this during development.

Hope this helps and glad you found smuggo useful,
Tim

SamirD · October 18, 2016

Very cool! I'm well versed in command line on windows, but am still having a bit of trouble with the syntax. Can you share an example of multiupload of a directory of 20 jpegs with 3 threads? Can wildcards be used? Thank you in advance!

movestill · October 20, 2016

multiupload

SamirD wrote: »

Very cool! I'm well versed in command line on windows, but am still having a bit of trouble with the syntax. Can you share an example of multiupload of a directory of 20 jpegs with 3 threads? Can wildcards be used? Thank you in advance!

SamirD, I think the last example under Uploading Files in the Readme.md covers your use case.

If you have your photos in say, c:\photos\Oct2016 and your album key is 7r2j1, then the command would like this:

smuggo multiupload 3 7r2j1 c:\photos\Oct2016\*.jpg

Hope this helps,
Tim

SamirD · October 22, 2016

movestill wrote: »

SamirD, I think the last example under Uploading Files in the Readme.md covers your use case.

If you have your photos in say, c:\photos\Oct2016 and your album key is 7r2j1, then the command would like this:

smuggo multiupload 3 7r2j1 c:\photos\Oct2016\*.jpg

Hope this helps,
Tim

Thank you! That helped a lot. I didn't have a readme.md file in the windows binary zip I downloaded. I downloaded the source and found it in that zip.

One question I have is where does smuggo store its auth information? Since it's multi-platform, I'm guessing that it creates some sort of file in the directory where the binary is (which would be perfect for me), but it would be nice to know for certain.

My platform is windows.

SamirD · October 23, 2016

Well I think this error message answers my question:

Saving API key: open C:\Users\Administrator/.smuggo/apiToken.json: The system ca
nnot find the path specified.

Now the question is how to fix it?

Is there a way I can specify the path?

SamirD · October 23, 2016

I just tried it on another system and got this message:

Saving API key: open C:\Documents and Settings\Administrator/.smuggo/apiToken.json: The system cannot find the path spec
ified.

It maybe looks like it's using the wrong slashes?

dunno

SamirD · October 23, 2016

Tried it on a 3rd system and got this:

Saving API key: open C:\Documents and Settings\Samir/.smuggo/apiToken.json: The
system cannot find the path specified.

It's either the slash or that the '.smuggo' directory doesn't exist. I'm going to try manually making the directory and see if that works.

SamirD · October 23, 2016

Manually making the '.smuggo' directory worked.

SamirD · October 23, 2016

I've been able to get smuggo to auth and SM shows it is authorized, but I'm getting the following error on the the 'smuggo albums' command:

getUser response: 401 Unauthorized
No Uri object found in getUser response.

I do have over a thousand albums, but it comes back with this error msg pretty quick.

SamirD · October 23, 2016

I've assumed that the key is the same key we see in the url when looking at an album. So I tried a single upload using the following command and got the following error message:

smuggo upload _KEY_ FILE.JPG
200 OK
{"stat":"fail","method":"smugmug.images.upload","code":5,"message":"system error
"}

Hopefully this gives you enough information to help me get smuggo working. This would be the absolute killer sm companion for me if it can work.

movestill · October 23, 2016

authorization

SamirD wrote: »
I've assumed that the key is the same key we see in the url when looking at an album. So I tried a single upload using the following command and got the following error message:
smuggo upload _KEY_ FILE.JPG
200 OK
{"stat":"fail","method":"smugmug.images.upload","code":5,"message":"system error
"}
Hopefully this gives you enough information to help me get smuggo working. This would be the absolute killer sm companion for me if it can work.

If you go to SmugMug's Account Settings and look at Authorized Services, what do you have for smuggo? It should say Access: Full and Permissions: Modify. If it doesn't, you'll need to revoke permissions and redo the auth. Let me know if you have to update the permissions. If so, I may need to change the way smuggo sets up auth.

smuggo couldn't create the .smuggo folder because you're running as the administrator user. It would have to run as administrator to write to that folder.

Thanks,
Tim

SamirD · October 24, 2016

movestill wrote: »

If you go to SmugMug's Account Settings and look at Authorized Services, what do you have for smuggo? It should say Access: Full and Permissions: Modify. If it doesn't, you'll need to revoke permissions and redo the auth. Let me know if you have to update the permissions. If so, I may need to change the way smuggo sets up auth.

smuggo couldn't create the .smuggo folder because you're running as the administrator user. It would have to run as administrator to write to that folder.

Thanks,
Tim

Thank you so much! I re-used an existing key that I had for a set of tools I was working on with the old sm. Since sm updated so much, I doubt any of the tools I made would work, so I reused the key, but as you noted, I had to update the permissions.

I just tried it with the updated permissions in SM, but no dice. I'll have smuggo redo its auth again and see what happens.

I'm not sure what you mean about the administrator user. Did you mean to say smuggo couldn't create because I was not running as administrator? But I was running as admin because I could manually make the directory whereafter smuggo was able to write its .json files there.

SamirD · October 24, 2016

So I tried redoing the smuggo auth and that didn't help. I noticed when getting a new key that the older api keys are depreciated (or at least that's how I read the message on the key application page). I just ran 'smuggo apikey' and 'smuggo auth' again with the new api key. Let's see how that works.

SamirD · October 24, 2016

I'm pretty puzzled now. Permissions in SM are correct, new API key shows in SM, smuggo has apikey and auth saved to its .json files. But I'm getting the following errors:

>smuggo albums
Requesting number of albums.
No albums found.

>smuggo upload aLBUMkEY FILE.JPG
200 OK
{"stat":"fail","method":"smugmug.images.upload","code":5,"message":"system error
"}

Thoughts?

movestill · October 24, 2016

no albums

SamirD wrote: »
I'm pretty puzzled now. Permissions in SM are correct, new API key shows in SM, smuggo has apikey and auth saved to its .json files. But I'm getting the following errors:
>smuggo albums
Requesting number of albums.
No albums found.

>smuggo upload aLBUMkEY FILE.JPG
200 OK
{"stat":"fail","method":"smugmug.images.upload","code":5,"message":"system error
"}
Thoughts?

Have you upgraded your SmugMug account to the new SmugMug (new as in a couple of years ago they revamped everything)? That's the only thing I can think of, right now.

As another test, login to SmugMug and try typing this into your browser: https://api.smugmug.com/api/v2/album/<album key>

Replace <album key> with the key of one of your albums. You should get lots of information back on the given album.

Also try this: https://api.smugmug.com/api/v2/user/<user name>!albums

This is what smuggo does to list your albums. Replace <user name> with your SmugMug user name.

Tim

SamirD · October 25, 2016

movestill wrote: »

Have you upgraded your SmugMug account to the new SmugMug (new as in a couple of years ago they revamped everything)? That's the only thing I can think of, right now.

As another test, login to SmugMug and try typing this into your browser: https://api.smugmug.com/api/v2/album/<album key>

Replace <album key> with the key of one of your albums. You should get lots of information back on the given album.

Also try this: https://api.smugmug.com/api/v2/user/<user name>!albums

This is what smuggo does to list your albums. Replace <user name> with your SmugMug user name.

Tim

There was a date back a few years ago when all old accounts automatically converted to the new SM. I actually lost a lot of customization that I never had a chance to save. :cry And I haven't even had a chance to try to redo anything, but that's another story...

I didn't realize the album key was different than the key that is in the web url (I'm glad too as that would have been a security issue otherwise). I was able to retrieve a list of albums, but it was only 50 of my albums and sadly the one I wanted wasn't listed. Do you know the api call to get the key for just a particular album?

SamirD · October 25, 2016

Once I had the right album key, it looks like it works!

barb

By far this is one of the most useful command line applications I've ever used for SM.

I did a 6 thread multiupload and the upload seemed to be faster than the same upload via the standard web-based uploader. Not to mention I didn't have to spend time queuing them (drag and drop, etc).

My use case is to put smuggo on a network drive and then have any system use it to upload. So far, I've run into a 'Error uploading: Post https://upload.smugmug.com/: x509: certificate signed by unknown authority' error when copying the .smuggo directory to the appropriate place on another system. Any ideas? This is the last thing I need working to make smuggo the key component in my use case.

One suggestion I have is for smuggo keep its .json files in the same path as itself or have an option to specify where the .json files are located. This would make the application truly portable and also make it easy to use with multiple SM accounts.

movestill · October 26, 2016

Glad you had some success. You're definitely persistent.

I'm not sure why you're getting that particular error when you copy the .smuggo folder to another system. Assuming the .json files weren't corrupted, then it should just work. I'll add specifying the folder to look for the .json files onto the to-do list.

Regarding finding an album's key, you can use the search functionality. I noticed that SmugMug changed the response data for search, so the current smuggo version's search is broken. I have a code fix, but need to find time to make a new release.

In the meantime, you can use the API web page to search: https://api.smugmug.com/api/v2/album!search Scroll down to the Options section and you can fill in the form to search. Put your user name in the Scope textbook to confine the search to your albums.

Tim

SamirD · October 27, 2016

movestill wrote: »

Glad you had some success. You're definitely persistent.

I'm not sure why you're getting that particular error when you copy the .smuggo folder to another system. Assuming the .json files weren't corrupted, then it should just work. I'll add specifying the folder to look for the .json files onto the to-do list.

Regarding finding an album's key, you can use the search functionality. I noticed that SmugMug changed the response data for search, so the current smuggo version's search is broken. I have a code fix, but need to find time to make a new release.

In the meantime, you can use the API web page to search: https://api.smugmug.com/api/v2/album!search Scroll down to the Options section and you can fill in the form to search. Put your user name in the Scope textbook to confine the search to your albums.

Tim

I never try to let a good thing go.

I'll do some more experiments with the .smuggo folder and the .json files. It's strange because the two systems are literally identical, so I didn't expect an error at all. Thank you for considering my feature request. I'm sure I"ll be your biggest beta tester in a short while.

I found a quick way using the api to find the album key. If I have a chance I'll post it here for others to use if need be.

Thank you for the terrific tool. As I type this, my originals are copying and once they're compared with the memory card, I just have to press enter and smuggo will use 6 threads and send them on their way.

thumb:ivar

lust

SamirD · October 27, 2016

One thing I was curious about was the multiupload and exactly how it work, especially when compared to multiple instances of smuggo uploading in parallel. Would multiple instances of smuggo be more or less efficent? Ie, 6 'smuggo upload', each in its own command prompt, vs a 'smuggo multiupload' with 6 threads. Where does the efficency of multi-uploads break down? 20+? 99+?

Thank you in advance for your responses.

movestill · October 27, 2016

performance

SamirD wrote: »

One thing I was curious about was the multiupload and exactly how it work, especially when compared to multiple instances of smuggo uploading in parallel. Would multiple instances of smuggo be more or less efficent? Ie, 6 'smuggo upload', each in its own command prompt, vs a 'smuggo multiupload' with 6 threads. Where does the efficency of multi-uploads break down? 20+? 99+?

Thank you in advance for your responses.

Multi-upload uses goroutines. It's similar to traditional threads except it uses Go's scheduler and less resources than traditional threading. I think multiple instances of smuggo would use more resources, but not be any faster. I wouldn't use any more goroutines or smuggo instances than you have cores. I think you'll run into your upload bandwidth limitation of your ISP before you hit the limit of your system.

However, this is all speculation. I haven't done any speed tests since I don't transfer more than say, 100 images, at a time.

Tim

SamirD · October 28, 2016

Thank you very much for the explanations. You're right about hitting upload bandwidth limitations before system limitations. Back in the day, I would use 3 different systems uploading 3 threads each in order to saturate my 3 cable modems.

These days, the upload bandwidth is there, and a single system could do the job if there was efficiency in the upload process. I thought I had noticed an improvement in upload speed with smuggo. I'll have to quantify it at some point by doing some tests.

In the meantime, I have another feature request. When an upload fails, smuggo displays the error message and condition, and I'm assuming that on the SM end, they discard the data since it wasn't a complete photo/video, correct?

It would be nice if there was a 'retry' parameter where you could tell smuggo to try x amount of times or 'U' for unlimited. This way, my set and forget upload really becomes set and forget. So far I have to run the SM uploader after smuggo to piece up any errored out.

There's a couple of other applications for smuggo that might be useful since they were needed back in the day. One was a duplicate file finder with delete capabilities. You could tell smuggo to check for duplicates in a particular gallery and it would return a set of 'duplicates' that you can select to delete or not delete, or delete all. Not exactly sure how to do this on command line besides the find duplicate part.

movestill · October 28, 2016

SamirD wrote: »

These days, the upload bandwidth is there, and a single system could do the job if there was efficiency in the upload process. I thought I had noticed an improvement in upload speed with smuggo. I'll have to quantify it at some point by doing some tests.

In the meantime, I have another feature request. When an upload fails, smuggo displays the error message and condition, and I'm assuming that on the SM end, they discard the data since it wasn't a complete photo/video, correct?

It would be nice if there was a 'retry' parameter where you could tell smuggo to try x amount of times or 'U' for unlimited. This way, my set and forget upload really becomes set and forget. So far I have to run the SM uploader after smuggo to piece up any errored out.

There's a couple of other applications for smuggo that might be useful since they were needed back in the day. One was a duplicate file finder with delete capabilities. You could tell smuggo to check for duplicates in a particular gallery and it would return a set of 'duplicates' that you can select to delete or not delete, or delete all. Not exactly sure how to do this on command line besides the find duplicate part.

Please let me know what you find if and when you do performance tests.

Right, SmugMug will discard any failed uploads. Retries sounds like a good idea.

Regarding duplicates. I think a hash can be sent for each image, as it's uploaded, so the upload will be rejected if the image already exists in the album. Would that meet your needs? That fits in well within the confines of a command line application.

Tim

SamirD · October 30, 2016

I definitely will. Right now ease of uploading is the primary objective, but optimal bandwidth saturation will be the next goal.

I saw the hash feature in the api documentation. That would allow a 'skip duplicates' type of feature in smuggo, which would be great.

But what I was talking about in terms of duplicates, was a way to find out if duplicates exist in a gallery. I don't know how well the SM uploaders work today, but a few years ago when I was regularly uploading 2000+ pieces of media every weekend, invariably there would be a duplicate file every so often that would be tedious to find. I had a script written that used the old api that would go through a gallery and check the filename of a particular file with the previous filename, thereby finding duplicates. These files would have different hashes if one of the files was partially uploaded to SM, and it was still processed (which seems to still happen as I saw this behavior recently). A simple command that would reveal each set of duplicate would help someone find that potential duplicate in a gallery of thousands of images.

I'll see if I can't dig up my old script and see if it still works or share the pseudocode with you.

movestill · October 30, 2016

SamirD wrote: »

I saw the hash feature in the api documentation. That would allow a 'skip duplicates' type of feature in smuggo, which would be great.

But what I was talking about in terms of duplicates, was a way to find out if duplicates exist in a gallery. I don't know how well the SM uploaders work today, but a few years ago when I was regularly uploading 2000+ pieces of media every weekend, invariably there would be a duplicate file every so often that would be tedious to find. I had a script written that used the old api that would go through a gallery and check the filename of a particular file with the previous filename, thereby finding duplicates. These files would have different hashes if one of the files was partially uploaded to SM, and it was still processed (which seems to still happen as I saw this behavior recently). A simple command that would reveal each set of duplicate would help someone find that potential duplicate in a gallery of thousands of images.

I checked my upload code and noted that I already include the MD5 hash as well as the number of bytes to expect. Given that I'm providing that, I would expect SmugMug to reject a failed upload. However, I also expected it to reject duplicates for the same reason.

Have you seen partial or corrupted uploads using smuggo?

BTW, I uploaded v0.2 to GitHub today. This release just has a fix for search.

Tim

smuggo v0.1 released

Comments