smugmug command line tool in python

rutt

I use links in separate galleries to get this effect.

firecat53

re: Links in separate galleries
*sigh* Please forgive my 'density', but I can't figure out what you mean by 'using links in separate galleries'.

Thanks a lot for your help!!

Scott

rutt

On Unix and Linux you can make separate directories (folders) which contain subsets of the very same image. For example, suppose you have all your family shots in one directory: ~/photos/family_shots and now you would like separate directories for pictures from 2005 and 2004. Just to make this easier, suppose your 2005 pictures are all named like: img_XXXX_2005.jpg and simlarly for the 2004 shots.

Then you can do this in a command shell:

mkdir 2004
cd 2004
ln -s ~/photos/family_shots/*2004*.jpg .
cd ..
mkdir 2005
cd 2005
ln -s ~/photos/family_shots/*2005*.jpg .

Now you can run sm_tool.py separately in the 2004 or 2005 directories, or in the root of the two with full_update.

gymshots

.smugmugrc
rutt,

I am new to python, and smugmug. I have some unix experience and plan on using sm_tool under windows 2000. It is not clear to me, what you
expect in the .smugmugrc file....I wondered if you could help me with the
syntax.

thanks
gymshots

rutt wrote:

I have written a command line tool that uses the smugmug labs API to create gallereis, upload photos to them, and keep them updated as images change and new images are added locally. This is very useful if you want to treat you smugmug account as a mirror of a local directory tree. It also is not subject to the 100MB limit of the various browser based upload tools.

Who is this for? The brave, people who want to deal with a lot of photos and galleries, rogrammers; in short people like me. If you are not comfortable with command line programs, you won't like this. If you are aggressively non-technical, this isn't for you. But if you like to tinker and you aren't afraid of what's under the hood of your computer, this might be just the thing for you.

I have been using variants of this program for almost a year, but this version has been radically revised in hopes that it can be more than personal software. As such, I'm sure there will be bugs. Please email me with these. (please don't PM.) The program text contains a disclaimer, but I suppose I'd better add one here as well. Use this thing at your own risk. I won't be responsible for any damage it inflicts.

I have used and tested on OS X and on Linux. I have not tried on windows, but there really is no reason that it shouldn't work there. I have tried to be as portable as possible (and python allows a lot of this.) I'd like to hear about experiences on windows. Best of all, I'd like a python programmer on windows to smooth out the rough corners there.

Download smugmug.py

Smugmug.py requires python 2.3 or better.

firecat53

gymshots wrote:

rutt,

I am new to python, and smugmug. I have some unix experience and plan on using sm_tool under windows 2000. It is not clear to me, what you
expect in the .smugmugrc file....I wondered if you could help me with the
syntax.

thanks
gymshots

login: login@email.com
password: smugmugpassword
public: True

That's all I used! Good luck....you'll probably have to try it a few times and experiment before you get it exactly how you want it. It's a good tool!

Scott

gymshots

all set
thanks scott! It's working like a charm. great tool!
:

firecat53 wrote:

login: login@email.com
password: smugmugpassword
public: True

That's all I used! Good luck....you'll probably have to try it a few times and experiment before you get it exactly how you want it. It's a good tool!

Scott

mugzy

Ok, trying to get this working under xp.

I start in the directory with the images I want to upload.
I type "c:\sm_tool.py create b17 *"
It creates the b17 album, but does not upload images.

It works if I give it a specific filename instead of *.

Are wildcards allowed?

rutt

Wild cards work on *nix, including OS X, but I'm not sure about XP. The various unix and linux shells expand wild cards before passing to the program, so the program doesn't actually have to do anything. What if you type "*.JPG" or "*.jpg" depending on what you have?

If you look in the script, you'll find a function like this:

def update_dir(smugmug,dir,opts,files):
  li = LocalInformation(dir)
  
  files_to_upload = []
  for f in files:
    if re.match(opts.filter_regular_expression,f):
      file = path.join(dir,f)
      if li.file_needs_upload(file):
        files_to_upload.append(file)
  if len(files_to_upload) > 0:
    files_to_upload.sort()
    smugmug.upload_files(li.gallery_id(),opts,files_to_upload,
                         local_information=li)

If you add line to print the files, like this, you may be able to figure it out:

def update_dir(smugmug,dir,opts,files):
  li = LocalInformation(dir)
  
  files_to_upload = []
  for f in files:
    print f
    if re.match(opts.filter_regular_expression,f):
      file = path.join(dir,f)
      if li.file_needs_upload(file):
        files_to_upload.append(file)
  if len(files_to_upload) > 0:
    files_to_upload.sort()
    smugmug.upload_files(li.gallery_id(),opts,files_to_upload,
                         local_information=li)

Check if there is a new folder SMUGMUG_INFO in the folder after you run. The files there might also be interesting.

mugzy

rutt wrote:

Wild cards work on *nix, including OS X, but I'm not sure about XP. The various unix and linux shells expand wild cards before passing to the program, so the program doesn't actually have to do anything. What if you type "*.JPG" or "*.jpg" depending on what you have?

I thought it might be something like this. I should have looked at the code first.

It looks like windows does not expand the wild cards. I will either brush up on my python, or throw together a quick .vbs to break up the wild card.

Thanks

rutt

Adding windows wild card expansion to the script has to be pretty easy and would be a boon to all the other 10s of thousands of people who use it. Catch the exception when you try to import the function and you'll know whether or not you have to use it.

zanthrus

Windows Tweaks
I had to do a couple things to get this to work in WindowsXP.

1.) I had to define a HOME environment variable to avoid a KeyError. I think you could resolve this by trying HOME first, and then falling back to USERPROFILE which I think is set by default in windows to C:\Documents and Settings\[Username]

2.) File uploads refused to work. When reading files in, only the first 58 bytes of my *.jpg would be read. Fixing this is as simple as opening the file in binary mode:

f = file(name,"rb")

I think both these fixes are simple enough that cross-platform compatibility won't be damaged.

Other than that, this thing looks fantastic. I might add some new features to it if I ever get the time. I'd like the ability to synchronize everything with Smugmug. Uploading is great, but if I change captions or titles on the website I'd like to get a local copy of that information.

rutt

The part about opening in binary mode should have been fixed already. Did you see:

http://www.dgrin.com/showpost.php?p=178628&postcount=28

Where did you pick up the script?

zanthrus wrote:

I had to do a couple things to get this to work in WindowsXP.

1.) I had to define a HOME environment variable to avoid a KeyError. I think you could resolve this by trying HOME first, and then falling back to USERPROFILE which I think is set by default in windows to C:\Documents and Settings\[Username]

2.) File uploads refused to work. When reading files in, only the first 58 bytes of my *.jpg would be read. Fixing this is as simple as opening the file in binary mode:

f = file(name,"rb")

I think both these fixes are simple enough that cross-platform compatibility won't be damaged.

Other than that, this thing looks fantastic. I might add some new features to it if I ever get the time. I'd like the ability to synchronize everything with Smugmug. Uploading is great, but if I change captions or titles on the website I'd like to get a local copy of that information.

zanthrus

rutt wrote:

The part about opening in binary mode should have been fixed already. Did you see:

http://www.dgrin.com/showpost.php?p=178628&postcount=28

Where did you pick up the script?

I got it from http://gate.chezrutt.com:8030/rutt/sm_tool/sm_tool.py. It says it's version 8. I updated to version 10, and the fix is there already.

Thanks

Edit, found a nice fix for the first problem, use os.path.join(os.path.expanduser("~"), ".smugmugrc") to find the file instead of bothering with environment variables.

tsk1979

Brilliant- and a question
Just what I was looking for. I have couple of questions. We have the option of "update" and "upload". Suppose I have many files in the dir.
I want to do update, but it will upload all image files which are not there in the gallery, so is it possible that I give it a file list. and it checkes if those files are there and then updates?
For example my dir may have files like img_a_001.jpg. img_b_001.jpg....
I want to upload only img_b_00*.jpg and nothing else. I can sure use wildcards, but if there is a possiblity of giving a text file as the list of files relative to the current dir tree this will be good. that I can use a complex script to create a txt list of files and then give this list to this script to upload!

rutt wrote:

I have written a command line tool that uses the smugmug labs API to create gallereis, upload photos to them, and keep them updated as images change and new images are added locally. This is very useful if you want to treat you smugmug account as a mirror of a local directory tree. It also is not subject to the 100MB limit of the various browser based upload tools.

Who is this for? The brave, people who want to deal with a lot of photos and galleries, rogrammers; in short people like me. If you are not comfortable with command line programs, you won't like this. If you are aggressively non-technical, this isn't for you. But if you like to tinker and you aren't afraid of what's under the hood of your computer, this might be just the thing for you.

I have been using variants of this program for almost a year, but this version has been radically revised in hopes that it can be more than personal software. As such, I'm sure there will be bugs. Please email me with these. (please don't PM.) The program text contains a disclaimer, but I suppose I'd better add one here as well. Use this thing at your own risk. I won't be responsible for any damage it inflicts.

I have used and tested on OS X and on Linux. I have not tried on windows, but there really is no reason that it shouldn't work there. I have tried to be as portable as possible (and python allows a lot of this.) I'd like to hear about experiences on windows. Best of all, I'd like a python programmer on windows to smooth out the rough corners there.

Download sm_tool.py

Smugmug.py requires python 2.3 or better.

[Edited to reflect the true correct home of the script.]

mpmcleod

tsk1979 wrote:

Just what I was looking for. I have couple of questions. We have the option of "update" and "upload". Suppose I have many files in the dir.
I want to do update, but it will upload all image files which are not there in the gallery, so is it possible that I give it a file list. and it checkes if those files are there and then updates?
For example my dir may have files like img_a_001.jpg. img_b_001.jpg....
I want to upload only img_b_00*.jpg and nothing else. I can sure use wildcards, but if there is a possiblity of giving a text file as the list of files relative to the current dir tree this will be good. that I can use a complex script to create a txt list of files and then give this list to this script to upload!

How fast should this be?

sm_tool.py works fast for about 20 files and then slows way down. From my office with T3+ speeds it only uploaded about 500 files in 10 hours. I need to upload 30,000 files.

thanks,

rutt

mpmcleod wrote:

How fast should this be?

sm_tool.py works fast for about 20 files and then slows way down. From my office with T3+ speeds it only uploaded about 500 files in 10 hours. I need to upload 30,000 files.

thanks,

I don't have that experience, but I've never had such a fast connection either. After each file uploaded, sm_tool.py prints a speed and estimate of time remaining. What does that look like for you?

Do you have any experience ftp'ing or ssh'ing large collections of files outward from to someplace with a good connection? What happens?

tsk1979

mpmcleod wrote:

How fast should this be?

sm_tool.py works fast for about 20 files and then slows way down. From my office with T3+ speeds it only uploaded about 500 files in 10 hours. I need to upload 30,000 files.

thanks,

I tried this for about 100 files on a 1Mbps link, the speed was always hovering near the limit. I think its some throttling at your office, when It detects a connection going at the limit for a prolonged period of time, poof it throttles. This tool is just pushing the data to smugmug, so does not mater 10 files or 10,000 files.

mpmcleod

rutt wrote:

I don't have that experience, but I've never had such a fast connection either. After each file uploaded, sm_tool.py prints a speed and estimate of time remaining. What does that look like for you?

Do you have any experience ftp'ing or ssh'ing large collections of files outward from to someplace with a good connection? What happens?

I have moved over 100GB of other data between the office and an offsite compute cluster in less than one hour (during lunch) using SFTP.

I will do some more digging and play some more.

thanks,

mpmcleod

mpmcleod wrote:

I have moved over 100GB of other data between the office and an offsite compute cluster in less than one hour (during lunch) using SFTP.

I will do some more digging and play some more.

thanks,

When I run this in a directory with 30,000 files I get an error.

using phython2.3
>../sm_tool.py full_update

Traceback (most recent call last):
File "../sm_tool.py", line 656, in ?
main()
File "../sm_tool.py", line 649, in main
full_update(Options(argv[2:]))
File "../sm_tool.py", line 625, in full_update
create(smugmug,name,root,opts)
File "../sm_tool.py", line 563, in create
album_id = smugmug.create_album(name,opts)
File "../sm_tool.py", line 227, in create_album
return self.sp.createAlbum(self.session,name,category,properties)
File "/usr/lib/python2.3/xmlrpclib.py", line 1029, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.3/xmlrpclib.py", line 1316, in __request
verbose=self.__verbose
File "/usr/lib/python2.3/xmlrpclib.py", line 1080, in request
return self._parse_response(h.getfile(), sock)
File "/usr/lib/python2.3/xmlrpclib.py", line 1219, in _parse_response
return u.close()
File "/usr/lib/python2.3/xmlrpclib.py", line 742, in close
raise Fault(**self._stack[0])
xmlrpclib.Fault: <fault 10="" missing="" title="">

using python2.4

Traceback (most recent call last):
File "../sm_tool.py", line 656, in ?
main()
File "../sm_tool.py", line 649, in main
full_update(Options(argv[2:]))
File "../sm_tool.py", line 625, in full_update
create(smugmug,name,root,opts)
File "../sm_tool.py", line 563, in create
album_id = smugmug.create_album(name,opts)
File "../sm_tool.py", line 227, in create_album
return self.sp.createAlbum(self.session,name,category,properties)
File "/usr/lib/python2.4/xmlrpclib.py", line 1096, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.4/xmlrpclib.py", line 1383, in __request
verbose=self.__verbose
File "/usr/lib/python2.4/xmlrpclib.py", line 1147, in request
return self._parse_response(h.getfile(), sock)
File "/usr/lib/python2.4/xmlrpclib.py", line 1286, in _parse_response
return u.close()
File "/usr/lib/python2.4/xmlrpclib.py", line 744, in close
raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault 10: 'missing title'>


am I missing something?

Is there a limit to the number of files sm_tool can handle?
</fault>

mpmcleod

This may indeed be some sort of problem on my end (or between SM and myself).

I threw together a very rough hack using perl to do the same job and it loads 20 files quite fast and smooth and then hangs. My code doesn't have a timeout so instead of breaking it just stays hung until I kill it.

I have sent an email to my IT people to see what the problem might be.

thanks,

tsk1979

Not working as expected
I have a gallery called files under the category "Other".
Now on the machine there is a directory called files with lots of jpg files(135M)
To upload all the files to the gallery files, should I give the following command line
./sm_tool.py upload files files/*jpg --category=Other
I did this and got the following erros. I have my ~/.smugmugrc file with login, password and public: True

./sm_tool.py upload files files/* --category=Other
Traceback (most recent call last):
  File "./sm_tool.py", line 656, in ?
    main()
  File "./sm_tool.py", line 645, in main
    upload(argv[2],Options(argv[3:]))
  File "./sm_tool.py", line 598, in upload
    smugmug = Smugmug(opts.login,opts.password)
  File "./sm_tool.py", line 178, in __init__
    self.login()
  File "./sm_tool.py", line 184, in login
    rep = self.sp.loginWithPassword(self.account,self.password,self.api_version)
  File "/usr/lib/python2.3/xmlrpclib.py", line 1032, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.3/xmlrpclib.py", line 1319, in __request
    verbose=self.__verbose
  File "/usr/lib/python2.3/xmlrpclib.py", line 1065, in request
    self.send_content(h, request_body)
  File "/usr/lib/python2.3/xmlrpclib.py", line 1179, in send_content
    connection.endheaders()
  File "/usr/lib/python2.3/httplib.py", line 715, in endheaders
    self._send_output()
  File "/usr/lib/python2.3/httplib.py", line 600, in _send_output
    self.send(msg)
  File "/usr/lib/python2.3/httplib.py", line 567, in send
    self.connect()
  File "/usr/lib/python2.3/httplib.py", line 987, in connect
    sock.connect((self.host, self.port))
  File "<string>", line 1, in connect
socket.error: (111, 'Connection refused')
Exception exceptions.AttributeError: "Smugmug instance has no attribute 'session'" in <bound method Smugmug.__del__ of <__main__.Smugmug instance at 0x40276c4c>> ignored

rutt

This looks to me as if you aren't getting through to smugmug at all. See the connection refused message at the bottom of the error report.

You can try adding some print statements near the apparent ssource of the problem:

print self.account
print self.password
print self.api_version
rep = self.sp.loginWithPassword(self.account,self.password,self.api_version)
print rep

tsk1979 wrote:

I have a gallery called files under the category "Other".
Now on the machine there is a directory called files with lots of jpg files(135M)
To upload all the files to the gallery files, should I give the following command line
./sm_tool.py upload files files/*jpg --category=Other
I did this and got the following erros. I have my ~/.smugmugrc file with login, password and public: True

./sm_tool.py upload files files/* --category=Other
Traceback (most recent call last):
  File "./sm_tool.py", line 656, in ?
    main()
  File "./sm_tool.py", line 645, in main
    upload(argv[2],Options(argv[3:]))
  File "./sm_tool.py", line 598, in upload
    smugmug = Smugmug(opts.login,opts.password)
  File "./sm_tool.py", line 178, in __init__
    self.login()
  File "./sm_tool.py", line 184, in login
    rep = self.sp.loginWithPassword(self.account,self.password,self.api_version)
  File "/usr/lib/python2.3/xmlrpclib.py", line 1032, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.3/xmlrpclib.py", line 1319, in __request
    verbose=self.__verbose
  File "/usr/lib/python2.3/xmlrpclib.py", line 1065, in request
    self.send_content(h, request_body)
  File "/usr/lib/python2.3/xmlrpclib.py", line 1179, in send_content
    connection.endheaders()
  File "/usr/lib/python2.3/httplib.py", line 715, in endheaders
    self._send_output()
  File "/usr/lib/python2.3/httplib.py", line 600, in _send_output
    self.send(msg)
  File "/usr/lib/python2.3/httplib.py", line 567, in send
    self.connect()
  File "/usr/lib/python2.3/httplib.py", line 987, in connect
    sock.connect((self.host, self.port))
  File "<string>", line 1, in connect
socket.error: (111, 'Connection refused')
Exception exceptions.AttributeError: "Smugmug instance has no attribute 'session'" in <bound method Smugmug.__del__ of <__main__.Smugmug instance at 0x40276c4c>> ignored

tsk1979

What is the port through which the connection goes to smugmug. I think that port may be closed as outgoing connections are allowed through port 80 only for now.

rutt wrote:

This looks to me as if you aren't getting through to smugmug at all. See the connection refused message at the bottom of the error report.

You can try adding some print statements near the apparent ssource of the problem:

print self.account
print self.password
print self.api_version
rep = self.sp.loginWithPassword(self.account,self.password,self.api_version)
print rep

rutt

tsk1979 wrote:

What is the port through which the connection goes to smugmug. I think that port may be closed as outgoing connections are allowed through port 80 only for now.

That would account for it. it's on 443, the standard port for https.

tsk1979

rutt wrote:

That would account for it. it's on 443, the standard port for https.

A clarification about the upload script.
suppose I alread have a gallery called files in category Other.
In the sm_tool.py upload option there is something called "Gallery Id". This will be the gallery name, ie "files"?

mpmcleod

tsk1979 wrote:

A clarification about the upload script.
suppose I alread have a gallery called files in category Other.
In the sm_tool.py upload option there is something called "Gallery Id". This will be the gallery name, ie "files"?

I'll jump in here. The gallery id is the number of the gallery. http://<username>USERNAME.smugmug.com/gallery/NUMBERS</username>

in this case the gallery id is 123456

I had /gallery/123456 in the example but it turns out
1) 123456 is a real gallery id
2) you can link there regardless of the username (I think this is a strange "feature")


Thanks Rutt for this tool.

-- Mike

gymshots

extension to rutts python tool
I needed to identify the original file name of images on smugmug and of images sold. The sales data only provides the image number from their database so that made it hard to figure out which original files sold.

I wrote two extensions to rutt's python tool to do two things:

1) getFileInfo 'photoId'

takes the photo ID from the URL (the last numbers before the '/' in the URL)
and returns the original file name and gallery where it is loaded. This helps me quickly locate the CD where I have backed up the image, if I wish to retrieve the original image from archive.

2) getInfoFiles filename

this is really a batch job that answers the same question as #1 above.

where 'filename' is a file that contains a list of the image URLs exported from the sales data. it retrieves the file name and gallery where these images came from and puts them into a '!' delimited file with the .out extension. This file in turn can be imported back into the sales data and used to sort the sales data on original file name and event/gallery.


if anybody is interested, I will post the extensions in this thread.

Susan Jones
gymshots.smugmug.com

jaimielan

Love the script. I am doing some major archiving and the full update feature is terrific.

I have run in to this error, it is a minor inconvenience in that I let the script run all night and find out that it errored out at 4 am. Any ideas? (BTW, I am using OS X 10.4.6)


File "smugmug.py", line 656, in ?
main()
File "smugmug.py", line 649, in main
full_update(Options(argv[2:]))
File "smugmug.py", line 626, in full_update
update_dir(smugmug,root,opts,files)
File "smugmug.py", line 579, in update_dir
local_information=li)
File "smugmug.py", line 293, in upload_files
self.upload_file(albumid,file,caption(file,opts))
File "smugmug.py", line 331, in upload_file
self.post_multipart("upload.smugmug.com","/photos/xmladd.mg",fields,[file])
File "smugmug.py", line 346, in post_multipart
h.send(body)
File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/httplib.py", line 579, in send
self.sock.sendall(str)
File "<string>", line 1, in sendall
socket.error: (32, 'Broken pipe')
</string>
<filename><filename></filename></filename>

rutt

I'm pretty sure it means that smugmug timed out or the connection was otherwise broken perhaps by some network flakiness.

jaimielan wrote:

Love the script. I am doing some major archiving and the full update feature is terrific.

I have run in to this error, it is a minor inconvenience in that I let the script run all night and find out that it errored out at 4 am. Any ideas? (BTW, I am using OS X 10.4.6)


File "smugmug.py", line 656, in ?
main()
File "smugmug.py", line 649, in main
full_update(Options(argv[2:]))
File "smugmug.py", line 626, in full_update
update_dir(smugmug,root,opts,files)
File "smugmug.py", line 579, in update_dir
local_information=li)
File "smugmug.py", line 293, in upload_files
self.upload_file(albumid,file,caption(file,opts))
File "smugmug.py", line 331, in upload_file
self.post_multipart("upload.smugmug.com","/photos/xmladd.mg",fields,[file])
File "smugmug.py", line 346, in post_multipart
h.send(body)
File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/httplib.py", line 579, in send
self.sock.sendall(str)
File "<string>", line 1, in sendall
socket.error: (32, 'Broken pipe')
</string>
<filename><filename></filename></filename>

jaimielan

rutt wrote:

I'm pretty sure it means that smugmug timed out or the connection was otherwise broken perhaps by some network flakiness.

That's entirely possible, I am in South America at the moment and flakiness is a way of life when it comes to internet connections.