new *BETA* API release - May 10, 2005

onethumb

Hey everyone...

The threatened revamped REST release is out, and with it, a bunch of new things for both REST and XML-RPC. Highlights include:

- new BETA endpoint to test your code against before it rolls out to the production endpoints. See REST docs and XML-RPC docs on the subject.

- REST is now quite a bit more verbose. All ID tags should now use the requested format, and all responses should now reside in a root container in addition to the RSP container. This is what I need feedback on more than anything else.

- new 'Heavy' flags for a few methods allow you to get heavier results, when desired, and get more data back. Methods include smugmug.albums.get, smugmug.images.get, and smugmug.users.getTree.

- smugmug.albums.getStats automatically uses a Heavy-like mode already, so you'll get images.getStats in the same call.

- new methods, including: smugmug.users.getTransferStats, smugmug.users.getTree

- renamed smugmug.accounts.getType to smugmug.users.getType. old method will, of course, continue to work but will be removed from the docs once this is out of beta.

- both interfaces now support gzip encoding (use standard HTTP header 'Accept-Encoding'). Use it, love it. I've seen responses go from 500K to 13K.

- both interfaces now support standard HTTP "If-Not-Modified" headers using ETags. Save even that 13K if you'd like if things haven't changed.

- doc bugs fixed, including smugmug.images.get, smugmug.albums.changeSettings, smugmug.albums.getInfo, and more.

- REST now properly responds in UTF-8, so foreign chars should propogate properly. Be interesting to see what happens to UTF-8 submissions. Anyone care to test a bit more?

- fixed bug in logout, should work properly now

- fixed bug in smugmug.images.delete, should work properly now

UPDATE 11am PST: XML-RPC now accepts a struct with named elements for all calls. No more worrying about the order of the elements, just make it a struct and name each one appropriately. I imagine this will probably become the standard way of doing things (no, the old way won't go away, it will just cease to be recommended or documented).

Whew. There might be more, but luckily, it's a BETA. Some of the docs have intentionally been left a little vague while we test and refine any responses to the methods. Suggestions welcome, but anything tagged with BETA in the docs is clearly not complete.

Enjoy!

Don

devbobo

Don,

Looks like some great stuff here, but...
smugmug.users.getTree ??

I am going to cry
Took to hours to write and debug my code to build that exact same tree

David

devbobo

Don,

With smugmug.users.getTree if Heavy is set, does that return all the images in an album ?

Thanks,

David

onethumb

devbobo wrote:

Don,

With smugmug.users.getTree if Heavy is set, does that return all the images in an album ?

Thanks,

David

No, it doesn't. It did for a little while in early testing, but since images aren't really part of the organization, I figured they didn't belong.

Additionally, if the call gets too heavy, it could fail to finish - I have checks for runaway calls. Some people have 60,000 photos (in who knows how many galleries), so it could get pretty enormous.

Combined with the other calls, though, this should dramatically reduce the # of calls required to accomplish almost anything.

I *might* be open to putting the images in there, but I'd have to hear a pretty compelling reason why you'd want all that stuff in a single call.

Don

devbobo

No, i didn't really want it in there. I was actually worried about how large the response my be, that was the original intent of the message.

Cheers,

David

devbobo

Don,

ok, a couple of questions regarding XMLRPC...

1. can't seem to retrieve the public flag, from any methods....smugmug.albums.getInfo, smug.albums.get with Heavy and smugmug.albumtemplates.get. Any ideas ?

2. using smugmug.images.get with the heavy flag, I can't retrieve ImageID ? Or is it called something else ?

Thanks,

David

Nikolai

Yay!!!
Great news, Don, thank a bunch!
Sorry being too late with my response, was postprocessing:-)
It looks like many of my personal wishes came to reality:-)
I'm looking forward to get my hands dirty again:-)

Cheers!

Nikolai

gzip encoding..

onethumb wrote:

- both interfaces now support gzip encoding. Use it, love it. I've seen responses go from 500K to 13K.

It's seems like a truly great feature, but I don't seem to be able to find it in the RPC-XML docs. What do I need to do to activate it?

Thanks!

onethumb

Nikolai wrote:

It's seems like a truly great feature, but I don't seem to be able to find it in the RPC-XML docs. What do I need to do to activate it?

Thanks!

It's a web-standard via HTTP headers, really has nothing to do with the API itself, and thus, isn't documented on smugmug.

Google probably knows all about it, though. Look for HTTP specs. 'Accept-Encoding' is the header in question, I believe. ETag & If-Not-Modified headers would be in those specs, too.

Don

onethumb

devbobo wrote:

Don,

ok, a couple of questions regarding XMLRPC...

1. can't seem to retrieve the public flag, from any methods....smugmug.albums.getInfo, smug.albums.get with Heavy and smugmug.albumtemplates.get. Any ideas ?

Hmm, I see Public in a few of these just fine. I'll keep looking, though. Can you post the XML-RPC response?

devbobo wrote:

2. using smugmug.images.get with the heavy flag, I can't retrieve ImageID ? Or is it called something else ?

Hmm, yeah, that would help.

Fixed, I think. Let me know.

Don

devbobo

onethumb wrote:

Hmm, I see Public in a few of these just fine. I'll keep looking, though. Can you post the XML-RPC response?


Hmm, yeah, that would help.

Fixed, I think. Let me know.

Don

ok...seems that the type of Public (and the other boolean flags) has changed back to INTs again....ggrrrrr

Can't seem to access the ImageID though...any thoughts ?

Thanks,

David

Nikolai

Gotcha, thanks!

onethumb wrote:

It's a web-standard via HTTP headers, really has nothing to do with the API itself, and thus, isn't documented on smugmug.

Google probably knows all about it, though. Look for HTTP specs. 'Accept-Encoding' is the header in question, I believe. ETag & If-Not-Modified headers would be in those specs, too.

Don

More grinder time:-)
Cheers!

onethumb

devbobo wrote:

ok...seems that the type of Public (and the other boolean flags) has changed back to INTs again....ggrrrrr

Can't seem to access the ImageID though...any thoughts ?

Thanks,

David

Woops, try now. Both.

Don

dsdee

getTree looks pretty slick; can't wait to get home from work and play with it.

Hey Don, any chance (still) of getting an interface into the print pricing into the API ??

--David

onethumb

dsdee wrote:

getTree looks pretty slick; can't wait to get home from work and play with it.

Hey Don, any chance (still) of getting an interface into the print pricing into the API ??

--David

Just so we're clear, the eventual goal with the API is to expose everything that A) is exposed in the web interface, and doesn't require SSL-grade security (ie, shopping cart, signing up for an account).

So, yes, pro pricing is on the TODO list. Who knows which year it'll get done, but it's coming.

Don

Nikolai

Good to know!

onethumb wrote:

Just so we're clear, the eventual goal with the API is to expose everything that A) is exposed in the web interface, and doesn't require SSL-grade security (ie, shopping cart, signing up for an account).

So, yes, pro pricing is on the TODO list. Who knows which year it'll get done, but it's coming.

Don

Pricing API would be soooo damn cool!

onethumb

As noted in the first post on this thread, it's been updated with the following info:

XML-RPC now accepts a struct with named elements for all calls. No more worrying about the order of the elements, just make it a struct and name each one appropriately. I imagine this will probably become the standard way of doing things (no, the old way won't go away, it will just cease to be recommended or documented).

Don

devbobo

onethumb wrote:

Woops, try now. Both.

Don

Don,

Thanks...both are working


David

Nikolai

Re: named structs

onethumb wrote:

As noted in the first post on this thread, it's been updated with the following info:

XML-RPC now accepts a struct with named elements for all calls. No more worrying about the order of the elements, just make it a struct and name each one appropriately. I imagine this will probably become the standard way of doing things (no, the old way won't go away, it will just cease to be recommended or documented).

Don

This is great news, I think it's a much more robust way of dealing with all those calls!
Thanks again!

studiosells

Persistent HTTP vs. Parallel Connections and Nice job on the Beta API
I'm working on a Perl library for the REST API and some small clients. I hadn't read the BETA API stuff so missed out several new features (getTree, heavy requests, ...) until today. My current implementation (that doesn't use the BETA API features) uses parallel HTTP GET requests over separate connections to do a series of albums.getInfo and images.getInfo.

Without knowing about the BETA API, I was thinking that I wanted a smugmug.albums.getAllInfo that's like smugmug.subcategories.getAll and will return info for all albums for a given user. This would save clients having to enumerate each album and doing a smugmug.albums.getInfo on each one. And similarly, when getting image information for a particular album, a client has to enumerate the ImageIDs returned by smugmug.images.get and do a smugmug.images.getInfo for each one. Similarly, I had wanted a smugmug.images.getAllInfo. These both seem to be covered by the BETA API through the Heavy parameter - well done!

The reason I post, however, is to let others know that the production REST API supports persistent HTTP connections which seem like a good alternative to doing parallel connections/requests. For those not framiliar with persistent HTTP:

RFC2068, describes an improvement to HTTP which maintains a continuous connection to a HTTP server for multiple requests, P-HTTP. This removes the inefficiency of continually reconnecting to a web server to download multiple images from the same page. The constant connection and reconnection results in a lot of unnecessary overhead.

Some advantages over the original HTTP protocol are:
1. By opening and closing fewer TCP connections, CPU time and memory are saved.
2. Multiple HTTP requests and responses can be sent without waiting for a response which would otherwise be necessary when opening and closing multiple connections.
3. Network congestion is reduced since there are fewer packets.

With the BETA API covering my above requests, I suspect it's still a performance enhancement to use persistent HTTP. Has anybody tested the performance of persistent connections vs. parallel ones? I've fiddled with both, but only a little. I suspect the SM admins would be happier if everything went through a persistent connection.

Beware!! I'll also note that responses from the BETA REST API aren't the same as the production one. I think this has happened with adding support for Heavy requests. For example, smugmug.albums.get returns just a series of Album entries whereas the beta one returns those entries inside an Albums entry.

-m

onethumb

studiosells wrote:

I'm working on a Perl library for the REST API and some small clients. I hadn't read the BETA API stuff so missed out several new features (getTree, heavy requests, ...) until today. My current implementation (that doesn't use the BETA API features) uses parallel HTTP GET requests over separate connections to do a series of albums.getInfo and images.getInfo.

Without knowing about the BETA API, I was thinking that I wanted a smugmug.albums.getAllInfo that's like smugmug.subcategories.getAll and will return info for all albums for a given user. This would save clients having to enumerate each album and doing a smugmug.albums.getInfo on each one. And similarly, when getting image information for a particular album, a client has to enumerate the ImageIDs returned by smugmug.images.get and do a smugmug.images.getInfo for each one. Similarly, I had wanted a smugmug.images.getAllInfo. These both seem to be covered by the BETA API through the Heavy parameter - well done!

The reason I post, however, is to let others know that the production REST API supports persistent HTTP connections which seem like a good alternative to doing parallel connections/requests. For those not framiliar with persistent HTTP:

RFC2068, describes an improvement to HTTP which maintains a continuous connection to a HTTP server for multiple requests, P-HTTP. This removes the inefficiency of continually reconnecting to a web server to download multiple images from the same page. The constant connection and reconnection results in a lot of unnecessary overhead.

Some advantages over the original HTTP protocol are:
1. By opening and closing fewer TCP connections, CPU time and memory are saved.
2. Multiple HTTP requests and responses can be sent without waiting for a response which would otherwise be necessary when opening and closing multiple connections.
3. Network congestion is reduced since there are fewer packets.

With the BETA API covering my above requests, I suspect it's still a performance enhancement to use persistent HTTP. Has anybody tested the performance of persistent connections vs. parallel ones? I've fiddled with both, but only a little. I suspect the SM admins would be happier if everything went through a persistent connection.

Beware!! I'll also note that responses from the BETA REST API aren't the same as the production one. I think this has happened with adding support for Heavy requests. For example, smugmug.albums.get returns just a series of Album entries whereas the beta one returns those entries inside an Albums entry.

-m

I've always assumed everyone is using persistent connections, but your post makes me realize that maybe they aren't. Thanks for bringing this up.

If you're opening and closing the port each and every time, you're wasting valuable time. We allow keep-alive requests and persistent connections. If you're idle too long, the connection will close, so you'll have to check your state, but that's fairly trivial.

I think the best way to do API connections is a mixture of persistant and parallel connections. For example, doing parallel uploads is very handy - while we're checking to make sure the file you've uploaded is a valid file, there's a pause. During this pause, there's no reason you can't be uploading the next photo on a different socket and thread. Go for it - it'll save a lot of time.

Likewise, getting info for many photos/galleries/etc can be spread across multiple persistent connections.

Don

devbobo

Don,

Just a quick note to let you know how cool some of the new features are. I really love the new heavy flag. It's made my work some much easier.

Love ur work

David

studiosells

getTree Doesn't Return Albums
http://api.smugmug.com/hack/rest/beta.mg?APIKey=FOO&method=smugmug.users.getTree&Version=1.1.0&SessionID=BAR&Heavy=true&NickName=studiosells

Doesn't seem to return Album entries.

I also tried it with Heavy=1

-m

onethumb

studiosells wrote:

http://api.smugmug.com/hack/rest/beta.mg?APIKey=FOO&method=smugmug.users.getTree&Version=1.1.0&SessionID=BAR&Heavy=true&NickName=studiosells

Doesn't seem to return Album entries.

I also tried it with Heavy=1

-m

First of all, the correct call would be:

http://api.smugmug.com/hack/rest/beta.mg?method=smugmug.users.getTree&Heavy=1&NickName=studiosells&SessionID=SESSIONID

Remember, Version, APIKey, etc, are all set during the login phase, not on subsequent calls.

Secondly, I get an Album there just fine. In the Motorcycles Category (id=14). You don't see it?

Don

studiosells

Gotcha on the version/APIkey/version not being needed for each request.

From an anonymous session it seems to never return the Albums. If I login as studiosells then getTree will return albums for that one user but not any others. Did you use some super-admin-session to be able to list the albums for studiosells ?

With the old API I noticed that albums.get would return private entries for which the albums.getInfo call would subsequently fail. Perhaps this is related somehow?

-m

studiosells

smugmug.accounts.getType for anonymous SessionID
Unless I'm doing something silly, accounts.getType for an anonymous session returns AccountType Power /AccountType.

I also noticed that the help page for smugmug.login.anonymously has

Latest Version is currently String "1.1.0".

So perhaps my problem is that the BETA APIs aren't enabled properly for anonymous connections, even if I specify a version of 1.1.1.

-m

rutt

gzip vs x-gzip
I'm trying the gzip feature. I add the header: "accept-encoding: gzip". I get back "encoding: gzip", which makes sense to me. But the python patches I dug up to support this are expecting "x-gzip". Which is right?

rutt

xmlrpc resuls changed?
Maybe I missed this in the thread, but it seems that the results for xmlrpc calls are pretty different. I used to get something like this from login:

{'SessionID': 'a8fa6666eca05179ccb6d4618', 'PasswordHash': '$1$sTTB3bPG$sRMm3o0hXqR/', 'UserID': 666}

But now I get something like:

{'Login': {'values': {'SessionID': 'f50d0d556519d9d72192', 'PasswordHash': '$1$ojiCMwYz.nqNSVZ.', 'UserID': 666}, 'parameters': {'arrayName': 'Login'}}}

Is this intentional? Perhaps my python xmlrpc parser has been fooled, though it does seem to have parsed correctly.

Anyway, I didn't see this documented.

dsdee

rutt wrote:

Is this intentional? Perhaps my python xmlrpc parser has been fooled, though it does seem to have parsed correctly.

Anyway, I didn't see this documented.

Nope, don't think it' python as i saw similar with Perl's xml-rpc.

it also seems that an extra layer (or two?) is put into the structure returned

I used to have:

smAPI.pm:	$smSessionID = $resp->{'SessionID'}->value;

but with the beta must have:

smAPIBeta.pm:	$smSessionID = $resp->{'Login'}->{'values'}->{'SessionID'}->value;

I just figured that the docs for the call hadn't been updated yet...

onethumb

rutt wrote:

Maybe I missed this in the thread, but it seems that the results for xmlrpc calls are pretty different. I used to get something like this from login:

{'SessionID': 'a8fa6666eca05179ccb6d4618', 'PasswordHash': '$1$sTTB3bPG$sRMm3o0hXqR/', 'UserID': 666}

But now I get something like:

{'Login': {'values': {'SessionID': 'f50d0d556519d9d72192', 'PasswordHash': '$1$ojiCMwYz.nqNSVZ.', 'UserID': 666}, 'parameters': {'arrayName': 'Login'}}}

Is this intentional? Perhaps my python xmlrpc parser has been fooled, though it does seem to have parsed correctly.

Anyway, I didn't see this documented.

Strange. No, the REST results are now wrapped in 'Login', but XML-RPC shouldn't be.

Are you seeing this with other calls, too?

Don

rutt

onethumb wrote:

Strange. No, the REST results are now wrapped in 'Login', but XML-RPC shouldn't be.

Are you seeing this with other calls, too?

Don

This seems to have become fixed since the last time I tried it.