API Key rejections

luke_church

EDIT: I now believe that this problem is not the responsibility of Smugmug, but rather an issue with my XML comms proxy. IMHO This should not be considered a bug in Smugmug, unless this edit is removed.


So tonight was the first time for a while that I've had anytime to commit to the Smugmug APIs. this was supposed to be a SmugTest Health Check report, but unforutantely, the sea of Fault Code 18, Invalid API Keys, makes such a report impractical, nearly every method except anonymous login fails. The automated test report produced 3200 API Key rejections and 7 successes, all of which concentrated at the start of the test.

Current success rate < 0.1% :-(

The problem was a pain a while back, but seems to have become a serious problem.

avonwyss has also commented on this problem:
http://www.dgrin.com/showthread.php?t=20140 (avonwyss confirms problem)
http://www.dgrin.com/showthread.php?t=18422 (My original report, back in the days when this was rare)

Unfortunately it now occurs at login, so there is no delay or caching that I can do to try and help matters. It occurs with both API keys registered to me, targetting 3 seperate accounts and both XML-RPC 1.1.1 and 1.1.0.

The only possible solution would be to authenticate using the old pre-API key APIs and then use the returned SessionID with the API key in a retry-until-succeed loop. I'm unwilling to do something this hacky, the structure of the application assumes at most 2 end points, one encrypted and one plain-XML.

Applications currently stalled on the API Bugs:

Flickr->Smugmug
Fotki->Smugmug
SmugTest
SmugTools.NET Library
+ others not currently publically anounced.

I've started getting akward emails asking me whether anything is ever going to be released... I don't like telling people that I can't answer their questions... :-(

As always, I can send XML traces from the wire (though I can't post the keys to the forum, obviously), I can send any other data that would be helpful.

If there's anything that I can do to help you fix these bugs, give me a yell, I look forwards to getting these issues fixed.

Luke

onethumb

luke_church wrote:

So tonight was the first time for a while that I've had anytime to commit to the Smugmug APIs. this was supposed to be a SmugTest Health Check report, but unforutantely, the sea of Fault Code 18, Invalid API Keys, makes such a report impractical, nearly every method except anonymous login fails. The automated test report produced 3200 API Key rejections and 7 successes, all of which concentrated at the start of the test.

Current success rate < 0.1% :-(

The problem was a pain a while back, but seems to have become a serious problem.

avonwyss has also commented on this problem:
http://www.dgrin.com/showthread.php?t=20140 (avonwyss confirms problem)
http://www.dgrin.com/showthread.php?t=18422 (My original report, back in the days when this was rare)

Unfortunately it now occurs at login, so there is no delay or caching that I can do to try and help matters. It occurs with both API keys registered to me, targetting 3 seperate accounts and both XML-RPC 1.1.1 and 1.1.0.

The only possible solution would be to authenticate using the old pre-API key APIs and then use the returned SessionID with the API key in a retry-until-succeed loop. I'm unwilling to do something this hacky, the structure of the application assumes at most 2 end points, one encrypted and one plain-XML.

Applications currently stalled on the API Bugs:

Flickr->Smugmug
Fotki->Smugmug
SmugTest
SmugTools.NET Library
+ others not currently publically anounced.

I've started getting akward emails asking me whether anything is ever going to be released... I don't like telling people that I can't answer their questions... :-(

As always, I can send XML traces from the wire (though I can't post the keys to the forum, obviously), I can send any other data that would be helpful.

If there's anything that I can do to help you fix these bugs, give me a yell, I look forwards to getting these issues fixed.

Luke

This sounds like a new problem.

For the record, I fixed the API Key rejection problem many weeks ago. As suspected, it was due to minor latency issues between our DB masters and our DB slaves.

That particular issue is no longer occurring. This must be something new.

What's more, I'm seeing thousands of API requests, from hundreds of different apps, with keys, every hour. They're not getting rejected.

So either you've found some corner case (most likely) or there's some bug in your code (less likely).

What can we do to narrow it down? Is anyone else experiencing these types of problems?

Don

luke_church

onethumb wrote:

This sounds like a new problem.

For the record, I fixed the API Key rejection problem many weeks ago.

Excellent. :

What's more, I'm seeing thousands of API requests, from hundreds of different apps, with keys, every hour. They're not getting rejected.

So either you've found some corner case (most likely) or there's some bug in your code (less likely).

OK, so this is good, I'd much rather deal with a corner case/bug than a general case.

What can we do to narrow it down? Is anyone else experiencing these types of problems?

If you could give me the name of one the applications, particulally if it isn't using HTTPS for the comms, then I could do an XML trace comparison? Or if you can post a sample XML emit-response pair missing the sensitive info. I'll grab one now from the test application.

SendToSmugmug is using the old API without keys

I didn't see any keys in FxFoto's XML trace, I think it's using HTTPS for authentication (as it should :-))

I don't currently have a licence for S*E, so I can't look at that anymore, I guess I should just buy a copy....

Does your server offer any justifications in the logs as to why it's rejecting them? I can instruct my client to start throwing requests with a couple of minutes notice, if it helps. (As long as I'm at my computer at the time).

Glad to hear this is a different issue,

Thanks for the quick reply,

Luke

luke_church

Successful anonymous login trace
So here we go, a login that works anonymously, I've left everything in, except the API key

[php]

POST /hack/xmlrpc/ HTTP/1.1
Content-Type: text/xml
User-Agent: XML-RPC.NET
Host: api.smugmug.com
Content-Length: 336
Expect: 100-continue
Connection: Keep-Alive

HTTP/1.1 100 Continue

<?xml version="1.0"?>
<methodCall>
<methodName>smugmug.login.anonymously</methodName>
<params>
<param>
<value>
<string>1.1.0</string>
</value>
</param>
<param>
<value>
<string>...</string>
</value>
</param>
</params>
</methodCall>HTTP/1.1 200 OK
Date: Tue, 01 Nov 2005 23:50:14 GMT
Server: Apache
X-Powered-By: smugmug/1.2.0
Cache-Control: private, max-age=1, must-revalidate
Pragma:
Set-Cookie: SMSESS=9b14c3870efb6552919bc7fed02d7efc; path=/; domain=.smugmug.com
ETag: sm-619396822d550839eb27200275ea2d47-sm
Content-Length: 301
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/xml

<?xml version="1.0" encoding="iso-8859-1"?>
<methodResponse>
<params>
<param>
<value>
<struct>
<member>
<name>SessionID</name>
<value>
<string>9b14c3870efb6552919bc7fed02d7efc</string>
</value>
</member>
</struct>
</value>
</param>
</params>
</methodResponse>

[/php]

I'll now get one that doesn't. The test result dump file says:

01/11/2005 19:56:11: Attempting anonymous login
01/11/2005 19:57:24: Login Suceeded, Session ID: d1427d2fff3cb09bed35f727b70d1abf
01/11/2005 19:57:24: In Logout
01/11/2005 19:57:24: Attempting Logout
01/11/2005 19:57:25: Warning Fault: SmugmugFaultCode: InvalidSession TrialCount: 0
01/11/2005 19:57:25: Attempting anonymous login
01/11/2005 19:58:25: Login Suceeded, Session ID: ddaa8301d8f6d7b457926170760ad1de
01/11/2005 19:58:25: In Get AccountType
01/11/2005 19:58:25: In Logout
01/11/2005 19:58:25: Attempting Logout
01/11/2005 19:58:26: Warning Fault: SmugmugFaultCode: InvalidSession TrialCount: 0
01/11/2005 19:58:26: Attempting Login With Password
01/11/2005 19:58:26: Warning Fault: SmugmugFaultCode: InvalidApiKey TrialCount: 0
01/11/2005 19:58:26: Attempting Login With Password
01/11/2005 19:58:27: Warning Fault: SmugmugFaultCode: InvalidApiKey TrialCount: 1... Until trialCount hits 50 and the exception is allowed to cascade up and terminate the test case.

So it seems that the problem only occurs on passworded logins. I could try getting a hash using REST and see whether that plays...

Anyway, I'll get a failed login dump next

luke_church

Sorry, that log was from the tests I did 5 hours ago. But exactly the same behaviour was repeated at 23:51, so ~10 minutes ago.

Luke

luke_church

So the bug lies in the comms between my test Application, the SmugmugLowLevelAPI that I've written and its XML-RPC generator. It's primarily my fault.

Appologies Don et al, this one looks as if it was my blunder.

Now to see whether I can coax it into providing the data in the order that I specify...

Sorry,

Luke

skuwamoto

I'm seeing exactly the same thing. I can reproduce the problem easily just hitting the REST URL from a browser like so:

http://api.smugmug.com/hack/rest/?method=smugmug.login.anonymously&apiKey=XXXXXXXX

luke_church

skuwamoto wrote:

I'm seeing exactly the same thing. I can reproduce the problem easily just hitting the REST URL from a browser like so:

http://api.smugmug.com/hack/rest/?method=smugmug.login.anonymously&apiKey=XXXXXXXX

Hi,

This is because the call isn't quite properly structured. If you look in the documentation at http://www.smugmug.com/hack/method-smugmug.login.anonymously

it specifies: APIKey not apiKey (note caps difference)

Calls to your url fail for me.

Calls to your url with s\apiKey\APIKey work fine.

Just to re-affirm: I do not believe that the issue I mentioned in this post is a bug, it was a mistake I was making.

Let us know whether the above change fixes matters.

Cheers,

Luke

avonwyss

onethumb wrote:

For the record, I fixed the API Key rejection problem many weeks ago. As suspected, it was due to minor latency issues between our DB masters and our DB slaves.

That particular issue is no longer occurring. This must be something new.

Great! But even though I try to read all the answers to my posts here (or in the API section of the forum in general), I didn't see a notice that this was fixed. Since I was quite frustrated after the problems I stubled across while developing (you may remeber that I posted some things about it), I put the whole thing beside, waiting for information about fixes for the reported bugs. But I didn't see anything, so I didn't continue my work...

Would it be possible to have some sort of bugtracking list which can be used to track open issues, or to at least put a quick notice when specific bugs were fixed (at least for such important ones)? This would be greatfor all devs out here I think...

Thank you!