Options
BETA API Testing
Nikolai
Registered Users Posts: 19,035 Major grins
Don, it was said before several times by several people (including yours truly): such testing requires immediate feedback from a human on the server side. I don't mind "getting my hands dirty", but my previous experience with such a testing - when bugs/issues were reported several times in a row with a zero response from "the API team" (i.e you, since nobody else is working on API) - did not provide a lot of encouragement.onethumb wrote:Of course, I can't get anyone to TEST the BETA API so we can't release it...
Don
I suggested this before, and I'm suggesting it again: let's set up a day (weekend, preferably) and let's guarantee your online presense, so we can actually test variaous API call, report the results, get the feedback, get the fixes if needed, flourish on each other's work - and all those API will be probably tested and deemed OK in one day.
Otherwise - I don't know about the others, but I got my fingers burnt too many times to initiate another "lone warrior" testing session on my own. I will eventually get through it, but no ETA..
Cheers!
"May the f/stop be with you!"
0
Comments
I think I agree with Nik's sentiment on this.
I spent considerable time developing code (which of course you guys did as well), but what I found was from one release to another some stuff would work and some would be broken. I would have to code to temporary fix until the next release when it would be fixed. I realise this is the nature of development.
From my perspective, there was a deafening silence on some issues that got asked again and again. I am pretty sure that the bug Luke is talking about I raised it several times on several releases (but I can't confirm at this point in time if it is fixed or not). I realise that you state that the API isn't supported. But the gate swings both ways...if you don't support the developers to some extent, the developers are going to lose interest...and the API won't end up getting tested.
Please don't take this the wrong way, I totally respect the effort you have personally put in to this project.
David
SmugMug API Developer
My Photos
With previous releases, I just rolled them out and hoped for the best. You, among others, asked for some sort of beta period where you could test against new versions. This made a ton of sense.
I re-structured the API to support two unique interfaces, one production, one Beta. I then blocked off the entire week when I released the BETA API to do nothing other than fix bugs and support you guys.
There was deafening silence. (Actually a few good bugs got found and fixed, but I never saw someone run anything major against it, like Star Explorer or Send-to-smugmug).
My inclination now is to go back to the "release it into production after internal testing and fix errors as they come up" rather than this open beta testing - no-one uses it.
If I'm coming off as bitter, I am. I feel like I wasted a ton of time coming up with the BETA structure, a ton of time sitting around waiting for people to test against it, and nothing useful has happened. It's held up some great tools, like JT's Mac OS X widget and matt's new Mac uploader because I can't get any legacy testing.
I'm 100% ready to give it one more try where I block of a day or week to do almost purely API release work, but I'm skeptical. It didn't work last time.
Don
Again, I don't want to beat a dead horse, but I made every attempt (including postponing other projects, putting off meetings, etc) to be available the entire week after the BETA was released. Only there was very little interest after the first day or two.
I'm happy to support it as much as I'm able, but only if our developers seem to WANT support. Which none of us seem to feel we got. (I've asked, and JT's asked, for the BETA to be tested a number of times since it was released, to no avail).
Don
Whoa, Don, hold your horses! I, for one, didn't have a slightest idea that after you have rolled out this beta you actually were "sitting there for a whole week waiting on us".
If I knew this - oh, man, I would definitely try to use this valuable time of yours and pound the $hit out of this API:-). But I didn't know it. Honestly.
What I knew was that all previous API features were rolled out and immediately after that we were on our own. Sometimes - very rarely - you were reachable, but most of the time you were not. "API is not supported, have fun, guys!"
OK, it was all in the past, and there is no use pointing fingers.
Let's do something constructive. You said you're willing "to sit with us". I suggested the same. We have to be prepared for this session, so we can use it in the most effective manner. Basically, we need to agree on some protocol, so we can do the most effective usage of yours and our time.
I can honestly say - I'm not ready right now. I can be ready in two weeks, though. I would prefer Sunday, since my Saturdays are pretty much booked every week.
So, how about Sunday September 25th? We can discuss more things in the mean time, kinda make sure all the nooks and crannies are covered.
What do you think?
To quote you from a previous thread... I don't know about other 3rd party developers, but I have time constraints too. I am working a full time job, studying a masters at Uni, doing additional contracting at home, trying to spend time with my family...and do a bit of photography (the main reason I am here). If you I have any available time, I was trying to do some work on my smugmug utility.
I use to be really excited about what I was developing, to the extent that I even sent an email to support addressed to you, outlining ideas for the API and my app..that went without reply. And after a while it makes you wonder...do these guys really care ?
David
SmugMug API Developer
My Photos
We can go round and round, but the bottom line is: Yes, we care. I should say I care, since the API is my baby, but everyone at smugmug loves all the wonderful apps that come out of the API. So it is "we."
But lately the sentiment inside smugmug has been "do our developers care?". It's a two-way street. We're trying to do what's best for our customers, and we have a limited amount of time. When they get excited about a new feature (say, CSS customization) and tell us so, we spend more time on it. When they don't seem excited (say, BETA API), we re-prioritize to things that they do seem more excited about.
We can't have it both ways, and it's a bit of a chicken-and-the-egg thing. I released the API, hoping people would get excited. Luckily, the egg grew into a chicken, and people did get excited. Magic happened, and massively useful things came out of it.
I released the BETA API, hoping people would get excited. They seemed to, and then interest vanished. From my point of view, that egg never hatched, and it's tough for me to want to give it any time when people are clamoring loudly for custom watermarks and such.
I still have no idea, for example, if people like the more verbose XML responses in REST *and* XML-RPC over the more terse responses there are now. Without knowing that, the API can't move forward.
What I'd *really* like to see happen is for the debate to end and the work to begin. I'm committed to making the API release cycle (BETA -> test -> fix bugs -> release) work. I'll give it my time and attention - but only if our developers will do the same.
Who's with me?
Don
Whats this thread about
Thanks again for the handwarmers OT ! Had you not helped me out i was looking straight down the barrel of another 2 hours sitting on a rock facing the sun with my tounge out.
Gus
SmugMug API Developer
My Photos
Imagine a village without an idiot.
And perhaps you guys can help me out of a bind. The original concept of my smugbrower firefox extension was to work on all platforms, but I have run into difficulty with files and pathing in MacOS and javascript. Is this something you guys could help me out with ?
David
SmugMug API Developer
My Photos
I haven't been in on any of these issues, this is the first time I serioulsy started prodding the API, but I agree lets look forward.
OK, so I've looked briefly at it, but shall we say that I don't think that the Send To Smugmug infrastructure is quite robust enough to use for testing a Beta interface (it randomly falls over)... To get serious relibability I'm going to have to re-write the thing.
Once I'm there, I'm game for working with you on testing. The problem is that in the initial learning stage it's kind of hard to tell whose mistakes are whose...
OK, so I would love more verbose error messages, as long as my XML-RPC (XML-RPC.NET) parser doesn't have a fit and choke on them, I'm afaid I simply don't have the time to write my own parser infrastructure.
I am.
I sorry you feel bitter, and I think some of the devs here seem to feel the same. However, I'm game for giving this some of my time. I'll have to ask for consideration (I'm a student, and am doing 2 other jobs and a major research project at the same time), so my time commitment will vary over the year.
Lets set the time for testing and go forwards.
Might I sugest something that Microsoft use on testing their Beta Development tools?
- People post 'bug' reports, description, sample code
- Other people confirm/disagree with the bug as to whether they can replicate it.
MS seem to respond pretty fast to confirmed bugs, as least they know it's less likely to be a clueless user. With this system people are also faster at posting stuff they think are bugs, but are not sure.
Any thoughts?
Luke
Oh yes. PS. If you want real time feedback, we'll have to arrange a date, I should usually be asleep when you're awake!
SmugSoftware: www.smugtools.com
The only thing is - since you and us want results back *fast* we need to agree on a schedule. We all live pretty busy lives, hence I can't imagine that we all can be availabele on any given day/time on a *short* notice. That may be especially hard on you, since some of us are in quite different time zones, but want to play, too.
I suggested Sunday in 2 weeks, Sept 25th. If we agree, I'll be ready with a testbed so we can test if not all, then at least a 90% of code (I'm talking XML-RPC, don't see the reason to switch to REST since I already have the basics covered this way).
How's that for a cooperation?
Cheers!
Portfolio • Workshops • Facebook • Twitter
I don't want results "fast". I just want results.
Sundays are rarely good for me, I'm usually busy with church things, and every once in awhile, it's the only day of the week I get off. (Ok, maybe that's wishful thinking, but it is the goal!)
We have any other time?
Again, I don't need this to be a fast, realtime sorta thing (though that would be fun and effective). A simple "oh, I think I found a bug, let me post a thread about it" would work.
1 bug = 1 thread (none of this "collect all the bugs in one thread so we can't see what we're doing" nonsense).
Don
I hate to repeat myself, but thus far my experience with bug reporting "your way" was as follows: 75%, if not more, of all the API bug reports (both mine and others') went unnoticed/unanswered by you - either at all, or for a very, very long time.
You're talking about valuable time - I totally agree. And if I set aside (read= stole from my family:-) couple of hours to implement a new feature for S*E, I, for once, value these two hours very highly. So, if by the end of those few hours I have zero advance simply because this or that API is not working the way it's described - I'm not only getting excited, but rather frustrated and disappointed. Yes, I would post the bug - only to find out the next day there are no responses. And the next day. And so on.. And then - out of desperation - I would contact Baldy and he'd tell me that you are "in a deep submersion mode" and are not to be touched.
How do you think would I feel after that?
The only reason I suggested real time session is that I see it as the only way to guarantee that our reports would *not* go unnoticed.
If Sunday is bad - well, let's do it some evening. Maybe several evenings, not necessarily in a row.
Another way to do it - if you can see it feasible to actually respond to our bug reports at least on a daily basis at least for some known time period. You were grieving about that one week that you lost waiting for our results - well, who knew that you were waiting for it? And who was ready? You can publish your availability times right here. "Hey guys, I'll be available this week". "Sorry guys, next week is busy, don't expect any responses/fixes".
We all agreed: "it swings/leads both ways".
Your time is valuable - so is ours.
You want to fix the problems with the API - so do we.
All we need to do is to agree on the protocol..
What do you think?
OK, I've now finished writing an XML-RPC description of the entirity of Smumug's 1.1.0 interface for SmugTools, I'm now writing excercise code and associated unit tests. The upshot of which is that I'm going to be prodding 95%+ of the code surface. I will systematically file any bugs that I come across.
If this ends up being a few, please don't think that I'm overloading you guys, I'm not trying to pressure, just throwing issues out as and when I come across them.
If I've time, once I've finished with 1.1.0 I'll write the version targetting 1.1.1.
Hope this helps.
I've started with a trivial documentation bug.
Luke
SmugSoftware: www.smugtools.com
You're a hero, man!
I was doing call-by-call... Hope your work will bring some bugfixes!
Muchas gracias, amigo!
Cheers!
Bear in mind that the next version (currently in beta) fixes *a lot* of bugs.
So testing against the current release might be useful, but I may end up spinning my wheels trying to find bugs that are no longer existant.
Don
OK, no problem, from now on I'll re-run all the tests against the Beta API as well and state in each report whether it reproduces on 1.1.1
But I'm not going to get round to testing the new functionality in 1.1.1, until I'm finished with 1.1.0
Cheers,
Luke
SmugSoftware: www.smugtools.com