Don, I have a performance question
Nikolai
Registered Users Posts: 19,035 Major grins
One user of mine (gosh, it's been out only about a week and I already have *u*s*e*r*s*:-) complained that my upload process (which simply loops through the prepared list of files using one of your upload APIs) is
1) more bandwidth-intensive compared to your IE uploader
2) somewhat slower compared to your IE uploader
Sounds contradictional, I know. Hogging more bandwidth shoul result into higher speed, but that what the guys said... and "customer is always right":-)
I haven't done any extensive benchmarking yet, but simply to get an idea and directions - what is the nature of IE uploader? Are you using essentially the same API and similar loop-throu-the-filelist approach, or do you have something else (which we probably can't use)?
Thanks!
1) more bandwidth-intensive compared to your IE uploader
2) somewhat slower compared to your IE uploader
Sounds contradictional, I know. Hogging more bandwidth shoul result into higher speed, but that what the guys said... and "customer is always right":-)
I haven't done any extensive benchmarking yet, but simply to get an idea and directions - what is the nature of IE uploader? Are you using essentially the same API and similar loop-throu-the-filelist approach, or do you have something else (which we probably can't use)?
Thanks!
"May the f/stop be with you!"
0
Comments
If you're using the XMLRPC upload method, you'll see a 25% reduction in speed. Base64 encoding increases the # of bytes. See the notes in the docs. It's highly recommended that this method is never used, it's only there for completeness.
If you're using the POST w/XMLRPC response, the speed will be nearly identical. The only difference is the processing time (just a second or two) after each photo occurs during the upload, instead of all loaded on the back.
For example, if someone uses the IE uploader to upload 200 images, it's done in a *single POST*. That sucks for all kinds of reasons, not the least of which is that we then process all 200 of those images simultaneously and sequentially afterwards. So instead of a 2 second pause between each photo, we have 2 seconds * 200 images = 400 seconds after the upload completes before our webserver replies. Ugh.
You can "bypass" the second or two pause between photos while we validate by threading a new upload process right as the old one finishes. Thus, bytes are continually going across the wire in a new thread even while waiting for a reply on the previous image in the old thread.
Other than that, I can't think of any optimizations you can make to speed things up. Total time, unless you're doing something strange, should be roughly identical (though marginally faster, I would think, since your POSTs are being spread around our cluster, which the IE uploader doesn't do) to the IE uploader.
Don
I have figured out that much ;-)
Currently a batch upload happens in a single for-loop, with synchronous POST commands following each other. Do I understand you correctly that I can use asynchronous POSTs, essentially shooting requests one after another without necessarily wating for reply. I mean, I still need to get the response, so in practice I simply can follow your advice and launch several (2-3) threads shooting asyncronuns POSTs and sharing a single "send" resource, which they release immediately after thier own send is done and they are waiting for the response.
Is this correct?
Thank you very much for your detailed answer, I do appreciate it!
Yes, exactly. Each thread will (of course) need to open it's own socket to our upload web servers, but there's no reason you can't have 2 or more uploads going simultaneously, if you think that will somehow help. But the scenario described above, where the threads "overlap" slightly is probably the most bandwidth efficient.
Don
Cheers!