Batch upload: insight info needed..
Nikolai
Registered Users Posts: 19,035 Major grins
I am about to start tesing batch uploads.:D
I know that smugmug's own windows/IE uploader features the following restrictions (per one upload session):
Technically, each single file upload can be considered as a "batch of one" from the server point of view. So at this point we're obviously "kosher". And if I upload next one in an hour - I don't think it will be a problem. So, if I pause for an hour between "appropiate" (type/size ok) single-file uploads, everything's great.
Now, what about 10 minutes pause? Most likely ok.
1 min delay? Probably ok, too.
So my question is - how fast can I go? 45 sec, 30 sec, 15 sec, 10 sec, 5 sec, 1 sec? No delay at all? At which speed I'm risking to "chock" the server?
And, if there is such a risk, is there a way to determine the server's "status", so I can dynamicaly adjust my delays according to the server's current ability to churn the upload?
Or, maybe, there is some sort of "buffer" (100Mb/300files) to which I should tune up my upload process. Yet again - how can I determine that my "buffered" batch has been processed and I can proceed with the next one..
Man, I do need that "NOOP"/"ping" call...
Thank you!
I know that smugmug's own windows/IE uploader features the following restrictions (per one upload session):
- 1 gallery
- 300 files max
- 100 Mb total max
- 8Mb per file max
Technically, each single file upload can be considered as a "batch of one" from the server point of view. So at this point we're obviously "kosher". And if I upload next one in an hour - I don't think it will be a problem. So, if I pause for an hour between "appropiate" (type/size ok) single-file uploads, everything's great.
Now, what about 10 minutes pause? Most likely ok.
1 min delay? Probably ok, too.
So my question is - how fast can I go? 45 sec, 30 sec, 15 sec, 10 sec, 5 sec, 1 sec? No delay at all? At which speed I'm risking to "chock" the server?
And, if there is such a risk, is there a way to determine the server's "status", so I can dynamicaly adjust my delays according to the server's current ability to churn the upload?
Or, maybe, there is some sort of "buffer" (100Mb/300files) to which I should tune up my upload process. Yet again - how can I determine that my "buffered" batch has been processed and I can proceed with the next one..
Man, I do need that "NOOP"/"ping" call...
Thank you!
"May the f/stop be with you!"
0
Comments
onethumb is the real mastah of the API and can give more authoritative answers than I can, but until he gets a chance to weigh in I'll venture a few answers that are probably correct.
I haven't heard of the 300 files limit. Mac Uploader users routinely upload about 2.63 gazillion images per batch, so I don't know what would create a 300-file barrier.
<img src="http://cmac.smugmug.com/photos/9262161-M.jpg" width="150" height="150" align="left" hspace="6" vspace="6">In terms of making the server choke, we have users who upload from their company's datacenter at horrendous speed. We accept the photos as they upload them without a bottleneck. However, there is some processing after receiving the photos, which includes making display copies. We place "processing image" thumbnails in your gallery as placeholders until our servers can process the images.
I hope this helps.
Thanks,
Baldy
Hi, Baldy,
very nice of you to reply!:-)
My information about existing restirctions was based on:
- 8Mb per file: Smugmug help/faq/guidelines
- 300 files per upload - IE Windows uploader
- 100Mb per upload - IE Windows uploader
So, my understanding is, that I can start playing with those mass uploads without a serious risk of chocking you guys up. Well, that's great news. I was simply trying to be "better safe than sorry" and didn't want to cause any trouble..I tried to upload like 500 small files and got error message about 300 limit.
Whenever I accidentally drop more than this amount, it will refuse to upload.
Thanks! You'll see some uplaod traffic from me soon then:-)
Just to be clear (and I'll try to update the API pages about this too), there are *no* speed or number limits other than a maximum of 8MB per photo. The limits you see with the windows uploader, quite frankly, are there because that thing sucks. We're working hard on new uploaders to replace them, but it's certainly not a server or smugmug limit.
You can upload batches of many thousands at once (and people have). You can upload them as fast as you'd like, and even multi-thread your app to upload multiple images simultaneously, if you'd like.
You'll notice there's a second or two pause after we receive a photo... That's us checking to make sure the photo/video looks ok. There's no reason you can't have another thread start another upload during this time, for example, so you don't have even a second of downtime to slow your uploads.
Don
Thank you for the clarification!