Multi threaded uploads

Frode

Any third party way to do this? I've always gotten horrible upload speeds to Smugmug, but with the increase in image size and now also video, the inability to upload multiple files simultaneously is killing me.

At 140KB/s upload (less than 1/20th of my available upload speed), and gigabytes worth of data to upload, it's taking forever and then some. The bandwidth is there all the way, there's nothing in particular wrong that me or support can see, it's just a factor of the protocols and latency limiting the potential throughput of a single connection. It's an issue I face quite often, and adding connections always solves it, but last I spoke to Smug support they had no plans to implement it.

I vaguely remember seeing mention of a third party application that might be able to do this, but I can't recall where I saw it or a name to google for.

Any suggestions would be welcome.

denisegoldberg

If you're running on Windows I'd recommend Star*Explorer.

http://www.starexplorer.com/
On dgrin at Star*Explorer Thread.

--- Denise

Frode

Thanks, looking at it now. Think this is the one I was almost able to remember. It does state it'll do multiple connections, but I'll have to dig around a bit it seems to figure out how many.

denisegoldberg

Frode wrote:

Thanks, looking at it now. Think this is the one I was almost able to remember. It does state it'll do multiple connections, but I'll have to dig around a bit it seems to figure out how many.

Post a question in the Star*Explorer thread I referenced above. Nikolai does check in often and answer them.

--- Denise

Frode

Think I may have to do that. I decided to download and test it, and it does use two upload threads, but that's it. Granted, that's double the speed, but still leaves me at a tenth of desired. Question is whether that's a Smugmug limitation or an application one.

I'll look around it a bit more the next day or two, then make a post once I feel confident I'm not overlooking an obvious option somewhere. Thanks again.

SamirD

I've been dealing with this issue for years. Unfortunately, you're not really going to find an easy answer. KomodoDrop is one of the uploaders that can do multiple threads, but I never got it to work correctly on any of my computers.

I have three cable modems, each capable of 25m/5m. These are connected to a Cicso rv016 router. This router is capable of handling 7 internet connections and using them all simultaneously. To do this, it basically checks which connection is most free and put the requested traffic on that connection. It's a rudimentary round-robin type of load balancing, but it's very effective.

To max out a single one of my modems at 5m upload, I have to have 4 uploads running. So to max out all 3 modems, I have to have 4*3=12 connections simultaneously. Manually setting up these uploads is a pain, but it's something I'm used to. What would be really nice is to use something like the Filezilla FTP client, which automatically uses as many connections as you specify. You don't have to manually load balance your uploads. This is why I want FTP. I've created an option for it here:
http://smugmug.uservoice.com/forums/17723-smugmug/suggestions/294159-ftp-uploading?ref=title

Vote for it if you think this would help you too.

Frode

You just made my Christmas card list. KomodoDrop works great. I pumped up the threads and it's flying!

I don't get why Smugmug doesn't implement this officially. This is a well known issue when combining today's broadband connections and geographic distance. Considering the sizes of the images nowadays (I'm struggling to stay below 12MB myself, for my Power account), it's long overdue.

I realize there's more complexity to doing video files properly, multithreading a single file is a lot more complicated than just doing a whole image in each thread. But I don't get why it hasn't been implemented for images in the standard uploader years ago.

BeachBill

Frode wrote:

Any third party way to do this? I've always gotten horrible upload speeds to Smugmug, but with the increase in image size and now also video, the inability to upload multiple files simultaneously is killing me.

Try SmugLoader. One of the advertised features is up to 7 concurrent uploads. Please report back your results. I'm sure the author would appreciate the feedback.

Frode

BeachBill wrote:

Try SmugLoader. One of the advertised features is up to 7 concurrent uploads. Please report back your results. I'm sure the author would appreciate the feedback.

Thanks, but KomodoDrop works great for me and supports 16 concurrent uploads. So I'm a happy camper now.

SamirD

Frode wrote:

You just made my Christmas card list. KomodoDrop works great. I pumped up the threads and it's flying!

Glad I was able to help! Santa, I want a Leica DLux4.

BeachBill wrote:

Try SmugLoader. One of the advertised features is up to 7 concurrent uploads. Please report back your results. I'm sure the author would appreciate the feedback.

Interesting. I'll have to check this out. Thank you for the link!

Nikolai

Frode wrote:

Think I may have to do that. I decided to download and test it, and it does use two upload threads, but that's it. Granted, that's double the speed, but still leaves me at a tenth of desired. Question is whether that's a Smugmug limitation or an application one.

I'll look around it a bit more the next day or two, then make a post once I feel confident I'm not overlooking an obvious option somewhere. Thanks again.

FWIW, S*E uses up to 7 threads. If you add only two files to the queue, it will use two, since there is no need for more. If you add more, it will automatically engage more. You can configure the range of threads from 1 to 7. From my experience, 7 threads saturates even the fastest connections (if you look at the network performance chart there is no pauses). I can increase amount of threads, but thus far I didn't really see a need for it.

Frode

Nikolai wrote:

FWIW, S*E uses up to 7 threads. If you add only two files to the queue, it will use two, since there is no need for more.

Strange, I could've sworn I added 10 files to the queue. I used tcpview to verify it only established two connections. If the trial doesn't run out by the time I remember to do another test, I'll do that and post back the results.

I need to use pretty much all 16 threads to saturate my bandwidth with KomodoDrop by the way, so I would assume the same to be true for S*E. If I had one of the faster connections even that would not be enough.

Edit: I should point out I did verify the amount of threads was set to 7 at the time. I concluded it meant something other than upload threads since I could only see two being used.

Nikolai

Frode wrote:

Strange, I could've sworn I added 10 files to the queue. I used tcpview to verify it only established two connections. If the trial doesn't run out by the time I remember to do another test, I'll do that and post back the results.

I need to use pretty much all 16 threads to saturate my bandwidth with KomodoDrop by the way, so I would assume the same to be true for S*E. If I had one of the faster connections even that would not be enough.

Edit: I should point out I did verify the amount of threads was set to 7 at the time. I concluded it meant something other than upload threads since I could only see two being used.

Would you mind posting a screenshot of the taskmanager network performance tab showing S*E uploading, say, 24 mid-sized files (1..5mb each) so I can see the pauses between the threads?
And also similar screen shot from KomodoDrop ( for the same files, naturally)?

You can post it here ot send it to me via email...
TIA!

SamirD

Nikolai wrote:

FWIW, S*E uses up to 7 threads. If you add only two files to the queue, it will use two, since there is no need for more. If you add more, it will automatically engage more. You can configure the range of threads from 1 to 7. From my experience, 7 threads saturates even the fastest connections (if you look at the network performance chart there is no pauses). I can increase amount of threads, but thus far I didn't really see a need for it.

Great info Nikolai! I need to download a new version of SE and also try it. I think the 7 threads won't be enough for my setup, but like you said, changing it doesn't take much.

Nikolai

Well, I just uploaded experimental "Terminator" edition.
Up to 100 threads, and you can configure how many of them are simultaneous. You'd have to play with your specific conditions, since the best results depends on many things (SM being busy or not, file size = response time, bandwidth). The idea is to kinda saturate your bandwidth with the minimal amount of simultaneous threads (if you add more than that, they will become crowded and overal performance will drop), yet have enough total threads so there is no wait period when the active ones are done with the upload and are simply waiting for results.

My own test results (for my new 35/35 FiOS): 19 files, 102Mb total.

7 total, 1 uploading (essentially, the current "Seven Dwarves" edition): 1m:24s, 15%..17% bandwidth saturation

8 total, 4 uploading (several tests): 0:50s..0:59s, 35%..40% saturation

30 total, 20 uploading (since there were only 19 files it means all threads were uploading at the same time):1:01s, ~40% saturation

25x4, 12x4 - about the same as 8x4.

So, generaly I could get 30% speed increase. Naturally, effect will be different on larger batch volumes and different file sizes.

At any rate - you are now in control. The magic settings are in the Upload tab.

EDIT: don't forget to tweak your registry: http://support.microsoft.com/kb/282402

SamirD

Nikolai wrote:

Well, I just uploaded experimental "Terminator" edition.

I should be working on processing images, but this is just too tempting. Downloading!

SamirD

The battle for the ultimate uploader has begun...

VERY NICE upgrade Nikolai! This is a very fast version of SE. My 2000+ albums load in a very acceptable time, the multiple threads (right now only set to 12) seem to max out pretty well, and it seems to address every issue that kept SE from being my uploader of choice. I'm testing a 380MB upload and it should be done by the time I post this. I like how adding images to an album takes less time than with SM's default uploader. This used to kill quite a bit of time. And now I can set all my album uploads at one time. Kudos!

Nikolai

I edited the original post, but just in case: all these extra threads will just eat CPU cycles without doing you any good UNLESS you tweak your registry settings.
By default IE is set to have only 2 connections to a server (IE8 increased this amount to 6). This means, that "out of the box" having more than 2 (6) simultaneous threads wouldn't do you any good, since they would be blocked at the OS level (S*E uses IE internals, called wininet)
Here is Microsoft article explaining how to overcome this obstacle
http://support.microsoft.com/kb/282402
There are plenty similar ones if you don't like this particular explanation.

So naturally, the amount of simultaneous threads should match this setting. If you want 20 upload threads going in parallel - set the registry to 21..22 (just in case you want to browse SM at the same time:-).

The total amount of threads should be chosen from the fact that once "uploading" threads reach the waiting point and thus stop being "uploading", there should be enough threads in the "pool" to start uploading and keep the bandwidth occupied while those in wait state get through (the latter depends mostly on file sizes and how busy SM servers currently are).

All in all, it does require some playing around to find the optimal configuration for your particular case. Obviously, 0.7mbps dsl and 3x20mbps cable modems would have very different optimal configurations. But it's all up to you.

I would say, in any case the amount of simultaneous threads should not exceed 30%, mayby 50% of total. Use Task Monitor Networking tab to observe the pattern and decide what is the best for you.

Remember: S*E automatically limits the total number of threads by the actual queue size. So even if you have thread counts set to 100:42, but uploading only 5 files - only 5 threads will be engaged (5< 100) and they all will go in parallel (5 < 42). So when testing, test wisely!

I hope this clears the situation a bit...

Frode

Nikolai wrote:

Would you mind posting a screenshot of the taskmanager network performance tab showing S*E uploading, say, 24 mid-sized files (1..5mb each) so I can see the pauses between the threads?
And also similar screen shot from KomodoDrop ( for the same files, naturally)?

Based on the subsequent posts, I take it this is no longer necessary? I'll be happy to do it if you still need it, but it doesn't look like I'll be able to find time until after Easter.

Nikolai

Frode wrote:

Based on the subsequent posts, I take it this is no longer necessary? I'll be happy to do it if you still need it, but it doesn't look like I'll be able to find time until after Easter.

Not necessary:-) I'd rather you try the new S*E version..:-)

Nikolai

Just for the clarification: 35% saturation in my 100mbps LAN means exactly 35mbps upload speed I have from my ISP (Verison Fios). So, in my case 4 uploading threads are typically reaching that point. So I think I'll leave my settings at 25:5 (25 total, 5 simultaneous), it should not overkill system sesources yet provide enough leeway for the bigger batches/larger files (longer response times).
HTH