The Ultimate SM Uploader

SamirD

I think SM is close to a really good native uploader. But it's missing one major feature--bandwith control.

I have 15mb of upload bandwidth available to me via three 5mb cable modems. Since each file being uploaded only uses a fraction of the bandwidth, I have to run multiple upload sessions if I want to use a majority of my bandwidth.

With the new SM uploaders uploading three files at a time, some of this work is being done automatically. But there's some problems introduced for those that don't want all their bandwidth used as three is enough to saturate smaller lines like 1mb. And with only three files, it still doesn't use all the bandwidth on multi-wan connections like mine.

So what to do? Well, we need a bandwidth control slider from 0-100%. But how would it know what 0% and 100% is? Simple, it learns as it uploads.

When you initially start an upload, it starts one file, and then adds another, and another, each time monitoring the total amount of data that's being sent. Once it gets to a point that adding files no longer speeds up the upload, the bandwidth is saturated. And now it knows what 100% is. Based on this information, the uploader could adjust to whatever the slider is set to.

For example, it takes about 16 simultaneous uploads to max out my bandwidth. The uploader would start with one file, and then add another, and another , and so on, until it it doesn't see an increase in overall bandwidth use. In my case, after about 16 files. This is the max bandwidth utilization, and it knows how many files it has to have uploading simultaneously to reach max bandwidth utilization.

Since 16 files corresponds with 100%, if I set the slider to 50%, it will know to only upload 8 files simultaneously. If I set it to 5%, it will know to only upload one file at a time.

Now, this doesn't address the problem of trying to upload multiple files to multiple albums. Well from what I've learned, it's easier to simply upload everything to one album and then move things to where you want them. By their nature, the uploaders will probably still miss files, upload dupes, etc, and it's easier to just upload everything into one place first and get the missing/dupes taken care of in one gallery rather than in 10 different ones.

I'd be interested in hearing some drawbacks or problems with this solution.

rainforest1155

Did you try StarExplorer? It has bandwidth control.
There's also a setting allowing you to set how many upload threads you wish to have going in StarExplorer. I just tried it and with multiple simultaneous threads and it uploaded at 5mbit (on my 10mbit connection here while a non-SmugMug upload was going on as well). If it's not using enough bandwidth try increasing the "at a time" setting in the upload section of the settings.

SamirD

I own a license to SE, but it crashes randomly on every hardware/software configuration I throw at it. Nikolai and I have spent a lot of time trying to diagnose it without luck. :cry

I used SmugLoader for a while, but it konks out after about 30 minutes. I'm now onto Smugftp, which is laser fast, but isn't a SM native application, so it too has drawbacks.

Instead of having to tweak the number of upload threads, which most people won't know how to do, it has to be smart. And the algorithm for bandwidth control isn't difficult to implement. We all use the same system manually, so automating it is the next logical evolution. It would be nice to see it in a SM native uploader.

jfriend

SamirD wrote: »

Now, this doesn't address the problem of trying to upload multiple files to multiple albums. Well from what I've learned, it's easier to simply upload everything to one album and then move things to where you want them. By their nature, the uploaders will probably still miss files, upload dupes, etc, and it's easier to just upload everything into one place first and get the missing/dupes taken care of in one gallery rather than in 10 different ones.

I'd be interested in hearing some drawbacks or problems with this solution.

It is definitely not easier for me to upload everything to one gallery and then sort things out on SM. I have my images keyworded and a few clicks in Adobe Bridge on a directory of generated JPEGs, gets me the images I want in a particular gallery. I drag/drop that set of images into an uploader. That is pretty quick on my computer. That is not quick on SM. Much better for me to sort images by gallery before upload rather than after upload. I like the bandwidth utilization algorithm in concept though. For me, the point would be to get the max upload bandwidth possible while maintaining usability of my internet connection for other things (the BackBlaze upload client has a similar sounding slider in it and it seems to work well).

SamirD

jfriend wrote: »

It is definitely not easier for me to upload everything to one gallery and then sort things out on SM. I have my images keyworded and a few clicks in Adobe Bridge on a directory of generated JPEGs, gets me the images I want in a particular gallery. I drag/drop that set of images into an uploader. That is pretty quick on my computer. That is not quick on SM. Much better for me to sort images by gallery before upload rather than after upload. I like the bandwidth utilization algorithm in concept though. For me, the point would be to get the max upload bandwidth possible while maintaining usability of my internet connection for other things (the BackBlaze upload client has a similar sounding slider in it and it seems to work well).

And I believe that most people are faced with your scenario rather than mine where sorting is more easily done outside of SM. And that's a serious challenge as it requires either an ability to have a 'smart queue' very similar to an ftp client or the ability to have multiple instances of uploaders without issues.

I specifically thought about the slider control from a post you made about BackBlaze's, and your need for the feature. And I think more people are in your situation with limited upload bandwidth and an easy way to manually allocate it for everyone on the network.

It's pretty amazing how close SM's native uploader is to addressing the bandwidth control/maximization feature. But I have no idea how to implement multi-gallery uploading or the UI for it. What's your UI thoughts for a multi-gallery uploader John? I know you know how to create great UIs.

jfriend

SamirD wrote: »

And I believe that most people are faced with your scenario rather than mine where sorting is more easily done outside of SM. And that's a serious challenge as it requires either an ability to have a 'smart queue' very similar to an ftp client or the ability to have multiple instances of uploaders without issues.

I specifically thought about the slider control from a post you made about BackBlaze's, and your need for the feature. And I think more people are in your situation with limited upload bandwidth and an easy way to manually allocate it for everyone on the network.

It's pretty amazing how close SM's native uploader is to addressing the bandwidth control/maximization feature. But I have no idea how to implement multi-gallery uploading or the UI for it. What's your UI thoughts for a multi-gallery uploader John? I know you know how to create great UIs.

The UI seems like no big deal. Let me specify the target gallery, drop some files in, specify a different gallery, drop some files in and so on. Then, just display a column in the uploader list of files to process that shows the gallery name. That's how SE does it and it works fine.

The issue, I think, is that Smugmug doesn't seem interested in supporting multi-gallery uploads. They've argued against me in previous threads essentially saying that it's not something most of their users need or ask about and one can use a 3rd party uploader for that.

I shoot a lot of sports and find it very useful to break the photos up into smaller units by gallery for more effective viewing (often by player, by game, by date, by activity, by race, by grade level, etc...). I use virtual galleries for some of this now that they've fixed many of the issues with virtual galleries, but I don't use virtual galleries for everything because they still have some limitations so I nearly always am still uploading to multiple galleries. My goal is to configure an upload, start it in the evening and let it run unattended overnight.

SamirD

jfriend wrote: »

The UI seems like no big deal. Let me specify the target gallery, drop some files in, specify a different gallery, drop some files in and so on. That's how SE does it. Then, just display a column in the uploader list of files to process that shows the gallery name.

Sounds pretty simple. So maybe a two-paned design? Where the top portion would be where you could drop in the files and specify the target gallery and the bottom shows the progress and the pause/resume and bandwidth slider control?

Since the current way to specify a gallery it to go to the gallery and upload from there, how would you design the target gallery selector? Keep in mind that for someone like me, it has to be able to deal with well over 2000 galleries efficiently. I can't use any of the gallery selectors on SM because the javascript kills the cpu on everything but the Athlon x2.

jfriend

SamirD wrote: »

Sounds pretty simple. So maybe a two-paned design? Where the top portion would be where you could drop in the files and specify the target gallery and the bottom shows the progress and the pause/resume and bandwidth slider control?

Since the current way to specify a gallery it to go to the gallery and upload from there, how would you design the target gallery selector? Keep in mind that for someone like me, it has to be able to deal with well over 2000 galleries efficiently. I can't use any of the gallery selectors on SM because the javascript kills the cpu on everything but the Athlon x2.

Why are you trying to design the UI here?

SamirD

Oh, I dunno, so maybe SM developers might actually use our ideas from the beginning rather than after they've made a solution that doesn't work for us.

jfriend

SamirD wrote: »

Oh, I dunno, so maybe SM developers might actually use our ideas from the beginning rather than after they've made a solution that doesn't work for us.

If I'd seen any receptivity to supporting multi-gallery uploads, I'd be all over suggestions for how to do the UI, but right now it feels like a brick wall so I'd rather put my energy elsewhere.

SamirD

I don't think it's a brick wall since they implemented your list of changes very, very quickly. It's the only thing that prompted me to make this 'wish list' thread. But I understand your reservation. It gets exhausting trying to effect change.

jfriend

SamirD wrote: »

I don't think it's a brick wall since they implemented your list of changes very, very quickly. It's the only thing that prompted me to make this 'wish list' thread. But I understand your reservation. It gets exhausting trying to effect change.

My list of changes to make a robust uploader started several years ago. Yes, they did implement them quickly once they decided to go, but it took years for them to feel it was important enough to actually do.

SamirD

jfriend wrote: »

My list of changes to make a robust uploader started several years ago. Yes, they did implement them quickly once they decided to go, but it took years for them to feel it was important enough to actually do.

Oh, I remember the list when it first came out. I still can't believe the fullscreen uploader vs windowed one hasn't been implemented. I mean, I don't even use the drag and drop as much and almost messed up yesterday.

Personally, I think your changes just coincided with work that was already done. I can count on half a hand all the times I've seen changes implemented that quickly over the years, sad to say.

Andy

SamirD wrote: »

I can count on half a hand all the times I've seen changes implemented that quickly over the years, sad to say.

http://news.smugmug.com has nearly 10 years of release notes. Hundreds and hundreds of features we implement bases on customer requests. We don't ignore our customers. Some happen fast, some happen slow. But we always try and do the right thing.

We're bootstrapped, independent, answer to nobody but our customers and us. And we're into our tenth year of running this thing we love called SmugMug.

SamirD

Andy wrote: »

http://news.smugmug.com has nearly 10 years of release notes. Hundreds and hundreds of features we implement bases on customer requests. We don't ignore our customers. Some happen fast, some happen slow. But we always try and do the right thing.

We're bootstrapped, independent, answer to nobody but our customers and us. And we're into our tenth year of running this thing we love called SmugMug.

I didn't mean to imply that you guys didn't listen--just that immediate responses to feature requests was extremely rare.