Wanna test our new uploading technology?

onethumb

We're just about to start rolling out a revamped uploading architecture, and are looking for a few brave souls.

This change should provide:

• Faster transfer rates, especially for those on the East Coast and Europe
• *Much* faster 'Verify times', providing for much faster overall upload performance, especially for large photos, videos, and SmugVault items
• More stability & fewer errors
• Much more scalable especially for those busy Sunday night uploading marathons you're all so fond of

If you'd like to test it, you have to first cookie your browser by going here while logged in: http://www.smugmug.com/hack/enable.mg?feature=uploadEndpoint

Got a custom domain? Use this:

http://www.example.com/hack/enable.mg?feature=uploadEndpoint and replace "example.com" with your custom domain.

Then use either the 'HTML5' or 'Simple' uploaders.

Done testing? Simply revisit that link above, and your cookies will be reset.

Holler on this thread with what you see. If you find a bug, we'll jump right on it - but please try it with the "old" uploading method, too, so we know whether this is an old bug or a new one.

Thanks!

devbobo

bump

Andy

Hey folks - try this - it's pretty amazing from where I sit - I'm in NY and seeing a huge (4x to 6x) speed increase in my upload speeds. We're standing by if you have any questions!

Quique_Schamann

Guys,
my feedback uploading 164 files (690Mb) using HTML5. Today will test Java.

Even the interface changed (visually improved, the possibility to use drag & drop and now can upload 3 files at the same time), I didn't see major changes. I'm using a ADSL connection 8Mb down /1Mb up.

Let me show two things that can be improved:
1- Possibility of upload (drag & drop) folders instead of just files;
2- Re-upload option for files that, for some reason, failed (my case, 15 warnings/upload fails over 164 files)


Cheers,
Quique

emtp563

I won't be uploading anything until the weekend. One question: Is this faster than Star Explorer?

jfriend

For one who is upload bandwidth limited (as I assume most people uploading from home are), is this really faster at pushing the bits up than what you had before? How is it possible to be "orders of magnitude" faster?

onethumb

jfriend wrote: »

For one who is upload bandwidth limited (as I assume most people uploading from home are), is this really faster at pushing the bits up than what you had before? How is it possible to be "orders of magnitude" faster?

Many of our customers are uploading from work or have high-end home connectivity (FIOS, etc). I think Andy was exagerrating a little - he didn't see 10-100X improvement that "orders of magnitude" would imply, but he is seeing above 4X improvement.

During this testing period, we're seeing people on home connections hitting 20Mbps on a single file uploads, which is pretty impressive considering our uploaders are multi-threaded, so that could compound.

And to answer your question, yes, it's even faster for people who are bandwidth limited because the verify step is so short now, we can keep the bits flowing more continously. We'll do our best to saturate your upstream bandwidth.

jfriend

onethumb wrote: »

Many of our customers are uploading from work or have high-end home connectivity (FIOS, etc). I think Andy was exagerrating a little - he didn't see 10-100X improvement that "orders of magnitude" would imply, but he is seeing above 4X improvement.

During this testing period, we're seeing people on home connections hitting 20Mbps on a single file uploads, which is pretty impressive considering our uploaders are multi-threaded, so that could compound.

And to answer your question, yes, it's even faster for people who are bandwidth limited because the verify step is so short now, we can keep the bits flowing more continously. We'll do our best to saturate your upstream bandwidth.

OK, that's what I expected. Might be slightly better with lower overhead at the ends, but I only have 1Mbps up so I'm probably not going to see much of a difference in total throughput as I'm upbandwidth limited.

Andy

onethumb wrote: »

Many of our customers are uploading from work or have high-end home connectivity (FIOS, etc). I think Andy was exagerating a little - he didn't see 10-100X improvement that "orders of magnitude" would imply, but he is seeing above 4X improvement.

Yeah - I was so excited I didn't state it correctly I am getting 4x-6x increase in speed, depending on time of day. I'm on Cablevision's top service, called "Ultra" (which is Docsis). My max upload is 15Mb/sec and before the change, I was getting 2-3 Mb/sec and now I'm getting 8-12Mb/sec sustained on large uploads.

Quique_Schamann

jfriend wrote: »

For one who is upload bandwidth limited (as I assume most people uploading from home are), is this really faster at pushing the bits up than what you had before? How is it possible to be "orders of magnitude" faster?

John,
I'll test HTML5 vs Simple (Java) vs Star Explorer, by Monday, and put my comment here.
Btw, I have a ADSL with 8Mbps DOWN / 1Mbps UP

Cheers,
Quique

devbobo

jfriend wrote: »

OK, that's what I expected. Might be slightly better with lower overhead at the ends, but I only have 1Mbps up so I'm probably not going to see much of a difference in total throughput as I'm upbandwidth limited.

John,

Replacing images should be orders of magnitude quicker, they will upload the same speed as new uploads. As an example, a replace that took 45 secs using the old method took 13 secs using the cloud.

Cheers,

David

tchwojko

HTML 5 uploader just sits there saying all my files are ready to upload.
Java uploader running at about 5Mb/s, then 20-30 seconds to verify an upload of a 7MB picture.

Mac OS X 10.6.7
Safari Version 5.0.4 (6533.20.27)
Frontier FiOS

Andy

tchwojko wrote: »

HTML 5 uploader just sits there saying all my files are ready to upload.
Java uploader running at about 5Mb/s, then 20-30 seconds to verify an upload of a 7MB picture.

Mac OS X 10.6.7
Safari Version 5.0.4 (6533.20.27)
Frontier FiOS

Did you cookie your custom domain on the new upload endpoint? Must do that

http://photos.chwojkofrank.com/hack/enable.mg?feature=uploadEndpoint

Try again, after doing that, and upload a bunch of files.

Allen

Andy wrote: »

Did you cookie your custom domain on the new upload endpoint? Must do that

http://photos.chwojkofrank.com/hack/enable.mg?feature=uploadEndpoint

Try again, after doing that, and upload a bunch of files.

I'm just guessing on what you're saying. I logged in using nickname then clicked
the cookie link. Have no idea what endpoint means. Saw no difference on
upload page in browser link. Used the Simple uploader and it did seem much
faster though.

Andy

Allen wrote: »

I'm just guessing on what you're saying. I logged in using nickname then clicked
the cookie link. Have no idea what endpoint means. Saw no difference on
upload page in browser link. Used the Simple uploader and it did seem much
faster though.

Al, we give both examples in post #1. You'll need to cookie this

http://www.photosbyat.com/hack/enable.mg?feature=uploadEndpoint

Then use the uploaders.

Allen

Andy wrote: »

Al, we give both examples in post #1. You'll need to cookie this

http://www.photosbyat.com/hack/enable.mg?feature=uploadEndpoint

Then use the uploaders.

I did. Then went back to my already logged in site and uploaded. Just the terminology is confusing.
The term endpoint means nothing here to me.

Andy

Allen wrote: »

I did. Then went back to my already logged in site and uploaded. Just the terminology is confusing.
The term endpoint means nothing here to me.

Good. Fasterer?

Allen

Andy wrote: »

Good. Fasterer?

Seemed much fasterer, yes.:D

tchwojko

Andy wrote: »

Did you cookie your custom domain on the new upload endpoint? Must do that

http://photos.chwojkofrank.com/hack/enable.mg?feature=uploadEndpoint

Try again, after doing that, and upload a bunch of files.

I thought I did. Tried eating the cookie again and this time got "We've encountered a problem with the HTML5 uploader."

I'm going to try to undo the cookie, and start all over. I'll also try from Firefox and Chrome. (And I do know what an endpoint is, so if this is not user error, let me know what kind of diagnostic info you need.)

tchwojko

Tried it again on Safari. HTML 5 uploader just sits there saying it's ready (I can e-mail screenshot, cookie info, etc.).
Obvious smugmug cookies I see:
brandNewUploader html5
uploadEndpoint upload-cloud.smugmug.com
Sreferrer

onethumb

Thanks for testing - we're now doing live bucket tests across everyone on SmugMug, so this part of the test is over. Faster & more stable uploads coming to everyone shortly...

jfriend

devbobo wrote: »

John,

Replacing images should be orders of magnitude quicker, they will upload the same speed as new uploads. As an example, a replace that took 45 secs using the old method took 13 secs using the cloud.

Cheers,

David

But, the difference is the verify step right - not the actual upload? The uploads where speed is most important to me are uploading 900 images across 25 galleries. I generally use StarExplorer for those with 9 consecutive threads so bandwidth is being fully used by another upload during the verify step anyway. I don't know if Star Explorer is maximally efficient with the upload bandwidth, but it will let me specify 900 images across 25 galleries in one upload that I can walk away from rather than having to do 25 separate uploads. The onesy/twosy uploads where you have to sit and wait for the verify step will be nice to be a bit faster, but they aren't what really cost my workflow. I don't mean to sound negative here - I will try out the new uploader when I get home (I'm on the road now on college visits) - just trying to understand what I should look for and what to expect.

It sure would be nice if you guys labeled each uploader in the corner of the uploader window so we know what we're really using. Pretty much impossible for an end-user to know what they're using. Hard to run tests. Hard to report bugs accurately. Hard to know you aren't stuck on something you don't want because of some previous test, or problem, etc... The label could either be meaningful (like HTML5, Java, Flash, etc...) or it could be coded (001, 002, 003, etc...). Seems like it would be really useful for support too.

Andy

jfriend wrote: »

It sure would be nice if you guys labeled each uploader in the corner of the uploader window

Thanks for the feedback. For now,

HTML5- browse button on the bottom of the window
Simple (Java) - browse button on the top of the window
Flashy - browse button on the bottom, big curvy arrow saying "upload photos and videos"

Baldy

There weren't many posts in this thread, but we saw tons of upload traffic as a result of it, which is what we really wanted. Thanks! We took the silence among so many uploads to be a good sign.

It gave us the chance to monitor error rates and see that they are lower when uploading directly to Amazon, meaning fewer retries from the uploaders, meaning faster uploads.

When we cut over completely to direct uploads, all uploaders should benefit unless they do something really nonstandard, which we aren't aware of them doing.

RogersDA

HTML5 + 209 files went pretty quickly w/ no errors that I could see.

onethumb

jfriend wrote: »

But, the difference is the verify step right - not the actual upload? The uploads where speed is most important to me are uploading 900 images across 25 galleries. I generally use StarExplorer for those with 9 consecutive threads so bandwidth is being fully used by another upload during the verify step anyway. I don't know if Star Explorer is maximally efficient with the upload bandwidth, but it will let me specify 900 images across 25 galleries in one upload that I can walk away from rather than having to do 25 separate uploads. The onesy/twosy uploads where you have to sit and wait for the verify step will be nice to be a bit faster, but they aren't what really cost my workflow. I don't mean to sound negative here - I will try out the new uploader when I get home (I'm on the road now on college visits) - just trying to understand what I should look for and what to expect.

If you're already saturated, you probably won't notice much - we can't enlarge your pipe for you.

Error rates should be lower, verification times should be faster, replaces will be dramatically faster if that's part of your workflow, etc... But speed-wise the big benefits will come for people who have fast connections.

SamirD

Simple uploader crashed again when trying to get two 4gb batches uploaded simultaneously on my fastest system (hp dc5750 3gb ram). I still want FTP. So I can sleep earlier rather than deal with these uploaders. *goes off to reboot system and start re-uploading again*

devbobo

SamirD wrote: »

I still want FTP.

:deadhorse

cmonroe

Uploads pending with Simple Loader
Using FF 3.6.16
Windows XP
1. Visited URL to set cookie
2. Tools->Upload->New
3. Created testUpload dir in Other folder
4. Once presented with "drag and drop" screen, selected "try different...."
5. Selected Simple
6. Opened explorer window to "my pictures" folder
7. Selected all and dragged to drag and drop location
8. Files immediately started to upload (quickly by the look of it)
9. Once the upload completed the last file, the overall progress stayed at the following: "overall progress 25/26......."

That status never changed from there although the upload was complete.

Craig

Dna

Overall there didn't seem to be any difference between upload-cloud and upload, both were around the 8 - 10 Mb/s range. The verify was a lot quicker with upload-cloud.

I upload a bunch of files several times via both methods and also did a 200 Mb movie.
thumb

Andrew

jfriend

Here's what I notice (I'm using Chrome):

1) When I first open the upload window, there's no way to select whether I want to skip duplicates or not. So, there appears to be no way to change that until the upload already starts. Doesn't that kind of defeat the purpose of the setting?

2) There is no obvious way to stop the upload. I press the pause button and nothing stops. I press the close button and it offers to close the whole window. When you're on home DSL and the uploader is saturated the upload bandwidth, there are many times when some other internet need in the house calls for the ability to suspend or pause the upload for a little while, then resume later.

3) If you miss the drop area when dragging/dropping (something that's easy to do when managing multiple windows on a small screen), the browser window opens and shows one of your images and everything you already had dropped into the uploader is "poof" gone. This is terrible user interaction. Missing the target, but still in the same window should either accept into the target or do nothing. It shouldn't kill your entire upload.

4) When I actually try an upload, I get errors:





There is nothing wrong with these images as they've all been uploaded successfully before.