The only time I'm mad at smugmug

jfriend

I'm generally a pretty big smugmug fan. But, there is ONE issue that makes me mad/frustrated at smugmug almost once a day.

That is the time that it takes for a new upload to finish processing. I am quite active in a number of online forums (dgrin, dpreview, nikoncafe, etc...) and the process of responding to a posting there often involves uploading an image to smugmug and then using the smugmug URL in a posting that I'm in the middle of composing. The problem with the current delay (today it was almost 5 minutes) is that I am just sitting and waiting for the image show up, often unable to finish my computer session and leave to go do the next thing I'm headed out to do. It's sometimes just purely wasted time. If I have other computer things to do, I can sometimes go do them, but even that is a pain to context switch and not be able to just finish the posting I was in the middle of.

Are we going to just have to live with uncertain upload conversion times (sometimes as long as 5 mins) or is there a plan to shorten this time and make it more predicatable?

Andy

jfriend wrote:

Are we going to just have to live with uncertain upload conversion times (sometimes as long as 5 mins) or is there a plan to shorten this time and make it more predicatable?

Hi John,

Thanks for telling us straight - and we'll certainly take this into consideration as we are always looking for at ways to improve performance across all areas of the site.

All the best,

JamesJWeg

You know, this brings up an interesting idea, could smugmug in the interest of an overall processing ballance tell us when the peak and off peak times are? It has crossed my mind that event photogs loading 1000+ shots at once could do other SM customers a favor and load them at off peak times. Personally when loading a ton of shots I wouldn't mind hitting go at an off peak time. I know they have big boxes but I am sure I am not the only one who has hit go and a couple thousand shots at once.

james.

JamesJWeg

FWIW, I have noticed an increase in processing times latly. Popularity is a @#!$ no?

James.

rainforest1155

JamesJWeg wrote:

You know, this brings up an interesting idea, could smugmug in the interest of an overall processing ballance tell us when the peak and off peak times are?

Great idea! What about a current server-load page and additionally statistics over the week/day indicating free resources. I could also adjust my general upload behaviour to not so busy times.
I'm not uploading much, but if more people who are able to shift to other times apply this we could help balancing the load without additional hardware.

Sebastian

gus

JamesJWeg wrote:

could smugmug in the interest of an overall processing ballance tell us when the peak and off peak times are?
james.

Noooooooooooooooooooooooooooooooooooooooooooo !!!!

0zzies are a tiny minority compared to 'mericans & when your rugged up against your cold cold nights at 3am dreaming of what would happen if a moose drank juice for a goose or if a goose drank juice for a moose...we have the BBQ going with some fine cheese/wine out & full speed on SmugMug.

JamesJWeg

Humungus wrote:

Noooooooooooooooooooooooooooooooooooooooooooo !!!!

0zzies are a tiny minority compared to 'mericans & when your rugged up against your cold cold nights at 3am dreaming of what would happen if a moose drank juice for a goose or if a goose drank juice for a moose...we have the BBQ going with some fine cheese/wine out & full speed on SmugMug.

Snork! :lolPoor boy, look at it this way, if we the users can help SM out by adjusting upload times and files sizes we are protecting SM as we know it, maybe we can keep the prices down/unlimited space available. One person doing it will not be the magic but many might help. Personally I have started trimming my online files, I need to anyway and by doing so we can help save our unlimited upload ability.

James.

Baldy

jfriend wrote:

I'm generally a pretty big smugmug fan. But, there is ONE issue that makes me mad/frustrated at smugmug almost once a day.

That's not the only thing that makes me mad at SmugMug, but it's one I hate too.

We have a new Director of Operations (translation: IT weenie) starting next week (Andrew) who we've known for years, and one of the first things on his list is to expand capacity.

Strange as it may seem, we have idle machines in our data centers not plugged in — like a fabulous and incredibly expensive database slave because we simply can't get enough power and cooling in the datacenters.

We get about 8,000 images an hour now but what's tough about the load is how it comes in fits and starts. When we get hit with a load of images from someone like this:

http://wesley.smugmug.com

who has an incredibly fast connection coupled with mega megapixels and mega megabytes, it's like filling the datacenters with a firehose.

Thanks,
Chris

onethumb

Baldy wrote:

We get about 8,000 images an hour now

That's actually average, so peak hours are probably at least 2X that.

Sunday evenings (US timezones) are the *worst* time for us, and anytime after midnight (US timezones) should scream.

We carefully track how long photos are taking to process, and the average is just a few seconds per photo, but sometimes someone floods the queue and you feel the pain.

There are plenty of corporate customers who can upload 1000 - 5000 photos in a matter of seconds to smugmug, and those photos can "clog up" the queue. If you're lucky enough to upload after them, or in the middle of their upload, you might have to wait awhile.

I'm afraid this isn't necessarily a solveable problem. The easy answer is "add more hardware", which we do on a regularly, continous basis, but even that just can't scale at a rate where we can continue to make money - processing 5000 8Mpix photos in less than 10 seconds costs a lot of CPU dollars.

Again, we track and watch the processing times closely, and we are due for some more CPUs in that cluster soon, but it's really an arms race. The faster we get, the more people just use their corporate connections to dump photos on us.

Don

jfriend

An idea

onethumb wrote:

That's actually average, so peak hours are probably at least 2X that.

Sunday evenings (US timezones) are the *worst* time for us, and anytime after midnight (US timezones) should scream.

We carefully track how long photos are taking to process, and the average is just a few seconds per photo, but sometimes someone floods the queue and you feel the pain.

There are plenty of corporate customers who can upload 1000 - 5000 photos in a matter of seconds to smugmug, and those photos can "clog up" the queue. If you're lucky enough to upload after them, or in the middle of their upload, you might have to wait awhile.

I'm afraid this isn't necessarily a solveable problem. The easy answer is "add more hardware", which we do on a regularly, continous basis, but even that just can't scale at a rate where we can continue to make money - processing 5000 8Mpix photos in less than 10 seconds costs a lot of CPU dollars.

Again, we track and watch the processing times closely, and we are due for some more CPUs in that cluster soon, but it's really an arms race. The faster we get, the more people just use their corporate connections to dump photos on us.

Don

Your description of how some large uploads clog up the queue makes me wonder if some queing priority could help with the customer expectation. If I as a customer, upload a single photo or a small group of photos (1-5), I'm pretty upset if it takes even multiple minutes to process those. I am usually wanting to do something right away when I upload only a couple photos. If I, as a customer, upload 50, 200 or 500 photos, I'm rarely needing or expecting instantaneous results. As long as I'm confident that they are being processed and that I can see progress, I'm OK if it takes a little while - heck the upload took awhile in the first place.

So, that makes me wonder if you could change your algorithm from what I'm guessing is FIFO (first in, first out) to something that gives some priority to small uploads so that they don't have to get stuck behind large uploads and get no processing until the large uploads get done. I suspect that's not a trivial thing to code, particularly since even in large uploads the images still come in one at a time, it's just that they keep repeating that process one after the other, but I'm sure a sufficient algorithm could be derived if you thought that would make a difference in the perception. I think it would.

Of course, if you can add hardware to keep processing times short with the current algorithm, even at peak, that's certainly fine with me. But, it might be worth thinking about an algorithm that matches the average customer need and perception more than FIFO.

JamesJWeg

Baldy wrote:

That's not the only thing that makes me mad at SmugMug, but it's one I hate too.

We have a new Director of Operations (translation: IT weenie) starting next week (Andrew) who we've known for years, and one of the first things on his list is to expand capacity.

Strange as it may seem, we have idle machines in our data centers not plugged in — like a fabulous and incredibly expensive database slave because we simply can't get enough power and cooling in the datacenters.

We get about 8,000 images an hour now but what's tough about the load is how it comes in fits and starts. When we get hit with a load of images from someone like this:

http://wesley.smugmug.com

who has an incredibly fast connection coupled with mega megapixels and mega megabytes, it's like filling the datacenters with a firehose.

Thanks,
Chris

Glad you arn't nameing names. Sounds like you understand the problem in my office, it was 100.8F in my office this am when I walked in, the AC had froozen up. Nothing like having to shut systems down to control heat. Seriously, some of us would be glad to move our large upload times, I get back from karting events which I plan to do more this next year, on sunday night, with a couple thousand frames. I would gladly use a tool that I set and it waited until your servers told it to send, managing upload realtime. James.

Andy

JamesJWeg wrote:

I would gladly use a tool that I set and it waited until your servers told it to send, managing upload realtime. James.

If Nikolai hasn't built this already into Star*Explorer, he should -- seems like a great feature, esp for event shooters, high-volume shooters.

jfriend

How does this help the event shooter?

Andy wrote:

If Nikolai hasn't built this already into Star*Explorer, he should -- seems like a great feature, esp for event shooters, high-volume shooters.

I think I'm missing something. How does this help the event shooter? I see why smugmug might like everyone to wait until things aren't busy, but I'm missing why the event shooter benefits from waiting on his/her upload?

JamesJWeg

jfriend wrote:

I think I'm missing something. How does this help the event shooter? I see why smugmug might like everyone to wait until things aren't busy, but I'm missing why the event shooter benefits from waiting on his/her upload?

a couple ways, he protects his low prices and features with SM, and he can set and forget an upload and not tie up his line until he is off to bed etc. Also SM *could* offer us a better reason James.

Andy

jfriend wrote:

I think I'm missing something. How does this help the event shooter? I see why smugmug might like everyone to wait until things aren't busy, but I'm missing why the event shooter benefits from waiting on his/her upload?

James nailed it. Maybe a guy has 1000 shots to upload....he offline queues everything to the new galleries, and sets the upload for 2 or 3am, or whatever, really - kinda like when I set the dishwasher to run in the middle of the night. Of course, there are some shooters that want those 1000 shots up now, and then send out the emails etc to tell the viewers & buyers that the shots are there. Just options, that's all

jfriend

An interesting thought experiement...

JamesJWeg wrote:

a couple ways, he protects his low prices and features with SM, and he can set and forget an upload and not tie up his line until he is off to bed etc. Also SM *could* offer us a better reason James.

I get the timed upload feature where you configure an upload and tell Star*Explorer to start the upload at a time when you know you won't be using your internet line. I do that manually, by setting everything up and just waiting to hit Ctrl-U until right before I go to sleep. But this is a different feature than telling Star*Explorer to start the upload when smugmug's servers say to which is what I read in the original suggestion.

But, the notion that any significant number of event shooters are going to altruistically inconvenience themselves in order to protect SM low prices with no direct or personal incentive or direct cause/effect link is a lot harder for me to believe. I'm more a believer in market forces influencing behaviors. Provide specific selfish incentives or disincentives in order to influence behavior.

One such market force that could influence people's behavior would be to say that large uploads at peak time will be prioritized lower than at other times. So, if you want your upload to go quickly and appear in your galleries quickly, pick a lightly loaded time to do it. If you don't care how quickly it appears, then don't worry about when you upload it, but smugmug will have the ability to de-prioritize it in order to maintain responsiveness for smaller jobs if the system is busy.

cmason

Perhaps this is well understood, so forgive me if it is...but for most of us on non-commercial broadband connections (Cox, Roadrunner, DSL), there is limited bandwidth for uploads: uploads are much slower than downloads.

To see this for yourself, click here and choose "Click to start MySpeedTest".

This of course would not explain variability in upload times of course, though QoS and peak times if you are on cable can have a significant impact (lots of users sharing the common "neighborhood pipe".

rainforest1155

Waiting until server says that it's ready could be a feature of S*E, but it'll require that the API supports this kind of command.
DON!?

Another feature still in works as far as I know is to upload just preview pictures instead of full-blown ones. When a customer orders a print - the user simply will upload the requested picture in full size. This is especially useful for event shooters and could decrease the server peaks a lot. Let's see when this comes out.

Sebastian

EDIT: @James: I created a new thread with your idea in the API forum for Don to see.

JamesJWeg

cmason wrote:

Perhaps this is well understood, so forgive me if it is...but for most of us on non-commercial broadband connections (Cox, Roadrunner, DSL), there is limited bandwidth for uploads: uploads are much slower than downloads.

To see this for yourself, click here and choose "Click to start MySpeedTest".

This of course would not explain variability in upload times of course, though QoS and peak times if you are on cable can have a significant impact (lots of users sharing the common "neighborhood pipe".

People like me are the other side of the coin, my up line is 3Mbps. James.

Baldy

Hey, a leetle discovery... Has anyone noticed that the replace photo funcitionality seems to trigger instant processing?

I suppose if you had a quick need to post something you could just replace one of your current photos.

jfriend

Neat trick

Baldy wrote:

Hey, a leetle discovery... Has anyone noticed that the replace photo funcitionality seems to trigger instant processing?

I suppose if you had a quick need to post something you could just replace one of your current photos.

Neat trick. I'll remember that next time I'm in a hurry for a single photo. I might even put some small dummy placeholder images in the gallery I use for direct linking to forums.

rainforest1155

Baldy wrote:

Hey, a leetle discovery... Has anyone noticed that the replace photo funcitionality seems to trigger instant processing?

I suppose if you had a quick need to post something you could just replace one of your current photos.

Coool! I'll try this the next time I'm in a hurry.

Thanks,
Sebastian

binghott

Baldy wrote:

Hey, a leetle discovery... Has anyone noticed that the replace photo funcitionality seems to trigger instant processing?

I suppose if you had a quick need to post something you could just replace one of your current photos.

hahaha, i have noticed that, but i always suspected it was just a coincidence. good to know!

jfriend

"processing image" speedup?

Wireless wrote:

This is a heads up to let everyone know that we're going to be doing work on this site tonight during our regularly scheduled Maintenance Window. Please expect SmugMug to be inaccessible for much if not all of the 5 hours we've allocated from 11:00pm to 4am PST.

We apologize for the inconvenience (hey, I always get some good work done on my photo galleries during this period ), however we're going to be doing some cool things behind the scenes to make SmugMug better and more reliable!

Here's to hoping that the "processing image" time after an image is uploaded is reduced significantly. As per my earlier posts, that is the one time that smugmug really annoys me when I have to just wait before I can finish a task that needs to refer to that image online.

Andy

jfriend wrote:

Here's to hoping that the "processing image" time after an image is uploaded is reduced significantly. As per my earlier posts, that is the one time that smugmug really annoys me when I have to just wait before I can finish a task that needs to refer to that image online.

Sundays/Mondays are busiest. How long is too long, John - I am curious as to your threshhold? Thanks in advance for the feedback!

Andy

jfriend wrote:

Here's to hoping that the "processing image" time after an image is uploaded is reduced significantly. As per my earlier posts, that is the one time that smugmug really annoys me when I have to just wait before I can finish a task that needs to refer to that image online.

And, John - do you know the "replace image" trick? It's instant... sorta like those people who walk around Disneyland with the special badges - VIPs and such - they cut straight to the front of the line!

jfriend

Here's what I do know

Andy wrote:

Sundays/Mondays are busiest. How long is too long, John - I am curious as to your threshhold? Thanks in advance for the feedback!

Here's the previous discussion on the topic which included thoughts from OneThumb and Baldy.

I don't know the maximum time that would cease to make me mad and I don't think I could guess it very accurately without some experimentation with different times.

Here's what I do know about my own perceptions:

• Unpredicatable times that sometimes extend beyond 5 minutes make you really mad. When it's completely unpredicatable you can't even plan out your work because you have no idea when it will be done.
• I don't mind if it takes a little while after I've posted lots of pictures. I rarely need immediate access to an individual photo after uploading lots of pictures. And, for some reason, I don't expect snappy access right after uploading lots of images. It may also be that since I'm on an asymmetric DSL line, uploading lots of images already takes lots of time so I already have to plan around the whole process taking awhile and nothing smugmug does really affects it one way or the other.
• When I only post one or two images, I expect the availability to be snappy. The upload time isn't very long and I know that some competing alternatives would make these available nearly instantly. So smugmug is missing my expectation and is not matching my perceptoin of some competing alternatives.
• Without knowing exactly what time would work, I could attempt to bound it by saying that I'm pretty sure that availability 30 seconds after upload of a single image would pretty much always feel fast enough and having to wait more than 2 minutes will pretty much always feel unacceptably slow. As I said earlier, I don't need or expect availability that quickly when I upload multiple images.

Since 30 seconds is probably a tough target to shoot for in peak times, in an attempt to match my own expectations, I suggested an algorithm idea in the previous thread that might give small jobs a higher priority so single uploads wouldn't have to wait behind large, multi-image uploads when things were busy.

Andy

jfriend wrote:

Here's the previous discussion on the topic which included thoughts from OneThumb and Baldy.

I don't know the maximum time that would cease to make me mad and I don't think I could guess it very accurately without some experimentation with different times.

Here's what I do know about my own perceptions:

• Unpredicatable times that sometimes extend beyond 5 minutes make you really mad. When it's completely unpredicatable you can't even plan out your work because you have no idea when it will be done.
• I don't mind if it takes a little while after I've posted lots of pictures. I rarely need immediate access to an individual photo after uploading lots of pictures. And, for some reason, I don't expect snappy access right after uploading lots of images. It may also be that since I'm on an asymmetric DSL line, uploading lots of images already takes lots of time so I already have to plan around the whole process taking awhile and nothing smugmug does really affects it one way or the other.
• When I only post one or two images, I expect the availability to be snappy. The upload time isn't very long and I know that some competing alternatives would make these available nearly instantly. So smugmug is missing my expectation and is not matching my perceptoin of some competing alternatives.
• Without knowing exactly what time would work, I could attempt to bound it by saying that I'm pretty sure that availability 30 seconds after upload of a single image would pretty much always feel fast enough and having to wait more than 2 minutes will pretty much always feel unacceptably slow. As I said earlier, I don't need or expect availability that quickly when I upload multiple images.

Since 30 seconds is probably a tough target to shoot for in peak times, in an attempt to match my own expectations, I suggested an algorithm idea in the previous thread that might give small jobs a higher priority so single uploads wouldn't have to wait behind large, multi-image uploads when things were busy.

Thanks for the feedback, John!

jfriend

Occasionally use replace image

Andy wrote:

And, John - do you know the "replace image" trick? It's instant... sorta like those people who walk around Disneyland with the special badges - VIPs and such - they cut straight to the front of the line!

Yeah, I read that in the previous thread. I've even used it occasionally.

But it hasn't seemed very efficient to me in terms of my workflow for uploading. First, it requires advanced planning to have a bunch of images you can replace. Second, I can't use my normal uplaoder (Star*Explorer populated by drag/drop from Bridge), so it's a lot less efficient to go through Replace Image. With two mouse clicks and two keystrokes, I can have an image uploading in Star*Explorer from Bridge. Third, I don't like having placeholder junk images sitting in my galleries.

Thanks for the suggestion though. I do use it occasionally, but not enough to make a difference. In my earlier posting, I was hoping that single image uploads could also cut in front of the line. Continuing with your Disneyland analogy, it would be like single image uploads getting a "Fast Pass" for upload.

Andy

jfriend wrote:

Yeah, I read that in the previous thread. I've even used it occasionally.

But it hasn't seemed very efficient to me in terms of my workflow for uploading. First, it requires advanced planning to have a bunch of images you can replace. Second, I can't use my normal uplaoder (Star*Explorer populated by drag/drop from Bridge), so it's a lot less efficient to go through Replace Image. With two mouse clicks and two keystrokes, I can have an image uploading in Star*Explorer from Bridge. Third, I don't like having placeholder junk images sitting in my galleries.

Thanks for the suggestion though. I do use it occasionally, but not enough to make a difference. In my earlier posting, I was hoping that single image uploads could also cut in front of the line. Continuing with your Disneyland analogy, it would be like single image uploads getting a "Fast Pass" for upload.

I just uploaded an image for Rutt's post here and by the time I typed my post, the image was ready (like 1.5 minutes) and today's a Monday. I upload all day long, every single day - 7 days a week and believe me, Wireless would be hearing from me if this we a huge deal for me. But both Don and Wireless have said, that image processing is something that they always look hard at, and try to improve upon. Personally, I think to expect that images be reading in a minute or so all the time is unreasonable. I think that if they are ready in "a few minutes" on average, that's cool with me. Remember, we're checking each image for viability, assigning an image number, we're making duplicate copy and validating, we're making the -L, -S, -M, -Th, and -Ti versions, we're making the exif link, man, there's a lot going on

Baldy

We installed a whole lot of new image processing hardware lately. Are you still seeing long processing times? We monitored it pretty closely last night and it seemed we hardly saw any backlog.