Weekly upload speeds

DJKennedy · October 7, 2006

OK. This getting really freaking rediculas Smugmug.

Yes, this IS a weekly issue. I have never, ever had this problem with any other host. Ever.

I'm trying to upload 15 images, total size 6.8megs.It took ONE image 22 minutes to upload - so far it hasn't processed.

WHY DO WE YOUR CUSTOMERS HAVE TO PUT UP WITH THIS????

and EVERY weekend????

I'm trying to post photos of my sons 4th birthday. He will be 6 before the images are ready.

This is a KNOWN issue. Seriously - what is SM doing about this?

I'm paying HOW much for this ??

C'mon - enough is enough.

cjyphoto · October 7, 2006

Oh great. I've got 116 photos to upload and it took 3 minutes to finally get to the "create new gallery page"

DJKennedy · October 7, 2006

cjyphoto wrote:

Oh great. I've got 116 photos to upload and it took 3 minutes to finally get to the "create new gallery page"

Well, because I thought the uploader was screwy, I cancelled the upload twice after what appeared to be only a couple images being uploaded - now it appears I'm uploading all images 3 times. The last one took 24 1/2 mins. Only about 35 to go thanks to cancelling the first 2 uploads.

Andy · October 7, 2006

DJKennedy wrote:

Well, because I thought the uploader was screwy, I cancelled the upload twice after what appeared to be only a couple images being uploaded - now it appears I'm uploading all images 3 times. The last one took 24 1/2 mins. Only about 35 to go thanks to cancelling the first 2 uploads.

The queue us a tad long right now, I've called our CEO and chief geek (pretty sure I interrupted movie date night with the missus)... so we're aware of the situation and on it. Please standby, your images will process, though a bit longer.

Sorry for the trouble!

DJKennedy · October 7, 2006

Andy wrote:

The queue us a tad long right now, I've called our CEO and chief geek (pretty sure I interrupted movie date night with the missus)... so we're aware of the situation and on it. Please standby, your images will process, though a bit longer.

Sorry for the trouble!

33.3 mins for the last one to upload. Thats 33 mins too long.

Andy · October 7, 2006

DJKennedy wrote:

Seriously - what is SM doing about this?

I'm not sure if you've followed the other threads, but we've spent a fortune on new processing servers, new data center, new super fast comms lines, and new load balancers. This has all occurred in the past weeks. There was a problem (vendor bug) with the load balancers, we think that was licked but there may still be side effects.

We know you and our customers deserve better, and shouldn't have to have the "weekend blues" regarding uploading. The issue is being dealt with, and I'm confident that things will improve. The road may be temporarily bumpy, but in the long run, Smug will be stronger and faster and better. So yeah, we're doing a lot. I know that doesn't help you right now, but I wanted to address your question directly.

Sorry for the hassle, and I hope this helps explain things a bit better, and restores some of your faith in us.

Holler back if we can answer anything else, anythign at all.

Andy · October 7, 2006

DJKennedy wrote:

33.3 mins for the last one to upload. Thats 33 mins too long.

Please see my post just above, thanks.

Sorry again, for the hassle.

onethumb · October 8, 2006

DJKennedy wrote:

OK. This getting really freaking rediculas Smugmug.

Yes, this IS a weekly issue. I have never, ever had this problem with any other host. Ever.

I'm trying to upload 15 images, total size 6.8megs.It took ONE image 22 minutes to upload - so far it hasn't processed.

WHY DO WE YOUR CUSTOMERS HAVE TO PUT UP WITH THIS????

and EVERY weekend????

I'm trying to post photos of my sons 4th birthday. He will be 6 before the images are ready.

This is a KNOWN issue. Seriously - what is SM doing about this?

I'm paying HOW much for this ??

C'mon - enough is enough.

It actually has nothing to do with uploading. The entire site got bogged down, and since everything is tied together, it caused upload processing to slow down. But it wasn't caused by, nor limited to, uploading.

Tonight's issue was sorta unique - it looked like someone was searching for all photos tagged with the keyword "christmas" over and over - to the tune of thousands of searches per second. I'm not sure whether it was an intentional DoS (denial of service) attack, or not, but the end result is the same.

Long story short, this wasn't a typical "weekend slowdown" but a new problem we hadn't seen before. I'm sorry we didn't anticipate it, and it'll take us awhile to analyze the data and come up with a strategy to eliminate or minimize it in the future, but that's our job and we'll do it.

Let me be really clear about something: I, and the rest of the team, are spending every waking second working on the problems you're screaming about above. I left my wife in the middle of the first date we've had in weeks to work on this problem tonight, and it's neither the first nor the last time I've done so. There's nothing humanly possible that we can do faster than we're already doing. So scream all you like - but it's not going to help. (Neither will a little courtesy, but it'd be more appreciated).

Oh, and to answer your question - you're paying somewhere between $29.95 and $149.95 per year for this. I'd love to have your business, and it's what I'm busting my butt trying to keep, but if we're not doing the trick, take your money somewhere that will. Money talks - we're a business.

Kudos to all of our customers who realize that there are real human beings, who care about you and your photos, over here at SmugMug HQ burning the oil all through the weekend. You're the reason we do it.

Don

mhilbush · October 8, 2006

Interesting that you would bring up keyword searching in your post last night. I've been seeing bahavior that I can't quite explain with some of my keyword searching. The behavior is described in this post:
http://www.dgrin.com/showthread.php?t=44337

The following URL:
http://hilbush.smugmug.com/search/index.mg?searchWords=fallball+hilbush&searchType=I nUser&NickName=hilbush
seems to hang for over a minute (usually about 100 sec, but not always) on the image search part of the transaction. This seems to suggest something wrong or inefficient in the image search. I know it's a stretch that this could be related to the problem from last night, because that does sound like a DoS attack (either intentional or not). Anyway, I thought I would at least mention this.

Mark Hilbush

rajah · October 8, 2006

I'm having all the same problems. Do we have any time frame on when this will be fixed? I'm assuming that a site like this is capable of having problems in the future as well. Other than these forums is there a URL that we can go to check the system status and features status? If there is not currently one, can you consider implementing that idea?

Thank you.

Andy · October 8, 2006

rajah wrote:

I'm having all the same problems. Do we have any time frame on when this will be fixed? I'm assuming that a site like this is capable of having problems in the future as well. Other than these forums is there a URL that we can go to check the system status and features status? If there is not currently one, can you consider implementing that idea?

Thank you.

Hi Rajah, thanks for posting. For now, you comem here, and we always have the updated status.

We're looking into other notification options.
We're having some temporary technical difficulties. We're working on them at the moment, and expect service to return shortly. We apologize for the inconvenience.
We're not happy about it, of course, but we are prepared for it and expect there to be no data loss or any long-term reduction in service.
Thanks for your patience and understanding!

rajah · October 8, 2006

Andy wrote:

We're looking into other notification options.
We're having some temporary technical difficulties. We're working on them at the moment, and expect service to return shortly. We apologize for the inconvenience.

OK. Thank you.

In case Smugmug does seriously look into other notification options, I'm posting a link to what my webhost does. I have no idea how the backend works or if it could even be configured to work for smugmug, but the concept and general look is very easy and clear. Here's the LINK. Thanks.

Maybe something from here could end up being a solution for smugmug.
http://www.tchstatus.com/
http://www.tchstatus.com/serviceproblems.gif

snapapple · October 8, 2006

Thanks Andy and One Thumb,

I was having trouble with my satelite connection, couldn't get IE to open at all, then some other sites were very slow. So, I thought that was the problem over at Smugmug too. Until I finally checked here.

It's so great to have this forum to go to! It beats beating my head against the wall wondering what's going on. And on a Sunday evening too! Sometimes I think everyone at Direcway goes home on the weekend and they turn the power down because they don't think it will be needed on the weekend.

Then I come here and find everyone working so hard to keep everything up and running. Hey, this high tech stuff has glitches from time to time, we understand. It's just so comforting to have a place to go to find out that somebody is on top of it.

Thanks again.

MPerdomo · October 8, 2006

Thanks for the 411 onethumb.

Dna · October 8, 2006

onethumb wrote:

I left my wife in the middle of the first date we've had in weeks to work on this problem tonight, and it's neither the first nor the last time I've done so.

Now that is above and beyond the call of duty. I hate to think of the penance to be paid ...

Kudos to all of our customers who realize that there are real human beings, who care about you and your photos, over here at SmugMug HQ burning the oil all through the weekend. You're the reason we do it.

Don

I read a quote one time, that computers are the most complex white goods in the typical house. When you consider the interaction of all the different parts, made by different manufacturers, it's actually amazing that it works.

I have no complaints about Smugmug. I was paying 4 times what I pay Smugmug, for a limited space and limited bandwidth and I had to keep the system going, install all the programs and made sure everything ran smoothly. It took away from my photography time and I was glad to switch to Smugmug and let them do all of that.

While I'm on a roll, let me comment on the Customer Service. It's awesome. It is the best that I have ever come across. The fact that a customer of mine can email the help desk and get her photo order corrected five minutes later while I'm out shooting elsewhere is priceless.

Thanks to all at Smugmug and keep up the wonderful work.

Dna

GraphyFotoz · October 8, 2006

I WAS about to post something about this processing bit but I see it's being addressed.

As for new upgrades and all......

Come on guys it's a friggin computer!! Gezzzz
You know how reliable a computer is.
Esp with all new stuff there are bound to be issues to address.

For the service Smugmug provides on a PROFESSIONAL Level I'd say a lil glitch here and there isn't gonna kill us!
Lets be adults and let the GEEK guys get things worked out.

Nothing in life is perfect esp a dang computer!
There is always something goofy with em that keeps us cuzzin up a storm!

After all that.....has anyone got some toilet paper?

System · October 8, 2006

Good luck guys, hope you get it sorted soon. Until then I'll try to do something productive...like laundry.

evoryware · October 8, 2006

I hate to complain as I see you guys are working on it. I just am feeling let down. I've been reading these forums since April and have been working on my own gallery since the end of August when I came back from Africa. I figured that this weekend, after I got a chance to shoot with my camera at a cousin wedding, would be a good time to let my family, friends, and coworkers in on my smugmug secret. Unfortunately, the sites been giving me a headache with uploading, deleting, moving, watermarking and other functionality that led me to pay for a pro site, to the point I'm kinda embarrassed that I gave the site out today to my family after the wedding. It's been processing some of todays images for the past 3 hours and I'm on a fiber connecion.

Oh well, hope it's fixed soon.

CatOne · October 8, 2006

Glad to see you're looking into it... I'm not having any problems with upload speeds, however preview generation and in general "getting things sorted" is taking a very long time. Took over 2 hours for one gallery, and we're on 2 1/2 and counting for the next one, with not a single preview generated yet.

I'm a bit anxious... they're a bunch of 30+ year old photos I scanned from the folks:

http://blloyd.smugmug.com/gallery/1959185

Ah well, guess we'll see it by tomorrow morning. Thanks Andy and Don!

Andy · October 8, 2006

evoryware wrote:

I just am feeling let down.

We're so sorry about this. Thanks for your understanding though, and we promise this isn't the norm, and that we ARE working to improve things.

Andy · October 8, 2006

CatOne wrote:

Glad to see you're looking into it... I'm not having any problems with upload speeds, however preview generation and in general "getting things sorted" is taking a very long time. Took over 2 hours for one gallery, and we're on 2 1/2 and counting for the next one, with not a single preview generated yet.

I'm a bit anxious... they're a bunch of 30+ year old photos I scanned from the folks:

http://blloyd.smugmug.com/gallery/1959185

Ah well, guess we'll see it by tomorrow morning. Thanks Andy and Don!

Thank you, Bill

chinabuddie · October 9, 2006

This 'processing image' happened to me last night for a few hours and it's also happening to me tonight. I'm trying to show a client the photos I took of her today and she's anxious to see them. However I uploaded the 75 photos almost 3 hours ago and we still can't view them. I pay for smugmug service and apparently this stuff happens all the time on the weekends. I think someone needs to fix this from happening continuously. It makes us photographers look bad when we can't show our clients/subjects their photos because our 'reliable' webhost isnt so reliable.

Andy · October 9, 2006

chinabuddie wrote:

This 'processing image' happened to me last night for a few hours and it's also happening to me tonight. I'm trying to show a client the photos I took of her today and she's anxious to see them. However I uploaded the 75 photos almost 3 hours ago and we still can't view them. I pay for smugmug service and apparently this stuff happens all the time on the weekends. I think someone needs to fix this from happening continuously. It makes us photographers look bad when we can't show our clients/subjects their photos because our 'reliable' webhost isnt so reliable.

Hi, and thanks for posting. Welcome to Dgrin, too!

Here are two posts that hopefully explain better what's going on:
http://www.dgrin.com/showthread.php?p=399214#post399214

http://www.dgrin.com/showthread.php?p=399337#post399337

We're sorry, and it's *definitely not* the norm. You are right, you and all of our customers deserve better! We're working on it.

Thanks.

Dna · October 9, 2006

chinabuddie wrote:

This 'processing image' happened to me last night for a few hours and it's also happening to me tonight. I'm trying to show a client the photos I took of her today and she's anxious to see them. However I uploaded the 75 photos almost 3 hours ago and we still can't view them. I pay for smugmug service and apparently this stuff happens all the time on the weekends. I think someone needs to fix this from happening continuously. It makes us photographers look bad when we can't show our clients/subjects their photos because our 'reliable' webhost isnt so reliable.

1. It doesn't happen every weekend.
2. It's only happening at the moment because they are upgrading to provide better service.
3. I can understand that it is frustrating when you can't show your client her photos. Last time it happened to me, my website went offline for months ... Which is why I switch to Smugmug.

Dna

Baldy · October 9, 2006

DJKennedy wrote:

Yes, this IS a weekly issue. I have never, ever had this problem with any other host. Ever.

This is a KNOWN issue. Seriously - what is SM doing about this?

First of all, no excuses. It kills our pros who are in a hurry to work on their photos to have to wait, who knows how long. And it makes our poor help desk people sick who don't control the datacenter and hate to see the anguish. They don't know what else to say except we're working on it, which is true. I hate to see so much engineering time and anguish go into this because it's less energy spent on features we all want to see.

I said no excuses but maybe I can shed some light on why this vexes us more than other hosts and some things we're doing to address it.

PBase, Flickr, and YouTube all have had their well-publicized upload processing delays, and when I read them I think, "man, if you guys only knew..."

I'm not aware of other popular hosts who cater to event photographers and offer unlimited storage and upload bandwidth. It's a problem of our own making, a challenge we're certain we will handle, but I honestly don't think any other host gets a flood of high-megapixel, low compression images in spikes like we do on weekends.

We've made things tougher on ourselves by converting images from other formats, such as bmp and tiff, which we're discontinuing. But we still feel we have to convert the rising tide of ProPhoto and Adobe RGB images we're receiving, which other hosts don't do.

For months we'll be fine but our history as you correctly point out is that someting will happen to cause us to get backups several weekends in a row. It happened when we introduced the new custom watermarks.

Once the backup starts, a terrible spiral begins because many people upload the same batch again. It doesn't take many photographers with super-speed connections to upload big batches repeatedly before we're wishing for more computing resources.

These are sounding like excuses... I think it boils down to this: we didn't fully understand how big and complicated these spikes can be.

Anyway, this is a big priority for us and Don has a plan he's working on, but I just can't say what it is.

Sorry I don't have a quick fix tonight.

Baldy

olegos · October 9, 2006

Baldy wrote:

These are sounding like excuses... I think it boils down to this: we didn't fully understand how big and complicated these spikes can be.

Anyway, this is a big priority for us and Don has a plan he's working on, but I just can't say what it is.

Did you guys look at Amazon EC2? I've read in Don's blog about your use of S3 for storage; you must love EC2 for processing. Especially as your load is uneven through the week.

olegos

jfriend · October 9, 2006

A couple ideas...

Baldy wrote:

First of all, no excuses. It kills our pros who are in a hurry to work on their photos to have to wait, who knows how long. And it makes our poor help desk people sick who don't control the datacenter and hate to see the anguish. They don't know what else to say except we're working on it, which is true. I hate to see so much engineering time and anguish go into this because it's less energy spent on features we all want to see.

I said no excuses but maybe I can shed some light on why this vexes us more than other hosts and some things we're doing to address it.

PBase, Flickr, and YouTube all have had their well-publicized upload processing delays, and when I read them I think, "man, if you guys only knew..."

I'm not aware of other popular hosts who cater to event photographers and offer unlimited storage and upload bandwidth. It's a problem of our own making, a challenge we're certain we will handle, but I honestly don't think any other host gets a flood of high-megapixel, low compression images in spikes like we do on weekends.

We've made things tougher on ourselves by converting images from other formats, such as bmp and tiff, which we're discontinuing. But we still feel we have to convert the rising tide of ProPhoto and Adobe RGB images we're receiving, which other hosts don't do.

For months we'll be fine but our history as you correctly point out is that someting will happen to cause us to get backups several weekends in a row. It happened when we introduced the new custom watermarks.

Once the backup starts, a terrible spiral begins because many people upload the same batch again. It doesn't take many photographers with super-speed connections to upload big batches repeatedly before we're wishing for more computing resources.

These are sounding like excuses... I think it boils down to this: we didn't fully understand how big and complicated these spikes can be.

Anyway, this is a big priority for us and Don has a plan he's working on, but I just can't say what it is.

Sorry I don't have a quick fix tonight.

Baldy

Here are a couple of ideas that came to mind when reading your post.

It seems like a high priority to make sure that people don't re-upload the same images when things are slow since that only makes a bad situation worse. Is there any clearer way to communicate to a customer that their images have been received successfully and they will be processed and it will only make things go slower if they re-upload them. Can you put some messaging in the uploaders after the upload when things are backed up? Can you put some temporary message in the gallery that's awaiting processing that it will complete and it won't help to re-upload? When things are running slow, can you put a warning message in the uploaders before uploading trying to fend off a re-upload?
If certain types of uploads are a lot more load intensive (color space conversions, file format conversions, etc...), can you put those either in a separate queue of their own or put them at the end of the queue to be processed only when the system is caught up (e.g. lower priority). If you could communicate this to the people doing the uploads, it would both protect the system from these high load uploads, but also would incent people to use the more efficient formats when the load was high.

onethumb · October 9, 2006

olegos wrote:

Did you guys look at Amazon EC2? I've read in Don's blog about your use of S3 for storage; you must love EC2 for processing. Especially as your load is uneven through the week.

olegos

We don't use EC2, yet, but we are in the beta and are evaluating it.

I'm not as impressed by the EC2 price, for one thing, since my math shows that I can get more CPU power for less dollars myself, which is significant.

But we're definitely looking into it. I'm sure I'll post on my blog about our findings once we're done evaluating it.

Don

saurora · October 9, 2006

I Could Lower My Blood-pressure...
Gosh...I could lower my blood-pressure if I would just remember to check this thread before I drive myself crazy wondering "what have I done now?????" I have had so many computer problems, DSL problems, and yes, Smugmug problems that I don't know where to look lately! I'm so inept at diagnosing problems that it makes me nuts.

I know you guys are only human and that you are doing all that's possible to address the issues. I'm sure sorry for the Pros that depend upon this site to be operational. It's frustrating to me with my measly 6 shots I attempted to upload! Now that I know my pc isn't corrupted and that I am only temporarily insane, I guess I will go do some housecleaning and relieve some stress! Good luck guys!

Baldy · October 9, 2006

jfriend wrote:

Here are a couple of ideas that came to mind when reading your post.

Good ideas. Don just added email notification that fires when we have to convert from Adobe RGB, ProPhoto, or CMYK — explaining why we do it and asking their patience.

Weekly upload speeds

Comments