Maintenance Window Sucks

jfriend · March 25, 2011

I'm just going to go on record here and say that a regular maintenance window that prevents us from uploading images sucks! Come on. You guys are not a new company - at least 6-7 years old. You can engineer around locking your customers out on a regular basis. I get it if every once-in-a-long-while (like only a couple times a year) you have a major upgrade that requires a little down time, but regularly interrupting, stopping and prevent image uploads. I don't care if it's an announced or planned time. That's immaterial. The rest of the world can't always plan around your schedule. Plus, the rest of the world doesn't go check your maintenance schedule before every upload.

I finished my image processing of a whole soccer season tonight (902 images across 20 galleries) and I started the upload at 10:30pm, hoping it would run overnight and finish sometime tomorrow (probably take about 10 hours to run). Fortunately, I came to check on it an hour later and I find every image in the upload is generating an error. Crap. You're probably in read-only mode. I try to stop the uploader gracefully, but it doesn't shut-down gracefully and I lose the entire upload queue. Great, one more thing to do over again because of your maintenance.

Now, I have to start all over and I have no idea when you're darn read-only mode will be done so I can actually get this upload going overnight. You don't even have a reliable all-clear message from the maintenance so I could know when to actually start the uploads and I wasn't planning on staying up that late either and just because uploads start being allowed again doesn't mean you're actually done and it's really clear (I've learned that before). So here I am. I'm stuck. I have to start my upload all over again. I have to repartition all the images into the 20 separate galleries into the uploader all over again. I have to delete the images that have uploaded so far so I don't end up with dups. I have no way of knowing when you're actually done with maintenance so I can start or whether to even plan on trying to stay up that long. The upload will take about ~11 hours and I can't have it going during the middle of the day tomorrow (for bandwidth usage reasons) so it looks like I just lost an entire day on having these images up. Have I said this yet. This sucks!

This is a solvable problem. You could engineer your systems to allow uploads to continue even though you were doing regular maintenance. Maybe once in a blue moon, uploads would be part of the actual maintenance, but most of the time they wouldn't be. If you had to, you could accept the uploads and just hold them in a queue for processing when the upgrade or maintenance was done. Does Google announce downtime when they need to do a system upgrade? No. They engineer around it so they don't take their system down when doing upgrades or maintenance. This is a solvable problem if you choose to solve it. Is it work - yes, but it's time for you to do that work. It appears that you aren't going to do that work until customers start complaining loudly to make you start thinking about it more seriously. This post is for more customers to start calling for less and less planned downtime. If you're a customer and you're tired of this scheduled downtime, then post your thoughts into this thread.

I don't want to have to plan my site uploads around your maintenance schedule. I don't want a 902 image upload that I planned to run overnight to be forced to be delayed an entire day. If I wanted to get it started tonight anyway after your maintenance, I have no idea when you'll actually be done or how to know when you're actually done. This sucks. It's time you started acting like a more mature company and engineer yourselves out of this regular downtime - particularly for uploads.

You guys do a lot of things right and you know my dedication to the site, but I've had it with this one. I've just wasted a bunch of time, have to start over on a bunch of work I already had in progress and my upload will probably be delayed by a day. I'm sure you can tell - I'm not happy.

Squeezer · March 25, 2011

Sucks even more when you delete your CSS to paste in some new stuff ( the HTML5 Jfriend Mods !! ) and after deleting the old stuff but before the new stuff has been pasted it goes into maintenance and leaves the site looking like crap and there's nothing you can do about it

Art Scott · March 25, 2011

Yes it sux....Nicely put JFriend & Squeezer........I have had to email several clients over the past to tell them their galleries would be empty on Friday Morning due to the hosting companies maintenance.....which sounds like a cop out to the clients and have been told so......I thought the norm was redundant systems so if one goes down the company stays alive.....

BigRed · March 25, 2011

And this week was a double whammy because there was no advance notification (well, unless you count 31 minutes).

Dgrin SmugMug Maintenance Window notification
Mar-25-2011, 01:28 AM EDT
using our maint window tonight

Status Log
March 25, 1:59AM EDT
We're rolling out some new changes to the site tonight, and expect to be in read-only mode for an hour or two while we complete the upgrades.

Andy · March 25, 2011

BigRed wrote: »

And this week was a double whammy because there was no advance notification (well, unless you count 31 minutes).

We put the notification on our http://status.smugmug.com site much earlier in the week.

Andy · March 25, 2011

Hi John, thanks for the feedback - I've made sure our Ops team has seen this.

Art Scott · March 25, 2011

Andy wrote: »

We put the notification on our http://status.smugmug.com site much earlier in the week.

Many of us do not have the time to go looking to see what the engineers are doing .....and we really should not have to.....if the back up system is brought on line before any work on the main system is done.....

thumb:D

jfriend · March 25, 2011

Art Scott wrote: »

Many of us do not have the time to go looking to see what the engineers are doing .....and we really should not have to.....if the back up system is brought on line before any work on the main system is done.....thumb:D

Art, in defense of Smugmug, it's not as simple as you make it sound. It's not like Smugmug has double the entire site capacity just sitting around and they can switch to it while working on the main system. But, upgrades without downtime can be engineered in more involved ways. My particular gripe was about uploads and something special could be done for them.

Allen · March 25, 2011

Not knowing anything about this, why can't they just leave the upload que
active and processing paused?

jfriend · March 25, 2011

Allen wrote: »

Not knowing anything about this, why can't they just leave the upload que
active and processing paused?

Something like that is what I'm asking for.

Andy · March 25, 2011

John, part of last night's maintenance was regarding uploading. I can't and won't get into any of it right now, but I will say we've just recently put an incredible amount of energy, resource, money and time into improving uploading yet again - and there should be noticeable improvements in speed.

jfriend · March 25, 2011

Andy wrote: »

John, part of last night's maintenance was regarding uploading. I can't and won't get into any of it right now, but I will say we've just recently put an incredible amount of energy, resource, money and time into improving uploading yet again - and there should be noticeable improvements in speed.

Glad you're working on uploading, but wouldn't my upload speed bottleneck be my own internet link and wouldn't that be the case for most consumers?

Or are you talking about the speed of processing them after they've been uploaded?

Andy · March 25, 2011

jfriend wrote: »

Glad you're working on uploading, but wouldn't my upload speed bottleneck be my own internet link and wouldn't that be the case for most consumers?

Or are you talking about the speed of processing them after they've been uploaded?

Everything from you--->us is up to you and your connection. Once it gets to us, we worked a lot of magic to make things fasterererererererer

ImageX Photography · March 27, 2011

I have also had countless uploads delayed or interrupted... and during prime computing hours too. I don't have a problem with the site making improvements but the frequency and time of the "maintenance windows" is ridiculous. Why are these not being done at 3-5am when most people are asleep? Yes, I know it's a global site but the majority of users are in America and are asleep during those hours. Every respectable company out there does maintenance during the wee hours so why does Smugmug do it at 10pm when everyone is on the computer?

jfriend · March 27, 2011

Andy wrote: »

Everything from you--->us is up to you and your connection. Once it gets to us, we worked a lot of magic to make things fasterererererererer

I was just curious what would we see as fasterer?

Andy · March 27, 2011

jfriend wrote: »

I was just curious what would we see as fasterer?

Image processing and everything that happens on our end. Especially for larger files.

cfrey6 · March 28, 2011

Andy wrote: »

Image processing and everything that happens on our end. Especially for larger files.

I sure hope you don't forget the "replace photo" part of this equation, as well. The speed of that is absolutely pathetic!

As for uploads, I want to be able to schedule uploads so they happen at a certain time for multiple galleries.

SamirD · March 28, 2011

jfriend wrote: »

I finished my image processing of a whole soccer season tonight (902 images across 20 galleries) and I started the upload at 10:30pm, hoping it would run overnight and finish sometime tomorrow (probably take about 10 hours to run). Fortunately, I came to check on it an hour later and I find every image in the upload is generating an error. Crap. You're probably in read-only mode. I try to stop the uploader gracefully, but it doesn't shut-down gracefully and I lose the entire upload queue. Great, one more thing to do over again because of your maintenance.

Now, I have to start all over and I have no idea when you're darn read-only mode will be done so I can actually get this upload going overnight. You don't even have a reliable all-clear message from the maintenance so I could know when to actually start the uploads and I wasn't planning on staying up that late either and just because uploads start being allowed again doesn't mean you're actually done and it's really clear (I've learned that before). So here I am. I'm stuck. I have to start my upload all over again. I have to repartition all the images into the 20 separate galleries into the uploader all over again. I have to delete the images that have uploaded so far so I don't end up with dups. I have no way of knowing when you're actually done with maintenance so I can start or whether to even plan on trying to stay up that long. The upload will take about ~11 hours and I can't have it going during the middle of the day tomorrow (for bandwidth usage reasons) so it looks like I just lost an entire day on having these images up. Have I said this yet. This sucks!

Ahhh...welcome to my world. :cry

Fortunately, I haven't faced too many issues with the maintenance window since my main bulk is uploaded on Sunday/Monday. But the broken upload scenario you describe is one that I've faced so many times since 2005, and has caused enough frustration to be furiously angry with SM on more than one occasion. I've expressed only a small percentage of this over the years.

It's why uploading, even when there is no maintenance window, needs vast improvement. I almost switched to Exposure Manager for their robust FTP uploading capability. If SM didn't introduce videos the exact same time I was testing with EM, I would've been gone. I was uploading albums to EM 3x faster. Not because of better bandwidth or processing, but simply because it was FTP, a protocol designed for file transfer.

I'm working very closely with the developer of the smugftp third party product doing heavy testing. And in my tests, I'm able to get 2-3x the transfer speeds--not to mention no more setting up multiple web-based uploaders to max out bandwidth, which can save up to an hour.

And FTP-based uploading system could know when to retry, what to retry, what's there, and what's not. You could just queue up and leave it alone, maintenance window or not.

As long as I've been a SM customer, I firmly believe there's a business reason behind SM's decision to make uploading a challenge. I'm sure there's a good number of customers who try the trial and then leave when they can't upload robustly. I guess if uploading was easy, it would stress something else, so it's good that it functions the way it does--from SM's point of view.

Just out of curiosity John, why weren't you using SE?

jfriend · March 28, 2011

SamirD wrote: »

Just out of curiosity John, why weren't you using SE?

I was using SE and it failed to shut-down properly when I tried to stop it and it lost my upload queue. Bugs in SE kept it from working properly in this case.

Andy · March 28, 2011

SamirD wrote: »

As long as I've been a SM customer, I firmly believe there's a business reason behind SM's decision to make uploading a challenge.

We've poured tons of money and people into uploading to make it as easy and friction-free as possible, with constant improvements over the years. We have zillions of customers and extremely, very extremely low complaints about uploading. And the ones we get are almost always connection-related. We absolutely don't want this to be a challenge (why would we? we're in the photo sharing and photo commerce business - we want customers to be able to upload).

jfriend · March 28, 2011

Andy wrote: »

We've poured tons of money and people into uploading to make it as easy and friction-free as possible, with constant improvements over the years. We have zillions of customers and extremely, very extremely low complaints about uploading. And the ones we get are almost always connection-related. We absolutely don't want this to be a challenge (why would we? we're in the photo sharing and photo commerce business - we want customers to be able to upload).

Andy, do you offer an uploader that can upload across multiple galleries robustly that will succeed 100% of the time without any manual intervention even though there are temporary connectivity issues or a maintenance window interruption? That's what both Samir and I are looking for.

Dave17 · March 28, 2011

Is there problem with uploads tonight? Got about half my gallery uploaded and then the systems hangs on "verfiying" and then the upload fails.
Thanks,

--Dave

Andy · March 28, 2011

Dave17 wrote: »

Is there problem with uploads tonight? Got about half my gallery uploaded and then the systems hangs on "verfiying" and then the upload fails.
Thanks,

--Dave

Dave, we had a problem a little bit ago, but it looks to be fixed now, can you try again?

Stash · March 28, 2011

Is there any chance of changing the maintenance window time? I realize that for most of the world, Thursday night is a quiet time. However some of us live in a different culture and Friday morning (Thursday night maintenance time in your time zone) is a day off from work, where we actually get quality time to work on our sites, until the Maintenance Window kicks in and we've lost another opportunity. A mid week downtime area would affect all of us equally.

Andy · March 28, 2011

Stash wrote: »

Is there any chance of changing the maintenance window time? I realize that for most of the world, Thursday night is a quiet time. However some of us live in a different culture and Friday morning (Thursday night maintenance time in your time zone) is a day off from work, where we actually get quality time to work on our sites, until the Maintenance Window kicks in and we've lost another opportunity. A mid week downtime area would affect all of us equally.

I wish we could - and I hear your pain, but we have to go with the time that is statistically the slowest time for uploads and other activity on the site - and thus, 10pm CA time Thursday nights. I'm sorry I don't have a better answer for you!

Dave17 · March 28, 2011

Just tried again. Uploads still hanging on "verfiying upload".

TalkieT · March 28, 2011

Stash wrote: »

Is there any chance of changing the maintenance window time? I realize that for most of the world, Thursday night is a quiet time. However some of us live in a different culture and Friday morning (Thursday night maintenance time in your time zone) is a day off from work, where we actually get quality time to work on our sites, until the Maintenance Window kicks in and we've lost another opportunity. A mid week downtime area would affect all of us equally.

Although Andy's already given the only answer he can, you need to remember a key thing about Smugmug.

They are a US BASED COMPANY, and they cater primarily for US BASED PHOTOGRAPHERS and viewers. Sure, they don't actually lock out foreigners from using the service, but everything they do, every design choice they make, it's always going to be primarily focussed on their US userbase.

This, unfortunately, is their only sensible option and I'd do it the same way in their position, but regardless of how they try to position themselves as a global service, they have always been, and will always be, massively US focussed.

So no, maintenance will always be when it's least inconvenient for US customers.

Cheers - N

Andy · March 28, 2011

TalkieT wrote: »

Although Andy's already given the only answer he can, you need to remember a key thing about Smugmug.

They are a US BASED COMPANY, and they cater primarily for US BASED PHOTOGRAPHERS and viewers. Sure, they don't actually lock out foreigners from using the service, but everything they do, every design choice they make, it's always going to be primarily focussed on their US userbase.

This, unfortunately, is their only sensible option and I'd do it the same way in their position, but regardless of how they try to position themselves as a global service, they have always been, and will always be, massively US focussed.

So no, maintenance will always be when it's least inconvenient for US customers.

Cheers - N

This isn't true, TalkieT.

We based our maintenance window on the lowest activity period, it had nothing to do with US or non-US customers.

As to the other things, I've replied in all the other threads, and on our feedback site, that good things are coming - you won't be unhappy, I promise.

Let's please keep it in perspective, thanks!

Allen · March 28, 2011

Stash wrote: »

...A mid week downtime area would affect all of us equally.

So rather then only effect some you want everyone to suffer?

They studied upload logs and are using the least used period so less are
effected.

TalkieT · March 28, 2011

Andy wrote: »

This isn't true, TalkieT.

We based our maintenance window on the lowest activity period, it had nothing to do with US or non-US customers.

As to the other things, I've replied in all the other threads, and on our feedback site, that good things are coming - you won't be unhappy, I promise.

Let's please keep it in perspective, thanks!

Quck response :-)

I'm not saying you do it arbitrarily, I'm saying because the bulk of your userbase is in the US, this makes the least inconvenient time for planned outages map to what it is today. As I say, I'd choose the same time, becuase I'd apply the same logic to determining the time. (Leaving aside for a moment the fact that pretty much every other site gets away without regular planned outages)

As for the other issues :-) I didn't even mention them this time!

Personally, I only care on principle at this point - I won't bother undoing the Paypal integration I have done. I got an email from Smugmug a couple of weeks ago about a sale and I was surprised because at this point, easily 95% of my sales are through Paypal in my local currency.

I love Smugmug's customisation and especially the support (both from the company and community - they are both exemplary) but You're only ever going to see my yearly subscription from now on. I forget how long we've been asking for foreign currency support - it's been LITERALLY years now.

I really hope it happens, if for no other reason than to appeal to more global users and even out your customer base which might have better long term impacts on your development priorities.

Cheers - Neil G

sssesq · March 28, 2011

As far as I can tell without running a realtime sniffer on your net, you do have intermittent uploading problems. Right now I have had 5 of 24 10mb jpgs fail to upload. I work in software QA and have a 60Mb/5Mb connection that is testing at 62.5 and 3.75 (while I am uploading). This is not meant to be critical as I know too well how many things can sneak up on you and degrade performance, but it is not on my end. An internet connection problem that is out of your hands is much more likely. That being said, you could redirect you incoming uploads to a buffer while doing Maint. How are you doing your backups???

Maintenance Window Sucks

Comments