Maintenance Alert: 8/28 10:00pm - 06:00am PDT

Wireless

This evening, we will be performing equipment moves that require us to take down SmugMug for a few hours. We will do everything possible to minimize the amount of time we will be down.

The work will be performed from 10:00pm to 06:00am Pacific time, Monday night 8/28 to Tuesday morning 8/29.

If tonight is a success, we will follow it up with a brief outage tomorrow, Tuesday night, to upgrade our load balancers. More information will be posted about that later.

Thanks for your patience, and sorry for the inconvenience!

Sheaf

This will also affect Dgrin I believe.

Andy

Bump.

Baldy

Here's some of the stuff we have to move:

http://zdnet.com.com/1606-2_2-6110331.html

THE TOUCH

Great video!

What will costumers see when they visit our Smugmug galleries while it's down? Should we put a notice up on our websites?

Thanks!

Wireless

We already have a couple of machines in the new datacenter that we plan to put a notice up on. Customers should see our standard Maintenance page when we completely shut down.

Andy

Bump

scwalter

Why doesn't the Amazon S3 thing pickup the slack during this move?

onethumb

As you can probably tell, it has begun.

There probably will be a 20-30 minute window where nothing may come up on any pages, but we'll try to minimize it as much as we can.

Thanks again for your patience!

Don

mcgil

I'd think that's because Amazon S3 only has a copy of the images ... not the whole rest (templates, CSS, software to call those images) ... all these things that are Smugmug and customers proprietary...

scwalter wrote:

Why doesn't the Amazon S3 thing pickup the slack during this move?

onethumb

mcgil wrote:

I'd think that's because Amazon S3 only has a copy of the images ... not the whole rest (templates, CSS, software to call those images) ... all these things that are Smugmug and customers proprietary...

Actually, it's because Amazon S3 is purely storage. It's not CPU or database or web servers or anything like that.

So if we have to take core parts of our infrastructure down, like we do tonight, we don't have the server logic necessary to fetch stuff from S3.

What is cool, though, is that we've been moving the storage nodes all month this month without taking down the site. And every storage node that has Amazon S3 redundancy hasn't had any problems and no customers have even noticed that we had stuff offline.

We managed to move 300TB of storage with almost no problems during primetime hours. Pretty amazing.

Don

ballentphoto

onethumb wrote:

Actually, it's because Amazon S3 is purely storage. It's not CPU or database or web servers or anything like that.

So if we have to take core parts of our infrastructure down, like we do tonight, we don't have the server logic necessary to fetch stuff from S3.

What is cool, though, is that we've been moving the storage nodes all month this month without taking down the site. And every storage node that has Amazon S3 redundancy hasn't had any problems and no customers have even noticed that we had stuff offline.

We managed to move 300TB of storage with almost no problems during primetime hours. Pretty amazing.

Don

I would suggest that before taking down the system, that an e-mail be sent to all account holders so at least we know that it is happening. I only found out after trying to log in, and was met with a maintenance screen, so I came here to find out what was happening. An e-mail would at least let me know that it was looming.

dcyphert

Redirect while Maintenance is being done
Just a thought....

When maintenance of any kind is being performed, can a redirect script be in place that will direct from our site's "The page cannot be displayed" page, which really should be http://www.propointmedia.smugmug.com/
to the "Down for Maintenance" page on SmugMug's site. This would save allot of emails from customers asking why they can't view as well as eliminate the need to post something on our own sites saying it's down.

onethumb

dcyphert wrote:

Just a thought....

When maintenance of any kind is being performed, can a redirect script be in place that will direct from our site's "The page cannot be displayed" page, which really should be http://www.propointmedia.smugmug.com/
to the "Down for Maintenance" page on SmugMug's site. This would save allot of emails from customers asking why they can't view as well as eliminate the need to post something on our own sites saying it's down.

That's exactly what we do.

Don

JohnR

Good point because I had no idea that Smugmug was going offline tonight. I just tried to upload some more photos and it wouldn't log in.

JohnR

onethumb wrote:

That's exactly what we do.

Don

Thanks! You guys are on the ball! :

I Simonius

ballentphoto wrote:

I would suggest that before taking down the system, that an e-mail be sent to all account holders so at least we know that it is happening. I only found out after trying to log in, and was met with a maintenance screen, so I came here to find out what was happening. An e-mail would at least let me know that it was looming.

yup!!!

Murphy's law dictates that just when you're about to do someting really important to your site and have just told someone to go look at it , that's when maintaenance hits

- then we go and see why at DGrin:smack

I have told two people of a shopping centre management to look at my site so they can determine whether or not my style is suitable for taking shots in their shopping centre so they can determine whether I can get permission and whether they are interested in talking to me a bout the possibility of setting up an exhibition on their site afterwards:bash

It would have helped me enormously not to be red faced if I had had a warning email of the outage:splat

suchit

e-mail sent ahead would have been better for planned shut downs

onethumb wrote:

That's exactly what we do.

Don

What is really needed is a simple e-mail message if anything major is happening.

Right now I came to my account http://photos.suchit.in and it won't work. After trying for 15 min I decided to check out DGRIN and found out what was happening. Went back and now I see the system maintainance message as well. Since we are in India (+5.5 GMT) its working hours for us. Behind the scene is fine but if something as major as couple of hours of shut down is going to take place it would help to let customers know (via e-mail) in advance so that they too can plan ahead.

We are about to hold a Photo Exhibition and sending out my URL to many who won't be able to access my site. If I had know ahead I would used my other domain - http://suchit.net which is hosted elsewhere.

~ Suchit

elyobelyob

ballentphoto wrote:

I would suggest that before taking down the system, that an e-mail be sent to all account holders so at least we know that it is happening. I only found out after trying to log in, and was met with a maintenance screen, so I came here to find out what was happening. An e-mail would at least let me know that it was looming.

I couldn't agree more. It's pretty poor that I now have no time to make any changes to stave off the effects. I'll be cancelling smugmug and going in-house as this isn't acceptable for my needs.

Andy

elyobelyob wrote:

I couldn't agree more. It's pretty poor that I now have no time to make any changes to stave off the effects. I'll be cancelling smugmug and going in-house as this isn't acceptable for my needs.

Hi,

We're sorry to hear this. We really hate excessive email, but in this case, I do agree with you, at least maybe we could have used our control panel messaging. Sorry about that. We do hope you won't be leaving, goodness - the whole point of this is to make SmugMug stronger, faster, better Sorry for the inconvenience, we hope to keep these sorts of things to a minimum. And we'll certainly talk about notification.

onethumb

elyobelyob wrote:

I couldn't agree more. It's pretty poor that I now have no time to make any changes to stave off the effects. I'll be cancelling smugmug and going in-house as this isn't acceptable for my needs.

We just can't win.

We've gotten complaints from people in the past that our outage notifications were 'spam' and we claimed we didn't spam. Some cancelled.

We also got complaints that our email was caught in spam filters, so they didn't see it.

Those people asked us to post it on our message forum instead, so they could read it at their leisure instead of being spammed. So that's what we did.

I believe this is our only maintenance outage for all of 2006. That's pretty unheard of for any online company. I wish we didn't have to do it - but we do. I'm sorry.

Don

I Simonius

onethumb wrote:

We just can't win.

We've gotten complaints from people in the past that our outage notifications were 'spam' and we claimed we didn't spam. Some cancelled.

We also got complaints that our email was caught in spam filters, so they didn't see it.

Those people asked us to post it on our message forum instead, so they could read it at their leisure instead of being spammed. So that's what we did.

I believe this is our only maintenance outage for all of 2006. That's pretty unheard of for any online company. I wish we didn't have to do it - but we do. I'm sorry.

Don

You CAN win:

Wouldn't a good solution for all be an email notification system that you have to opt IN for?

That way, part of the opting-in process would ensure that your spam filter didn't nab it ( e.g. ' Someone, possibly you has requested email notification of maintenance outages - click on this link to confirm that you want to have email notification etc')

BTW eyyobelyob - I don't think leaving SM for this one single outage makes sense because there is no guarantee that any new system you set up will be 100% outage free surely?

Notification is the key.

elyobelyob

Simon King wrote:

You CAN win:

Wouldn't a good solution for all be an email notification system that you have to opt IN for?

That way, part of the opting-in process would ensure that your spam filter didn't nab it ( e.g. ' Someone, possibly you has requested email notification of maintenance outages - click on this link to confirm that you want to have email notification etc')

BTW eyyobelyob - I don't think leaving SM for this one single outage makes sense because there is no guarantee that any new system you set up will be 100% outage free surely?

Notification is the key.

At the moment I have two points of potential outage. One is smugmug, the other is my host. Actually, my domain registrar is another. However, this has just proved to me that I shouldn't really host my photos seperately, there's not enough of them to completely justify it, and also adds more latency when displaying pages. At least if either my registrar or webhost went down, the whole lot would be out. This just adds another point of failure which I can do without.

Nothing personal, but it's just tipped me over the edge. It'll take some work moving it all back, but I probably shouldn't have outsourced in the first place.

The good news is that I will still recommend smugmug for people wanting to do heavy photography work, it's just not the right solution for me anymore.

Manny

Oh my goodness, anyone complaining about an outage is a dweeb amateur ... get a grip and learn a thing or two about systems. Even your On Line Bank, the big old rich guys (Chase, Citibank, BoA) all go down. Only the really big powerful ones, the ones in the real know, like Amazon and eBay... they also go down.. a lot less, but they do. They have learned a lot of lessons and do a lot to minimize and eliminate downtime.

Sometimes it has nothing to do with having redundancy but just with how things are setup.... sometimes databases need TLC and it means doing a re-build and it means downtime.

Can it be zero? Well, never, but you can get it down to 99.99999 (that is 5 nines) which I can tell you costs a ton of money. I know because I am a Technical Architect at a large IT shot in NYC and when we look to go from 4 9's availability to 5 9's, like the telcos have to, it is exponentially more money.

So I go the cheapo route. I have a separate domain and website hosted outside SMUGMUG. This way these guys can do their maintenance and I am still up. It costs close to nothing. If they had to give me 5 9's reliability, my yearly fee would be at least $1,000.

Test it out, my website is up... http://www.mannyphoto.com/

My pictures are mostly at http://pictures.mannyphoto.com/ and that is down... but I can easily communicate with my customers/family/whatever through my domain site. If my main domain site goes down, Smugmug is up... win/win.

I hate downtime too, but you need to get a reality check and get a clue about how systems run, how they operate, what it takes to run a real database, especially one with millions of records, 24 hrs a day, 7 days a week, 365 days a year.

To the guys at Smugmug: Not that you need more work but one thing that has helped me in my shop is total disclosure. People now don;t complain as much. We post our entire outage windows and graph them against year to date, month to date and week to date... it clearly shows how we are doing in pretty colored charts. Once you hit that wonderful 364 days up, 1 day down, the graph speaks volumes.

Another thing we do on major apps/websites is post a countdown timer estimating when the outage will be over/site comes back... that helps MILLIONS more than a static page. I know it sucks but it works great for customer service. The number can easily be pushed back or forth as needed.

Good luck with your move, I am sure a lot of bullets were sweat already. Go get a lemonade and enjoy the success of the move. I'll be patiently waiting for my pics to come back clap

Cheers!

Manny

elyobelyob wrote:

At the moment I have two points of potential outage. One is smugmug, the other is my host. Actually, my domain registrar is another. However, this has just proved to me that I shouldn't really host my photos seperately, there's not enough of them to completely justify it, and also adds more latency when displaying pages. At least if either my registrar or webhost went down, the whole lot would be out. This just adds another point of failure which I can do without.

Nothing personal, but it's just tipped me over the edge. It'll take some work moving it all back, but I probably shouldn't have outsourced in the first place.

The good news is that I will still recommend smugmug for people wanting to do heavy photography work, it's just not the right solution for me anymore.

Can I see your pictures? Post your website URL... thanks...

Wireless

Almost there folks, sorry for the delay.

marlinspike

Manny wrote:

Oh my goodness, anyone complaining about an outage is a dweeb amateur ... get a grip and learn a thing or two about systems. Even your On Line Bank, the big old rich guys (Chase, Citibank, BoA) all go down.

Heh, my banks online banking went down just last night in fact. The servers of another guy I know, who maintains his own site, have been down for about 4 days now, so I don't think running your own site helps things.

Andy

Manny wrote:

Good luck with your move, I am sure a lot of bullets were sweat already. Go get a lemonade and enjoy the success of the move. I'll be patiently waiting for my pics to come back clap

Cheers!

Thanks Manny, words like this go a long way to all the guys pulling the all-nighter.

cdhames

Simon King wrote:

yup!!!

Murphy's law dictates that just when you're about to do someting really important to your site and have just told someone to go look at it , that's when maintaenance hits

amen. rofl

So can we expect any glitches, minor problems like pages or images not being found for the next few days, after the transfer is complete? I'd just like to forwarn my customers if possible.

bwg

I told them printing out all the pictures and re-scanning them back in was a bad idea. But NOOOOO, nobody listens to the new guy.

Barb

bigwebguy wrote:

I told them printing out all the pictures and re-scanning them back in was a bad idea. But NOOOOO, nobody listens to the new guy.