Maintenance Alert: 8/28 10:00pm - 06:00am PDT

bham

bigwebguy wrote:

I told them printing out all the pictures and re-scanning them back in was a bad idea. But NOOOOO, nobody listens to the new guy.

Yeah like 90,000,000 images in one night, that's amazing. They should be helping google with their book scanning project.

chn

Hope your move is going well and that no nuclear reactor move is getting in your way. When our data center's mainframe was moved to a new building we unexpectedly had to deal with road closures since some other department of the university was getting rid of their small research reactor.

Don't forget to take some pictures of the move and share them with us.

samwise

chn wrote:

Don't forget to take some pictures of the move and share them with us.

Flash

Wireless wrote:

This evening, we will be performing equipment moves that require us to take down SmugMug for a few hours. We will do everything possible to minimize the amount of time we will be down.

The work will be performed from 10:00pm to 06:00am Pacific time, Monday night 8/28 to Tuesday morning 8/29.

If tonight is a success, we will follow it up with a brief outage tomorrow, Tuesday night, to upgrade our load balancers. More information will be posted about that later.

Thanks for your patience, and sorry for the inconvenience!

Few hours my butt

dcyphert

Manny hit the nail on the head!

Manny wrote:

Oh my goodness, anyone complaining about an outage is a dweeb amateur ... get a grip and learn a thing or two about systems. Even your On Line Bank, the big old rich guys (Chase, Citibank, BoA) all go down. Only the really big powerful ones, the ones in the real know, like Amazon and eBay... they also go down.. a lot less, but they do. They have learned a lot of lessons and do a lot to minimize and eliminate downtime.

Sometimes it has nothing to do with having redundancy but just with how things are setup.... sometimes databases need TLC and it means doing a re-build and it means downtime.

Can it be zero? Well, never, but you can get it down to 99.99999 (that is 5 nines) which I can tell you costs a ton of money. I know because I am a Technical Architect at a large IT shot in NYC and when we look to go from 4 9's availability to 5 9's, like the telcos have to, it is exponentially more money.

So I go the cheapo route. I have a separate domain and website hosted outside SMUGMUG. This way these guys can do their maintenance and I am still up. It costs close to nothing. If they had to give me 5 9's reliability, my yearly fee would be at least $1,000.

Test it out, my website is up... http://www.mannyphoto.com/

My pictures are mostly at http://pictures.mannyphoto.com/ and that is down... but I can easily communicate with my customers/family/whatever through my domain site. If my main domain site goes down, Smugmug is up... win/win.

I hate downtime too, but you need to get a reality check and get a clue about how systems run, how they operate, what it takes to run a real database, especially one with millions of records, 24 hrs a day, 7 days a week, 365 days a year.

To the guys at Smugmug: Not that you need more work but one thing that has helped me in my shop is total disclosure. People now don;t complain as much. We post our entire outage windows and graph them against year to date, month to date and week to date... it clearly shows how we are doing in pretty colored charts. Once you hit that wonderful 364 days up, 1 day down, the graph speaks volumes.

Another thing we do on major apps/websites is post a countdown timer estimating when the outage will be over/site comes back... that helps MILLIONS more than a static page. I know it sucks but it works great for customer service. The number can easily be pushed back or forth as needed.

Good luck with your move, I am sure a lot of bullets were sweat already. Go get a lemonade and enjoy the success of the move. I'll be patiently waiting for my pics to come back clap

Cheers!

I couldn't agree more...and having been involved with and have seen IT evolve firsthand (since my Haze Gray Navy Daze of the 80's), I realize what a huge task it must be to manage. I too don't like down time but it's a necessary evil of the business. Good job!

flyingdutchie

Andy wrote:

Hi,
... at least maybe we could have used our control panel messaging ....

That would not work
If you can't log in, you can't view your control panel messages.

I would say: Why not an opt-in e-mail notification (much like whether you want to be notified when comments are posted)?

BTW: The move seems to take more than just a few hours.... :uhoh

chn

flyingdutchie wrote:

BTW: The move seems to take more than just a few hours.... :uhoh

It's probably due to the fact that they have to calm their customers, create a new notification system for downtimes, and take pictures. All things they didn't calculate for.

Andy

flyingdutchie wrote:

That would not work
If you can't log in, you can't view your control panel messages.

Hi Anton, I was referring to "advance notification."

Scott_Quier

I really am not complaining...

Wireless wrote:

This evening, we will be performing equipment moves that require us to take down SmugMug for a few hours. We will do everything possible to minimize the amount of time we will be down.

The work will be performed from 10:00pm to 06:00am Pacific time, Monday night 8/28 to Tuesday morning 8/29.

If tonight is a success, we will follow it up with a brief outage tomorrow, Tuesday night, to upgrade our load balancers. More information will be posted about that later.

Thanks for your patience, and sorry for the inconvenience!

By my calculations, it's now 7:08AM PDT. As with all things computers (and I should know, it's how I pay for my addiction), this is taking just a mite longer than predicted. Oh well. Dgrin is working much better for me, much faster....

papajay

Decisions, Decisions!...tough to win hearts and minds!

Andy wrote:

Thanks Manny, words like this go a long way to all the guys pulling the all-nighter.

Andy, as you know, I'm an amateur #1 (not a pro photog), and a tinkerer #2 (not a real "techie")...so feel free to consider my"observations" with both facts in mind.

I've been a Smugmug customer for about 8 months now, and while I've probably done more than my share of complaining about relatively minor stuff in the overall scheme of things, I can say this with conviction:

I HAVE NEVER EXPERIENCED MORE COMMITTED, PROACTIVE, EFFECTIVE, AND RESPONSIVE CUSTOMER SERVICE FROM ANY OTHER COMMERCIAL ENTERPRISE IN MY LIFE THAN THAT WHICH IS ROUTINELY PROVIDED BY SMUGMUG....BAR NONE!

Having said that, I'm one of those in the "opt-in" camp for notification (funny thing is I probably would have chosen NOT to opt-in initially, but would chose to do-so now that I know how infrequently these interruptions occur). I "invited" several folks early last evening to go check out my latest photographic endeavors. I'm certain most of them felt a sense of "let down" when they tried to access my site, only to see the Maintenance message.

I, too, felt even more dissapointment as recently as just this minute (7:00 AM PACIFIC on Tuesday, a full nine hours after I first saw the message myself). There is no indication of actual dates/times in the message, so anyone visiting right now (on Tuesday) might assume the plan is to be down until Wednesday morning:

"Tonight, we're moving a lot of servers to a new datacenter. We'll likely be offline most of the night. "

Careful, cautious, "one-size-fits-all" wording often conveys unintended meaning.

Scott_Quier

papajay wrote:

I HAVE NEVER EXPERIENCED MORE COMMITTED, PROACTIVE, EFFECTIVE, AND RESPONSIVE CUSTOMER SERVICE FROM ANY OTHER COMMERCIAL ENTERPRISE IN MY LIFE THAN THAT WHICH IS ROUTINELY PROVIDED BY SMUGMUG....BAR NONE!

Was three of them enough?

chuckice

Scott_Quier wrote:

Was three of them enough?

I'll 4th-ify it...

nickphoto123

Smugmug is Down
Hello,

New York, 10:26 AM, No Smugmug, what is the status?

Thank you.

DJOrourke

I'm one of those who really would have benefited from an email notification. Opt-in or forced would be fine with me. It just so happened that today was a day that I'm away from my personal backups and need to access originals sitting on Smugmug. Worked out to be terribly inconvenient for me. Some notice would have helped me avoid the situation.

Andy

nickphoto123 wrote:

Hello,

New York, 10:26 AM, No Smugmug, what is the status?

Thank you.

Hello Nick, the info is here in this thread, and keep an eye out for updates from Wireless or Onethumb.

http://www.dgrin.com/showthread.php?t=41408

http://www.dgrin.com/showthread.php?p=373401#post373401

grannyrobin

Right you are

papajay wrote:

Andy, as you know, I'm an amateur #1 (not a pro photog), and a tinkerer #2 (not a real "techie")...so feel free to consider my"observations" with both facts in mind.

I've been a Smugmug customer for about 8 months now, and while I've probably done more than my share of complaining about relatively minor stuff in the overall scheme of things, I can say this with conviction:

I HAVE NEVER EXPERIENCED MORE COMMITTED, PROACTIVE, EFFECTIVE, AND RESPONSIVE CUSTOMER SERVICE FROM ANY OTHER COMMERCIAL ENTERPRISE IN MY LIFE THAN THAT WHICH IS ROUTINELY PROVIDED BY SMUGMUG....BAR NONE!

Having said that, I'm one of those in the "opt-in" camp for notification (funny thing is I probably would have chosen NOT to opt-in initially, but would chose to do-so now that I know how infrequently these interruptions occur). I "invited" several folks early last evening to go check out my latest photographic endeavors. I'm certain most of them felt a sense of "let down" when they tried to access my site, only to see the Maintenance message.

I, too, felt even more dissapointment as recently as just this minute (7:00 AM PACIFIC on Tuesday, a full nine hours after I first saw the message myself). There is no indication of actual dates/times in the message, so anyone visiting right now (on Tuesday) might assume the plan is to be down until Wednesday morning:

"Tonight, we're moving a lot of servers to a new datacenter. We'll likely be offline most of the night. "

Careful, cautious, "one-size-fits-all" wording often conveys unintended meaning.

I saw the message about moving the servers earlier this morning. i knew what it meant, and so it made perfect sense to me. :

I read your message here, wondered what on earth? Went back to look at the message, and the light dawned. Oh! No wonder someone emailed to ask whether we'd be down until tomorrow.

Whether you're patient or impatient, we appreciate your bearing with us.

Thanks for the warning about wording! I think it may have been Baldy who said that SmugMug would not be the site that it is without the comments and suggestions we receive from customers.
-grannyrobin

XHawkeye

Making progress, the maintenance screen is gone and the cannot find server or DNS Error screen is now visible. Web page should be just around the corner.

dancorder

XHawkeye wrote:

Making progress, the maintenance screen is gone and the cannot find server or DNS Error screen is now visible. Web page should be just around the corner.

Hurrah, mine is back up (rather slow at the moment but I'm sure that won't last)

Well done smugmug

leebase

FYI -- my photos are back online (hurray) -- but it's running real slow (awww).

Count me in as one who really appreciates the service -- and one who'd love an "opt in" notification of outages.

Lee

jfriend

Opt-in email setting in the control panel

onethumb wrote:

We just can't win.

We've gotten complaints from people in the past that our outage notifications were 'spam' and we claimed we didn't spam. Some cancelled.

We also got complaints that our email was caught in spam filters, so they didn't see it.

Those people asked us to post it on our message forum instead, so they could read it at their leisure instead of being spammed. So that's what we did.

I believe this is our only maintenance outage for all of 2006. That's pretty unheard of for any online company. I wish we didn't have to do it - but we do. I'm sorry.

Don

Why not add a couple checkbox items to the control panel:

[X] Notify me via email before Smugmug site maintenance
[X] Notify me via email when significant new Smugmug features become available

And, then include a paragraph at the bottom of each email that explains that you can opt-out of these emails by changing the settings in your control panel.

It seems that you'd get the best of both worlds. Those who want notification can have it. Those who don't don't opt-in.

dogwood

another vote for notification

leebase wrote:

Count me in as one who really appreciates the service -- and one who'd love an "opt in" notification of outages.

Lee

I check dgrin regularly so this outage wasn't a huge surprise (so don't ask me why I started an upload at 9:40 p.m. PT yesterday... just wanted to see if SM really would go down at 10-- you guys were a few minutes behind schedule )

Anyway, check the forums over at pbase-- wow. The concensus there is that pbase goes down randomly (it's been up and down for four days now), no one involved with pbase warns anyone anywhere (even on the forum), then refuses to respond to inquiries (even on the pbase forums). So just to add my two-cents to this discussion, I'll take SM service anyday over the lack of service at pbase (which I basically use as backup and forum linking).

rdlugosz

leebase wrote:

FYI -- my photos are back online (hurray) -- but it's running real slow (awww).

Count me in as one who really appreciates the service -- and one who'd love an "opt in" notification of outages.

Lee

Seems to be alive, but my vanity domain is going to the SM main page and not my galleries...

nickphoto123

Gallies link Direct to SM main page

rdlugosz wrote:

Seems to be alive, but my vanity domain is going to the SM main page and not my galleries...

Me too....And very SLOOOOOOW

Nicholas

JamesJWeg

rdlugosz wrote:

Seems to be alive, but my vanity domain is going to the SM main page and not my galleries...

Vanity Domain? yeah all co-brands seem screwed, I'm sure they'll have it sorted out soon. Sounds like they might be eyeing the backups in the corner at this point.

James.

NaturalEye

grannyrobin wrote:

I saw the message about moving the servers earlier this morning. i knew what it meant, and so it made perfect sense to me. :

I read your message here, wondered what on earth? Went back to look at the message, and the light dawned. Oh! No wonder someone emailed to ask whether we'd be down until tomorrow.

Whether you're patient or impatient, we appreciate your bearing with us.

Thanks for the warning about wording! I think it may have been Baldy who said that SmugMug would not be the site that it is without the comments and suggestions we receive from customers.
-grannyrobin

Also bear in mind that some of us are in significantly different time zones - it may be night for you, but is probablt isn't for us!

Gary

rdlugosz

JamesJWeg wrote:

Vanity Domain? yeah all co-brands seem screwed, I'm sure they'll have it sorted out soon. Sounds like they might be eyeing the backups in the corner at this point.

James.

yea - as an IT guy myself I can empathize... I certainly don't envy the SM team right now!

photofreak

I, too have done my share of complaining...can I blame it on the fact that I have teens...oh, wait that explains the facial tic...the complaining, I think comes from my impatience. But, I have to say...I was in the middle of uploading last night when the maintenance screen came up. I took a deep breath, a technique I've learned by having teens, and decided to call it a night. I'm beginning to understand the vaule we all have here in SM. I came straight to DG to find out what was going on.
So...thank you, SM for all the quick responses, the never-ending patience, and the incredible way you all are willing to help those of us out here in "pictureland." I, for one, have learned a little patience goes along way.
I love my smugmug

JimM

As a guy that is a partner in a web hosting company, I certainly feel their pain. To help matters, we had a hardware failure on one of our servers too. So I understand the complaints and frustrations from both sides of the fence.

Hang in their SM guys! No pressure from me, I have a customer waiting to see a retouched photo (that I haven't had time to do yet), so the site being down works as a good excuse for me!

Manny

bigwebguy wrote:

I told them printing out all the pictures and re-scanning them back in was a bad idea. But NOOOOO, nobody listens to the new guy.

Good one... made my morning

Tony Bonanno

Geesh, Some Advance Alert to Account Holders Would Have Helped Immensely !!
Geesh, I've got an event that my clients are scrambling to see and this morning I'm getting emails that they can't access the gallery, etc. y@#$.

I had no idea that this was scheduled. I finally arrived here at dgrin and found out what's going on. I don't think that I should have to lurk on this forum to get a head's up about the system being down. Many of us with PRO accounts are working photographers, and don't have time to be cruising DGRIN on a regular basis. I am very enthused about Smugmug, often recommend it to colleagues, but really.. there's got to be a better way so that we know this kind of thing is being scheduled.

Thank you.

Tony Bonanno