Unscheduled downtime, new feature Tuesday night 1/26

Baldy

I am so sorry for the late notice and deserve 1000 lashes for re-scheduling what we normally do on Thursday to Tuesday.

My only defense is this is for a really big and important feature that is very near the top of the list of what our customers have long asked for. And it has very big implications for site performance and scaling. We can rally all the critical resources tonight but there were unavoidable conflicts on Thursday.

This feature requires a hard outage that at times will just leave a white page in the middle of the night.

I pinged some customers, who said, "Oooooo!!! For this we'll take the downtime! (Even though we hate it.)"

I hope you'll agree that this feature is worth some middle-of-the-night pain.

All the best,
Chris

Andy

Status blog updated: http://smugmug.wordpress.com/

DustinFinn

Baldy wrote:

I am so sorry for the late notice and deserve 1000 lashes for re-scheduling what we normally do on Thursday to Tuesday.

My only defense is this is for a really big and important feature that is very near the top of the list of what our customers have long asked for. And it has very big implications for site performance and scaling. We can rally all the critical resources tonight but there were unavoidable conflicts on Thursday.

This feature requires a hard outage that at times will just leave a white page in the middle of the night.

I pinged some customers, who said, "Oooooo!!! For this we'll take the downtime! (Even though we hate it.)"

I hope you'll agree that this feature is worth some middle-of-the-night pain.

All the best,
Chris

All of that and I don't get to know the new feature ?

agunther

You guys
I just read it on twitter, come here full of excitement, read two paragraphs and still don't know the feature. I can hardly contain my excitement.

jchin

now you've made me impatient to see what is coming!
any more hints?

Kagan

I hope the upgrade is nice new PACKAGE of goodness!!!!!!

Sheaf

jchin wrote:

any more hints?

Baldy wrote:

...This is for a really big and important feature that is very near the top of the list of what our customers have long asked for

Let the guessing begin!

Linh

My guess Virtual Galleries. 2nd would have been privacy groups, but that's not even started on that list...

agunther

Sheaf wrote:

Let the guessing begin!

I upload bad pictures and Andy magically turns them into works of art.

agunther

Linh wrote:

My guess Virtual Galleries. 2nd would have been privacy groups, but that's not even started on that list...

I think its blog-style galleries (highest voted planned feature)

TalkieT

Baldy wrote:

[snip]I hope you'll agree that this feature is worth some middle-of-the-night pain.

All the best,
Chris

I approve of new features, but ONCE AGAIN, your US Centrism is showing up badly.

It's NOT THE MIDDLE OF THE NIGHT EVERYWHERE. Not everyone lives in US Time Zones. It's like the way only some of the world uses US Dollars.

Time and time again I hear SM claiming that they are not US Centric, but every single communication and every new feature is focussed on US customers. There may be occasional and small exceptions, but tell me I'm wrong. No, don't tell, me - PROVE to me I am wrong.

Regards
Neil Gardner
NEW ZEALAND
www.nzsnaps.com

mbrady

Very near the top of the list, or AT the top of the list?

Only 3 of the items on that first page are marked "Started" and the only one of those that could affect performance and scaling like you said is Virtual Galleries.

Baldy

Kagan wrote:

I hope the upgrade is nice new PACKAGE of goodness!!!!!!

Hahaha, good guess, but no. Packages is in test and is being held up by this feature, which has been consuming all testing resources. Another reason to get it out tonight so we can get packages out the door.

My guess is this feature will be more popular for a broad audience, but many commerce-oriented pros will say packages mean more to them.

Both of them were difficult engineering projects but tonight's feature has special implications for scaling and server load.

Baldy

TalkieT wrote:

I approve of new features, but ONCE AGAIN, your US Centrism is showing up badly.

It's NOT THE MIDDLE OF THE NIGHT EVERYWHERE. Not everyone lives in US Time Zones. It's like the way only some of the world uses US Dollars.

Time and time again I hear SM claiming that they are not US Centric, but every single communication and every new feature is focussed on US customers. There may be occasional and small exceptions, but tell me I'm wrong. No, don't tell, me - PROVE to me I am wrong.

Regards
Neil Gardner
NEW ZEALAND
www.nzsnaps.com

I know it's especially hard for Earth's most beautiful country... We have a lot of Aussie customers too and they suffer.

But we do it at the low point of traffic, which is 15% of peak. And we drink caffeine to stay awake.

pabloconrad

Wellll?????
What would be these fine improvements everyone's all-a-pitter about?

Hmmmmm???

We're picking your brain:

Brains-2.jpg

TalkieT

Baldy wrote:

I know it's especially hard for Earth's most beautiful country... We have a lot of Aussie customers too and they suffer.

But we do it at the low point of traffic, which is 15% of peak. And we drink caffeine to stay awake.

Oh I completely understand that downtime when unavoidable needs to be done at your offpeak times, and I would do it at the same time if I were you - but it's the tiny things that help.

"It's the middle of the night for the vast majority of our customers - we're sorry to those on the other side of the world, this is necessary work. We'll keep the outage as short as possible and we'll set up a static redirect to a brief note explaining that we're upgraing the infrastructure so none of your clients get too much of a shock."

Too hard? Too long? It's just that I feel like sometimes you DO only see a 15% of peak value, and not all my customers getting angry that the site won't load.

I know the work has to be done, I know downtime is unavoidable, but don't forget about us as well and think of us only as a pesky 15%.

Are we really going to get a blank page at times? How long is this expected for? What's the actual time window for dead air? Is it possible to have a static page up to explain what's happening?

Cheers - N

Baldy

TalkieT wrote:

Oh I completely understand that downtime when unavoidable needs to be done at your offpeak times, and I would do it at the same time if I were you - but it's the tiny things that help.

"It's the middle of the night for the vast majority of our customers - we're sorry to those on the other side of the world, this is necessary work. We'll keep the outage as short as possible and we'll set up a static redirect to a brief note explaining that we're upgraing the infrastructure so none of your clients get too much of a shock."

Too hard? Too long? It's just that I feel like sometimes you DO only see a 15% of peak value, and not all my customers getting angry that the site won't load.

I know the work has to be done, I know downtime is unavoidable, but don't forget about us as well and think of us only as a pesky 15%.

Are we really going to get a blank page at times? How long is this expected for? What's the actual time window for dead air? Is it possible to have a static page up to explain what's happening?

Cheers - N

Those are all great points and we appreciate them. Especially since two of our favorite countries on earth are Oz and NZ and we have lots of customers there.

We have three levels of outages: read-only, which carries us through most product releases, a static page, which carries us through almost everything else, and a white page, which is awful and we almost never have to do it.

But this time we need some white page time... I'd like to promise it will be short because that's our expectation, but it's a complex upgrade.

Sheaf

TalkieT wrote:

I approve of new features, but ONCE AGAIN, your US Centrism is showing up badly.

Sorry, it was unfortunate wording. We make the decisions to do these things simply on traffic and site load. We have far less of both between the hours of 10 PM and 4 AM PST. We looked at the numbers tonight and here is how the traffic numbers break down:

Midnight to 4 AM PST: 5.61% of daily traffic
4 AM to 8 AM PST: 13.28% of daily traffic
8 AM to noon PST: 21.55% of daily traffic
Noon to 4 PM PST: 23.08% of daily traffic
4 PM to 8 PM PST: 22.77% of daily traffic
8PM to midnight PST: 13.71% of daily traffic

I assure you, we would much rather do this during the daytime rather than stay up into the wee hours of the morning, especially because we are all exhausted from building and testing this feature. But our decision has to be made according to how many of our customers (and yours) will be affected by the downtime.

BELphotos

Good luck with the upgrade.... Been there, done that in another life.

Andy

BELphotos wrote:

Good luck with the upgrade.... Been there, done that in another life.

IBM?

BELphotos

Andy wrote:

IBM?

Bell System - 25 years. NYT, ATT, WECO, etc.

Linh

mbrady wrote:

Very near the top of the list, or AT the top of the list?

Only 3 of the items on that first page are marked "Started" and the only one of those that could affect performance and scaling like you said is Virtual Galleries.

that's pretty much why I said it. Virtual galleries would need a big overhaul.

I hadn't seen that feature request thread in a while, some of the comments are hilarious... it's like they think SM is not implementing it because they are lazy. I don't think those people understand the amount of work that has to go into that, especially if it's to retrofit it into an existing platform-and do it well so it doesn't fell like a retrofit.

Anyway, tomorrow will be a big day, SM big update, and finally see what this Apple tablet is about, heh (though, I'd only likely buy it if it were under $500).

BradfordBenn

Ya ain't lived until a firmware & software update takes down the audio for a nationally televised event. DAMHIK, IJK So Break Leg!

richpepp

I really hope it's not virtual galleries, that would ruin my day...


...as I've got work to do and I just would be able to help myself from seeing what I could do with it

brecklundin

Well, looks like I picked a good week to up the Xanax dose!! looking forward to something new and fun. Then again a couple vicodin for the RA and I really won't care either way...

JVo

My website is not running! http://vanosten.smugmug.com
I just send this weekend our links to lots of people we know and we had regular visitors each day.
All my pictures are gone.
I am not at all amused with this...

Sheaf

JVo wrote:

My website is not running! http://vanosten.smugmug.com
I just send this weekend our links to lots of people we know and we had regular visitors each day.
All my pictures are gone.
I am not at all amused with this...

We're sorry, but we are working hard at getting the upgrade done and getting everything back to normal. We'll update this thread when we're finished.

JVo

Sheaf wrote:

We're sorry, but we are working hard at getting the upgrade done and getting everything back to normal. We'll update this thread when we're finished.

OK, just hope everything will be back normal soon...
For the future: it would be better to show a screen that says "website is down for maintenance, we'll be back soon"

pjspics

JVo wrote:

OK, just hope everything will be back normal soon...
For the future: it would be better to show a screen that says "website is down for maintenance, we'll be back soon"

Never seen a maintenance/feature downtime do this before... my website is a mess at the moment.
Lost all my front page - code where the slideshow and main image should be.
Doesn't look good to visitors at all.
All my popular pics gone and lots more....

Is this being caused by you guys? just so I know its going to come back some time soon...
Cheers

Sheaf

pjspics wrote:

Is this being caused by you guys?

Yes, sorry! We're frantically trying to deal with some unexpected problems that were not occurring in our testbed.

denisegoldberg

I know, I know, you are working on it.

I find it pretty scary that the only photos that are showing on my site are on the popular photos page. Everything else shows as text-only.

This seems like it would be a good time for you (smug) to put a message on our sites that work is in progress.
I hate to think of people coming to my site in it's current state with no indication that there is a problem.

--- Denise