As we've had several hours with minimal functionality, the site should be shut down with a message indicating that the host is having difficulty. I look pretty foolish with the quirky operation... backgrounds changing, slideshows not running, slow load times... I'd rather my clients see nothing than what is presently displayed.
+1. By my watch, this outage has now lasted about an hour (minimum, that's when I tried to upload my latest shoot.) Still not a word on the status blog. C'mon guys ... please try!
As we've had several hours with minimal functionality, the site should be shut down with a message indicating that the host is having difficulty. I look pretty foolish with the quirky operation... backgrounds changing, slideshows not running, slow load times... I'd rather my clients see nothing than what is presently displayed.
Just read the post above mine... we seem to be on the same wavelength...
Ug. That's this same outage? When I looked at 8 pm, I saw as the last post:
18:14 PDT we identified the problem and are working to restore the site, hopefully not much longer.
... thinking "hopefully not much longer" would be less then two hours, and that this would be a "new outage."
The next post, at 21:06 (three hours later!) is a bit of an understatement. "Some images aren't loading" is not the same as what we're experiencing, which includes an inability to UPLOAD new images, as well.
Don't mean to be beating up on SmugMug, but this is kind of bad. Along with the others, I urge y'all to re-evaluate your policies for dealing with outages, and implement some more proactive actions (more frequent posts to the status blog or twitter feed would be good, as would a re-direct to a "Sorry, but we're temporarily broken" page instead of inconsistent scatter-shot delivery of content.)
Allow pros a custom 'sorry, we're down' message instead of things just not working or working extremely sporadically when SM is having trouble.
...since this is becoming all too regular.
We don't normally have the site in this state - normally we have RO mode or a Site Down mode. I'm finding out why we don't have one of those right now.
Allow pros a custom 'sorry, we're down' message instead of things just not working or working extremely sporadically when SM is having trouble.
PLEASE. Even the maintenance window sign would be an improvement.
And what's up with being logged out 3 times within the last hour? dgrin, not my smuggy.
The world needs more patience. I love my smugmug site and wouldn't give it up for any other hard to use / ugly / cheesy site. Keep working on it! (As I am sure you didn't need me to tell you!)
Smugmug's engineers are not sitting on their hands while the site is having issues. There is so much engineering behind the scenes (databases, files servers, file systems, network connections, operating systems, disk storage, etc.) it can be mind-boggling. The dependencies are multi-layered.
Computers and the networks that connect them are prone to failure even without human intervention. The original problem must be determined and then the appropriate solutions must be implemented with the lowest amount of risk. Complaining about the issue won't make the engineers work any harder. I know this because I'm a software engineer for a Fortune 50 company. Management inquiries about the problem can't make me work any faster than I already do to fix a sev 1 issue.
Should SM go into RO mode? Maybe. But complaining about the outage won't help anyone. SM has a VERY good uptime record...just not this week.
Smugmug's engineers are not sitting on their hands while the site is having issues. There is so much engineering behind the scenes (databases, files servers, file systems, network connections, operating systems, disk storage, etc.) it can be mind-boggling. The dependencies are multi-layered.
Computers and the networks that connect them are prone to failure even without human intervention. The original problem must be determined and then the appropriate solutions must be implemented with the lowest amount of risk. Complaining about the issue won't make the engineers work any harder. I know this because I'm a software engineer for a Fortune 50 company. Management inquiries about the problem can't make me work any faster than I already do to fix a sev 1 issue.
Should SM go into RO mode? Maybe. But complaining about the outage won't help anyone. SM has a VERY good uptime record...just not this week.
Should SM go into RO mode? Maybe. But complaining about the outage won't help anyone. SM has a VERY good uptime record...just not this week.
I think we're beyond RO mode and have been well into "site down" scenario.
I'm sure we all want the fix to come faster but realizing that it does take time, we also want our customers to be greeted with a friendly notification that the site will be back up shortly.
Even if means taking the engineers away from doing valuable work of getting the site fixed we have to remember that Smugmug isn't just sharing vacation photos. This is a professional ecommerce site and needs to act that way.
I won't pile on since it serves no purpose. What concerns me is the missing "Read Only" / "Site Down" that we've seen in the past and the fact that I have a slew of broken image links throughout my entire blog.
Along the lines of the "Read Only" image, I'd like to see a similar image that gets inserted in place of linked images housed on your Smugmug site (similar to what you've seen from places like PhotoBucket when a person's bandwidth limit has been exceeded).
Oh- and it's the last day of the month and I'm about to loose my monthly stats since all I see on my gallery stats page are broken links. Sigh.
Same here, needing to upload ~200 photos from this week, but hang in there. I'm just happy / relieved that it's down at night and not mid-day when the staffers are trying to view the pics of their 3-star General's that I took this week (have an event to photograph tomorrow with a 4 star).
Our site soft-opened at the beginning of the year and has had a million photo views so far
And as for the explanations; I can only imagine that posting details would get the SM staff absolutely INUNDATED with suggestions and 'all you have to do is this' fixit plans from us Monday mornin' quarterbacks.
For some of us it is the middle of the day & also the middle of the day when maintenance is always done
Smugmug's engineers are not sitting on their hands while the site is having issues. There is so much engineering behind the scenes (databases, files servers, file systems, network connections, operating systems, disk storage, etc.) it can be mind-boggling. The dependencies are multi-layered.
Computers and the networks that connect them are prone to failure even without human intervention. The original problem must be determined and then the appropriate solutions must be implemented with the lowest amount of risk. Complaining about the issue won't make the engineers work any harder. I know this because I'm a software engineer for a Fortune 50 company. Management inquiries about the problem can't make me work any faster than I already do to fix a sev 1 issue.
Should SM go into RO mode? Maybe. But complaining about the outage won't help anyone. SM has a VERY good uptime record...just not this week.
As an IT professional, then you should also know that 7-8+ hours of downtime over a one week period (major outages nearly every day as well...not just one big one) would result in a change of staff at most companies. My employer is fairly forgiving with people and their mistakes, but cannot imagine what would happen to my team if we had this level of instability. Ok, well, I do know what would happen...
That said, it is obvious (and discussed in previous outage threads) that SM is a small operation with limited resources. Call it growing pains, inexperience, lack of budget to allow for necessary DR strategy, etc...it's all the same in the end: downtime.
I can only guess what would result in an outage for this long. If it was a hardware failure, there should be hot or cold spares of each type of switch, router, fiber switch, etc. If it was ISP related, there should be redundant paths and multiple providers. If it was database related, there should be proper backups of appropriate sizes to allow quick restores. The list goes on. Now, if it is the Amazon S3 backend causing this (which I doubt), then it's out of their hands.
Fortunately, I am just using this site as part of my hobby, so it's just a minor annoyance. I've now seen enough downtime on here to feel the pain for the pros relying on it for their businesses.
After this outage and reading everyone's comments, I too would be interested in seeing an uptime report. It is most definitely not "VERY good" by any standard I'm aware of for at least the past six months.
Canon 7D and some stuff that sticks on the end of it.
funcrunch summed this up best: Smugmug is a Mom'n'Pop operation masquerading as a professional host.
The downtime for my site began at least 3 hours before the "site issues" on the Status Update blog indicate. My site has not loaded for at least 8 hours now. Sometimes, my url redirects to the Smugmug homepage, occasionally to an "error" page, but most often it just sits on my page, not loading. No error message, no nothing. Just a dead address. I cannot express how unprofessional that looks.
The Status Blog is not accurate: Smugmug's downtime is considerably greater than indicated, and the platitudes offered in the dgrin forum, "we're sorry" punctuated by lots of dancing smilies, are effective for about 8 seconds. If Smugmug is to continue as a professional photo host site, I'd suggest using words and forthrightness to describe situations.
And then I'd suggest Smugmug prioritize the stabilization of existing features before patting themselves on the back for inventing new features that also do not work well.
I am fortunate in that I have never taken the leap into letting Smugmug touch my professional endeavors. I pity you who have. Smugmug's general unreliability has no doubt cost you money. Their smug attitude has absolutely cost you more money.
Computers and the networks that connect them are prone to failure even without human intervention. The original problem must be determined and then the appropriate solutions must be implemented with the lowest amount of risk. Complaining about the issue won't make the engineers work any harder. I know this because I'm a software engineer for a Fortune 50 company. Management inquiries about the problem can't make me work any faster than I already do to fix a sev 1 issue.
You're right -- the issue here is not technical. The issue is customer relations. SmugMug needs to provide more info during the outage, provide information about the cause of outages after the fact, and provide an SLA for Pro customers. Will any of these make it possible for engineers to fix issues faster? Maybe not, but it will still make a lot of people happier. (Though in my experience, SLAs make it easier to assign a dollar value to downtime, and therefore make it easier to cost-justify purchases and headcount that improve availability.)
funcrunch summed this up best: Smugmug is a Mom'n'Pop operation masquerading as a professional host.
The downtime for my site began at least 3 hours before the "site issues" on the Status Update blog indicate. My site has not loaded for at least 8 hours now. Sometimes, my url redirects to the Smugmug homepage, occasionally to an "error" page, but most often it just sits on my page, not loading. No error message, no nothing. Just a dead address. I cannot express how unprofessional that looks.
The Status Blog is not accurate: Smugmug's downtime is considerably greater than indicated, and the platitudes offered in the dgrin forum, "we're sorry" punctuated by lots of dancing smilies, are effective for about 8 seconds. If Smugmug is to continue as a professional photo host site, I'd suggest using words and forthrightness to describe situations.
And then I'd suggest Smugmug prioritize the stabilization of existing features before patting themselves on the back for inventing new features that also do not work well.
I am fortunate in that I have never taken the leap into letting Smugmug touch my professional endeavors. I pity you who have. Smugmug's general unreliability has no doubt cost you money. Their smug attitude has absolutely cost you more money.
my site seems to be working now...crossing my fingers that it stays that way! Way to go SM heros!
Maybe we should save the "Way to go" for when we make it through a day with a fully functional site. Let me guess tomorrow there will be scheduled downtime that doesn't count as downtime.
Comments
actually it's been over 3 hours now
Just read the post above mine... we seem to be on the same wavelength...
3/28 - down one hour (morning)
3/28 - down one hour (evening)
3/30 - down ~45 minutes
3/31 - down 4+ hours and counting...
wow.
Ug. That's this same outage? When I looked at 8 pm, I saw as the last post:
18:14 PDT we identified the problem and are working to restore the site, hopefully not much longer.
... thinking "hopefully not much longer" would be less then two hours, and that this would be a "new outage."
The next post, at 21:06 (three hours later!) is a bit of an understatement. "Some images aren't loading" is not the same as what we're experiencing, which includes an inability to UPLOAD new images, as well.
Don't mean to be beating up on SmugMug, but this is kind of bad. Along with the others, I urge y'all to re-evaluate your policies for dealing with outages, and implement some more proactive actions (more frequent posts to the status blog or twitter feed would be good, as would a re-direct to a "Sorry, but we're temporarily broken" page instead of inconsistent scatter-shot delivery of content.)
http://ssklar.smugmug.com
Let me prepare my response to them.
Allow pros a custom 'sorry, we're down' message instead of things just not working or working extremely sporadically when SM is having trouble.
...since this is becoming all too regular.
Portfolio • Workshops • Facebook • Twitter
PLEASE. Even the maintenance window sign would be an improvement.
And what's up with being logged out 3 times within the last hour? dgrin, not my smuggy.
Galleries here Upcoming Ranch/Horse Workshop
Thanks guys.
I Live at http://www.alaskamountainforum.com
Good luck.
Oh and thanks for making me look stupid tonight.
Computers and the networks that connect them are prone to failure even without human intervention. The original problem must be determined and then the appropriate solutions must be implemented with the lowest amount of risk. Complaining about the issue won't make the engineers work any harder. I know this because I'm a software engineer for a Fortune 50 company. Management inquiries about the problem can't make me work any faster than I already do to fix a sev 1 issue.
Should SM go into RO mode? Maybe. But complaining about the outage won't help anyone. SM has a VERY good uptime record...just not this week.
Keep us up to date as always
I dont care if there are millions behind the scenes working.
I dont care how complicated it is.
I dont care how sorry they might be that it isn't working.
The only thing I care about and pay for is the site to be up and working properly. Its harsh but thats life.
I think we're beyond RO mode and have been well into "site down" scenario.
I'm sure we all want the fix to come faster but realizing that it does take time, we also want our customers to be greeted with a friendly notification that the site will be back up shortly.
Even if means taking the engineers away from doing valuable work of getting the site fixed we have to remember that Smugmug isn't just sharing vacation photos. This is a professional ecommerce site and needs to act that way.
Along the lines of the "Read Only" image, I'd like to see a similar image that gets inserted in place of linked images housed on your Smugmug site (similar to what you've seen from places like PhotoBucket when a person's bandwidth limit has been exceeded).
Oh- and it's the last day of the month and I'm about to loose my monthly stats since all I see on my gallery stats page are broken links. Sigh.
Blog: http://blog.scolephoto.com
www.kimages.com.au
Truly there's worse things happening to people on the planet right this second than Smugmug being down.
Bear
Some of my photos on Flickr
My Facebook
True, true.
Blog: http://blog.scolephoto.com
As an IT professional, then you should also know that 7-8+ hours of downtime over a one week period (major outages nearly every day as well...not just one big one) would result in a change of staff at most companies. My employer is fairly forgiving with people and their mistakes, but cannot imagine what would happen to my team if we had this level of instability. Ok, well, I do know what would happen...
That said, it is obvious (and discussed in previous outage threads) that SM is a small operation with limited resources. Call it growing pains, inexperience, lack of budget to allow for necessary DR strategy, etc...it's all the same in the end: downtime.
I can only guess what would result in an outage for this long. If it was a hardware failure, there should be hot or cold spares of each type of switch, router, fiber switch, etc. If it was ISP related, there should be redundant paths and multiple providers. If it was database related, there should be proper backups of appropriate sizes to allow quick restores. The list goes on. Now, if it is the Amazon S3 backend causing this (which I doubt), then it's out of their hands.
Fortunately, I am just using this site as part of my hobby, so it's just a minor annoyance. I've now seen enough downtime on here to feel the pain for the pros relying on it for their businesses.
After this outage and reading everyone's comments, I too would be interested in seeing an uptime report. It is most definitely not "VERY good" by any standard I'm aware of for at least the past six months.
So it's not just me ... ok F-it going to bed, sadly its down now that I just got back from WMC and I'm getting 20k/unique hits a day *sad*
Yeah, and the harsh reality is that computers suck sometimes. That's life.
The downtime for my site began at least 3 hours before the "site issues" on the Status Update blog indicate. My site has not loaded for at least 8 hours now. Sometimes, my url redirects to the Smugmug homepage, occasionally to an "error" page, but most often it just sits on my page, not loading. No error message, no nothing. Just a dead address. I cannot express how unprofessional that looks.
The Status Blog is not accurate: Smugmug's downtime is considerably greater than indicated, and the platitudes offered in the dgrin forum, "we're sorry" punctuated by lots of dancing smilies, are effective for about 8 seconds. If Smugmug is to continue as a professional photo host site, I'd suggest using words and forthrightness to describe situations.
And then I'd suggest Smugmug prioritize the stabilization of existing features before patting themselves on the back for inventing new features that also do not work well.
I am fortunate in that I have never taken the leap into letting Smugmug touch my professional endeavors. I pity you who have. Smugmug's general unreliability has no doubt cost you money. Their smug attitude has absolutely cost you more money.
You're right -- the issue here is not technical. The issue is customer relations. SmugMug needs to provide more info during the outage, provide information about the cause of outages after the fact, and provide an SLA for Pro customers. Will any of these make it possible for engineers to fix issues faster? Maybe not, but it will still make a lot of people happier. (Though in my experience, SLAs make it easier to assign a dollar value to downtime, and therefore make it easier to cost-justify purchases and headcount that improve availability.)
Several people have said this, but I've yet to see any hard data to back this assertion up. What *is* SM's uptime record?
And the harsh reality is people find something else. Buy better computers.
agreed.
Maybe we should save the "Way to go" for when we make it through a day with a fully functional site. Let me guess tomorrow there will be scheduled downtime that doesn't count as downtime.
10:32pm status update indicates some "stability enhancements" getting worked on before the all clear..
Blog: http://blog.scolephoto.com