Another downtime....
l4pi
Registered Users Posts: 57 Big grins
Gentlemen,
Looks like you have had another period of time that the site has crashed....
Two critical pieces of SmugMug’s infrastructure had failures within minutes of each other (with different root causes) which temporarily brought the site offline. All systems should be go now, though. Sorry again for the inconvenience.
At some point I think we are owed an explanation to all of the issues that have effected this site over the past couple of months. The "we are sorry for the inconvenience" is growing old and really not acceptable.
I've not had a chance to upload any photos for about a week, as I had to leave town, only to come back to finish the loads that I could not upload because of issues on your end and half way though smugmug crashes again.
As a result I will actively look for another host, and will move with a half a year left on my account if this does not settle down.
Greg
Looks like you have had another period of time that the site has crashed....
Two critical pieces of SmugMug’s infrastructure had failures within minutes of each other (with different root causes) which temporarily brought the site offline. All systems should be go now, though. Sorry again for the inconvenience.
At some point I think we are owed an explanation to all of the issues that have effected this site over the past couple of months. The "we are sorry for the inconvenience" is growing old and really not acceptable.
I've not had a chance to upload any photos for about a week, as I had to leave town, only to come back to finish the loads that I could not upload because of issues on your end and half way though smugmug crashes again.
As a result I will actively look for another host, and will move with a half a year left on my account if this does not settle down.
Greg
0
Comments
Canon 5D Mk.2/Grip || Canon 7D Backup
17-40 f/4L || 70-200 f/2.8L IS || 100mm f/2.8L Macro || 24-70mm f/2.8L
Wedding Photographer
www.cwphotos.net
Is there any place on the SM site that lists 'known' outages?
Has anyone received any feedback as to when this outages is expected to be resolved?
http://oaktreephotos.com
Pentax K20 User
http://smugmug.wordpress.com/
Greg,
There's absolutely no excuse for SmugMug's recent operational issues. They're an embarrassment to me personally, an embarrassment to the company, and worst of all, an embarrassment to you and your customers. Every company has operational issues, even the largest of Internet sites like Google, eBay, and Amazon, and SmugMug is no exception. Just as we have throughout our history, we'll have occasional outages when unforeseen and unusual circumstances occur. But the frequency and severity of our recent outages is completely unacceptable. There is no higher priority for me or the rest of the company than to see them resolved and put behind us, once and for all. The bad news is that things cannot simple change overnight. There isn't simply a "fix it quickly" or "throw money at the problem" or any other easy solution. The good news is that the solutions aren't difficult, they just take careful planning and implementation, plus thorough monitoring. Some major pieces of our infrastructure are "fixed" or nearly fixed already. We continue to work through the areas that aren't yet resolved and ensure that they are.
While there's no excuse, I can offer a bit of an explanation. For security and a variety of other reasons, I can't go into great detail, but I can fill in the broad strokes.
It's my fault. As stupid as it sounds, I took my eye off the ball. With very few exceptions, all of the recent outages were preventable with the proper care, and those that weren't, were fixable in a more timely manner than they were. I've been building Internet sites professionally since 1994, so I know how this is done. We haven't been doing it right this year because I've been focused on the extremely difficult portions of scaling a site of SmugMug's size and neglected to focus on the mundane bread-and-butter portions of typical Internet operations. If SmugMug was a jet fighter, I've been focused on the jet engine and the flight controls, and neglected to make sure the bolts holding the sheet metal were torqued down properly. It really is that simple and that stupid.
Now, mind you, building jet engines and flight controls is my job. Without it, SmugMug wouldn't be what it is today. But my job also entails making sure the bolts are torqued down and all of the other mundane stuff. It's less glamorous, less difficult, and less fun - but someone needs to do it. SmugMug has an operations team, of course, and over the years I've given them more and more leeway because there have been more and more jet engines to build. At some point that I missed, the current team got in over their head. They needed more of my help than they were getting, and they needed more assistance (in the form of a larger team with more huge site experience) than they realized themselves. By the time they noticed and I noticed, bolts were starting to shear off and we had to take drastic evasive maneuvers. We're still recovering, and like I mentioned before, it'll take some time before everything is worked out - but internally we're already seeing some significant improvements in our metrics, and we hope and expect those to continue.
So there you have it. It's not a glamorous explanation, but it's the truth. And it's certainly no excuse.
But I'm completely engaged with fixing the problems, staffing up operations with more manpower and more experience at this scale, and applying both procedural and technical solutions to manage these problems going forward.
More than anything, though, I'm truly sorry. This company is my baby, you and the rest of our customers are family, and I'm distraught every time there's a service hiccup - especially when it's one that we could have foreseen.
Please stick with us. I promise we'll get things fixed, piece by piece, and that our internet service will once again match our legendary customer service.
Don MacAskill
Founder, CEO & Chief Geek
Thanks so much for the explanation; it's greatly appreciated.
--- Denise
Musings & ramblings at https://denisegoldberg.blogspot.com
Well said Don. Thanks for posting. I'm glad you're focusing on getting it rock solid again.
Homepage • Popular
JFriend's javascript customizations • Secrets for getting fast answers on Dgrin
Always include a link to your site when posting a question
Agreed and thanks. I also work work in software and web development for large scale systems (very prominent company you'd know of) and completely understand things break, stuff slips through, etc.
For me (and many customers) to most important and best way to address these issues is to honestly acknowledge them and and provide regular updates as to when systems will be back online. Of course, BCP, systems are crucial for us, but less so for SM.
I'll check the site posted above; hopefully basic upload will work soon.
Thanks for the honest reply.
Larry
http://oaktreephotos.com
Pentax K20 User
I'm confused as I still cannot upload any images... is there still an issue and the blog site just isn't current?
Larry
http://oaktreephotos.com
Pentax K20 User
Thanks Don. Perhaps AWS could help you scale your site(s). Thank you for your honesty and humble apologies. GoDaddy!
That sums up my feeling
I just opened my 2nd smugmug account
Trust me, I shopped around a lot. I looked at price,"look and feel", customizablity, user support. If I had found a site w/more value and a user support forum like this, I'd be gone.
That being said, these outages are, as Don said, embarrassing to me.
My sites are not heavily trafficked and I don't sit here constantly looking at them or making changes. But this morning I was with someone I traveled far to see who I'm trying to collaborate with on some photo tours. I had a few prints and a CD with me, but not my whole portfolio.. "Sure, I'll show him my site."...blam....not there... I spend a lot of time on the web, and it's rare that other sites I visit are down...So this is frustrating.
I hope you guys get the bugs worked out. It sounds like you grew too fast...that your size out grew your capacity to deliver with high reliability. Luckily, no one dies as in your jet plane analogy.
I know you didn't want to have problems and appreciate your honesty, but many service oriented companies are done in by this type of scenario. Like the song says "it's a jungle out there" and the same competitors you try to stay ahead of by adding new features will get you if they deliver any aspect of the service that you can't.
good luck to the company and all the users
rt
I didn't lose sales or have hundreds of people not be able to see my galleries, but I did have family and friends email that they could not see the pics and video of my son's stay (and surgery) at Vanderbilt Children Hospital.
I jumped on friendfeed and saw that Leo mentioned SM on his radio show, so I figured that his listeners took the site down, but when it didn't get better I emailed support and they said that the CEO was working on it.
I always appreciate the quick responses from support and only hope the worst is behind us as many have said the last few months have been very bad.
Fuji X shooter
Thoughts and Images
I thought this was related to the outage at the beginning of the thread, but remains an issue. I've tried now 4 computers (one being the same PC (with no changes) that has been used for almost a year. Computers that worked yesterday, stopped working a few hours later.
Im thinking some site change on SM is now conflicting with my customizations (is this possible) and prevents uploading?
I'd be glad to have ANYONE assist (I'll share my PW if your a SM admin) as I have images I need to upload tonight for a potential client -- I thought this was a site issue and was being patient for last day.
Anyone?
Larry
http://oaktreephotos.com
Pentax K20 User
Have you contacted their support, they usually answer very quickly (24/7).
I can see your galleries if that helps any.
Have you tried their trace route to see if the problem lies there?
Fuji X shooter
Thoughts and Images
Running that now; I have no problem connecting, logging in our accessing galleries.. just can't upload.
I have the current version of Java (I think its a new Java version and wonder if its not compatible?).
http://oaktreephotos.com
Pentax K20 User
Try one of the other uploaders and see if they work. That might narrow the issue to java
Fuji X shooter
Thoughts and Images
Can you go here, and go ahead and download and update your java installation?
http://www.java.com
Even though it says you have the recommended version, java installations can get corrupted. Try just updating the one you have (even if it's the same version). First, clear your browser cache and cookies. Then download and install latest version.
http://oaktreephotos.com
Pentax K20 User
Had all the functionality intact with the exception of the upload. Tried the MacDaddy upload app and it works, so I guess I'll stick with that from now on.