Options

Down again?

13567

Comments

  • Options
    funcrunchfuncrunch Registered Users Posts: 14 Big grins
    edited March 31, 2010
    Andy posted:
    Yes, we're having some Ops issues this week - overall our availability and performance has been amazingly high.
    I'm sorry but I really do not consider the large number of downtimes SmugMug has suffered this week - not to mention in previous months - qualifies as "amazingly high" availability and performance. I'm a pro photog, put up two galleries with hundreds of photos this week, managed to get two sales before the downtime came again.

    I'm glad to have status updates more frequently than in the past, and customer support is very responsive. But something is going very wrong with the underlying infrastructure of the system. This host is providing services on a professional level, at professional prices, but operating like a Mom-and-Pop operation. I understand that the new features being put out are independent of the operations issues as different staff are involved. Regardless, more resources must be devoted to site stability and uptime or it will not be a viable site for pros to host and sell photos.
    Concert and event photography by Julie Bernstein
    funcrunchphoto.com
  • Options
    JAGJAG Super Moderators Posts: 9,088 moderator
    edited March 31, 2010
    I just had to put an out of order sign on my home page.:cry I certainly hope that the many many of galleries that have disappeared will return soon. Some of the galleries say "empty gallery" others are x'ed out. I litterally have over 1200 images missing at this point. Please don't disappoint me Smugmug heros!eek7.gif
  • Options
    4cornerscrystals4cornerscrystals Registered Users Posts: 6 Beginner grinner
    edited March 31, 2010
    exploregon wrote:
    I totally agree with 4corners. The majority of the bids usually come within the last couple hours of an auction end. If people can't see the product, then no more bids. An item that would generally sell for $150 that ends up selling for $25 really hurts the bottom line. Multiply that by 50 listings...........ouch.

    Yep, and fixed price listings get their most views in the last couple hours too.

    People see a bunch of pizza boxes and they move on to the next listing.

    :cry
  • Options
    brecklundinbrecklundin Registered Users Posts: 121 Major grins
    edited March 31, 2010
    Andy wrote:
    I hope we can improve things between now and then so we can change your mind - we'd hate to see you go. And if you do decide that, please email me attn Andy at our help desk.

    Andy,

    I realize you are sincere here, and trying to help, and platitudes are all that remain right now...but as the poster above mentions, for some of us ebay sellers, as well as others who have planned business happening now, not tomorrow and not 3-days from today, but today, this sort of outage truly costs us a LOT of money due to the time sensitive nature of our businesses. We have to plan and need to depend on server reliability. And as much fun as SmugMug is and the kind nature of the staff, that does not pay my bills. And in this case, does not over my expenses for my first vacation in nearly 15yrs that I have planned for next week. This weekend's sales were supposed to cover the expenses for that trip. Now, it is all questionable. What I have to do NOW is edit every listing, move pictures to my webserver then modify the code for the app I wrote to create my listings.

    And to top it off...I have the flu and I guess that is SM's fault as well...:hide hehehe...but know that while I jest a bit, I am completely serious in my decision. I have worked with far too many hosting services over the decades not to see the futility in remaining "loyal" as this is a two way street. It's just a business decision on my part as I cannot have this sort of issue even once a year.
  • Options
    JeffroJeffro Registered Users Posts: 1,941 Major grins
    edited March 31, 2010
    JAG wrote:
    Some of the galleries say "empty gallery" others are x'ed out. I litterally have over 1200 images missing at this point. eek7.gif

    Everytime this happens it scares the crap out of me...to even think that all of my images are gone, makes me...:cry
    Always lurking, sometimes participating. :D
  • Options
    R.LeonardoR.Leonardo Registered Users Posts: 188 Major grins
    edited March 31, 2010
    JAG wrote:
    I just had to put an out of order sign on my home page.:cry I certainly hope that the many many of galleries that have disappeared will return soon. Some of the galleries say "empty gallery" others are x'ed out. I litterally have over 1200 images missing at this point. Please don't disappoint me Smugmug heros!eek7.gif

    While I would rather the root cause fixed, you bring up a good point. I would like to have visitors know that there is maintenance. Right now, it simply looks like MY website of busted or poorly built. If I, or in this case a client did not check, neither would know the real issue.
    Perhaps this is not possible with today's issue.
    R. Leonardo
    www.RobArtPhoto.com
    Whether he is an artist or not, the photographer is a joyous sensualist, for the simple reason that the eye traffics in feelings, not in thoughts. -Walker Evans
  • Options
    JAGJAG Super Moderators Posts: 9,088 moderator
    edited March 31, 2010
    Jeffro wrote:
    Everytime this happens it scares the crap out of me...to even think that all of my images are gone, makes me...:cry

    Although I have all my images somewhere on external hardrives in my safe at home....it would be horrible to have to start over. Most of my sales are done locally here so the site is more of a showcase for them to see the work and ask for specific images.
  • Options
    JeffroJeffro Registered Users Posts: 1,941 Major grins
    edited March 31, 2010
    I'm going to bed, hopefully tomorrow, when I wake up, all the red x's will be gone.
    Always lurking, sometimes participating. :D
  • Options
    ssklarssklar Registered Users Posts: 22 Big grins
    edited March 31, 2010
    Please update the status blog more frequently!
    I don't want to jump on the dog pile here, but the frequent outages are of concern for those of us relying on SmugMug for some portion of our livelihood.

    I think that more frequent updates to the status blog might at least help the the feeling of "WTF" ... I'm a systems administrator, and in my organization, we've found that letting the users know that (a) we know there is a possible problem, and (b) we're working on it, and (c) regular updates (even if just one line) throughout the outage up until the resolution, lead to much happier users.
  • Options
    R.LeonardoR.Leonardo Registered Users Posts: 188 Major grins
    edited March 31, 2010
    Jeffro wrote:
    I'm going to bed, hopefully tomorrow, when I wake up, all the red x's will be gone.


    Humm, Visine eye drops for SmugMug?
    R. Leonardo
    www.RobArtPhoto.com
    Whether he is an artist or not, the photographer is a joyous sensualist, for the simple reason that the eye traffics in feelings, not in thoughts. -Walker Evans
  • Options
    gecko0gecko0 Registered Users Posts: 383 Major grins
    edited March 31, 2010
    I thought this would be of interest to those that are not familiar with the "five nines" standard. When people say they have 99% or 99.9% uptime, that is still actually quite a substantial amount of downtime.

    The unformatted chart (sorry about that...copy/paste destroys it):

    Availability % Downtime per year Downtime per month* Downtime per week
    90% 36.5 days 72 hours 16.8 hours
    95% 18.25 days 36 hours 8.4 hours
    98% 7.30 days 14.4 hours 3.36 hours
    99% 3.65 days 7.20 hours 1.68 hours
    99.5% 1.83 days 3.60 hours 50.4 minutes
    99.8% 17.52 hours 86.23 minutes 20.16 minutes
    99.9% ("three nines") 8.76 hours 43.2 minutes 10.1 minutes
    99.95% 4.38 hours 21.56 minutes 5.04 minutes
    99.99% ("four nines") 52.6 minutes 4.32 minutes 1.01 minutes
    99.999% ("five nines") 5.26 minutes 25.9 seconds 6.05 seconds
    99.9999% ("six nines") 31.5 seconds 2.59 seconds 0.605 seconds

    from here: http://en.wikipedia.org/wiki/High_availability

    So, in the end, even with 99.9% uptime, there is 8.76 hours of downtime, which could be quite a few 15-30 min stretches over the year.
    Canon 7D and some stuff that sticks on the end of it.
  • Options
    gecko0gecko0 Registered Users Posts: 383 Major grins
    edited March 31, 2010
    ssklar wrote:
    I don't want to jump on the dog pile here, but the frequent outages are of concern for those of us relying on SmugMug for some portion of our livelihood.

    I think that more frequent updates to the status blog might at least help the the feeling of "WTF" ... I'm a systems administrator, and in my organization, we've found that letting the users know that (a) we know there is a possible problem, and (b) we're working on it, and (c) regular updates (even if just one line) throughout the outage up until the resolution, lead to much happier users.

    +1 on the abc's there. I'm a network engineer and we also are required to provide updates (every 15 min) during a critical outage. This isn't in the form of anything technical, but simply an update...even if there isn't an update to give. It's purely political at that point and gives management (or customers in this case) a better feeling about the situation.

    While I would personally love to hear the technical details of what is going on, it would not serve SM much usefullness in letting everyone know exactly what causes the outages (mainly because most would not understand enough about it to have it be a benefit to them).

    .02
    Canon 7D and some stuff that sticks on the end of it.
  • Options
    ssklarssklar Registered Users Posts: 22 Big grins
    edited March 31, 2010
    gecko0 wrote:
    +1 on the abc's there. I'm a network engineer and we also are required to provide updates (every 15 min) during a critical outage. This isn't in the form of anything technical, but simply an update...even if there isn't an update to give. It's purely political at that point and gives management (or customers in this case) a better feeling about the situation.

    While I would personally love to hear the technical details of what is going on, it would not serve SM much usefullness in letting everyone know exactly what causes the outages (mainly because most would not understand enough about it to have it be a benefit to them).

    .02

    I'm sure SmugMug considers it a "blessing" that so many of their users are professional nerds. It must be like cooking the meal for a convention of the best chefs in the world. ;-)
  • Options
    R.LeonardoR.Leonardo Registered Users Posts: 188 Major grins
    edited March 31, 2010
    I think we should be asking for a proper, professional CAR (corrective action report).
    R. Leonardo
    www.RobArtPhoto.com
    Whether he is an artist or not, the photographer is a joyous sensualist, for the simple reason that the eye traffics in feelings, not in thoughts. -Walker Evans
  • Options
    Turbo V6 CamaroTurbo V6 Camaro Registered Users Posts: 29 Big grins
    edited March 31, 2010
    gecko0 wrote:
    +1 on the abc's there. I'm a network engineer and we also are required to provide updates (every 15 min) during a critical outage. This isn't in the form of anything technical, but simply an update...even if there isn't an update to give. It's purely political at that point and gives management (or customers in this case) a better feeling about the situation.

    While I would personally love to hear the technical details of what is going on, it would not serve SM much usefullness in letting everyone know exactly what causes the outages (mainly because most would not understand enough about it to have it be a benefit to them).

    .02

    but use network geeks would like to know, hey maybe i can help Laughing.gifeek7.gifclap.gifD

    just power it off for 50 seconds then power it back on, works every-time

    ///
    Former Air Force tech Controller
    Andrew - www.wideshutphotography.com
    Cameras: D300S w/MB-D10 grip
    Primes: 35mm F1.8 AFS| 50mm F1.8D | 300mm F2.8 AFS I
    Teles/flash: TC17E II | SB400
    www.turbov6camaro.com 1997 3800 II V6 Camaro
    11.23 1/4 mile 20psi :D
  • Options
    brecklundinbrecklundin Registered Users Posts: 121 Major grins
    edited March 31, 2010
    but use network geeks would like to know, hey maybe i can help Laughing.gifeek7.gifclap.gifD

    just power it off for 50 seconds then power it back on, works every-time

    ///
    Former Air Force tech Controller

    nope it's the muffler-bearings....ya need some 30-weight oil and some gauze pads to fix it... :D
  • Options
    Turbo V6 CamaroTurbo V6 Camaro Registered Users Posts: 29 Big grins
    edited March 31, 2010
    nope it's the muffler-bearings....ya need some 30-weight oil and some gauze pads to fix it... :D
    Laughing.gif don't forget about the headlight fluid
    Andrew - www.wideshutphotography.com
    Cameras: D300S w/MB-D10 grip
    Primes: 35mm F1.8 AFS| 50mm F1.8D | 300mm F2.8 AFS I
    Teles/flash: TC17E II | SB400
    www.turbov6camaro.com 1997 3800 II V6 Camaro
    11.23 1/4 mile 20psi :D
  • Options
    JAGJAG Super Moderators Posts: 9,088 moderator
    edited March 31, 2010
    nope it's the muffler-bearings....ya need some 30-weight oil and some gauze pads to fix it... :D

    are you guys talking football again??headscratch.gifrolleyes1.gif
  • Options
    Uncle FrankUncle Frank Registered Users Posts: 17 Big grins
    edited March 31, 2010
    I'm a photographer, not a mushroom.
    Andy wrote:
    Yes, we're having some Ops issues this week - overall our availability and performance has been amazingly high. I'm in your shoes, too, and I promise that we'll get better!

    Andy, I realize you're in a tough position, but you really can't make that commitment. It can only come from someone who knows what the exact problem is and has the resources and savvy to fix it.

    Changing hosting sites is a giant PITA. I put up with outages from pBase for 8 years before I was convinced I needed a high performance professional site. But I'm beginning to wonder if I picked the right one. I'm not ready to bolt yet, but you can be assured I'm not recommending Smugmug to anyone looking for a hosting site right now.

    It would be really cool if Don issued a state of the union address after the current crisis is over, and discussed the cause of the outages we've been experiencing and his solution. I'd like be on your side and to be optimistic, but it's hard to do when I'm getting the Mushroom Treatment.
    Warm regards,
    Frank
  • Options
    brecklundinbrecklundin Registered Users Posts: 121 Major grins
    edited March 31, 2010
    Laughing.gif don't forget about the headlight fluid

    hahahaha....thanks because I compeltely overlooked that possibility. I hear Ronson makes the "best" headlight fluid.
  • Options
    OscarcOscarc Registered Users Posts: 46 Big grins
    edited March 31, 2010
    R.Leonardo wrote:
    I think we should be asking for a proper, professional CAR (corrective action report).

    Seconded. I get one from my colo provider after each and every issue; SM must do the same if it wants to play in the big leagues.
  • Options
    AndyAndy Registered Users Posts: 50,016 Major grins
    edited March 31, 2010
    funcrunch wrote:
    Andy posted:

    I'm sorry but I really do not consider the large number of downtimes SmugMug has suffered this week - not to mention in previous months - qualifies as "amazingly high" availability and performance. I'm a pro photog, put up two galleries with hundreds of photos this week, managed to get two sales before the downtime came again.

    I'm glad to have status updates more frequently than in the past, and customer support is very responsive. But something is going very wrong with the underlying infrastructure of the system. This host is providing services on a professional level, at professional prices, but operating like a Mom-and-Pop operation. I understand that the new features being put out are independent of the operations issues as different staff are involved. Regardless, more resources must be devoted to site stability and uptime or it will not be a viable site for pros to host and sell photos.
    We've increased our Ops team just recently.

    My earlier statement was meant to be - overall, over the years. It always looks bad in a bad week - and believe me, we know it.

    We're working at it.
  • Options
    AndyAndy Registered Users Posts: 50,016 Major grins
    edited March 31, 2010
    Jeffro wrote:
    Everytime this happens it scares the crap out of me...to even think that all of my images are gone, makes me...:cry
    Your images are safe.
  • Options
    AndyAndy Registered Users Posts: 50,016 Major grins
    edited April 1, 2010
    ssklar wrote:
    I don't want to jump on the dog pile here, but the frequent outages are of concern for those of us relying on SmugMug for some portion of our livelihood.

    I think that more frequent updates to the status blog might at least help the the feeling of "WTF" ... I'm a systems administrator, and in my organization, we've found that letting the users know that (a) we know there is a possible problem, and (b) we're working on it, and (c) regular updates (even if just one line) throughout the outage up until the resolution, lead to much happier users.
    More than what we do now would cause me to interrupt those that are working to fix things. That'd be bad.
  • Options
    OscarcOscarc Registered Users Posts: 46 Big grins
    edited April 1, 2010
    Andy wrote:
    overall our availability and performance has been amazingly high.

    Stats, please. What has SmugMug's availability been over the past month, over the past quarter, and over the past year? Is it closer to 99% or 99.9%?

    While we're at it, let's discuss SLAs for Pro accounts. They're fairly common in mission critical environments; SmugMug should offer them, with credits. For example, here's what Google promises for their paid Google Apps customers: http://www.google.com/apps/intl/en/terms/sla.html
  • Options
    R.LeonardoR.Leonardo Registered Users Posts: 188 Major grins
    edited April 1, 2010
    Oscarc wrote:
    Stats, please. What has SmugMug's availability been over the past month, over the past quarter, and over the past year? Is it closer to 99% or 99.9%?

    While we're at it, let's discuss SLAs for Pro accounts. They're fairly common in mission critical environments; SmugMug should offer them, with credits. For example, here's what Google promises for their paid Google Apps customers: http://www.google.com/apps/intl/en/terms/sla.html

    So that is one SLA and a CAR request for customers.
    R. Leonardo
    www.RobArtPhoto.com
    Whether he is an artist or not, the photographer is a joyous sensualist, for the simple reason that the eye traffics in feelings, not in thoughts. -Walker Evans
  • Options
    ssklarssklar Registered Users Posts: 22 Big grins
    edited April 1, 2010
    Oscarc wrote:
    Stats, please. What has SmugMug's availability been over the past month, over the past quarter, and over the past year? Is it closer to 99% or 99.9%?

    While we're at it, let's discuss SLAs for Pro accounts. They're fairly common in mission critical environments; SmugMug should offer them, with credits. For example, here's what Google promises for their paid Google Apps customers: http://www.google.com/apps/intl/en/terms/sla.html

    +1. By my watch, this outage has now lasted about an hour (minimum, that's when I tried to upload my latest shoot.) Still not a word on the status blog. C'mon guys ... please try!
  • Options
    capturingimagescapturingimages Registered Users Posts: 66 Big grins
    edited April 1, 2010
    I was told by a client a few hours ago that she checked my website to find that it wasn't working, which is always a delight to hear from a client, and although the site is now somewhat working, the front screen image has disappeared and doesn't load in the gallery it was previously in, and I can't log in or upload a new image to replace it apparently because of the maintenance... so any clients who visit over the next few hours will be greeted by an empty front screen with a few links and a mesage saying 'Select a Photo or Video and it will be displayed here'. Is that right? I'm expecting email correspondence from international stage companies based on them viewing my website. Not at all happy.
  • Options
    AndyAndy Registered Users Posts: 50,016 Major grins
    edited April 1, 2010
    ssklar wrote:
    +1. By my watch, this outage has now lasted about an hour (minimum, that's when I tried to upload my latest shoot.) Still not a word on the status blog. C'mon guys ... please try!
    Hi, the blog's been updated since it began tonight.
    http://status.blogs.smugmug.com/
  • Options
    geohsiageohsia Registered Users Posts: 31 Big grins
    edited April 1, 2010
    Andy wrote:
    More than what we do now would cause me to interrupt those that are working to fix things. That'd be bad.

    You need a better fall back plan.

    When the site is having significant issues like its having now you need to divert all traffic from the website to a page that informs users that there is a problem, and you have to replace all images with an image indicating there are issues that will be resolved shortly.

    Watching the site gimp along in a half working fashion is simply unprofessional. Please consider this as you plan for the next outage.

    Because usage of the site includes more than just surfing to *.smugmug.com sites pulling images need to be properly informed as well.
Sign In or Register to comment.