January 28th - Site Down

2

Comments

  • KvPhotoKvPhoto Registered Users Posts: 364 Major grins
    edited January 28, 2007
    No problem Andy. I posted before reading the last page of posts ;)
  • SouwalkerSouwalker Registered Users Posts: 245 Major grins
    edited January 28, 2007
    It's back up now.
    pat
  • DodgeV83DodgeV83 Registered Users Posts: 379 Major grins
    edited January 28, 2007
    and we're back!
  • DJKennedyDJKennedy Registered Users Posts: 555 Major grins
    edited January 28, 2007
    KvPhoto wrote:
    No problem Andy. I posted before reading the last page of posts ;)


    Seems back up...:16 past the hours
    http://www.djkennedy.com

    What did Cinderella say when she left the photo shop? "One day my prints will come."

  • SystemSystem Registered Users Posts: 8,186 moderator
    edited January 28, 2007
    Concur; I'm back up too. Nice pickup, guys
  • belinebeline Registered Users Posts: 90 Big grins
    edited January 28, 2007
    hell yeah
    Covering It at D:D
    coverthis.smugmug.com

    SmugWP -
    now 30% more smug!

    No Steam Shovel

    Feature Requests:
    External Slide Show Support
    Black/White and Sepia Color Options
    Photo/Discount Packages/Coupons
    Electronic Payout for Pro accounts via PayPal
    Self-Fulfilment (Custom "Items")
  • dancinkatedancinkate Registered Users Posts: 267 Major grins
    edited January 28, 2007
    clap.gif
  • CleeFeltyCleeFelty Registered Users Posts: 35 Big grins
    edited January 28, 2007
    Thank you!
  • onethumbonethumb Administrators Posts: 1,269 Major grins
    edited January 28, 2007
    KvPhoto wrote:
    Any ideas on when this will be fixed?

    We are back up.

    The usual caveats apply: I believe we're fine, everything looks good, but you never know. Please bear with us if the patient requires more surgery.

    For the technically curious, I do have a snapshot of both the "dead" server and a live replica frozen in time at the exact second of death, so I'll be analyzing what's up in the hopes of preventing it in the future. No idea if it'll actually shed some light on anything, though. :)

    Don
  • onethumbonethumb Administrators Posts: 1,269 Major grins
    edited January 28, 2007
    onethumb wrote:
    The original mangled one of it's disks, but it's not related to replication. Or, at least, at this point doesn't appear to be.

    Restoring from an old backup would take a lot more time, and since I'm not seeing any evidence of corruption in any of the replicated copies, we're going with a live copy for now.

    Thanks for your interest, though.

    Don

    I just got accused by one of our customers (in private) of being condescending and "a snot" because of this post. Since I'm not a private kinda guy (I think you, our paying customers, deserve to get as much information and transparency out of me as possible, hence this forum and my blog), I hit reply to apologize publicly. I was sure that, in the heat of trying to fix our major problem, I had inadvertently snapped at someone and said something I shouldn't.

    Alas, though, I can't for the life of me see how that post was condescending, and certainly not how I was being "a snot." Further, I'm not sure I've ever seen the CEO of any of the companies I pay for actually get down and dirty about exactly what the problems are that the company is dealing with. To me, that's truly being condescending - not taking the time or caring enough about ones' customers to keep them informed.

    For the record, I was truly happy to see some of customers interested in our problems and thinking about the technical ramifications. The geek in me likes to see other geeks thinking about our problems, too - it keeps us on our toes. :)

    Can someone explain to me if I was being condescending, and if so, how, so I can make a proper apology?

    Thanks!

    Don
  • DJKennedyDJKennedy Registered Users Posts: 555 Major grins
    edited January 28, 2007
    A snot??

    I for one did NOT find it condesending in any way

    Derek

    EDIT: Sorry Don - I find the snot remark kinda funny. Rude...but kinda funny
    http://www.djkennedy.com

    What did Cinderella say when she left the photo shop? "One day my prints will come."

  • Iron CreekIron Creek Registered Users Posts: 59 Big grins
    edited January 28, 2007
    Onethumb - I for one have never had any problems with any post that you have ever posted (have I made my point?). Thank you very much for keeping us (me) posted. It's very easy for some people to find fault with others. You offer a (very good) service for which people pay (at a very reasonal cost) for. Things happen and they need fixing. I pay a lot more for cable than I do for smugmug and I don't get near the service from them that I do from you guys.

    Once again thank you for all you've done.
    Don Libby,
    Iron Creek Photography
    httP://www.Ironcreekphotography.com
    Tucson, Arizona

    Member NAPP; PPA
  • SystemSystem Registered Users Posts: 8,186 moderator
    edited January 28, 2007
    I'll step up. It was me. I'd have PMd it to you directly, but, well... we can't.

    "Thanks for your interest, though".

    That's the comment that set me off.

    I wasn't at all impressed with the tone it held for me. Perhaps I'm just provoked because I've been being handwaved here on the forums all week on 3 or 4 different topics.

    But I've been on the net a long time, and I know that transparency is hard to do. And I know *that* it is hard to do because it's painful, and that's mostly because Not All The Smart People Work For You.

    And, for my tastes, I've seen a lot of translucence the last few weeks, and not much *actual* transparency. I've had to do it myself; I know what it looks like -- I was chief engineer for a local ISP back in 1996, when we had to run 60 modems off of 768Kbs of uplink. Things didn't always go well. But my client base was largely techies, and they wouldn't *let* me get away with glossing and handwaving.


    I'm not the one who decided to run your company in public.

    I'm just the one who decided to give you money.

    You keep saying you want to hear what we have to say. But the reactions I've seen lately suggest that ... I don't actually know what they suggest. They suggest to me that maybe I've made a mistake.

    I don't like that feeling.

    Think about it over night, read the other things I've had to say this week, and tell me why I'm wrong.

    Or tell me I'm a kook. :-)

    Oh, and get some sleep.

    And contemplate that if this upgrade had been opt-in by client, perhaps you wouldn't have had to stay up all week?
  • DJKennedyDJKennedy Registered Users Posts: 555 Major grins
    edited January 28, 2007
    Biggest problem with the net (email, blogs, web pages, forums) is that it's easy to read into the text something that isnt there. I'm refering to tone of voice.

    SInce there is no 'voice', it's easy to read one into the text that wasn't actaully there.
    http://www.djkennedy.com

    What did Cinderella say when she left the photo shop? "One day my prints will come."

  • jfriendjfriend Registered Users Posts: 8,097 Major grins
    edited January 29, 2007
    No problem here
    onethumb wrote:
    I just got accused by one of our customers (in private) of being condescending and "a snot" because of this post.

    I didn't see or feel anything condescending. I thought you were just being frank and tellin' it like it is and obviously didn't have a lot of time to talk. No problem here.
    --John
    HomepagePopular
    JFriend's javascript customizationsSecrets for getting fast answers on Dgrin
    Always include a link to your site when posting a question
  • BaldyBaldy Registered Users, Super Moderators Posts: 2,853 moderator
    edited January 29, 2007
    imageinc wrote:
    "Thanks for your interest, though".

    That's the comment that set me off.
    In all honestly, I think he was grateful that you were interested.
  • onethumbonethumb Administrators Posts: 1,269 Major grins
    edited January 29, 2007
    imageinc wrote:
    I'll step up. It was me. I'd have PMd it to you directly, but, well... we can't.

    You have the direct email to both Chris and I, the founders of the company. It's sent to all new customers, and it's posted publicly on the website. We read each and reply to each one we get.

    PM'ing is far less efficient and convenient for me, and I'm far less quick-on-the-draw with PMs, so I turned them off. Just send me an email.

    It's founders at smugmug.
    imageinc wrote:
    Think about it over night, read the other things I've had to say this week, and tell me why I'm wrong.

    Or tell me I'm a kook. :-)

    Oh, and get some sleep.

    And contemplate that if this upgrade had been opt-in by client, perhaps you wouldn't have had to stay up all week?

    I'll try to get some time to read the other things you've said this week, but it may take me awhile. A summary post or email would help greatly, I'd think.

    I don't know about you, but I've been getting plenty of sleep all week. This week was great - we had a great new release, fueled by thousands of our customers giving us feedback over the last few months, and I've slept quite well.

    Every single customer had plenty of time to comment on the new release, and while I may have missed it, I don't believe anyone asked for an "opt-in" feature. Certainly not many.

    I'm sorry if SmugMug is no longer the place for you, and I certainly hope that's not the case, but I refuse to halt our forward progress for a few people who don't like change. The vast majority of our customers are ecstatic about the new release, and our signups have gone up dramatically as a result.

    We're getting flooded with emails from people telling us how they can finally get the rest of their family signed up for SmugMug because it's now [easy enough | pretty enough | fast enough] for them to recommend.

    The way is onward and upward. If it wasn't, SmugMug would have a single super-happy customer and everyone else would be elsewhere. That's not our destiny.

    Don
  • onethumbonethumb Administrators Posts: 1,269 Major grins
    edited January 29, 2007
    Baldy wrote:
    In all honestly, I think he was grateful that you were interested.

    I was (though it was someone else who was interested). :(

    Don
  • yoyoboyyoyoboy Registered Users Posts: 42 Big grins
    edited January 29, 2007
    imageinc wrote:
    Or tell me I'm a kook. :-)
    Yup you're a kook:D
    You've go some nice photos btw.
  • oddstuffoddstuff Registered Users Posts: 54 Big grins
    edited January 29, 2007
    The fix was a quickie. My site was up before I could finish reading the threads here. Cheers to Smugmug engineering team. thumb.gif

    Would it be possible for Smugmug to issue email to Smugmug users to notify all on expected/unexpected downtimes besides having them posted inside Smugmug? I mean, if a downtime hits, we can't get into Smugmug to be aware of it - can preempt/save a lot of "helpdesk" calls. I appreciate the strength of Smugmug's anti-spam policy, but mails on server issues would be helpful than anything, and should not be deemed spamful by virtue of it being sent. Just a wild thought.


    Anicca - the Theory of Impermanence: camera changes, photos stay (in Smugmug).
    About me - one of the the junior siblings in Dgrin family (DOB May 2006).
  • PixoulPixoul Registered Users Posts: 97 Big grins
    edited January 29, 2007
    onethumb wrote:
    Can someone explain to me if I was being condescending, and if so, how, so I can make a proper apology?

    I'm the new guy here so I don't know many of the regulars, but all I saw in your post was a geek trying to put out a fire at 11:00 PM on a Sunday evening but still taking the time to give a quick explanation after being poked at with sticks by the natives. rolleyes1.gif
  • PixoulPixoul Registered Users Posts: 97 Big grins
    edited January 29, 2007
    Looks like we may be having similar problems again...I just saw the same incorrect person's gallery pop up when I looked at photos.pixoul.com.
  • cabbeycabbey Registered Users Posts: 1,053 Major grins
    edited January 29, 2007
    Pixoul wrote:
    Looks like we may be having similar problems again...I just saw the same incorrect person's gallery pop up when I looked at photos.pixoul.com.

    If so it didn't last long, I see your images.
    SmugMug Sorcerer - Engineering Team Champion for Commerce, Finance, Security, and Data Support
    http://wall-art.smugmug.com/
  • KvPhotoKvPhoto Registered Users Posts: 364 Major grins
    edited January 29, 2007
    Don,
    I just wanted to say that I think you guys have done a phenomenol (sic) job with Smugmug. I will admit the downtime has been inconvenient at times and at first I was not thrilled with some of the changes but we must progress. If it was not for progress we would all still be running to get our rolls of film developed!
  • ThusieThusie Registered Users Posts: 1,818 Major grins
    edited January 29, 2007
    I probably should keep my mouth shut..but I'm not.

    I have been on Sm 3 years now and yes there probably has been more down time lately than use to be and yes it can be frustrating. BUT SM runs on computers and last time I checked no matter how careful you are computers can have issues. Look at our ISPs and our own personal computers, ISP's do an upgrade and things can tank, we do an upgrade on our PCs and things can go haywire, it is the nature of the beast.

    Tons of folks use SM and Sm has been upgrading to handle the load and improve service (seems like quite a bit of that has been going on lately) and that opens up the potential for 'issues'. Computers are computers and the folks at SM are human and yes we do pay for the service. That said we pay for the electric that comes into our homes, water, gas, cable, phone and not one gives 100% uptime and SM is way better than AEP:D
  • HCPHCP Registered Users Posts: 41 Big grins
    edited January 29, 2007
    Need to reload gallery after downtime last night
    Hi all,

    Seems like no one else had this happen but me... I am mentioning it just in case I am not the only one, just the only one reporting it. Also- should I be reporting this to the help desk as well as an FYI, or is mentioning here enough?

    Last night I created a new gallery on hcphoto.smugmug.com
    category: Winter Training Camps
    subcategory: Discovery Channel
    gallery: Solvang - January 2007

    I uploaded using the IE uploader, uploaded 23 files, and smugmug replied "Upload Complete" and then the site went down. I checked this forum, saw that others had reported the down problem. I am on the road and wasn't able to check the site again until this morning.

    I found an empty gallery and so needed to reload the images (they are up there now).

    Anyway- thought I would report this since an earlier message from Don in this thread indicated that all should be available via the backup disk. I must have just had incredibly bad timing with my upload. ne_nau.gif
    hcphoto.smugmug.com
  • AndyAndy Registered Users Posts: 50,016 Major grins
    edited January 29, 2007
    HCP wrote:
    Anyway- thought I would report this since an earlier message from Don in this thread indicated that all should be available via the backup disk. I must have just had incredibly bad timing with my upload. ne_nau.gif
    That's entirely possible, sorry for the hassle.
  • PixoulPixoul Registered Users Posts: 97 Big grins
    edited January 29, 2007
    cabbey wrote:
    If so it didn't last long, I see your images.

    It appears to be intermittent. I haven't seen the problem since 2:15 AM Eastern this morning but it was happening after the site came back after the database restore. Here's a screenshot of the incorrect gallery showing on photos.pixoul.com:

    126217848-L.jpg
  • PixoulPixoul Registered Users Posts: 97 Big grins
    edited January 29, 2007
    HCP wrote:
    I found an empty gallery and so needed to reload the images (they are up there now).

    I did an upload last night and there was one missing image from the following gallery: http://photos.pixoul.com/gallery/2407061

    I haven't figured out which image is missing yet; I just noticed that the gallery is reporting 63 images while I know I uploaded 64 images. headscratch.gif
  • AndyAndy Registered Users Posts: 50,016 Major grins
    edited January 29, 2007
    Pixoul wrote:
    It appears to be intermittent. I haven't seen the problem since 2:15 AM Eastern this morning but it was happening after the site came back after the database restore. Here's a screenshot of the incorrect gallery showing on photos.pixoul.com:

    126217848-Th.jpg

    Try emptying your Safari Cache :) I can see your site just fiine,
    I see you wrote the help desk too - that causes us double work, if you don't mind in the future, stick to one or the other :D thanks wave.gif
Sign In or Register to comment.