January 28th - Site Down

jfriend

Can't reach either of my sites for the last 5 minutes (one site is in my sig). DGrin, dpreview, yahoo, cnn and other internet sites working fine. Tracert to my smugmug looks normal.

Pixoul

jfriend wrote:

Can't reach either of my sites for the last 5 minutes (one site is in my sig). DGrin, dpreview, yahoo, cnn and other internet sites working fine. Tracert to my smugmug looks normal.

Ha, looks like they're down. And I'm still in my Pro trial!

chaberlack

jfriend wrote:

Can't reach either of my sites for the last 5 minutes (one site is in my sig). DGrin, dpreview, yahoo, cnn and other internet sites working fine. Tracert to my smugmug looks normal.

Yep, its down. At least I can't get to my site either.

JWilbur26

it must be...

I am getting "Http/1.1 Service Unavailable"

Pixoul

OK, I can now hit SM but someone else's gallery is coming up under my domain. http://photos.pixoul.com/ is showing up as either "macaskill's photos" (not me) or "Http/1.1 Service Unavailable".

CleeFelty

I can't get on either. Hope it's back up soon or I'm going to be up LATE uploading. I've been on Smugmug for over three years - we've been down a whole lot as of late.

*Just checked before hitting SUBMIT and got a Http/1.1 Service Unavailable

Didn't this happen last week, too?

Pixoul

And there's the official notice...

Temporary Service Interruption
The site you are trying to access is hosted by SmugMug.

We're having some temporary technical difficulties. We're working on them at the moment, and expect service to return shortly. We apologize for the inconvenience.

We're not happy about it, of course, but we are prepared for it and expect there to be no data loss or any long-term reduction in service.

Thanks for your patience and understanding!

yoyoboy

Sorry guys, it's all my fault, trying to upload 8 gigs of photos.

But seriously, whats going on?

2K9s4me

Pixoul wrote:

And there's the official notice...

boy was I ever shocked to see that notice......I was at my site not 1/2 hour ago and it was fine........

am sure they will have it up and running soon......

Manny

Must have just happened... I was uploading a few minutes ago :-(

Too bad, I was gonna enter some pics into a forum. Hopefully we'll be back before I go to beddy bye :-)

MG

j-bo

yeup... happened 2 weeks ago for sure! Now again. Makes it hard after I pronounce all photo's uploading. Bummer.

I'm only uploading 3gb's this time. Seems like Sundays are "the" day for a lot of event people to upload. Hope they get it figured out!

Oh well.. just a day, like before.

The only bummer is....... my uploads are half way through. Last time, it showed "complete" and only 15 photo's of 256 actually uploaded. Guess I'll just cancel the whole thing and start fresh in the morning.

oddstuff

Service Availability
Does Smugmug have an "uptime guarantee" or "service availability" clause anywhere in the terms? The frequency of downtimes seems to be increasing lately.

Honestly, is it under DDOS or something?

onethumb

Core Master Database failure
Hi guys,

Can't talk long, too busy trying to get your sites back up.

One of our core master databases has a data corruption error (possibly related to the problems we were having a week or so ago), and is having trouble coming back up.

I'm moving one of it's backups into place as I type this, and hopefully we can get service restored shortly.

Sorry for the downtime, and I'll post more if/when I have it.

Don

JWilbur26

yoyoboy wrote:

Sorry guys, it's all my fault, trying to upload 8 gigs of photos.

But seriously, whats going on?

8gigs??? Unrelated to the outage...but what is your site. Interested in seeing what you have for 8 gigs worth.

beline

your moving a backup in... soo, high chance of a loss of photos then I assume? awesome.

and for those of you mentioning the frequency of downtime. unfortunately, it sucks, but if you figure 99% uptime guarantee, that leaves 3.65 days of downtime... 99.5% = 1.825 days. and those are pretty reasonable numbers... when you consider just how much data is being thrown around, you kind of have to understand.

also... why are you uploading so much data? create a preset in Aperture for PROOF photos, delay shipping by 7 days, and upload full sized images when needed...

onethumb

beline wrote:

your moving a backup in... soo, high chance of a loss of photos then I assume? awesome.

and for those of you mentioning the frequency of downtime. unfortunately, it sucks, but if you figure 99% uptime guarantee, that leaves 3.65 days of downtime... 99.5% = 1.825 days. and those are pretty reasonable numbers... when you consider just how much data is being thrown around, you kind of have to understand.

also... why are you uploading so much data? create a preset in Aperture for PROOF photos, delay shipping by 7 days, and upload full sized images when needed...

No, I doubt we'll have any loss of photos or any other data. The backup I'm moving into place is a realtime replicated live backup. It should be up-to-the-second. But I'll know more soon.

Don

Pixoul

onethumb wrote:

No, I doubt we'll have any loss of photos or any other data. The backup I'm moving into place is a realtime replicated live backup. It should be up-to-the-second. But I'll know more soon.

I'm still in my Pro trial (8 days left) but I'm highly impressed with the fact that you guys are actually on the Forums to give updates and help. I run the Pepper Pad Community Forums so I know what it's like to try to balance Forums with work.

Thanks for the quick update...now get back to work!

Andy

oddstuff wrote:

Does Smugmug have an "uptime guarantee" or "service availability" clause anywhere in the terms? The frequency of downtimes seems to be increasing lately.

We do not, but that doesn't mean we're happy about this, either. It sucks for all of us. Onethumb and his team are working constantly to make the site as highly available as possible. The problems this month, have been database related, and proving to be particularly difficult to overcome. But overcome it, they will.

Thanks for your patience, and sorry for the hassle.

yoyoboy

beline wrote:

also... why are you uploading so much data? create a preset in Aperture for PROOF photos, delay shipping by 7 days, and upload full sized images when needed...

Backup.

beline

then if it is "up-to-the-second" it will still be corrupt, which would explain what it is happening again.

you said yourself that the original was corrupt. why would you want to use a backup of a corrupted database? that what a backup is for, to get back to the point before the corruption... I think...

Andy

Pixoul wrote:

I'm still in my Pro trial (8 days left) but I'm highly impressed with the fact that you guys are actually on the Forums to give updates and help. I run the Pepper Pad Community Forums so I know what it's like to try to balance Forums with work.

Thanks for the quick update...now get back to work!

Thanks Pix - holler if you need anything at all.

onethumb

beline wrote:

then if it is "up-to-the-second" it will still be corrupt, which would explain what it is happening again.

you said yourself that the original was corrupt. why would you want to use a backup of a corrupted database? that what a backup is for, to get back to the point before the corruption... I think...

The original mangled one of it's disks, but it's not related to replication. Or, at least, at this point doesn't appear to be.

Restoring from an old backup would take a lot more time, and since I'm not seeing any evidence of corruption in any of the replicated copies, we're going with a live copy for now.

Thanks for your interest, though.

Don

beline

yoyoboy wrote:

Backup.

Ah... Well, I must say, you picked a stable server...

I'm just poking fun. No hard feelings, I just like to pet my NAS backups... :baldy

CleeFelty

I ADORE Smugmug, and the amazing staff. There is no organization/community I have ever come across that has this level of service and support.

That said, it bums me out that we've hit this rocky road. I use the site for personal photos, but can certainly empathize with someone trying to make their living here. Maybe we've been spoiled for too long?

Thank you Don for keeping us posted. I imagine this isn't what you planned on spending your Sunday working on.

beline

onethumb wrote:

The original mangled one of it's disks, but it's not related to replication. Or, at least, at this point doesn't appear to be.

Restoring from an old backup would take a lot more time, and since I'm not seeing any evidence of corruption in any of the replicated copies, we're going with a live copy for now.

Thanks for your interest, though.

Don

Ah... Ok, now I follow. My bad.

Well, thank you for your help and hard work.

I'm going down to the casino floor. I'll have a drink for you.

JWilbur26

I agree...everyone here is so helpful. I wish I could get our IT team at work to react as quickly as the smugmug folks do. you guys do a great job.

djs@usa.com

What tha?!
Now, I'm not one to complain (lie) and I'm amazed that SmugMug works at all (truth), but I do hope that this "downtime thing" is not going to be something that will soon be considered normal.

quark

I personally sleep easily knowing my photos are in good hands and interruptions are always minor. The smugmug team has the best uptime of any site I have ever worked with. At one time I even maintained my own server and it drove me crazy how often I had to drop everything to go fix stuff. I don't think we pay these guys close to enough to deal with these headaches for us.

KvPhoto

Any ideas on when this will be fixed?

Andy

KvPhoto wrote:

Any ideas on when this will be fixed?

Don will post when he knows. Sorry I don't have a better answer for you.

evoryware

KvPhoto wrote:

Any ideas on when this will be fixed?

bleh I was in the middle of explaining to Verizon that I'm getting 256kbps instead of my normal 1500kbps and was going to show them by downloading some of my originals from smugmug (among other things). Have to download files somewhere else I guess... :cry