System issue updates

Baldy

First, a disclaimer : I haven't had time to go through these threads because I've been immersed in upload issues, but for people who want to hear my personal perspective (which could be different from others at SmugMug), in the spirit of openness, here goes:

1. We installed new database servers some months back and most people loved the performance boost. They were big, gnarly, power-hungry, and expensive -- but worth it.

With the stunning growth in images, users and the increasing demands new features place on database servers, it was surprising how quickly we began to load those servers. So we went to order a lot of new ones.

The issue is the manufacturer was on the verge of releasing the new hotness. You know the story -- faster processors & disks & more memory -- much better. The question was to settle for the old or hold for the new.

We held a few more weeks to get the new hotness for the long term good.

2. Memorial weekend happened. Aaaiiieee we got hit with volume. And if an upload takes time to process, customers automatically upload it again (and again), causing the situation to spiral.

3. Something happened with uploaders that I'm not sure we completely understand because we can't replicate it on our test servers, only on the live site. That suggested to me that it was load-related. It was very hard to diagnose because for the customers who were reporting the issue, it generally wasn't repeatable and was sometimes mixed in with Comcast's packet loss problems or switching to a new uploader.

We knew a significant part had to do with us and we've been immersed in what it could be. I get the feeling that today has been good for uploading, no? So perhaps we've got it and Don may be able to say with authority what it was.

We're proceeding with the new hotness servers as quickly as the manufacturer can get them to us. We're sickened by the problems some of you saw and hope never to see them again.

Back to work for me and a hundred lashes.

:whip:whip:whip

Thanks,
Chris

I Simonius

Baldy wrote:

We're proceeding with the new hotness servers as quickly as the manufacturer can get them to us. We're sickened by the problems some of you saw and hope never to see them again.

Back to work for me and a hundred lashes.

:whip:whip:whip

Thanks,
Chris

anyone can have problems and as long as they are flat out resolving them then no more can really be asked - as long as solutions are found within a reasonable time

There is certainly no point in jumping ship because if Smugmug can have the problems then so can anyone else. On top of that the smugmug openness and customer service are top notch and they will learn more with time not less as how to avaoid problems

So just my POV of how I take glitches, even major ones.

however - Where I am concerned - Being in the UK, is the question of slow access to smugmug generally, for my self and my friends + clients, that ISN'T getting addressed ( probably not for lack of trying i.e. problems beyond SMs control?). I have friends in this country and abroad who have all complained of the very slow access to my pics:cry. Sadly if that issue continues SM won't be for me long term.:cry :cry :cry

colourbox

Baldy wrote:

I get the feeling that today has been good for uploading, no? So perhaps we've got it and Don may be able to say with authority what it was.

All I can tell you is that I did not do any uploads during the unfortunate slow period, but I uploaded 40 or so images last night and it went very smoothly and without incident.

Ben

Simon King wrote:

however - Where I am concerned - Being in the UK, is the question of slow access to smugmug generally, for my self and my friends + clients, that ISN'T getting addressed ( probably not for lack of trying i.e. problems beyond SMs control?). I have friends in this country and abroad who have all complained of the very slow access to my pics:cry. Sadly if that issue continues SM won't be for me long term.:cry :cry :cry

Hi Simon,

The first step is to see whether it is networking related, or something else. Would you mind posting a traceroute from where you are to our servers? Just hit "Start" and then 'Run" in windows, and type "command". When the command prompt opens, type "tracert smugmug.com" and then use the c:\ icon to open a menu and copy and then paste it over here. That is a good place to get started figuring out what is going wrong.

Thanks!

-Ben

Cason

SmugMug:

I too know what it is like to have a DB or server go down in production and everybody crawls you for it. The customers want answers and the fix right away. Of course you know that slows you down even more. Things happen. People have to get over it.

I wanted to take the time to let you know that the business you are running is outstanding, the help from Team SmugMug is awesome.

Thank you for taking the time to explain what happened and all that you do to help us share our photos.

Now give me a hug.

I Simonius

Ben wrote:

Hi Simon,

The first step is to see whether it is networking related, or something else. Would you mind posting a traceroute from where you are to our servers? Just hit "Start" and then 'Run" in windows, and type "command". When the command prompt opens, type "tracert smugmug.com" and then use the c:\ icon to open a menu and copy and then paste it over here. That is a good place to get started figuring out what is going wrong.

Thanks!

-Ben

tried that before;

http://www.dgrin.com/showpost.php?p=288980&postcount=13

Nothing changed since then (beginning of April)

any thoughts?

Ben

Simon King wrote:

tried that before;

http://www.dgrin.com/showpost.php?p=288980&postcount=13

Nothing changed since then (beginning of April)

any thoughts?

Hi Simon,

I missed that thread, but I will grab Wireless and see what we can come up with. He is out of the office today, but give us a day or two and hopefully I can get a better answer for you.

-Ben

Wireless

Ben wrote:

Hi Simon,

I missed that thread, but I will grab Wireless and see what we can come up with. He is out of the office today, but give us a day or two and hopefully I can get a better answer for you.

-Ben

I never did get follow-up on the Limelight/BTN issue in the London area, and I believe our local ISP closed the case. I'll move back over to that other thread and follow-up from there.

Andy

RTP wrote:

Now give me a hug.

:feelgood

Thanks for the very kind words - everyone here appreciates that!

Baldy

RTP wrote:

Now give me a hug.

And a kiss?