Stuck in processing - when will the problem be fixed?

Sventekoz

I'd very much like to know when this long-standing problem is going to be fixed once and for all. It's certainly a part of Legacy SmugMug that I can do without. I shouldn't have to re-load and delete images almost every single time.

mbonocore

Can you please provide me with Your OS, Browser, and system information? We can't really recreate this, and from many reports, if it does process, it always does wind up finishing, there is no need to delete and reload. Thanks!

Sventekoz

I use Windows 8, and Firefox/Chrome/IE (i.e. it doesn't make a difference). What sort of system information are you after?

Lou Gonzalez

I have been having this same issue for YEARS. Something like 2 or 3 years at least. It's annoying as heck. I JUST uploaded 250 files and 6 are stuck processing. Way back in the day it never did this. All your photos were just there 1st time around. No re-uploading... but now it happens almost 100% of the time. I have to go through all my galleries and check them. What a waste of time. Some of my "processing" images NEVER materialize. They're just stuck in limbo forever. Oy.

Funny thing is I came here just now to check on this very issue and here we are still talking about. Frustrating to say the least.

I use Win 7 with FF and Chrome. I experience the same issues with both browsers. And again, it has been happening for a very long time. Its about time SM squashed this long standing bug.

RKnecht

Having this issue as well. It takes me about 4 times when uploading a proof to actually have it show up.

denisegoldberg

I'm consistently having this problem as well. Sometimes the stuck in processing images complete, sometimes not. I can't understand why I can upload a set of photos (the number doesn't seem to matter) and one or two either don't complete or take much longer than the others.

I'm running on Windows 8.1, have tried IE, Chrome, and Firefox.
Same problem occurred on Windows 8.

--- Denise

Ferguson

I have this problem almost every time I upload, one or a few will take 10 to 100 times as long as the others. It is truly annoying, as it thows off numbering (e.g. "look at #132" which for me is 132, but for someone else it's 129 because 3 are processing). It makes it very difficult to use Smugmug as a reliable conveyance in near real time.

Note that I don't need it to process instantly -- I get it that sometimes it's 1 minute, sometimes 5 minutes. What is very difficult is the 1 minute for most, 30 minutes for 3.

I can't imagine how you cannot find this problem -- people have frequently provided explicit images. You MUST have data logs showing when it processed. And it's almost impossible to AVOID it happening. Most people probably don't care because they aren't trying to deliver something quickly.

The last time I pressed with the help desk I was told this was normal and expected behavior. How is that that SMugmug can't decide if it is normal or not? Sometimes you say "normal" and othertimes "we can't reproduce the problem". Very frustrating, getting people's hopes up that maybe someone is paying attention to it.

Which is it? Bug or feature?

OffTopic

Same here, almost every time I upload more than one photo, using Win7 and Chrome.
What's really annoying is that sometimes that last one hangs for 20 minutes or more and I have to keep checking every few minutes to see if it ever goes through or if I need to delete and re-upload.

phaserbeam

I do have that problem too, just not that often. Maybe once every 100 photos (Linux, unsupported Opera-browser).

If it is stuck for a few mins i just replace the photo. The replaced photo seem to be processed like a normal upload, just takes a few seconds to complete processing.

Ferguson

phaserbeam wrote: »

I do have that problem too, just not that often. Maybe once every 100 photos (Linux, unsupported Opera-browser).

That's probably about right. I typically upload from 75-150 at a time, and typically have 1-5 photos go into this state. Though yesterday it was 4 out of 10!

phaserbeam wrote: »

If it is stuck for a few mins i just replace the photo. The replaced photo seem to be processed like a normal upload, just takes a few seconds to complete processing.

Yes, but that's something of a pain with the lightroom plugin.

Hey, maybe that's a fix -- we could ask for a feature-request to have the plugin have a "republish those not processed".

picturebike

yes same here, happens all the time and is very irritating !

thenickdude

denisegoldberg wrote: »

I can't understand why I can upload a set of photos (the number doesn't seem to matter) and one or two either don't complete or take much longer than the others.

Almost certainly, the work of generating the multiple size versions of your photos is divided amongst several servers, so some servers process some of the photos, some servers process others. If one server was busier than another, it'll take longer for it to get to your job, so some of your photos will lag behind in processing.

Sorry that the explanation isn't very helpful though... either way you're still stuck waiting for your photos when you don't want to be.

Ferguson

Lamah wrote: »

Almost certainly, the work of generating the multiple size versions of your photos is divided amongst several servers, so some servers process some of the photos, some servers process others. If one server was busier than another, it'll take longer for it to get to your job, so some of your photos will lag behind in processing.

Possible, but that doesn't seem to fit the observed results. Let's say that were true, either randomly allocated or a queue for different sizes in different servers, you would still expect more consistency across lots of images, and that's the opposite of this problem.

What I see is most images take an amount of time that's a nice, seemingly normal distribution of time. Depending on how busy SM is, that might be centered around 10 seconds or 50 seconds, but it's a pretty tight grouping, then WAY out on the edge one will be 20 minutes. Or 3 will be 40 minutes.

Unless there's a HUGE number of servers with a tiny number over-committed badly (and no intelligence in queuing), I would expect the pain to be more spread out if it were that.

While we are hypothesizing, though... I've often wondered if they get an error occasionally and re-queue them with a lower priority. If I were designing something to be fault tolerant (but allow some retries), I might make sure the retries always played second to the initial tries from others. So let's say that (again, making this up) there's a deadlock that kills try #1, then it requeues it for "much later, much slower", and try #2 works but takes 20 minutes.

Something like that could explain the randomness. I think what everyone has demonstrated is that it's not the photo content at fault, I think universally people find if they delete and re-upload it goes right through.

But... why are we speculating. I really wish SM would just look at their logs and tell us!

Lou Gonzalez

^^^ +1

cfrey6

I get this still, and it's February, and this issue is still unresolved. I am on Windows 7 and it does not matter how I upload, either with your new crappy uploader, which has it's own bugs, or with the Lightroom plugin, and it does not matter which browser I try.

cfrey6

Hello? Any answers?

Sventekoz

This is never going to be fixed, is it?

Ferguson

Sventekoz wrote: »

This is never going to be fixed, is it?

Interestingly I have not seen it lately. Maybe just haven't noticed, but I was thinking the other day "maybe they finally fixed it".

Is it still happening?

I've got some for an opening game that I will be in a hurry for today. That's when it usually strikes, so I may find out then. :cry

leftquark

If you have examples of this I'd love to see some examples galleries where this issue is happening. I've uploaded thousands of photos into hundreds of galleries and never seen one be completely. Sometimes I have to refresh a few times but it always finishes processing and posts.

Ferguson

Ferguson wrote: »

I've got some for an opening game that I will be in a hurry for today. That's when it usually strikes, so I may find out then. :cry

I've done three uploads since my last posting. In the past (when I was complaining) it would never have done all three immediately, at least one would have been stuck. This time none were.

It's hard to prove a negative (that it won't get stuck), but it's certainly better.

leftquark wrote: »

If you have examples of this I'd love to see some examples galleries where this issue is happening. I've uploaded thousands of photos into hundreds of galleries and never seen one be completely. Sometimes I have to refresh a few times but it always finishes processing and posts.

I think many of us provided a lot of evidence in the past. Perhaps Smugmug could comment on whether they FIXED it or not?

Or whether anyone ever investigated the prior reports?

leftquark

Ferguson wrote: »

Perhaps Smugmug could comment on whether they FIXED it or not?

Or whether anyone ever investigated the prior reports?

We're constantly monitoring and investigating issues when users post here and email our Support Heroes. We're always trying to optimize and improve our tools for monitoring server load and being able to process your images faster.

Ferguson

leftquark wrote: »

We're constantly monitoring and investigating issues when users post here and email our Support Heroes. We're always trying to optimize and improve our tools for monitoring server load and being able to process your images faster.

Thanks, but this was a specific, old, ongoing issue. I assume you have some sort of historical record of things fixed (even if only internal), and a support desk that might list problems investigated and resolutions.

I was hoping perhaps you had a record of this problem and what its resolution was?

leftquark

Ferguson wrote: »

Thanks, but this was a specific, old, ongoing issue. I assume you have some sort of historical record of things fixed (even if only internal), and a support desk that might list problems investigated and resolutions.

I was hoping perhaps you had a record of this problem and what its resolution was?

Unfortunately I can't dive into specifics because none of the examples in this thread specify a specific date and time in which the long processing took place. However, from a more generic standpoint: A lot of the time the resolution is to just apply more CPU resources to the queue of photos to process. We now have an automated process for determining a large queue and adding additional CPU resources to clear the queue quicker.

Ferguson

leftquark wrote: »

Unfortunately I can't dive into specifics because none of the examples in this thread specify a specific date and time in which the long processing took place. However, from a more generic standpoint: A lot of the time the resolution is to just apply more CPU resources to the queue of photos to process. We now have an automated process for determining a large queue and adding additional CPU resources to clear the queue quicker.

OK, first... if the problem isn't occurring (and I don't see people jumping in with examples), it's not worth a lot of effort. And thank you for at least listening and looking now, really. But in case you want to look further for curiosity (or if you start hearing it is happening)....

Support ticket #203496 was opened by me as far back as December 2010. They "fixed" the one photo stuck at the time.

Here's a more recent thread ( Mar -> Oct 2013) that has several examples, including one where I went through every image in an upload showing how some would take 20-30 minutes (note there is a typo in the stats where it says "131 were under 10 minutes" it means seconds, should be obvious when you look).

http://www.dgrin.com/showthread.php?t=232910

This thread shows a part of the problem - SM would occasionally know it had general processing problems, but those were conflated with the one-off outliers:

http://www.dgrin.com/showthread.php?t=234491

The issue that drove people nuts were not the periodic ones where SM was just slow, but the cases where in an upload of a few dozen or hundred shots, one or a few would be 'selected' to take 20-60 minutes to process (maybe more) instead of seconds.

It had been reported off and on for literally years, frequently with very specific examples, but there was never a real "we understand why this happened and we have fixed it".

I am delighted to, in recent memory, not see it happening any more. Unless it is.... anyone?

Sventekoz

Right now, there are quite a few images stuck in processing here: http://www.metrophoto.com.au/organize/Sport/Athletics/201415-Season/21-November-2014

Sventekoz

And still one processing.

Sventekoz

Just finished.

Ferguson

Yeah, it is still doing it for me also.

I just uploaded 126 photos to this gallery: http://www.captivephotons.com/Events/Other-Events/LipscombNKU112114/

125 of them processed quickly, one is stuck as I write this. the 31st (in order) is the one seemingly stuck. I'm going to wait it out and will update with details....

Wait...... still processing at 20 minutes (most took 3-6 seconds).....

Still going at 30 minutes (not seconds, minutes)....

Woah.... at about an hour it failed. A screen shot attached shows the image ID that failed. And indeed, the image is not there -- it is as though it was removed.

So it UPLOADED fine, but would not process.

And if I had not checked, I would not know, as it shows published in Lightroom's Smugmug plugin.

I marked it for republish. It uploaded, and processed in 3 seconds per the new display (I'll show that in the next reply as I can't post two images in one reply).

So.... WHY? Exact same image, no changes made, just a mark for republish.

And more importantly -- is this happening to others who may not know -- in the past it just delayed the processing but eventually (at least for me) completed. To fail silently (who looks at the stats regularly?!?!) is really, really bad.

Ferguson

Here's the success screen. Note that the screen itself is badly formatted so the date and image key overlay each other. That's a smugmug error as best I can tell.



Here's the direct link to the image ID after upload.

http://www.captivephotons.com/Events/Other-Events/LipscombNKU112114/i-dMBfKZM/A

Leftquark -- does this give you enough to go on? Can you look at the log from this image and figure out where it went wrong, and why it worked perfectly the second time (meaning it's clearly not content related, at least in a reproducible way).

Also note another bug -- the screen shot shows the "all uploads". The error is NOT in the "all". Shouldn't "All" include errors?

leftquark

Ferguson wrote: »

Leftquark -- does this give you enough to go on? Can you look at the log from this image and figure out where it went wrong, and why it worked perfectly the second time (meaning it's clearly not content related, at least in a reproducible way).

Perfect -- thanks! This will give me a start. I'll PM you if I have more questions.

Ferguson

leftquark wrote: »

Perfect -- thanks! This will give me a start. I'll PM you if I have more questions.

I uploaded another gallery last night and no delays. So at least for me it is rare now; it used to be very frequent.

Another coming after a shoot tonight.