We installed a whole lot of new image processing hardware lately. Are you still seeing long processing times? We monitored it pretty closely last night and it seemed we hardly saw any backlog.
I will pay closer attention to the actual time now that I know you've made some improvements. I did a bunch of uploads last night in support of this dgrin posting and some today for dpreview postings. It seemed like I waited longer than I wanted to (multiple minutes), but I wasn't clocking it.
We installed a whole lot of new image processing hardware lately. Are you still seeing long processing times? We monitored it pretty closely last night and it seemed we hardly saw any backlog.
I uploaded two single images in the last hour in support of some online postings I'm working on.
The first took 2:50 (5:44:43 to 5:47:33 PST) from the time the upload was complete until the image was available in my gallery. The second image took 8:20 (6:14:00 to 6:22:20 PST).
Edit: FYI, these images were 824Bb and 775Kb in size.
I've been following this thread mainly because I was curious - the processing times have never been much of a problem for me. I happened to upload a single image just now and thought "what the heck, I'll time it". As the 1 minute mark passed, I was surprised. At the 2 minute mark I got pretty annoyed. Here's the result:
600K jpeg, uploaded in 4 seconds, processed in 6 minutes 41 seconds. Ended around 9:45 Eastern time.
I have a pro account (not that I use it that way yet) and I have to say I was shocked at the time required. I never would have thought it would take that long for a single photo.
Anyway, just thought I'd throw in another data point for you guys.
I too find that it takes a while for a single photo...but it takes no longer to do 100 photos, and it seems to start going to work on them after the first one is uploaded, so by the time everything is uploaded they are all ready for viewing. It's odd in a way.
I too find that it takes a while for a single photo...but it takes no longer to do 100 photos, and it seems to start going to work on them after the first one is uploaded, so by the time everything is uploaded they are all ready for viewing. It's odd in a way.
That's a good point, and it's probably why I've never really noticed the lag much before this.
I uploaded two single images in the last hour in support of some online postings I'm working on.
The first took 2:50 (5:44:43 to 5:47:33 PST) from the time the upload was complete until the image was available in my gallery. The second image took 8:20 (6:14:00 to 6:22:20 PST).
Edit: FYI, these images were 824Bb and 775Kb in size.
In the interest of editorial balance, I thought I'd report some good news. Uploads have been very fast all day today. I've not seen an upload take more than a minute to appear from the time the upload finished until it showed up in the gallery. I don't know if you changed anything to make it go faster or if I'm just hitting it at a good time today, but it's been fast today.
In the interest of editorial balance, I thought I'd report some good news. Uploads have been very fast all day today. I've not seen an upload take more than a minute to appear from the time the upload finished until it showed up in the gallery. I don't know if you changed anything to make it go faster or if I'm just hitting it at a good time today, but it's been fast today.
In the interest of editorial balance, I thought I'd report some good news. Uploads have been very fast all day today. I've not seen an upload take more than a minute to appear from the time the upload finished until it showed up in the gallery. I don't know if you changed anything to make it go faster or if I'm just hitting it at a good time today, but it's been fast today.
Wish I could take credit, but I think you're just seeing a less busy upload time. We haven't made any changes in a week or two that would affect this.
Sorry!
Don
0
Options
BaldyRegistered Users, Super ModeratorsPosts: 2,853moderator
We installed a whole lot of new image processing hardware lately. Are you still seeing long processing times? We monitored it pretty closely last night and it seemed we hardly saw any backlog.
Wish I could take credit, but I think you're just seeing a less busy upload time. We haven't made any changes in a week or two that would affect this.
Sorry!
Don
Feeling like you're getting conflicting info from the SmugMug team?
We did indeed install more hardware a couple of weeks ago but during that week we saw posts on dgrin which indicated image processing times were sometimes still long. That motivated my post, which was to understand what you were seeing.
Last week we had some image processing hardware fail, so we actually had less horsepower at work, but the posts seem to be indicating that things had gotten faster.
So Don was careful not to give too much credit to extra hardware given the feedback we had gotten the week before.
Feeling like you're getting conflicting info from the SmugMug team?
We did indeed install more hardware a couple of weeks ago but that week the posts we were seeing on dgrin seemed to indicate that image processing times were still long at times. That motivated my post, which was to understand what you were seeing.
Last week we had some image processing hardware fail, so we actually had less horsepower at work, but the posts seem to be indicating that things had gotten faster.
So Don was careful not to give too much credit to extra hardware given the feedback we had gotten the week before.
Clear as mud?
Yep, clear as mud. There is something inconsistent going on. I uploaded single photos a whole bunch of times last night (Sunday - usually one of your busiest times I'm led to believe) and got nothing but a quick response. So, unless nobody was uploading like normal because of the superbowl, something is clearly different this week than last. As with all of these kinds of issues, it's a whole lot more comforting if observed differences can be explained rather than not explained, even if the current state is now good. Most unexplained issues tend to come back again sometime in the near future (at least that's my decades of experience with software issues).
So ... don't get me wrong, I'm not complaining that the uploads are now quick, just pointing out that not having a good explanation for why it was slow and is now fast tends to mean there's significant risk that whatever made it slow last week will come back again.
Yep, clear as mud. There is something inconsistent going on. I uploaded single photos a whole bunch of times last night (Sunday - usually one of your busiest times I'm led to believe) and got nothing but a quick response.
Yes, Sunday is our busiest time -- and holidays. My post explains why most of the time we're not seeing a backlog (more hardware thrown at the problem) but Don's explains that no amount of hardware can protect against every spike, and we haen't implemented a sophisticated batch prioritization algorithm or some such.
This is an interesting topic. When I was uploading my library of photos last week, there were times when I would check and find that the last three galleries that were being uploaded were still processing. But then recently, I'm amazed at how fast some things are. They're almost faster than processing the same action locally using irfanview.
The algorithm for something like this would be tough. This starts getting into branch prediction, out of order execution and all sorts of stuff that you find in computer programming and computer engineering. Makes me want to pull out my algorithms and data strucutres books and see what I can dig up, lol.
I'd start by splitting off part of the processing cluster and semi-dedicating it to smaller batches. If the batch is smaller than x, then it goes to the smaller cluster. If the smaller cluster is overwhelmed and the main cluster is idle, some of the smaller cluster jobs will be routed to the main cluster. If the main cluster is completely clogged, it can send images one at a time to the smaller cluster, but only if the smaller cluster is idle or has a queue less than y. Or something like that. There's probably already stuff like this in place...
This is an interesting topic. When I was uploading my library of photos last week, there were times when I would check and find that the last three galleries that were being uploaded were still processing. But then recently, I'm amazed at how fast some things are. They're almost faster than processing the same action locally using irfanview.
The algorithm for something like this would be tough. This starts getting into branch prediction, out of order execution and all sorts of stuff that you find in computer programming and computer engineering. Makes me want to pull out my algorithms and data strucutres books and see what I can dig up, lol.
I'd start by splitting off part of the processing cluster and semi-dedicating it to smaller batches. If the batch is smaller than x, then it goes to the smaller cluster. If the smaller cluster is overwhelmed and the main cluster is idle, some of the smaller cluster jobs will be routed to the main cluster. If the main cluster is completely clogged, it can send images one at a time to the smaller cluster, but only if the smaller cluster is idle or has a queue less than y. Or something like that. There's probably already stuff like this in place...
Actually, the concept of "batches" isn't something we do anymore. All of the modern uploaders (Send-to-smugmug, Star*Explorer, the Drag-and-Drop uploader, Mac Uploader, etc) send photos one at a time, not in a batch.
So we actually just brute-force FIFO out to a cluster of machines. No point in waiting for the "batch" to finish and be classified when you can (usually) have most of the photos processed when the "batch" finishes anyway.
I can't think of a more fair or efficient way of doing it without being unfair to some class of user.
Generally, our processing is very fast. We lagged for a month or two there simply because our datacenter couldn't get us more power - we had the CPUs sitting around, not plugged in. It was a shame, really, but I think we've solved that problem - more datacenters, and pre-allocation of power.
Comments
I will pay closer attention to the actual time now that I know you've made some improvements. I did a bunch of uploads last night in support of this dgrin posting and some today for dpreview postings. It seemed like I waited longer than I wanted to (multiple minutes), but I wasn't clocking it.
Homepage • Popular
JFriend's javascript customizations • Secrets for getting fast answers on Dgrin
Always include a link to your site when posting a question
I uploaded two single images in the last hour in support of some online postings I'm working on.
The first took 2:50 (5:44:43 to 5:47:33 PST) from the time the upload was complete until the image was available in my gallery. The second image took 8:20 (6:14:00 to 6:22:20 PST).
Edit: FYI, these images were 824Bb and 775Kb in size.
Homepage • Popular
JFriend's javascript customizations • Secrets for getting fast answers on Dgrin
Always include a link to your site when posting a question
600K jpeg, uploaded in 4 seconds, processed in 6 minutes 41 seconds. Ended around 9:45 Eastern time.
I have a pro account (not that I use it that way yet) and I have to say I was shocked at the time required. I never would have thought it would take that long for a single photo.
Anyway, just thought I'd throw in another data point for you guys.
That's a good point, and it's probably why I've never really noticed the lag much before this.
In the interest of editorial balance, I thought I'd report some good news. Uploads have been very fast all day today. I've not seen an upload take more than a minute to appear from the time the upload finished until it showed up in the gallery. I don't know if you changed anything to make it go faster or if I'm just hitting it at a good time today, but it's been fast today.
Homepage • Popular
JFriend's javascript customizations • Secrets for getting fast answers on Dgrin
Always include a link to your site when posting a question
Good to hear - thanks - and I noticed it, too...
Portfolio • Workshops • Facebook • Twitter
Wish I could take credit, but I think you're just seeing a less busy upload time. We haven't made any changes in a week or two that would affect this.
Sorry!
Don
We did indeed install more hardware a couple of weeks ago but during that week we saw posts on dgrin which indicated image processing times were sometimes still long. That motivated my post, which was to understand what you were seeing.
Last week we had some image processing hardware fail, so we actually had less horsepower at work, but the posts seem to be indicating that things had gotten faster.
So Don was careful not to give too much credit to extra hardware given the feedback we had gotten the week before.
Clear as mud?
Yep, clear as mud. There is something inconsistent going on. I uploaded single photos a whole bunch of times last night (Sunday - usually one of your busiest times I'm led to believe) and got nothing but a quick response. So, unless nobody was uploading like normal because of the superbowl, something is clearly different this week than last. As with all of these kinds of issues, it's a whole lot more comforting if observed differences can be explained rather than not explained, even if the current state is now good. Most unexplained issues tend to come back again sometime in the near future (at least that's my decades of experience with software issues).
So ... don't get me wrong, I'm not complaining that the uploads are now quick, just pointing out that not having a good explanation for why it was slow and is now fast tends to mean there's significant risk that whatever made it slow last week will come back again.
Glad it's fast now.
Homepage • Popular
JFriend's javascript customizations • Secrets for getting fast answers on Dgrin
Always include a link to your site when posting a question
I will report I uploaded 110mb of photos this afternoon and as soon as they were finished uploading, they were all there!!!
Nice job guys!
Glass: >Sigma 17-35mm,f2.8-4 DG >Tamron 28-75mm,f2.8 >Canon 100mm 2.8 Macro >Canon 70-200mm,f2.8L IS >Canon 200mm,f2.8L
Flash: >550EX >Sigma EF-500 DG Super >studio strobes
Sites: Jim Mitte Photography - Livingston Sports Photos - Brighton Football Photos
The algorithm for something like this would be tough. This starts getting into branch prediction, out of order execution and all sorts of stuff that you find in computer programming and computer engineering. Makes me want to pull out my algorithms and data strucutres books and see what I can dig up, lol.
I'd start by splitting off part of the processing cluster and semi-dedicating it to smaller batches. If the batch is smaller than x, then it goes to the smaller cluster. If the smaller cluster is overwhelmed and the main cluster is idle, some of the smaller cluster jobs will be routed to the main cluster. If the main cluster is completely clogged, it can send images one at a time to the smaller cluster, but only if the smaller cluster is idle or has a queue less than y. Or something like that. There's probably already stuff like this in place...
Want faster uploading? Vote for FTP!
Actually, the concept of "batches" isn't something we do anymore. All of the modern uploaders (Send-to-smugmug, Star*Explorer, the Drag-and-Drop uploader, Mac Uploader, etc) send photos one at a time, not in a batch.
So we actually just brute-force FIFO out to a cluster of machines. No point in waiting for the "batch" to finish and be classified when you can (usually) have most of the photos processed when the "batch" finishes anyway.
I can't think of a more fair or efficient way of doing it without being unfair to some class of user.
Generally, our processing is very fast. We lagged for a month or two there simply because our datacenter couldn't get us more power - we had the CPUs sitting around, not plugged in. It was a shame, really, but I think we've solved that problem - more datacenters, and pre-allocation of power.
Don