Bulk Delete on very large gallery fails

DrDavid

I'm trying to delete about 1600+ photos from a gallery that has about 11,000 photos. I was using S*E to upload about 30GB of pics, and, apparently, it kept 're-uploading' the same pictures.. Now, I want to remove the dupes, and upload the remaining images.

Firefox will show the whole Delete (Bulk) display (all 11,000 photos!), and I can choose what I want to remove, but, when I hit apply, it just waits and waits and waits and eventually I get a "Timed Out" error. I've tried to do it in batches too... If I do only a few at a time, it works, but, it'll take me forever. I tried 300 photos at a time, but that failed too.

Yes, I know, I should have organized them better, but, it was meant to be an archive repository for *all* my photos--in case disaster strikes my computer, etc; I simply wanted to get them up for safe keeping. I guess I could re-do the whole upload, but, it seems like such a waste of 15GB of bandwidth for you guys. This next upload will use my new system of year\month to make it easier to manage. No more 11,000 files in one gallery.. LOL

Anyways, can you help?

David

Sheaf

DrDavid wrote:

I'm trying to delete about 1600+ photos from a gallery that has about 11,000 photos. I was using S*E to upload about 30GB of pics, and, apparently, it kept 're-uploading' the same pictures.. Now, I want to remove the dupes, and upload the remaining images.

Firefox will show the whole Delete (Bulk) display (all 11,000 photos!), and I can choose what I want to remove, but, when I hit apply, it just waits and waits and waits and eventually I get a "Timed Out" error. I've tried to do it in batches too... If I do only a few at a time, it works, but, it'll take me forever. I tried 300 photos at a time, but that failed too.

Yes, I know, I should have organized them better, but, it was meant to be an archive repository for *all* my photos--in case disaster strikes my computer, etc; I simply wanted to get them up for safe keeping. I guess I could re-do the whole upload, but, it seems like such a waste of 15GB of bandwidth for you guys. This next upload will use my new system of year\month to make it easier to manage. No more 11,000 files in one gallery..

Anyways, can you help?

David

Wow, that's an interesting dilemma. Maybe you could try using a different tool, like the Move Photos tool? Then delete them in the gallery they get moved to?

DrDavid

Sheaf wrote:

Wow, that's an interesting dilemma. Maybe you could try using a different tool, like the Move Photos tool? Then delete them in the gallery they get moved to?

I just tried again with a smaller number to delete... It looks like to goes through the entire album with each file. So, basically, the delete procedure seems to be taking forever for each photo needing to be deleted. I would have assumed it was a simple SQL statement, something like:

delete from photos where photoid=1234567; (repeat as needed.. Yes, I know, it's not my best SQL ever, but, you get the point )

Instead, it seems to be doing a very intensive SELECT first, then some other lengthy procedure, then, finally, removing the photo. Repeat for each photo to be deleted.

I'm going to try to MOVE the photos; but, if it's the same basic algorythm, I'm not hopeful

Will report back on success/failure.

David

Allen

The delete function probably has to find all the display copies on the different
servers they're on and backed up on also.

DrDavid

Moving the images seems to be just as slow. It's definately the size of the original gallery; 11,000+ files, and the sql statements on the back end choke...

Can I include a feature request? I'd love to be able to construct SQL statements to move/alter/etc.. the photos.. I can imagine that's a huge security issue waiting to happen, but, I can dream, can't I?

David

DrDavid

Allen wrote:

The delete function probably has to find all the display copies on the different
servers they're on and backed up on also.

A CRON job to do large deletes on large galleries might be a useful thing.. Basically, just queue the changes up, and do it over the next x+y minutes. I totally understand that I shouldn't expect totally real-time changes on large moves, deletions, etc..

Oh, and on the move; I did about 136 photos, 5 moved in about 5 minutes or so, before firefox timed out and gave an error.

David

DrDavid

DrDavid wrote:

A CRON job to do large deletes on large galleries might be a useful thing.. Basically, just queue the changes up, and do it over the next x+y minutes. I totally understand that I shouldn't expect totally real-time changes on large moves, deletions, etc..

Oh, and on the move; I did about 136 photos, 5 moved in about 5 minutes or so, before firefox timed out and gave an error.

David

Odd... The job seems to be continuing to process, even though firefox gave me an error: "The connection to the server was reset while the page was loading"

I've had two photos moved to the new gallery, even after firefox gave up waiting... But, it's *super* slow!

Andy

DrDavid wrote:

Odd... The job seems to be continuing to process, even though firefox gave me an error: "The connection to the server was reset while the page was loading"

I've had two photos moved to the new gallery, even after firefox gave up waiting... But, it's *super* slow!

Could be a browser limitation, or your computer's resource limitation, too - that's a lot of stuff to cache?

DrDavid

Andy wrote:

Could be a browser limitation, or your computer's resource limitation, too - that's a lot of stuff to cache?

I might have thought that too; but, firefox is still completely responsive.. AND, the fact remains that some of the photo moving occured after firefox told me that the connection was reset. My computer is fairly beefy--while 11,000 photos is a lot to handle, it wasn't stressing over it. I could move around the page quickly, no time waiting to refresh, etc...

Anyways, I'm pretty sure there's nothing I can do on my end other than re-do the upload (in a much smarter way.. lol..)...

I have FIOS here; but, it's still only 2mbps up... SoCal doesn't get the cool 20/20mbps that NJ has :cry