Merging image directories (duplicate images)

georgec · December 30, 2009

Hi everyone,

A couple of years ago, I had a disk crash on me. I managed to recover most of the images. In the meantime, life moved on and we kept taking pictures and saving them to a new computer. While recovering the 10 GB or so of images, I made several copies as backups. So now I've got the active image directory (My Pictures), several backups of the recovered images, an "official recovered directory ("Pictures from 2001-2006"), and then a few smaller image directories smattered about that I think are just copies that I made in the past for one reason or the other, but I'm not 100% positive.

so I've got images scattered about all over 3-4 different disks.

I'm looking for software and/or a methodology to consolidate all of those images into one big directory, from where I can do proper backups. The software would need to help me identify and eliminate duplicates, since there would be need to save duplicates. I'm thinking that I would start with "My Pictures" and then test anything incoming to see if there are some unique images, and if so, then save those into My Pictures.

Any suggestions on software that might be able to help me in this task? I think that there is probably about 40GB of unique pictures, to give an idea of the volume of data.

Thanks!

BradfordBenn · December 31, 2009

One of the big questions is what is the file naming structure? As long as the name is the same there are a few options. I had a similar situation at work with lots of files after a coworkers left.... but that is a whole nother story. The files I was working with were not solely pictures, there were all sorts scattered and duplicated files on multiple locations. I found a solution that worked for me; not sure it is the best.

if it is pictures only, it might work to use Lightroom to consolidate them all and set it to not reimport suspected duplicates (it is done by filename) to a new location. Even if you don't own LR you can down load a 30 day trial and use it to clean up the directories and images.

On the Windows side I use "Where Is It" to catalog the files I have on various media. It is a database program that will list files and allow for searching etc... it has various tools in there for things such as reading IPTC info, creating thumbnails, showing dates ...etc. I used its File Management tool to go in an copy all of the files meeting my search criteria to one directory. It would then ask me if I wanted to overwrite or rename the file. So I would decided as I saw the files. It worked pretty well for me. I also use it to catalog my 100s of DVDs of Data and word documents and the like. It freaks people out when I say hang on, let me get that file.... it is a cooler tool for that but it does work with helping me find duplicates. It also has reporting tools.

georgec · December 31, 2009

BradfordBenn wrote:

One of the big questions is what is the file naming structure? .......

On the Windows side I use "Where Is It" <http: www.whereisit-soft.com=""> to catalog the files I have on various media.

My file naming structure has been pretty consistent. Directories are MMMM DD, YYYY, for example, December 31, 2009/ and then images are the same as the diretory name, but with an incrementing number, so the first image taken today would be My Pictures/December 31, 2009/December 31, 2009 001.

I haven't been prefect with that. Sometimes I use something like December 25, 2009 (xmas) as a directory name, and there are some AVIs in the directories, but for the most part, it's not bad.

The problem is that I have 2 sets of image directories; let's call them A and A'. Well, I *think* they are probably, maybe the same ... I'm pretty sure. But not 100% positive. Out of 10GB of images, there is a chance that there are a few unique images (files) scattered about, or a whole directory.

I've tried "uniquefiler". It's got some fundamentally good technology, but the interface is very old and clumsy for this purpose.

I'll go look at "Where Is It". Thanks.
</http:>

eur0ed · January 4, 2010

I have this problem regularly as I move between 2 laptops and a desktop.. once a month or so I condense my files and run this program.

http://www.visipics.info/index.php?title=Main_Page

And it's freeware :P

georgec · January 4, 2010

The Visipics looks a lot like uniquefiler, which I've used for years, but it was always a little clumsy to use. I did download and tried using the beta version of uniquefiler, which seems a lot better. Note: uniquefiler has not been updated since 2001 time frame. So Visipics might be a better direction.

Here's what I've started to do, and it seems to work pretty well:

Assuming two directories of images: MAIN and INCOMING, where MAIN has most of the images, and INCOMING might or might not exist already within MAIN.

make a backup copy of MAIN (to relieve paranoia)
make a backup copy of INCOMING
With uniquefiler, create the CRC database for MAIN
With uniquefiler, create the CRC database for INCOMING
Compare and then delete duplicates from INCOMING
- Note: The problem in the past has this always left a bunch of empty directories, or they had a few AVIs here and there, and always thumbs.db, and with possibly 50 or so directories, it was really tough to go through each one of them looking for remnants
Now, use "Remove Empty Directories" (see CNET downloads.com ... it's free) to remove any empty directories.
Those that aren't empty, copy to MAIN
Refresh the MAIN database for uniquefiler
Remove INCOMING
repeat ad nauseum...

BradfordBenn · January 4, 2010

I came up with another thing that might work as well in Lightroom. (As I went through this process last week.) Since LR will recurse directories, what I did while combining is changed the file names on import (yup it might mess up your system) to be Date (yyyymmdd) and time (hhmmss). If there are two pictures that have the exact same time date stamp it will append it with a -2, or -3, or -4... etc. So then what I did was sort by name/time, and I even did a quick little check search for -# to kind of cut it down quickly.

I also did the paranoid backup (46 4G DVDs)

Merging image directories (duplicate images)

Comments