I need advice on completely finishing the workflow process

kgarrett11

Currently once I have finished my processing I copy all of my RAW files to a DVD. I have talked to a couple of people at work and have gotten mixed information. Does anyone delete their RAW files from their hard drives at any point? I have a real problem with the "DELETE" button. I am a real pack-rat. Do you delete or keep buying more and more hard drives?

System

kgarrett11 wrote:

Currently once I have finished my processing I copy all of my RAW files to a DVD. I have talked to a couple of people at work and have gotten mixed information. Does anyone delete their RAW files from their hard drives at any point? I have a real problem with the "DELETE" button. I am a real pack-rat. Do you delete or keep buying more and more hard drives?

from the pack rat king-

delete?-

hell no-

externals for me-


but one of the key phrases is 'pack rat'-

not always a good idea to take advice from a freakin' neurotic such as me-

Stustaff

The basic/obvious rule is have at least two copies of everything.

So a RAW file will be on a DVD and on my external HD same for my JPEG's but they are also on smugmug.

kgarrett11

Thanks. so being a packrat isn't such a bad thing. This hasn't been a major problem up until now but I am going to Kenya and Tanzania in July and anticipate a lot of pic that have to be stored. So I will invest in another HD to cover my butt when I get back.

jfriend

My RAW backup workflow

kgarrett11 wrote:

Currently once I have finished my processing I copy all of my RAW files to a DVD. I have talked to a couple of people at work and have gotten mixed information. Does anyone delete their RAW files from their hard drives at any point? I have a real problem with the "DELETE" button. I am a real pack-rat. Do you delete or keep buying more and more hard drives?

I never delete my RAW files. I thiknk of them like my negatives from the film days. They are my originals and, if in the future, I want to reprocess any images for a different purpose, I may want to go back to the RAW image and start there. Particularly image destructive editing operations like cropping. You may be interested in cropping your post processing effort for an 8x10 print, but what if later on, you want to make a 20x30 (that has a different crop ratio). You may very well want to start with the whole original. Or imagine, your post processing skills or the RAW editing skills improve and you want to get "more" out of one of your originals to do something special with it. It sure would be nice to be able to start with the original RAW file.

As for copying your RAW files to a DVD, I don't consider DVDs a good form of permanent storage as the medium is anything but permanent (it has a finite lifetime). Further, relative to what I've spent on camera/lens gear, hard disk space is pretty cheap. For $0.50/GB or less, you can get a Firewire/USB2 external hard drive that makes an excellent and fast backup option (it's also orders of magnitude faster than DVDs). I personally have a 400GB external drive that I use for immediate backup purposes. When it fills up, I'll just put it on the shelf (or perhaps at my sister's house) and then just get another.

The external hard disk does cost a little up-front money ($100-$200 depending upon size), but you might want to think about what you're spending on writable DVDs for a similar amount of storage and think about the advantage of having the backup go so much faster.

System

kgarrett11 wrote:

Thanks. so being a packrat isn't such a bad thing. This hasn't been a major problem up until now but I am going to Kenya and Tanzania in July and anticipate a lot of pic that have to be stored. So I will invest in another HD to cover my butt when I get back.

I think this begs another question-

are you taking a laptop to transfer to-

or the type of external hd you can upload from your camera/card-

do you backup while on a trip-

kgarrett11

jfriend wrote:

I never delete my RAW files. I thiknk of them like my negatives from the film days. They are my originals and, if in the future, I want to reprocess any images for a different purpose, I may want to go back to the RAW image and start there. Particularly image destructive editing operations like cropping. You may be interested in cropping your post processing effort for an 8x10 print, but what if later on, you want to make a 20x30 (that has a different crop ratio). You may very well want to start with the whole original. Or imagine, your post processing skills or the RAW editing skills improve and you want to get "more" out of one of your originals to do something special with it. It sure would be nice to be able to start with the original RAW file.

As for copying your RAW files to a DVD, I don't consider DVDs a good form of permanent storage as the medium is anything but permanent (it has a finite lifetime). Further, relative to what I've spent on camera/lens gear, hard disk space is pretty cheap. For $0.50/GB or less, you can get a Firewire/USB2 external hard drive that makes an excellent and fast backup option (it's also orders of magnitude faster than DVDs). I personally have a 400GB external drive that I use for immediate backup purposes. When it fills up, I'll just put it on the shelf (or perhaps at my sister's house) and then just get another.

The external hard disk does cost a little up-front money ($100-$200 depending upon size), but you might want to think about what you're spending on writable DVDs for a similar amount of storage and think about the advantage of having the backup go so much faster.

Thank you. Currently that is what I have been doing. I have a 300GB external where everything is. I keep nothing on my internal. My external is about 3/4 full. I guess when you put it the way you did, if I spend over $1500 on glass then $200 on storage isn't so much.

kgarrett11

gefillmore wrote:

I think this begs another question-

are you taking a laptop to transfer to-

or the type of external hd you can upload from your camera/card-

do you backup while on a trip-

I am taking a laptop, going to buy a 120GB USB 2.0 external and dual-layered DVD withme. I have 14GB in CompactFlash (don't expect to use that up everyday) but want to make sure that I don't run out of storage in the middle of nowhere and then all I would be able to do is smile at the elephants as they charge me.

System

kgarrett11 wrote:

I am taking a laptop, going to buy a 120GB USB 2.0 external and dual-layered DVD withme. I have 14GB in CompactFlash (don't expect to use that up everyday) but want to make sure that I don't run out of storage in the middle of nowhere and then all I would be able to do is smile at the elephants as they charge me.

yeah-

I'm not sure which would suck worse-

having the elephants charging me or not being able to take a pic of it!-

kgarrett11

gefillmore wrote:

yeah-

I'm not sure which would suck worse-

having the elephants charging me or not being able to take a pic of it!-

As long as I am standing next to someone who is slower than I am then it would be worse to not be able to get the picture of the elephant and the slow person.

System

kgarrett11 wrote:

As long as I am standing next to someone who is slower than I am then it would be worse to not be able to get the picture of the elephant and the slow person.

hah!-

man, I would be sol!-

Stustaff

kgarrett11 wrote:

Thank you. Currently that is what I have been doing. I have a 300GB external where everything is. I keep nothing on my internal. My external is about 3/4 full. I guess when you put it the way you did, if I spend over $1500 on glass then $200 on storage isn't so much.

So you also have a backup of the external 300GB hard drive then yes?

If not what if that gets fried?

claudermilk

I look at it this way: hard drives are cheap, reshoots are not (or are impossible).

I add redundant hard drives as necessary (that means two drives with copies of the files on each). If you really want, I can get going on my anti-DVD-as-archive soapbox, too. soapbox

Duffy Pratt

I keep anything that was worth working on in PS. If its not good enough to make it presentable to someone, then I delete it.

Periodically, I try to delete a bunch of my old pictures, hopefully culling out the worst and thereby improving the overall collection. Basically, I think I would be lucky to get 10 really good shots/year. Then there are a bunch that I keep for sentimental reasons, even though I know that other people probably wouldn't be all that interested.

Duffy

Art Scott

How many TRUE pros ever toss any type of file (whether film or digital)...very few.....

Every book or article I have read said to NEVER...I repeat....NEVER delete a file...and that means out of focus, out of exposure..etc etc....a couple of exceptions are...files with no discernable images...so almost totally black or burned out white....cause your creative side just might get an idea that the over/under expsed shots from 3 yrs back just might make an interesting line drawing or abstract.....

But if you toss files and that idea hits....YOU'RE SCREWED......so now I have a full 180gb and 320gb Hard drives...moving to 500 gb real soon and will sell the smaller ones.....

I just basically set my drives with 2 folders...the keepers and the "junk"...both of those have several subfolders for the different types of photos....and I usually don't mess with anything in the junk folder unless I have an idea for different meduim (anything not percieved as a normal photo print), then once the Post processing is done it moves to the kepper file and proper subfolder.......I also renumber all my files once in the proper sub folder.......

xris

But how about back-up media? I agree that CDs have a limited lifetime, but I'd also think a CDR or DVD is FAR more reliable and longer-lived than an external drive! I mean hard disks are so much more prone to bumps and knocks, magnets, nuclear fallout and such. A DVD is good for many years, and the files can easily be transferred to new media in the future.

One other thing. I'd suggest you also store a ful-res TIFF of each photo. RAW files depend on propretary software you may not be able to run in ten years.

http://www.dl-c.com/ has some good threads on archival file formats.

jfriend

I do delete some photos from event shoots

Art Scott wrote:

How many TRUE pros ever toss any type of file (whether film or digital)...very few.....

Every book or article I have read said to NEVER...I repeat....NEVER delete a file...and that means out of focus, out of exposure..etc etc....a couple of exceptions are...files with no discernable images...so almost totally black or burned out white....cause your creative side just might get an idea that the over/under expsed shots from 3 yrs back just might make an interesting line drawing or abstract.....

I don't disagree in principle. Hard disk is cheap. Keep most everything.

But for my non-pro use, I've settled on a workflow for event shoots that does delete some things so I thought I'd explain why I don't think "never delete" is quite that black and white. When I'm shooting kid sporting events, I only have one output in mind. I'm going to put a set of shots on the web for parents and I may also be creating a "highlights slideshow". If a photo isn't going to make one of these two output vehicles, I will never use it. I will not look back after the photo processing for the event is done (other than the shots of my own kids, if any). If I keep it and never use it, it's just going to get in my way, so I do make a judgment call and delete some files.

So, I have a culling workflow that results in assigning a rating of 0-4 stars to every photo in a shoot. 1-4 stars means it's potentially good enough to use in the final output. No stars means that it's a technically OK photo (in focus, OK exposure), but not what I would consider a good subject capture. Photos that are technically flawed (out of focus, missed the subject, too much motion blur, something important obscured, bad lighting, etc...) I just delete. Even if it's the best photo I have of that player, there's no way I'm going to use it so I get rid of it.

So, now I've got photos with 0-4 stars on them that are all are technically OK. Now, I start figuring out what the output is going to look like. In the case of a sports team, I keyword the photos by player so I can see how many decent shots of each player I have. Since my goal is often some number of good shots of every player I look at how many of each player I have and sort by rating and if I'm looking for 10 shots of each player, I pick the 10 best of each player (using ratings as my previous guideline).

I'm hoping to not have to use any shots with a rating of 0 stars, but if I'm critically short on some players, I either note that I need more good shots of that player in the next game or I have to decide if the 0 star images are worth putting in the gallery.

At this point, I'm done deleting images. Even the ones that I don't use in the final output, but are technically OK images, I find it more work to go try to safely delete them than to keep them and they aren't in my way anymore.

jfriend

It's hard disks for me, not DVDs

xris wrote:

But how about back-up media? I agree that CDs have a limited lifetime, but I'd also think a CDR or DVD is FAR more reliable and longer-lived than an external drive! I mean hard disks are so much more prone to bumps and knocks, magnets, nuclear fallout and such. A DVD is good for many years, and the files can easily be transferred to new media in the future.

One other thing. I'd suggest you also store a ful-res TIFF of each photo. RAW files depend on propretary software you may not be able to run in ten years.

http://www.dl-c.com/ has some good threads on archival file formats.

I disagree. It's way more practical to buy a new drive every few years and copy all your photos over to it from the old drive than it is to reburn them on DVD every few years. I wouldn't only have one copy of my photos on a hard drive (I have copies on three separate drives).

I have 400GB of photos. That's 100 DVDs. Just think about how much time it would take to reburn all those DVDs. I can tell you from my own practical experience that it's more time than I will put into it so I just won't refresh my DVDs either. But, I will refresh my hard drives because I can just start a copy operation and walk away.

And, from what I hear, you can't assume that DVDs will last more than a few years, so you're faced with this monumental amount of work to keep your DVD backups viable every few years. What a nightmare.

Neither of these choices is something you can put in a closet for 40 years and have your grandkids pull out and hook up to their computer. I'm quite confident that computers 40 years from now won't be able to read today's DVDs and that the USB2 interface will probably be long gone. So, you're going to have to keep your backup technology fresh. The question is which technology makes it more practical to keep it fresh. DVDs doesn't work for me in this regard. Hard disks do.

As for storing RAW or TIFF, I haven't decided what to do their yet. I am currently storing RAW and if they were images I used somehow there are generated JPEGs as part of the things I backup. DNG is the other choice there which tries to have the longevity and durability of TIFF, but maintain the advantages of true RAW data. I'm starting to like the idea of DNG more and more, but haven't really decided yet.

gus

kgarrett11 wrote:

Currently once I have finished my processing I copy all of my RAW files to a DVD. I have talked to a couple of people at work and have gotten mixed information. Does anyone delete their RAW files from their hard drives at any point? I have a real problem with the "DELETE" button. I am a real pack-rat. Do you delete or keep buying more and more hard drives?

I delete everything..no questions asked. Yesterday i cleared 90 odd gig of old raw files that had built up.

Remember ..one morning you will turn your computer on & it will simply have shit itself as hard drives have a habit of doing thus it 'deletes' itself with no input from you at all.

Rhuarc

It'll be interesting to see how backup technology progresses with the new Blueray discs. I realize that it is still expensive, but the possiblity of being able to put 100GB on one disc is pretty appealing. That 100 DVD backup is suddenly down to 4 discs. If a company comes out with archival quality blueray discs (similar to the Kodak Gold, I think it is) this may become a viable long term backup. (or at least 5 or 10 years)

xris

jfriend wrote:

...
As for storing RAW or TIFF, I haven't decided what to do their yet. I am currently storing RAW and if they were images I used somehow there are generated JPEGs as part of the things I backup. DNG is the other choice there which tries to have the longevity and durability of TIFF, but maintain the advantages of true RAW data. I'm starting to like the idea of DNG more and more, but haven't really decided yet.

Good comments about the hard drive. You're making me think about it. Regarding DNG format, the same link has some very interested points regarding it too. I was strongly considering it, but after a few search seasons and a lot of reading I've decided I won't be touching DNG. TIFF, it turns out, is far closer to RAW. And far more open.

Rhuarc

xris wrote:

Good comments about the hard drive. You're making me think about it. Regarding DNG format, the same link has some very interested points regarding it too. I was strongly considering it, but after a few search seasons and a lot of reading I've decided I won't be touching DNG. TIFF, it turns out, is far closer to RAW. And far more open.

What doyou mean that TIFF is far closer to RAW than DNG? From my understanding TIFF is just another lossless file format similar to PSD. You can still do RAW conversions on DNG files, you can't on TIFF's. Right now I am archiving all my actual RAW files, then converting to DNG for my workflow. My Working directories are backed up as well, so I have my RAWs 3 ways, once as the original RAW file, and twice as DNG. Maybe I'm going too much by the book. And by the book I mean The DAM Book by Peter Krogh. I'm not following everything he says, but I really like the DNG format, mainly for the embedded JPEG preview. I understand the fears about DNG because it isn't a truly "open" format, but if it turns out anything like PDF then I'm not worried.

jfriend

Rhuarc wrote:

It'll be interesting to see how backup technology progresses with the new Blueray discs. I realize that it is still expensive, but the possiblity of being able to put 100GB on one disc is pretty appealing. That 100 DVD backup is suddenly down to 4 discs. If a company comes out with archival quality blueray discs (similar to the Kodak Gold, I think it is) this may become a viable long term backup. (or at least 5 or 10 years)

These types of storage technologies go in jumps. When DVDs first came out and we were upgrading from CDs, the 4GB of DVDs seemed huge. Now I have 12+MB RAW images to backup and I can't even get one soccer game on a DVD, so the content just blew past the capacity increase of DVDs.

Now, I'm starting to deal with video, slideshows of stills, personally shot video (which is moving to HD) and store bought digital video and I figure it won't be long before the Blueray discs don't feel that large either. It's a fairly constant race.

A lot of it comes down to the balance between convenience, discipline and cost.

If you have lots of discipline to keep your backups current, you can forgo convenience and save money by using media like DVDs.

If you are either less cost sensitive or don't have the discipline to actually do what's required when it's not very convenient, then you are better off paying more money and buying a system that will work quickly and easily (which is the camp I'm in).

The absolute worst backup system is the one you spent money on, are counting on, but aren't using because it's too much work. So, it ends up depending upon what you will actually DO.

System

"I understand the fears about DNG because it isn't a truly "open" format, but if it turns out anything like PDF then I'm not worried."



support for dng:

http://www.adobe.com/products/dng/supporters.html

xris

Rhuarc wrote:

What doyou mean that TIFF is far closer to RAW than DNG? From my understanding TIFF is just another lossless file format similar to PSD. You can still do RAW conversions on DNG files, you can't on TIFF's. ...

Well, first I'd suggest you check out the link I provided. Do a bit of background work. (http://www.dl-c.com/) I'm not going to go through it all again right now, but the gist (as I remember it at the moment.) is that tiff is capable of keeping the original raw channels in their original bit depth, simply wrapping a TIFF wrapper around them. Thus, when you open the TIFF in an app that can handle the full depth, you are, in effect, getting your RAW file data back.

The DNG issue is far more complex and there are several papers out there that do a good job of it. Start with the link above, do DNG and RAW searches and see what you come up with. I'll see if I can find sometime tomorrow or over the weekend.

Part of the problem is that DNG files seem to react differently in differemt apps. Another is that, I believe, DNG does not keep the original bit depth, but you'll have to check on that one. Anyway. The Digital Light and Magic threads go into a lot of detail, so it's a good resource to anyone here who's interested.

Rhuarc

gefillmore wrote:

"I understand the fears about DNG because it isn't a truly "open" format, but if it turns out anything like PDF then I'm not worried."

support for dng:

http://www.adobe.com/products/dng/supporters.html

Err...is that list of supporters to prove or disprove my statement? I'm actually pretty impressed with the size of the list, but maybe I'm issing something? Please enlighten me if so.

System

I think it basically shows that you're in good shape using dng-

it seems with all the software and cameras listed that it's moving quickly towards an open format-

I don't use dng (maybe I should) and I'm not much of an expert on workflow-

just saw your comment and wanted to provide the link; basically fyi-

Rhuarc

gefillmore wrote:

I think it basically shows that you're in good shape using dng-

it seems with all the software and cameras listed that it's moving quickly towards an open format-

I don't use dng (maybe I should) and I'm not much of an expert on workflow-

just saw your comment and wanted to provide the link; basically fyi-

Ahh, ok. Thank you!! I do archive one version of my CR2 files, but I have to copies of the equivalent DNG's, so even if I lose the original RAW files it isn't the end of the world. I jsut figure for now as long as storage isn't an issue for me I might aswell keep the original RAWs.

Zanotti

Wouldn't it be nice of Smugmug stored RAW files as well?

It supposed to be a photographers site - why not?

Z

jfriend

Zanotti wrote:

Wouldn't it be nice of Smugmug stored RAW files as well?

It supposed to be a photographers site - why not?

Z

An excellent idea (my opinion). Here are some previous threads on the same topic:

Interested in RAW File Backups on SmugMug?
Storing RAW files on smugmug

The conclusion I came away with the last time I saw it discussed here was that RAW stroage was not a business that Smugmug as a company wanted to be in.

Rhuarc

Just a little followup on archiving RAWs online. Was taking a look at different sites and came upon this one called MediaMax.

https://www.mediamax.com/Brands/MediaMax/home/pricing_comparison.aspx

Basically they offer:
25GB - Free
50GB - 9.95/month or 89.95/year
1,000GB - 29.95/month or 289.95/year

I find this hard to believe. That's a pretty good deal. There are some limitations on how much you can download in a month, but assuming you are just archiving you wouldn't be downloading too much too often. I'm going to give it a try, and I'll keep everyone notified of how it goes.