Big Files (too big)

Antonio Correia · January 5, 2006

I shoot CR2 with the 20D.
Files are 6.000MB to 9.000MB large.
I open the pictures in PS CS2.
1. If I save the files for future editing in PSD format (native format for Photoshop) some files are 280.000MB or bigger and hard to handle !...
2. If I save the files to JPG (with no possibility of future editing) they get 3.000MB to 5.000MB.

Instead of shooting CR2, when I shoot JPG everything is more reasonable and I can keep the PSD under acceptable sizes and good future editing.

Thought I have noticed that the pictures in CR2 are better than the others...:):

Can anyone give me a thought about this, please ?:D
Thank you :thumb

DavidTO · January 5, 2006

Check out this help page.

Andy · January 5, 2006

Antonio Correia wrote:

I shoot CR2 with the 20D.
Files are 6.000MB to 9.000MB large.
I open the pictures in PS CS2.
1. If I save the files for future editing in PSD format (native format for Photoshop) some files are 280.000MB or bigger and hard to handle !...
2. If I save the files to JPG (with no possibility of future editing) they get 3.000MB to 5.000MB.

Instead of shooting CR2, when I shoot JPG everything is more reasonable and I can keep the PSD under acceptable sizes and good future editing.

Thought I have noticed that the pictures in CR2 are better than the others...:):

Can anyone give me a thought about this, please ?:D
Thank you

I save only the PSDs that I must - the ones I've put a lot of time into and may edit in the future. I save all RAW files. I put my JPGs on SmugMug and of course save local copies, backedup.

Have you looked at Adobe DNG format?

DavidTO · January 5, 2006

I believe the answer lies in how you save your jpegs. PS is saving them at a higher quality than your camera is...and probably wasting megabytes along the way. What quality setting are you using in your save dialog box?

Antonio Correia · January 5, 2006

Thank all
for the fast comments....

Antonio Correia · January 5, 2006

JPG in 10

DavidTO wrote:

I believe the answer lies in how you save your jpegs. PS is saving them at a higher quality than your camera is...and probably wasting megabytes along the way. What quality setting are you using in your save dialog box?

I save my JPGs files with 10 for the Quality; Format Options Baseline and Size in 56,6Kbps.
May be 10 is too much ?! ...
http://antoniocorreia.smugmug.com/gallery/938400/1/51156916
for example.
I usually print some of what I consider to be the best pictures...
I do like specialy this one:
http://antoniocorreia.smugmug.com/gallery/951370/1/50515290
Comments please... ThU

Antonio Correia · January 5, 2006

Good point

DavidTO wrote:

Check out this help page.

Thank you

DavidTO · January 5, 2006

When I save at 10 my file sizes are all under 3MB. That's with my 20D. You could try 8, that might work. But 10 should be fine, I wonder what the difference is?

Antonio Correia · January 5, 2006

DavidTO wrote:

When I save at 10 my file sizes are all under 3MB. That's with my 20D. You could try 8, that might work. But 10 should be fine, I wonder what the difference is?

Actually, I do print only at 26,7cms by 18,3cms which is the golden rule with a margin of 1cm all around.
This size is sufficient for a good look at say 1,5 meters away.
May be I have to experiment !... with 6; 8 and 10 !...

:
Thank you for the tips

7th January : WRONG OF ME - The golden rule is 1,618 and not the one I have been using !! Shame ...

Andy · January 5, 2006

Antonio Correia wrote:

Actually, I do print only at 26,7cms by 18,3cms which is the golden rule with a margin of 1cm all around.
This size is sufficient for a good look at say 1,5 meters away.
May be I have to experiment !... with 6; 8 and 10 !... :
Thank you for the tips

Photoshop Save As - 10 is sweet, and yes, equal to what the jpgs out of camera are (on the 20D)

Antonio Correia · January 5, 2006

Thank you

Andy wrote:

Photoshop Save As - 10 is sweet, and yes, equal to what the jpgs out of camera are (on the 20D)

Good 2006 with HEALTH !...

Andy · January 5, 2006

Antonio Correia wrote:

Good 2006 with HEALTH !...

Obrigado, Antonio, Obrigado! E o mais melhor a voce, demasiado.

luke_church · January 5, 2006

Hi Antonio,

Antonio Correia wrote:

I shoot CR2 with the 20D.
1. If I save the files for future editing in PSD format (native format for Photoshop) some files are 280.000MB or bigger and hard to handle !...
2. If I save the files to JPG (with no possibility of future editing) they get 3.000MB to 5.000MB.

Instead of shooting CR2, when I shoot JPG everything is more reasonable and I can keep the PSD under acceptable sizes and good future editing.

I strongly suspect that this because of the bit depth of CR2 vs JPG. CR2 is a 16bits per channel format, JPG from most cameras is only 8bits per channel.

As the newer versions of Photoshop can do a lot in 16bit mode, you generally won't have to convert. However Photoshop seems to store the layers at a cost related to the bit depth, rather than a construction action set.

The upshot of this being, when I did a test with a CR2 file from a Canon 350D/XT, if I worked with the CS2 file in 16bit mode the PSD file was ~44Mb, reducing it to 8bit mode it was ~22Mb. Working with the JPG file in the same manner was ~21Mb.

As the number of layers go up, the x2 factor on the file will continue. Your 280Mb file would probably only have been 140Mb if it had been 8bit.

Could this be your difficulty?

In which case, you could try just setting your images into 8bit mode in Adobe Camera Raw? Even better, just work with the image in 16bit mode, and set it to 8bit just before you save it.

Please test this and let me know, if it isn't the problem I'll have a more detailed look.

My other comment is that I would strongly recomend keeping all your CR2 files.

As for JPG quality, if you're not going to work on the image anymore after you've saved it, then Q10 is just fine. If you're saving JPEGs as Q12, then you should probably be using a different format

:

Hope this helps, please feel free to yell if my above suggestion doesn't work.

Luke

luke_church · January 5, 2006

Andy wrote:

Photoshop Save As - 10 is sweet, and yes, equal to what the jpgs out of camera are (on the 20D)

Sorry Andy, I hate to disagree but if the 20D uses the same JPG algorithms as the 350D/XT which it probably does, this isn't quite true, at least not with Photoshop CS2.

Firstly: I agree, that the visual quality may be the same, and I think that JPEG Q10 is the right setting for most 'final distribution' purposes.

However the problem comes if this is used for tempory file saving, the word equal becomes kind of a bit dangerous.

For the luminance channel:

The quantization matricies that the 350D appears to use are in the range 1-6, whereas Photoshop Q10 Baseline uses 1-12. This would be so bad, expect that the numbers don't align, so in many places there is a 5 in Canon's file and a 12 in Adobe's

For the chroma channels:

350D: 1-5, Photoshop: 1-15, in a fairly odd pattern...

So there are a few things to note:

1. You won't easily be able to visually tell the difference between these because of the way JPEG works

2. If you took a 350D image, compressed it with Photoshop a value corruption will occur, hence I would be very reluctant to describe them as equal. The compressed curve will look blocky compared to the original.

What you would see for example, if the 'original in camera' DCT value was 21, you would see

21 -> 350D's JPEG -> 21/5 = 4 -> 4*5 = 20 when loaded in Photoshop
Then 20/15 = 1.... = 1 -> 1*15 = when loaded the second time

If this process is repeted with more quantization matricies that are relatively prime, then you're going to loose values fairly fast.

It's actually going to be even worse than this because DCTs generate floating point numbers that are then truncated to ints.

Hence after a few of these compressions, you'll start to see banding and other artefacts occuring in the image, I could notice some difference, expecially to areas of relatively low intensity after only a few rounds. Looking at the histogram it can be seen, it isn't exactly a big deal, but I wouldn't do it if I was saving and loading the file say 20 times.

Hence I would argue that saving at higher quality for intermediate files was reasonable (but you should probably be using PSD for this where possible), but for the final save, yeah sure, Q10 seems like a good balance.

I think you can probably use Optimised as well. It depends on exactly what Adobe means by 'Optimised Huffman Encoding' but if they mean what is sensible, then everyone should probably be using Baseline Optimised. Saves a bit more.

Hope this helps, and that you don't think I'm being tooooo much the techno geek

:

Any questions, just holler.

All the best,

Luke

Mike Lane · January 5, 2006

luke_church wrote:

I strongly suspect that this because of the bit depth of CR2 vs JPG. CR2 is a 16bits per channel format, JPG from most cameras is only 8bits per channel.

How can that be if the camera itself is only capable of 8 bits per channel? Yes you can in ACR push your images to 16 bits per channel but that's not the default setting.

DavidTO · January 5, 2006

Mike Lane wrote:

How can that be if the camera itself is only capable of 8 bits per channel? Yes you can in ACR push your images to 16 bits per channel but that's not the default setting.

12 bits, I thought....

Andy · January 5, 2006

luke_church wrote:

Sorry Andy, I hate to disagree but if the 20D uses the same JPG algorithms as the 350D/XT which it probably does, this isn't quite true, at least not with Photoshop CS2.
Luke

Stay tuned....

luke_church · January 5, 2006

Mike Lane wrote:

How can that be if the camera itself is only capable of 8 bits per channel? Yes you can in ACR push your images to 16 bits per channel but that's not the default setting.

Hi Mike,

So there is a servere possibility of me being an idiot here... However according to Rob Galbraith:

To keep complexity and manufacturing costs down, the Digital Rebel XT reads out data across 2 channels, compared to the 20D's 4-channel readout. Capture is 12-bits per colour, processed to 8-bits per colour or 16-bits per colour depending on file format.

ACR can either or 8 bits or 16, so 16 would be the logical option and pay the 4 bit padding cost.

So it could be that Antonio is using 16 bits. Or am I missing something?

Good point though, I had assumed that ACR was just being sensible offering me 16bit by default, but it could well be that I've changed it to that...

Cheers,

Luke

luke_church · January 5, 2006

DavidTO wrote:

12 bits, I thought....

So it seems, and this is actually more important than I realised immediately.

It means that if Antonio is using 16bits can downsample to 8 bits per image, half the file size and decrease the quality by even less than would be the case 16bits -> 8 bits.

All to the good 'ey?

:

Cheers,

Luke

Andy · January 5, 2006

Hi Luke

The answer is, it depends

Same file, Camera Raw (auto applied and saved as jpg), Camera Raw (no auto applied, saved as jpg), and the straight-from-camera JPG (highest setting)

luke_church · January 5, 2006

Andy wrote:

Hi Luke

The answer is, it depends

Duh! Sorry Andy, when you said that Photoshop Q10 was equal to the 20D, I thought you meant in quality, not file size.

Yeah, I'm sure the file size alignment depends on the image and is in the same order, but the quality is fixed, and slightly different from an editing perspective, if not from a viewing perspective.

Cheers,

Luke

PS. Nice obvious, slap in the face demo of the advantages of RAW

Andy · January 5, 2006

luke_church wrote:

Duh!

PS. Nice obvious, slap in the face demo of the advantages of RAW

R4w r0(k$, 1t r34££¥ Ð03$.

luke_church · January 5, 2006

Andy wrote:

R4w r0(k$, 1t r34££¥ Ð03$.

Andy,

Have you been reading this too much:

http://www.microsoft.com/athome/security/children/kidtalk.mspx

Apple just don't provide this kind of service

Luke

cabbey · January 6, 2006

Ok, so this thread brought out something I've been curious about lately... given a 20D, with the following data points:

sensor = 12bpp (bit per pixel, effective, there is some double blue magic happening such that it's really 16 bpp in the sensor and 12 bpp after digicII is done with it)
cr2 = 16bpp (as it comes out of the camera)
jpg = 8bpp (as it comes out of the camera)

For any given pixel, there are 2^12 different values for each of RGB.

In the case of the jpg, we know a few ways they could do the math to work that down to 2^8... simple division by 2^4 or 16 would be the simplest. (so thereby 0x000 = 0x00 for black pixels, and 0xFFF = 0xFF for full on pixels). (In reality, it's quite likely that they have some far more fancy alg than that, but emprically, I don't see too very much divergence from that simple model.)

But in the case of cr2... what do they do with it? leave 4 bits unused at the top? scale it up? I kinda get the feeling from some of my experimentation that they simply leave the topmost bits unset. In some conversion processes I've played with, this results in the image being *incredibly* dark. Because what should be "fully exposed white" shows up as 0x0FFF instead of 0xFFFF. Now admitedly, you truely have the full range of the camera's possible image at that point. But here's my question... given that data... what the heck do you do with it to get it properly "normalized" so that white is white, and black still remains black. I can think of a few different ways of doing this myself, but none that I really like. Forexample the equally nieve method that goes along with the above is to simply multiply every bit value by 16. Then "full on white" value of 0x0FFF comes out to 0xFFFF... but just one value below it... 0x0FFE comes out as 0xFFE0... and the entire range between those is totally devoid of any pixels. (making for one of the fuglies histograms I can think of.) At which point I started thinking about interpolating from the neighboring pixels....... yuck.

ok, i'm rambling... i should go to bed now....

luke_church · January 6, 2006

cabbey wrote:

In the case of the jpg, we know a few ways they could do the math to work that down to 2^8... simple division by 2^4 or 16 would be the simplest. (so thereby 0x000 = 0x00 for black pixels, and 0xFFF = 0xFF for full on pixels). (In reality, it's quite likely that they have some far more fancy alg than that, but emprically, I don't see too very much divergence from that simple model.)

Maybe, I wouldn't be suprisied if they actually were just bit shifting by 4, the reason being that a 4 bit bitshift can be done asynchronously, without using a CPU cycle (though having said which my knowledge of CPU design a amateur at best, I've only worked with very simple ones)

scale it up? I kinda get the feeling from some of my experimentation that they simply leave the topmost bits unset. In some conversion processes I've played with, this results in the image being *incredibly* dark. Because what should be "fully exposed white" shows up as 0x0FFF instead of 0xFFFF.

IMHO: If people are doing that, they're mad. I can't see any justification for this. If you're using a clever internal representation, but mapping the two colour spaces by zero padding the top bits is plain wrong, as you point out. Suddenty you have a system that maps white to a dull gray. Great....

The only possible advantage I can see is that it allows the user to decide in what manner they would like the colour-space stretch to be performed, i.e. they could use a weighted mapping to achieve a curve live effect.

But never the less, I would argue that this is a colour space deformation to maintain tiny bits of information that should be preservable anyway, and so is just plain wrong.

(If there is an obvious reason I'm missing, I'm of course happy to have it pointed out, Imaging is a relatively new interest, I'm still enjoying the learning curve

: )

Incidentally, do you have an example of anyone doing this? I certainly don't see ACR doing it.

Forexample the equally nieve method that goes along with the above is to simply multiply every bit value by 16. Then "full on white" value of 0x0FFF comes out to 0xFFFF... but just one value below it... 0x0FFE comes out as 0xFFE0...

Hate to be picky, but I'm afraid that your maths doesn't quite work out:

0x0FFF * 16 = 0xFFF0. 0x0FFF is actually 4095 not, 4096, hence the multiplication gives you 65520, not 65536 as you were looking for.

This can be fairly irritiating, the issue being that you're aligning the values to the bottom of their corresponding ranges. You might choose to align to the centre, or you might choose to do a shifted mapping so that both ends saturate, but I suspect if you do that you'll end up hurting yourself because of the floating point round off problems. So you probably just choose which end you want to saturate and you'll deal with the rest later.

and the entire range between those is totally devoid of any pixels.

Yep, but that's pretty much what your information space looks like, you can't really help it.

(making for one of the fuglies histograms I can think of.) At which point I started thinking about interpolating from the neighboring pixels....... yuck.

I don't really see the problem here. If you interpolate the values on your histogram between your pixels, it will be wrong. Now it might be that a wrong histogram is easier to use because you can seem the distribution more smoothly, but I would suggest that just shrinking your histogram display would be a more appropiate action. Remember that to display a full histogram on this colour space you're going to need a 65,536 pixel long histogram, if you can display that, I want your monitor

So the common method of subsampling would be to sum or average (it really doesn't matter) the number of pixels in each category, performing the interpolation 'automatically' and without actually having to conciously break the data. I know of very few people who even use a 1000 pixel histogram, so this spatial compression is going to be done anyway, by a factor of at least 64, which will map at least 4 actual values onto the histogram interval.

I don't think I'm really understanding your concerns?

Hope this helps at least to some degree, if there's anything further I can do, just shout.

Luke

cabbey · January 7, 2006

luke_church wrote:

Maybe, I wouldn't be suprisied if they actually were just bit shifting by 4, the reason being that a 4 bit bitshift can be done asynchronously, without using a CPU cycle (though having said which my knowledge of CPU design a amateur at best, I've only worked with very simple ones)

I wouldn't be surprised for a millisecond if the digicII has exactly that hardware optimization.

IMHO: If people are doing that, they're mad. I can't see any justification for this. If you're using a clever internal representation, but mapping the two colour spaces by zero padding the top bits is plain wrong, as you point out. Suddenty you have a system that maps white to a dull gray. Great....

I can think of a couple raw conversion kits that do *exactly* that. dcraw is the one I've been playing with, since it is open source. Note that you do have to pick an option that has a big disclaimer explicitly stating that it produces dark images, and is there solely for people who know what they're doing in photoshop. As near as I can tell however, it's the only way to get "all the bits" out of the image... any other output format is pre converted to 8bpp for you.

The only possible advantage I can see is that it allows the user to decide in what manner they would like the colour-space stretch to be performed, i.e. they could use a weighted mapping to achieve a curve live effect.

But never the less, I would argue that this is a colour space deformation to maintain tiny bits of information that should be preservable anyway, and so is just plain wrong.

(If there is an obvious reason I'm missing, I'm of course happy to have it pointed out, Imaging is a relatively new interest, I'm still enjoying the learning curve : )

Incidentally, do you have an example of anyone doing this? I certainly don't see ACR doing it.

dcraw is the main tool I've been using to explore the data format, simply because it is open source, and I'm an open source developer. I don't have ACR, I've been unsuccessfull in getting any of the canon software to actually talk to my 20D, and I've not purchased any 3rd party software for this. (heck, I've got a copy of photo shop elements 2.0 that came with my camera, still in the shrink wrap... suppose some day I should try it out.) Up untill recently when I finally moved my primary workstation over to the mac, my only tool for raw conversion was dcraw... now at least I can use iPhoto as well.

Hate to be picky, but I'm afraid that your maths doesn't quite work out:

That's what I get for doing math at 12:30 at night. You're math is correct of course.

Remember that to display a full histogram on this colour space you're going to need a 65,536 pixel long histogram, if you can display that, I want your monitor

It's a samsung syncmaster 213t, which you can't buy any more. But it's only 1600 across... but that's all you need to display all of it. Heck, you can do it on an 800x600 display. You see there is this neato-keen invention called a scroll bar.

I don't think I'm really understanding your concerns?

I'm not sure I have a concern. I have a curiosity. I have a problem I'm playing with. I have an image that comes out of the raw conversion process "really really dark" and I want to get it normalized to something that doesn't look like flies would find it delicious. And I have insomnia again, so here it is at now 2:20 in the morning after a hellish long week and i'm trying to make a rational post. Thanks for bering with me.