Help needed: Potential Photoshop bug RE: Flattening of Layers

Richard

Edit: This thread has been retitled. It started as a question I had about a problem I was having with a specific pic. Further investigation showed that I had stumbled on what now appears to be a fundamental bug in Photoshop. Read on, and if you are a Mac user, please try to replicate the little tests that Nik and I describe later and report your results to this thread. Thx.

-R.

---

I don't understand what's going on here. The appearance of a pic is changing when I flatten or merge to a new layer using ctl-alt-shift-E. Here is what I see before I do the merge:



This is the layers palette:



The top (merged) layer has been turned off. The color fill layer is 50% gray with added noise. It is in hard mix mode with fill at 64%. This gave me the grainy, high contrast effect I wanted. But when I do a merge, the result does not look the same...it looks like the hard mix effect is lost:



The result is the same if instead of creating a new merged layer on top I just flatten the image. I don't get it. Why should merging change the appearance?

Thanks,

pathfinder

Richard, have you tried collapsing one layer at a time with ctrl-E , rather than all at once? That might help identify where the problem is occurring.

Richard

pathfinder wrote:

Richard, have you tried collapsing one layer at a time with ctrl-E , rather than all at once? That might help identify where the problem is occurring.

Good thinking, Jim, but now I am even more perplexed. I tried it two ways: first, I merged down, starting from them Background Copy layer and moving up in the stack. Everything looked fine till I merged the Color Fill layer. It took a rather long time to complete the final merge, and the appearance changed to the low contrast image. When I started from the top (Color Fill layer), the merge down option was not available on the layers menu and ctrl-E did nothing. So I selected both the Color Fill and B&W layers and from the Layers menu, I hit Merge. The result had the noise, but the contrast seemed to be reduced and the image was in color. My conclusion is that the Color Fill layer is the source of the problem, but I'll be damned if I know why. Smart filter issue? Hard mix issue?

SloYerRoll

It's your blending options working togther to make strange things. Smart filters don't act that wway (that I know of)
PF has you on the right path.

If you still have problems. Shoot me a PM w/ your email and we'll get this sorted out.

Richard

SloYerRoll wrote:

It's your blending options working togther to make strange things. Smart filters don't act that wway (that I know of)
PF has you on the right path.

If you still have problems. Shoot me a PM w/ your email and we'll get this sorted out.

Thanks for the offer, Jon, but it might be helpful to others if we keep the discussion public.

I did another experiment which makes it look like the problem is the smart filter. I took the same image and followed the same steps, except that this time, I did not convert the color fill layer for smart filter and just applied the noise to the fill. With the same blending options, the merge was correct.

I only recently upgraded to CS3, so I don't have a lot of experience with smart filters yet. It may be the interaction of the smart filter with the specific blending mode that caused the problem here, or perhaps the order that I did stuff in. Regardless, I was used to having confidence that when you do a merge what you see is what you get, and I find it disconcerting that this wasn't true here.

Regards,

SloYerRoll

rsinmadrid wrote:

Thanks for the offer, Jon, but it might be helpful to others if we keep the discussion public.

Hey Richard,

I was just going to exchange emails via PM so I could look at the file then post results in here.

I'm interested in seeing what it is regardless. My invite still stands if you want.

Cheers,
-Jon

Richard

SloYerRoll wrote:

Hey Richard,

I was just going to exchange emails via PM so I could look at the file then post results in here.

I'm interested in seeing what it is regardless. My invite still stands if you want.

Cheers,
-Jon

I'm interested. PM sent.

pathfinder

rsinmadrid wrote:

Thanks for the offer, Jon, but it might be helpful to others if we keep the discussion public.

I did another experiment which makes it look like the problem is the smart filter. I took the same image and followed the same steps, except that this time, I did not convert the color fill layer for smart filter and just applied the noise to the fill. With the same blending options, the merge was correct.

I only recently upgraded to CS3, so I don't have a lot of experience with smart filters yet. It may be the interaction of the smart filter with the specific blending mode that caused the problem here, or perhaps the order that I did stuff in. Regardless, I was used to having confidence that when you do a merge what you see is what you get, and I find it disconcerting that this wasn't true here.

Regards,

I was going to ask what happens when you add the noise filter as the final step in your image, rather than in the smart layer as you have it. I am not sure I understand what is going on here either. Like you, I am still learning about Smart Objects.

Nikolai

Interesting... I'd like to know what's the end of this..
EDIT1: I reproduced your original laeyrs and got exactly the same effect
EDIT2: I rasterized fill color layer yet still got the same effect

SloYerRoll

Good idea on checking if recreating the layers produced the same results Nik!
Now I don't have to FTP the original 115MB file. I can just use the smaller one!

Richard

Nikolai wrote:

EDIT1: I reproduced your original laeyrs and got exactly the same effect

Cool...that means that I probably didn't do anything stupid without realizing it. Curiouser and curiouser... I'll play with it some more tomorrow unless someone else figures it out first.

Cheers,

pyrtek

Why don't you simply create a similar layer set without the use of Smart Objects and try to merge that?

Richard

pyrtek wrote:

Why don't you simply create a similar layer set without the use of Smart Objects and try to merge that?

Yes, I did that and it solved the immediate problem. But now I'm hung up on why the merge behaved badly using the smart filter. I use an action to flatten, convert mode and save two different JPGs when I am done processing pics, so it is important to me to know that flattening will not change the appearance. I don't like surprises later.

pyrtek

rsinmadrid wrote:

Yes, I did that and it solved the immediate problem. But now I'm hung up on why the merge behaved badly using the smart filter.

This could be either a bug (even Photoshop has them), or simply a known limitation
of Smart Filters. Perhaps a visit to the Adobe forums would yield the answer?
I've tried googling the problem, but it doesn't seem to come up.

Nikolai

pyrtek wrote:

Why don't you simply create a similar layer set without the use of Smart Objects and try to merge that?

As I mentioned, I tried no-smart-as...object approach. Final effect is the same.
Looks like a combo of a fill color layer + blending mode + stamping...

Richard

Nikolai wrote:

As I mentioned, I tried no-smart-as...object approach. Final effect is the same.
Looks like a combo of a fill color layer + blending mode + stamping...

I think I misunderstood your earlier post, Nik. Are you saying that even when you didn't use the smart filter, the hard mix effect disappeared when you merged? If so, your result was different than mine...with a plain old fill layer, the final merge kept the hard mix. If it's not too much trouble, maybe you could post your results.

I've had enough for today, but I'll pick up this thread tomorrow.

jfriend

rsinmadrid wrote:

Yes, I did that and it solved the immediate problem. But now I'm hung up on why the merge behaved badly using the smart filter. I use an action to flatten, convert mode and save two different JPGs when I am done processing pics, so it is important to me to know that flattening will not change the appearance. I don't like surprises later.

It sounds to me like a bug with smart objects. You might try posting on the Adobe forum and see if anyone there can speak to it. If it's reproducible with a set of fairly simple steps, someone will probably jump on it. There's also a bug report form on Adobe's site which I have used before, though I'd admit that I have no idea whether anyone actually reads it or not. A reproducible case is probably the most likely bug report to get paid immediate attention to if it isn't already known.

Nikolai

rsinmadrid wrote:

I think I misunderstood your earlier post, Nik. Are you saying that even when you didn't use the smart filter, the hard mix effect disappeared when you merged? If so, your result was different than mine...with a plain old fill layer, the final merge kept the hard mix. If it's not too much trouble, maybe you could post your results.

I've had enough for today, but I'll pick up this thread tomorrow.

Yes.

My layers:
--> color fill adjustment layer (gray) + mask removed + noise added directly to layer + hard mix + 50% fill (layer selected)
--- background

Stamping leads to disappearance of the hard mix effect (noise stays), as if it was Normal mode.
Merging down keeps everythign intact.

CS3 (10.0.1, all recent updates).

Richard

Update:

I looked at Adobe's forums as John suggested, but could not find any mention of this problem. This may be because the site is really poorly organized and the search function does not work well. Ironic that the leading provider of Web and graphics tools has such an awful site. But I digress... I did find a few references on Google to similar problems with hard mix, but nobody seemed to have a clear solution.

Now for the really bad news: I put my debugging hat on and tried to create the simplest possible case that showed the problem. It turned out to be easy. I opened a JPG, duplicated the background layer and added noise to it. Then I changed the duplicate layer blending mode and tried two different methods of merging: first I used ctrl-shift-alt-E to create a new merged layer on top. Then I hid that layer, selected the background copy layer and used ctrl-E to merge down. I repeated this process for every blending mode in CS3.

The majority of the blending modes changed appearance after merging and the merge method made no difference. Only the following modes worked correctly: normal, multiply, screen, overlay, exclusion, saturation and color. In each of the other 19 modes, the image changed.

Perhaps my mental model of Photoshop has been wrong all along. I have always assumed that what you see is what you get when merging visible layers. Is there something I am missing here or is this really a serious bug?

The results I saw were so shocking that I suspect there may be something wrong with my system (if not my brain). It is hard to believe that nobody has noticed this before. I am running CS3 10.0.1, which I installed less than a month ago on a WinXP Pro SP2 machine. I sincerely hope others who have been following this thread will attempt to duplicate my findings. For everyone's sake, I hope you can't.

jfriend

rsinmadrid wrote:

Update:

I looked at Adobe's forums as John suggested, but could not find any mention of this problem. This may be because the site is really poorly organized and the search function does not work well. Ironic that the leading provider of Web and graphics tools has such an awful site. But I digress... I did find a few references on Google to similar problems with hard mix, but nobody seemed to have a clear solution.

Now for the really bad news: I put my debugging hat on and tried to create the simplest possible case that showed the problem. It turned out to be easy. I opened a JPG, duplicated the background layer and added noise to it. Then I changed the duplicate layer blending mode and tried two different methods of merging: first I used ctrl-shift-alt-E to create a new merged layer on top. Then I hid that layer, selected the background copy layer and used ctrl-E to merge down. I repeated this process for every blending mode in CS3.

The majority of the blending modes changed appearance after merging and the merge method made no difference. Only the following modes worked correctly: normal, multiply, screen, overlay, exclusion, saturation and color. In each of the other 19 modes, the image changed.

Perhaps my mental model of Photoshop has been wrong all along. I have always assumed that what you see is what you get when merging visible layers. Is there something I am missing here or is this really a serious bug?

The results I saw were so shocking that I suspect there may be something wrong with my system (if not my brain). It is hard to believe that nobody has noticed this before. I am running CS3 10.0.1, which I installed less than a month ago on a WinXP Pro SP2 machine. I sincerely hope others who have been following this thread will attempt to duplicate my findings. For everyone's sake, I hope you can't.

The Adobe forums are a mess. They actually tried a major upgrade and it went sooo bad that they had to back it out. I think they're trying to fix it but apparently don't have their best people on it.

There have always been lots of situations where you cannot just merge two particular layers when there are other remaining layers, particularly when using blend modes and/or adjustment layers. You should, however, always be able to flatten or do merge to a new layer on top without any change. I've never considered this a bug, just a limitation of what merge two layers can do.

The reason you can't always merge two particular layers is something I understand why it is, but I'm not sure I know it well enough to explain it - but I'll try anyway.

Here are some examples:

Things you should always be able to merge (and it's a bug if you can't do it without the image changing):

• One regular pixel layer into another when both are in normal blend mode
• Flatten all layers with no change to the image
• Merge all to a new layer on top with no change to the image
• Marge an adjustment layer into a pixel layer that is set to normal blend mode

Things you usually can't do and get a good result:

• Merge two adjustment layers
• Marge two smart filters
• Merge any two layers where the bottom layer (which determines the merged blend mode) has a blend mode other than normal
• Merge any two layers where the bottom layer is not a pixel layer (e.g. it's an adjustment layer or a smart filter)
• Merge any two layers where the bottom layer has a mask that isn't the same mask as all the other layer layers in the merge

Nikolai

Interesting... Ctrl+Alt+Shift+E (stamp) did fail me in the past, so it was not a big surprise.
But I always thought the Ctrl+E (merge down) or Ctrl+Shift+E (merge visible), or Flatten works no matter what...

Richard

jfriend wrote:

Things you should always be able to merge (and it's a bug if you can't do it without the image changing):

• Flatten all layers with no change to the image
• Merge all to a new layer on top with no change to the image

My test covered the two cases above (though I did use merge down rather than flatten). In all but the modes I cited above, the image did change.

I did a sample test (not all the modes, just vivid light) and found that flatten also fails, just like merge down. I also did a sample test and found that this problem is present on CS2 as well as CS3.

Edit: I went back and tested all of the modes using flatten instead of merge down. The results were slightly different from the original test in that luminosity mode seemed to work while saturation did not. This may be because I used monochrome noise in the original test, but color in this last one--I'm not sure. In any event, most modes failed. What you see is not what you get in this very simple case.

Nikolai

Richard,
something's wrong...

I did what you said: open a JPEG, duplicate layer, add noise, change mode.
Ctrl+E works perfectly regardless of the mode.

Now the funny part: Ctrl+Shift+E (Merge Visible), Ctrl+Alt+Shift+E (Stamp Above) and even Flatten - all lead to the same "wrong" result (that is if the mode is not in your safe list).

Can you please give Ctrl+E (merge down one layer) another go?

BTW: if there is a chain of adjustment layes in non-trivial modes, the order in which you apply the MergeDown command *should* matter, so please be careful with that.

Nikolai

Well, FWIW, I submitted the bug report via the link John provided (thx, man!).

jfriend

rsinmadrid wrote:

My test covered the two cases above (though I did use merge down rather than flatten). In all but the modes I cited above, the image did change.

I did a sample test (not all the modes, just vivid light) and found that flatten also fails, just like merge down. I also did a sample test and found that this problem is present on CS2 as well as CS3.

Edit: I went back and tested all of the modes using flatten instead of merge down. The results were slightly different from the original test in that luminosity mode seemed to work while saturation did not. This may be because I used monochrome noise in the original test, but color in this last one--I'm not sure. In any event, most modes failed. What you see is not what you get in this very simple case.

A few other discussions of this issue, though nothing conclusive:
Merging Layers: Image Changed by Merge
Merge layers loses effects
Sometimes PS CS3 is unable to merge or flatten layers
CS3 major layer issues
Why merge down sometimes doesn't work

FYI, you can use Google to search the Adobe forums by appending "site:adobeforums.com" to your Google search.

Richard

Nikolai wrote:

Richard,
something's wrong...

I did what you said: open a JPEG, duplicate layer, add noise, change mode.
Ctrl+E works perfectly regardless of the mode.

Now the funny part: Ctrl+Shift+E (Merge Visible), Ctrl+Alt+Shift+E (Stamp Above) and even Flatten - all lead to the same "wrong" result (that is if the mode is not in your safe list).

Can you please give Ctrl+E (merge down one layer) another go?

BTW: if there is a chain of adjustment layes in non-trivial modes, the order in which you apply the MergeDown command *should* matter, so please be careful with that.

Interesting...I was able to duplicate your result with just ctrl-E (I used vivid light and did not try any other mode, but I'm guessing it will work with all of them). I was able to figure out why this result differed from what I saw yesterday. Try the following sequence:

Open image
Dup layer
Add noise
Set to one of the "dangerous" modes
Ctl-shift-alt-E (image changes)
Turn off visibility of top (stamped) layer
Select Background copy layer
Ctrl-E (image changes)

So it looks like the presence of the top stamped layer screws things up even if it is not visible. This was actually how I was doing stuff yesterday.

So that leaves us with the following:
Flatten fails,
Stamping fails
Merging down works if other layers are not present.

Has anyone verified these results on a Mac?

Nikolai

rsinmadrid wrote:

So it looks like the presence of the top stamped layer screws things up even if it is not visible. This was actually how I was doing stuff yesterday.

Just the presence of the extra hidden layer (does not have to be stamped, I used another copy, in any mode, including normal) screws things up, even for Ctrl+E

Richard

Making progress. I posted on the Adobe user forum (thanks, John) and got pointed in the right direction. Thread is here.

The problem was my viewing size. When the image is viewed at greater than 50%, the difference in appearance disappears. Since I work on a laptop, I generally work at around 33%, except when I am doing sharpening.

While I understand that distortions are inevitable when an image is downsized, it is still not clear to me why the two 33% views should be inconsistent with each other. More important, I still don't know which of the two is the truer representation of the full image. I tried to figure that out, but couldn't decide.

I should point out that this problem may be very specific to images with added noise, though I am not certain of that. Experiments with curve and saturation adjustment layers set to different blending modes showed no difference in the before and after flattening views at 33%.

So I guess the bottom line is that now I have a good justification for buying a huge, very high resolution external monitor. Right?

Cheers,

Nikolai

rsinmadrid wrote:

Making progress. I posted on the Adobe user forum (thanks, John) and got pointed in the right direction. Thread is here.

Aaaaaaaa....
Gotcha! So "it's only in your mind" effect... I still don't get it why hidden layers or different merge methods should lead to different results, even if only "mentally".
Anyway, thank you for getting to the bottom of this!

Richard

Nikolai wrote:

... I still don't get it why hidden layers or different merge methods should lead to different results, even if only "mentally".

In my mind, there really is a bug, but it is not what I thought it was. There is no inconsistency in the previews at 100% using any of the methods. The inconsistency is in the smaller sized previews, which must either be calculated differently or be based on different data sets depending on the method.

SloYerRoll

I've never heard of this in Photoshop. This has always been in the case of differennt zoom percentages in Illustrator and vector based PDF's though.

Thanks for that link Richard. Anyone that reads this will be saved allot of frustration if they know about this ahead of time!

Cheers,
-Jon