Getting clean masks in Photoshop (from 3ds Max)

Playdo

I posted this question in a render specific forum, and whilst I got replies from pros, it's five days later and I'm still stuck with this. I'm hoping one of you guys can help me to get it.

I want to get clean cut-out alpha passes (ObjectID or Matte passes specifically) out of Max, to use as masks for adjustment layers in Photoshop. I can't get rid of the hairline halo that is noticeable when the hue is altered.

I've tried everything that I've heard of; straight .tga, premultiplied .tga, defringing, remove black matte, remove white matte, duplicating the alpha and multiplying, duplicating the alpha and adding (linear dodge), and every combination of. I've still got the problem. I've read loads of threads and tutorials and I'm nearly exhausted of this.

Below is the test scene that I made. Image 1 is a master render. Image 2 shows how the close up looks, image 3 is the alpha (objectID pass) and image 4 shows the close up when the alpha's applied and the hue is greatly shifted. I've attached the scene file and all relevant render passes that I made.

By shifting the hue to the other side of the colour spectrum, it's made the halo very apparent. If you know how to solve this, then please take a look at the files and let me know where I'm going wrong. Please try it first with this test scene though as with each bit of info that I've been given, I've felt like I'm going around in circles. It's caused me difficulty long before this and I really need a solution.

Thank you

Image 1


Image 2


Image3


Image 4


http://www.fileswap.com/dl/W1gs9bq/Alpha_Issue.zip.html

Ark

Looks like its to do with the alpha being pre-multiplied.

Playdo

:poly127:

Then please try the scene and post how to solve it. I've tried rendering all different settings in the pass and tried everything I know of to get rid of the premultiplication in PS; eg rendering out straight .tga file, then tried multiplying (and adding: linear dodge) in PS. Increasing the brightness/contrast to make it opaque (causes aliasing).

I've literally spent more than 40 hours this week trying to solve this. I just don't get it; everyone's saying just do this and just do that, but none of it's working.

..now where's an imploding head emoticon when you need one.

Ark

When you come to saving out the image file, there a option in the dialog box to get a straight alpha instead on the pre-multiplied one.

Try doing that, or Photoshop and most Compositing have tools to help in remedying this issue.
http://www.polycount.com/forum/showpost.php?p=1290902&postcount=3

Playdo

Re-read my first post Arc.

dur23

The problem is that photoshop doesn't handle the alpha's correctly. I know that after effects does work properly.
Thankfully theres a workaround in photoshop.

http://www.digitalartform.com/archives/2005/10/compositing_pre.html

Edit: never mind, i see you tried this method.

dur23

I'm convinced that something is messing up in MAX. The alpha and beauty don't line up. In any of your files.

Playdo

Dur23: Yes, I tried that other method. It's for compositing objects separately. I tried to copy that method over and use it to just mask a master pass, but it didn't work. I think it's because an isolated object render can be brought in unpremultiplied (aliased) whereas a master pass can't. Maybe, I was doing it incorrectly or maybe it can't work for this. I don't know for sure.

You've looked at the files - great, thanks. You say they don't line up? The image I posted shows that there is an equal halo on all sides of the object. If the alpha was misaligned it would be one sided. Or do you mean that the Alpha outline doesn't perfectly match the beauty outline? Any idea why it's happening?

dur23

Playdo wrote: »

the Alpha outline doesn't perfectly match the beauty outline? Any idea why it's happening?

It appears no matter how i look at it, the alpha is 1 pixel bigger on all sides. I don't actually use max, so i'm at a loss of understanding what it is that is going on. I'm sorry i couldn't be any more help. Did you try the area forums or maybe the max forums on cgtalk?

Playdo

Thanks for taking a look. How can you tell that the alpha is 1 pixel bigger and that it is not the result of transparency happening at the edges instead?

If it is, as you say, a 1 pixel larger alpha, then it should easily be reduced in Photoshop. But as much as I've tried, I haven't resolved it.

I've tried importing into A.Effects and exporting to Photoshop but still no luck. I'm not totally sure if my procedure was correct here though. If you know how, do you have time to test if it works?

I haven't tried The Area but I put a thread up on the main page of CGTalk. Lots of ideas given but nothing's fixed it for me and no one's tested the file to see if it works. Really strange, considering that a clean alpha in Photoshop must be a common requirement, yet no advice I've tried has solved it.

Playdo

I've rendered both the master pass and the alpha with no anti-aliasing. It clearly looks like the alpha pass is 1 pixel larger than the master pass.

Does anyone know how to prevent this?

Snader

I dunno what the exact workings of this problem but how about this:
make a black material, 000,000,000 no specular no gloss no nothing
make a white material, 255,255,255 100% self illuminated.
assign materials to objects you want layered out,
render
use as layer mask.

Ben Apuna

I've had good experiences with rendering 32 bit PNGs, out of Max. IIRC they came into Photoshop (CS2) with perfect transparency, no fringing/background colors.

Perhaps this post by kodde from this thread will help a little.

leechdemon

Ben Apuna wrote: »

I've had good experiences with rendering 32 bit PNGs, out of Max. IIRC they came into Photoshop (CS2) with perfect transparency, no fringing/background colors.

Yeah, I second this. I've always used TGA's for alpha maps, but changing the file format can help a ton. Also, ignore that it's alpha altogether; save the Alpha out as an RGB image, and then just paste it manually into a Layer Mask inside of Photoshop.

Photoshop handles Alpha differently in different file formats, and handles opacity different from full layer information, as well. Spitting it out as a different format, or splitting out the alpha as RGB, might help.

Mark Dygert

Yea we switched from rendering tga's to png's and had some pre-multiplied alpha issues. Photoshop and AfterEffects handled them differently than max does and if you open up a max render in photoshop and do nothing more than hit save, it will rewrite the file and fubar the alpha. AE didn't seem to have that problem but had other settings that caused issues, I forget the specifics it was 2 years ago...

We avoided saving any alpha out of photoshop unless we had to, then they had to be tracked and processed separately, it was a giant pain until we upgraded our tools to be smart and reprocess all alphas bit by bit to be a special format we use.


Have you tried using render elements to output alpha masks?

Playdo

I'll give everything I haven't tried a go, and post back.

Playdo

Just in case there's any confusion as to what I was trying to do. I wanted to render a single master pass and then matte passes for adjustment layer masks. Some of the suggestions would only work if the objects are rendered as separate objects on black, then composited.

Ok, below's my PNG testing (was no 32-bit) / RGB Alpha / render element alpha masks / ... Still the same problem ..just varying degrees of it. The only working solution I have is to render aliased at double res and resize. Unfortunately this leaves me with massive file sizes. I've tried every single suggestion on all forums, and no-one has ever posted a working image of the scene. I don't think it's possible.

I'm done with this now. I can't bare to think about it anymore with the amount of time I've spent on it without finding a simple solution. There's also a link to my summary if anyone's interested in what I found out.

http://forums.cgsociety.org/showthread.php?f=2&t=959439&page=3

http://www.fileswap.com/dl/NenqFgF/Halo_PNG.zip.html

swytch

Playdo,

I may be overlooking exactly what you are trying to do, but would old school alpha pass technique of white 100% self-illuminated, and solid black, materials work any better?