another normal map problem :(

disanski

Hey there. I wil try to be short. I read a lot of stuff about normal maps - here and in other forums but there is still a lot more left.
Currently I am trying to bake normal map for my character and i have a lot of cylindrical shapes. I read Ben Mathis tutorial about the normal maps and simply to prevent the wavy lines when baking normal maps for cylinders I need to pull the verts on the cage closer to the cap of the cylinder. well i did that and the edges are looking better with less wavy lines but then the cap of the cylinder is picking up a lot of stuff that is not supposed to(see attached image #3)
why is it doing this. I tried moving the verts on the inner part up and down but I eiather get a lot of black or even more "leaking"


for now I am going to chamfer that top edge and I think this will fix my problems ( I hope) , but I would love to know why is it doing that. Second day trying to adjust the cage .... was it that bad when you started dealing with normal maps ? it is really driving me nuts
Thanks folks

Ben Apuna

My technique for cylinder baking with a cage goes like this:



Hope that helps.

disanski

Thanks a lot Ben. I read about this too but have never tried. As I am very new to this how do you combine them afterwords in PS? As i imagine it there will be some overlapping on the edge.
Thanks a lot

Ben Apuna

You'll end up with two maps one will have good ends the other will have good sides. It's really just a matter of layering them on top of each other in Photoshop then erasing out the bad parts from the top layer.

On more organic shapes it can be better to just push the cage outward evenly just enough so that it encompasses both the low and high poly models. The "wavy lines" can be good sometimes because they help the edges of the ends of the cylinder look more round when viewed at an angle.

EDIT:

Another trick would be to do two bakes one with a evenly pushed cage to capture the edge normals and one bake using the raycast method for the rest. Then combine in via erase in Photoshop.

A even more convoluted way is to use the UV matching technique, that requires your high poly model and low poly model to have their UVs set up and matching each other. Best bakes I ever experienced were using that method, it's really troublesome to set it up, and usually not necessary though...

EDIT EDIT:

I forgot to mention when doing these kinds of things it helps to render out a selection mask texture to help with the erasing in Photoshop.

Just apply a black material to your low poly model (100% self illumination IIRC) and a white material (also 100% self illumination IIRC) to the faces you want to select in Photoshop.

Then render to texture. Now you have a black and white texture that you can place into a new channel layer in Photoshop. You can now use load selection with that channel layer to isolate the faces you want to erase.

I think renderhjs's TexTools scripts has a pick mask function that might make it even easier to set up, though it might be limited to UV shells rather than face selections, still it's very usefull.

disanski

Thank you so much Ben. This is a lot of great info. I have never tried raycast or the match to uv so not sure yet how to use it and I will try your first suggestion. Very helpful reply. I cant thank you enough. Friend of mine saw something else in the problem- the renderings goes true the cap of the cylinder and captures the other high poly geometry penetrating inside the cylinder. I did not know this was possible. Now I am trying to do the 2 renders separately and to merge them. I am also going to try to cut the part of the geometry that is intersecting inside the cylinder to see if that helps.
Thanks a lot again

disanski

So last question before my head explodes .. what are the little dots caused from and can I do something to fix it or take it to PS
thanks a lot guys

Carl Brannstrom

I'm not completely sure but it might have to do with the super sampling. A time ago when I tried to bake a normalmap in max with super sampling. I got some weird, dotted wireframe all over the normalmap. You could try turning it off or bake in another program.

disanski

Thanks Corv. you kind of need to use it dont you? I will give it a try to see what comes out.

MoP

I thought you got the dots when not using supersampling?
Also I have a vague memory that it might be related to the texture filtering settings in the renderer when baking.

disanski

Thanks Mop.
I tried turning the supersampleler off and it reduces them but they still exist. I also changed the size of the filter and also tried different supersamplers - pretty much the same
the funny part is if I rotate the object the dots appear on a different spot weird ...
Nothing can surprise me any more today
Here is an image with he settings I am using but I used them last time and everything was ok .
Thanks for the help guys really appreciate it

MoP

Try turning off "Filter Maps" in that 2nd dialog window you show in the screenshot, see if that helps.

I'd always leave supersampling on when baking, it creates a better result.

disanski

I did try that. The very same result .... it looks like I will have to photoshop them but I need to know why are they happening for the next time...
Bed time now but I will be very happy tomorrow morning if someone knows what this is
Thanks MoP

Ben Apuna

What does your material look like? Is it a standard material with default settings?

I vaguely remember having issues with baking materials set to "display as DirectX" (pink box/something, I don't remember...).

disanski

Hmm not sure where that box is but it is just a standard material

Bokaja

I've had the same bake-dot-issue and the only thing that worked for me was to give the cage's Push Amount one or two click upwards...then hit render - voil

JohnnyRaptor

tried reset xform on low and high poly mesh? :S

disanski

Thanks Bokaja And Johny
i did reset the xforms several times on everything. I am past it now so I hope I dont get the same error again next time because I still dont know why am I getting it.
Colors are being applied at the moment so tomorrow should be ready