Normal Map Errors in Max

doc rob

As I remember we've discussed these kinds of smoothing errors backwards and forwards before, but I can't remember if there was ever any resolution besides adding more detail to the low poly or using smoothing groups (blech).

MoP

first off, awesome gun model.
is that a render or viewport? is there a difference? max 7 or 8?
tried a different shader? tried viewing/generating normals from a different app like xNormal?

cholden

I've had this problem before. Is the lighting error visible in the normal map as well? If so, you acn simple add a bit more triangles to to low poly where the problems occur, and re-render the normal.

poopinmymouth

I've found it's always been a difference in how Max renders the normals and then displays them. Potential problems you might be having:

1. Don't use the cage. At all. Using the cage alters the tangents of the normals, but doesn't bake them there. So it sees them one way based on your cage, and then displays them back in the default. If you don't want overlapping, move all your sub elements and their corresponding high poly elements, away from each other, and then just use ray distance.

2. Sometimes max displays those errors, but then in engine they look fine. This might be the case. I'd test in your target engine (which if it's personal art, might be max unfortunately, in which case, try the various normal map shaders other than the max default)

3. Like Cholden said, it might be too low poly. I know there really shouldn't be such a thing, but on in organic models, I've found sometimes those 90 degree angles can really bite you in the butt. Try chamfering an edge or two and see if that fixes it.

I don't use smoothing groups any more at all on my models, nor the cage. Make sure you're updated to SP3 of Max 8. If that doesn't work still, try baking with a different normal map generator.

Sorry, lots of potential solutions, but hopefully one will work.

MoP

Nice tip Poop, I didn't know the cage (by which you mean Projection right?) screwed with the normals. I'll have to try that out and see if there's a difference.

Also you should be able to use 1 smoothing group for everything. Poop's got some good ideas.

fritz

hey poop. can you elaborate a bit more on not using the cage. i remember on your tut that you used the cage. is ray distance still something you click in the render to texture box?

poopinmymouth

Using a cage is a way to specify ray distance per vertex. However you can also move it so it isn't pointing straight out along the normal. It's that part of the operation (moving from the normal) that hurts the final normal map. There are isolated incidences where you want this (at the end of a cylinder with a very low poly, this makes "scallops" in the normal map, but that can be painted out easily with photoshop, ridding yourself completely of the need for a cage)

Ray distance is a number in the options panel.


In the above image, I now uncheck use cage, and the offset box beside it becomes available. An easy way to get a good number for this, is to reset the cage in the projection modifier, tick "shade" under cage options in the modifier, then scroll upwards in push till it encompasses all. Round up and use that number in the offset box. I need to rewrite this portion of my tutorial, when I get time.

Also forgot to mention, when moving the elements away from each other. Hit F12 to get the transform type-in box and move the elements in 10 and 20 increments so they can be easily moved back. However I just make a save file right before prepping for baking, when all UV's are layed out. Then move the elements away, arrange light's, etc. Save this file as _bake and then bake the actual normal map. Then I can close it, open the pre bake file, and apply the maps to the geometry that is still all in one piece.

doc rob

Thanks guys, I tried all that stuff so I guess I'll just have to add polies. MoP - it was showing up in lots of different shaders/renderers with lots of different normal map generation alternatives.

I guess I never heard a rule stated before about why this happens. Maybe you can't have adjacent vertex normals with relative angles near or greater than 90 degrees. I always thought you could make the low poly as low-poly as you wanted and the normal map generator/renderer would take care of the math. Guess not.

ironbearxl

I agree with you poop on the cage, I've had normals turn yellow/orange when using it. Also, I have to change the resolve hit to closest in some cases.

Poop said: [ QUOTE ]
"If you don't want overlapping, move all your sub elements and their corresponding high poly elements, away from each other, and then just use ray distance."

[/ QUOTE ]

Poop, there's a way to avoid overlapping without moving anything:

Select all your lowpoly elements, if you want to adjust the cage apply the projection mod, adjust the cage and hit make unique. Then in the render to texture dialog choose individual settings, and it will let you individually link each lowpoly element to its highpoly counter part. When you hit render it'll create a seperate normal map for each element, so you'll have to combine them in photoshop.

fritz

geez....see? this is why next gen stuff is frustrating sometimes. oh well...i guess it's new and still evloving.

Noren

Hi DocRob,
if you have the time, could you try to create the Normalmap in Max8 and then render it with the Normal Bump ?
That dents usually occur, when the Tangents/Binormals (or what ever they are called- I'm no programmer ) of the geometry while creating the Normalmap are different from those used at rendertime. E.G when you are using the Metalbump shader that calculates them differently, or even Ben Clowars Shader. It seems the documentation/SDK of Tangentspace-Normalmaps was faulty in Max7.
I don't think that moving around the cage should change anything but the place where things are displayed on the normalmap. It will be calculated corresponing to the Tangents/Binormals and will therefor e.g change colors.
But when Tangents/Birnormals are calculated correctly at rendertime it should look fine, no matter how distorted the cage was. Perhaps a programmer could clear that up and use the right terms.

edit2: Of course, if you move vertices of the cage around, the details will move around on the lowpoly or be distorted later on, since the rays are casted now from a different postition. But the Normals should be ok.


edit:
Then there is the case where just too many edges/different angles meet in one point and the smoothing gets fucked up in general, (like two Cubes joined at a corner, e.g.). But this doesn't seem the case here.

MoP

I think it's actually an error with the way Max renders normals, I did some tests using objects in Max7 and xNormal, and xNormal produces a perfectly clean and nice normalmap, while Max makes one that has these shading artifacts from the lowpoly normals actually baked into the normalmap.
I'll post some pics soon.

doc rob

Noren, I did create and render in Max 8 - its not a package disparity. Up until this case, I agreed with you that anything should look correct as long as it's calculated the same way both times.

Noren

I did some quick tests in 8 and normals were perfect.
Mirroring and rotating uv-chunks worked fine.
There is also one Realtimeshader that ships with Max, without specular unfortunately, that seemed to work fine.
Of course I could have missed something.
As stated those were some quick tests and unfortunately I didn't keep them.

edit:
@ doc_rob.
Then I'm clueless I'm afraid.
I was testing even with 90 degree angles and it seemed fine.

doc rob

MoP, I had some trouble with xNormal. I'm going out of town for a few days but I'll give it another try when I get back next week.

Noren, if that doesn't work maybe I'll send you the model and you can check it out. Only if you're interested, of course.

MoP

Sure thing!

Noren

I´d be very interested. Better than trying to reproduce it myself.
Though I´m starting to get afraid that my enthusiasm about how great things work now might have been unjustified.
I´ll send you a pm.

Whargoul

I can't believe Max casts the rays in the direction of the cage: that totally skews your normals!

Noren

See it positiv.
Max gives you the control from which direction it casts the normals. If you don't want to use that feature you can just move the cage-vertices along their normals or uncheck the checkbox for the cage alltogether.
Unless you were ironic of course, english is obviously not my mothertongue.

Noren

Due to my impatient nature I did a quick mockup of your model. Hope you don't mind and it's of course a very reduced testcase so it's most likely not valid for all situations. I simply put some chamfered boxes and cylinders together for the "High"-poly.Please ignore the lazy uv-layout also
Though the highpoly is lacking detail and is quite boxy/flat I think it's a good test to check for bumpiness/dents in the normalmapped lowpoly.
I'm also tempted to say the outcome is pretty much perfect.
BUT.. (there is always a but )
up till now there seems to exist no puplic viewport/realtimeshader which is properly configured for the normalmaps produced by Max. You'd think Autodesk would provide something like that after choosing to create their own normalmapping-standard, but I'm afraid they didn't so far. Could also be that their implementation is too slow for realtime-rendering. But I'd be happy to learn that the opposite is the case.