xNormal Normal Map ray distance error

timotronprime

I am generating normal maps using an exploded high and low poly meshes. The cage ("external cage file") is a duplicate of the exploded low poly, except with all edges set to be soft edges.

https://i.imgur.com/JstHX.jpg
The normal maps seem to be catching details from pieces extremely far away that I would not expect at all.

The attached images show a few examples of my problem area (in red circles) and where they are projected from (in green circles).

I have a normal map to show my UV layout. I do use overlapping textures but that shouldn't be causing the problem as my projection setup (the exploded mesh) only includes one instance of every UV.

So... I tried lowering the maximum frontal/rear ray distance value in xNormal's "Low Definition Mesh" tab and it didn't fix the issue. Unchecking "closest hit if ray fails" yields me a flat normal map.

I'm wondering if anyone has any insight on the issue.

Thanks in advance.

Ace-Angel

Did you try and see if using the 'Ray Distance Calculator' in one of the other panels helps with the distance rays?

Try and 'ray bake' without the cage and see if that changes anything?

Also, try and use the cage directly as an SBM export (I think both Max and Maya should allow this as of the latest version properly?).

Are you offsetting the UV's which are overlapping by 1 unit to either the left or right? Always do that so they don't capture the wrong detail at the wrong parts.

Try and upscale the model size, maybe there is a unit size issue between XN and your App?

Lastly, what does your cage look like? Sometimes Max and Maya will not show the cages accurately at all, what looks 'covered' is actually still poking out.

However, with all of that said, I can confirm that I have been having this issue and I ended up manually fixing the normal maps by hand in PS. I was lucky since the model that was having the issue was an organic body, so alot was forgiven.

I baked 2 normal maps, one caged for the details and to avoid any overlapping bakes, and one with rays and no cage to correct the large details of the mesh.

timotronprime

The "Ray Distance Calculator" didn't seem to change anything. Also, I thought the ray distances get overriden when there is an external cage file present, is that a misconception on my part?

When I tried another bake with the calculated ray distances without using a cage, it did change, but not for the good. But if this were to work, wouldn't I still need a method to get an averaged projection?

The overlapped parts are not used in the projection, so all the meshes used to create the normal map have no overlapping.

The model size is scaled to fit the UDK UTDeathmatch player size, which as far as I know, is pretty big already (around 150 maya units in height).

My cage is an exact duplicate of the low-poly mesh but with all edges as soft edges instead (I believe that is "1 smoothing group" in 3DS Max terms, I'm a Maya/Softimage user so I may be incorrect on that).

I saw in the xNormal Master Thread you were suspecting that the current xNormal may have ray distance bugs but nobody was to confirm/deny it. I really hope that is the case as I really want to get a "correct/proper" bake this time without hand editing in Photoshop.

Ace-Angel

Hmm, I don't know what else to say since everything seems to be in order, and yes, you don't need rays when you're using a cage, I just asked for you to try it out, just to make sure XN is working and didn't bug out or something (it happens sometimes after mutiple bakes and long times in memory).

And yes, I still have these problems with XN, but it seems like it's only an 'issue on certain mesh', which I have been unable to figure out as to why or where or how to know which mesh is doing this issue.

Um, what about your vertices? Stuff like Freeze Transformations (in Max it's called Reset XForm), reset pivots, etc, basically cleaning up the mesh before an export down to the smallest detail?

Also, how big is the cage around your mesh? Did you try and see if an extra push or two changes anything?

Try a quick flip normals on your low poly and flipping them again (unify them in the same direction)?

If you bake your model with the cage, inside your app, do you get proper bakes out?

I know it sucks, and I have no idea on why this is happening, but I'm guessing until we get a fix from XN and it's author/users, the next best solution is an internal bake in your app?

timotronprime

Upon further testing, it doesn't seem to be xNormal's problem. Using Maya and Softimage's baker/mapper gives the same error.

Giving my cage mesh an even further push value was it.

Thank you for the help, Ace-Angel.