Xnormal Messy Textures

moldy

Hello!
I have been learning to use Xnormal recently, and it seems really nice and simple to use. Unfortunately, not for me...

Here is an example of my normal and AO maps.


So what's the problem? I am getting strange artifacts in some parts of my model. Some bits look just fine, other are really messy.
I have set smoothing groups inside of max, I have checked for overlapping verticies/ faces, I have tried using Xnormal/ external cages and nothing seems to fix the issue.
I have been searching quite a bit, and I can't find (or think of) any other solutions.

Anyone knows what could be causing this distortions?
Oh.. And also the AO pass looks a bit too faded, and I am not too sure why.

Baking always was giving me a lot of problems

Eric Chadwick

You can use a cage to fix many of these errors.

The AO requires some tweaking, this might help...
http://www.donaldphan.com/tutorials/xnormal/xnormal_occ.html

moldy

I have used both: the cage inside Xnormal, as the one from max but exported. I believe I have set up the cages correctly, covering all the geometry. No matter what I did, the results are the same.

And thank you for the AO tutorial

JL-Art

Which modeling software do you use? Maya, Max?

Try flipping your green channel. When Xnormal bakes a normal map the Y axis needs to be flipped (for example if you use 3dsmax).

Vysuki

It looks to me that the ends of that bar are attached in the UV and at a 90 degree angle. Add that to the thick edge padding you have and I think that's whats causing it. Try detaching those ends and moving them slightly further away?

moldy

Hey there, so I tried all of your suggestions and it still doesn't like me


Here I tried moving away the particular parts that don't get properly baked.


Flipping Y channel doesn't help as well.


And here are the UV's in case you are wondering what they look like.


Oh yeah, I want to make that clear as well- I used 3ds Max and Zbrush to create low/ high poly.

Eric Chadwick

Extreme gradients don't matter, as long as the model is shading correctly in your target app. Are you going out to UDK, or Unity, or something else? Have you tested it there? The preview in Max is not an accurate portrayal of your end result, as the tangent basis is likely different.

If it doesn't look good in your target app...

Are the end caps on your lowpoly set to a different smoothing group? And are the smoothing groups being imported correctly into Xnormal? (check in the viewer)

You could also add an edge loop on the sides, down near the caps. This will help ease the strain on the normals there.

Quack!

Make sure you are exporting to a file format that xnormal is not at odds with. If you export to FBX and it isn't synched with xnormal you will get funky radial gradients in certain spots. Make sure FBX and xnormal are on the same versions, and try exporting using a different format like xnormal's SBM.

As for you 'light' AO. It looks good to me. That shape won't produce much AO on it's own.

moldy

Okay, thanks for the tips everyone!
Eric- I will be using UDK as my final app. I just have tested my mesh in there and you can clearly see those gradients.

As for the smoothing groups, I have set them all inside 3ds Max, but as I checked in Xnormal, they don't appear in there at all. I think that might be the issue!

Quack- I am using obj files, as this is what seems to be the most common file everyone is using in Xnormal (I have watched quite a few of tutorials, and I think all of them use obj)


Thanks again! I will check out those smoothing groups problem.



EDIT:
So I ran a few test and here are the results:


SM stands for smoothing groups, and it kind of sorted out the issue I had, but at the same time it introduced new problems.

This time I played around with different files, I used FBX, SBM and both with the same results. Also tried using external cage but that doesn't change much.

Now I think that smoothing groups fixed the bottom part, but I got this strange boarders alongside the model. Once again, I'm stuck and can't think of anything else to try out fixing.