Hey,
If this is a common question I'd like to apologize in advance but I haven't found an answer to this by myself and it's driving me crazy.
I made a high-poly model and low poly in maya, unwarped and baked in XNormal. Most of the model is working well except on a flat surface where you can actually see some triangle and I have no idea how to fix it. It is meant for Unreal 4 but I don't have my license yet so I'm testing in Marmoset for the time being.
Here's the issue :


I have no idea if my baking is the problem there, or my low poly or the smoothing. Honestly I'm lost :I
Thanks in advance.
Replies
I bake using a cage, I make sure the offending faces (and their cage) are planar, but nothing I've been experimenting with has helped.
It appears the side and inner portion of the model are sharing a smoothing group, creating harsh normals near the top.
However, that doesn't explain the bright spot you have on that side. Perhaps a you baked on mirrored UVS?
A friend of mine came with a theory that somehow could fix it. This is a completly smoothed object getting baked, for most of it it's working great except these weird offending faces of sort. I baked the same model with a few more hard edges, edited some errors that could kinda break the render and this problem disappear. Of course these faces appears only on the smoothed baked.
I think I might have simply done something wrong in my workflow.
As a general rule of thumb anything angles over say... 80 degrees should have a hard edge between it and the adjacent geometry in addition to a UV split. Sometimes you can get away with more, sometimes you'll see issues from less. It's all a balancing act between the low polies normals and the normal map. The more you bake the more you'll get a feel for where/how errors like this will occur :thumbup:
Edit: If the hard edges/edited smoothing groups don't help, ensure those verts are welded. This is still a strange issue to see from harsh normals, so I'm kind of guessing it may be a broken weld.
You should bake with a cage, if you want to use hard-edges. This won't give you any of the errors you had to edit out.
Basically, you duplicate the model, soften all the edges, and inflate it until it completely encapsulates your high. Load this in as a cage mesh in xnormal.
Then for the low, harden all UV border edges. Any edges more than 80 degrees or so should have both a hard edge and a UV split. Any hard edges without a UV split will give you errors.
This thread explains it all. It's well worth reading through, especially any posts by Earthquake.
http://www.polycount.com/forum/showthread.php?t=107196
May as well read this one too:
http://www.polycount.com/forum/showthread.php?t=81154
Sense his UV's already had a split I didn't really feel the need to mention he needs a split, but you're absolutely right. I often forget other people stumble on threads like these that might not know
GREAT set of links Joopson posted here OP, give them a read and all things baking/normal related will become much easier.