xnormal bakes highlights edges

Duckmamoll

Hello,

I have been trying to bake some high poly rock normals onto low poly version of themselves using xnormal, but whatever I do the normal map ends up making edges more visible than they should be, here's an exemple in UE4.



It does the same thing in blender.
Those are not Uv seams as there is no uv cut there. I've looked at a lot of forum but I can't find anything. I've tried baking with a cage (did close to nothing), making the low poly entirely smooth and baking (which didn't work at all), scaling the mesh by a lot. Nothing does it.
It does the same thing on my other bakes.
I've tried baking it with blender same result.
Here's how most of my bake result look

If someone knows what does this please tell me, I am completly lost.
Thank you . 

Neox

Your normals should be soft. If you have hard edges, you need to split the UVs and safe some.space for padding. That said, in this case just make them all smooth

Duckmamoll

making them all smooth totally broke the normal map and it looked awful once applied, how do I know which I'm supposed to make smooth?

poopipe

You need to smooth them before you bake

Duckmamoll

yes I now understand that. I tried smoothing the whole object and baking with that, but it makes the normal map looks awful


This how it looked if it bake with the low poly set as smooth, I tried doing the same on a more complex mesh and it looked even worse.

poopipe

Assuming your high res mesh is smooth those artefacts are consistent with the low poly mesh having hard normals so  It's either the mesh or some setting in the Baker that's causing it.

The mesh you bake to and the mesh you apply the map to must have identical normals so ensure they're being treated the same in every tool 

Ue4 can override normals at import, most bakers can override normals at bake time and your modelling package can produce total garbage and have the mesh look fine. One of those things is likely causing the problem - which one I couldn't say

Duckmamoll

so does the high pol or low pol need to be smooth? i thought it was the low poly.
In blender is the difference between hard and soft normal just the soft and flat button in object mode? I don't know too much about those things. How do I make sure I have identical normals ?

edit : Baking with both set to smooth gave this result 



so still didn't work I feel completly lost


Double edit, I tried this normal map in blender and it looked right, which I think, means it's your last theory of UE4 making it look like trash, what provokes this ? (I obviously already inverted the green channel)

triple edit : I finally found how to do it, since I realized the bake with smooth low poly and hard high poly worked. It made me realized I also need the smoothed model to be imported in ue4 and not the hard one, so the 2 combined together worked.

Thanez

Only to clarify: Hard or smooth highpoly doesn't matter other than that the visual change in it's surface will be transferred to your lowpoly.
What matters is that within one uv shell on the lowpoly you need that entire shell to be in one smoothing group, or soft edges.
The lowpoly you export to engine needs to be exactly the same you baked to, as tangent space normals basically only are the difference between the lowpoly and the highpoly's normals per pixel.
In engine, reflections per texel will be [lowpoly's vertex normals] + [normalmap] = [highpoly estimation].
It stands to reason that if you display the normals on a different model or change the normal map, the result won't match your highpoly.