Triangulate when baking normals?
polycounter lvl 14
This may be a silly question...If i build a lowpoly mesh in quads, should i triangulate the mesh before i bake the normals? I ask this because ive baked my normals for my lowpoly as quads, it looks great on the quad version, but if i triangulate the mesh the normal map starts creating small distortions along the newly added edges. Is this what would happen if i plugged my quad version into a game engine? would it triangulate the mesh and cause my quad based normal map to produce distortions?
Replies
In case of 3dsMax, even though you are working with editable poly and quads, there is a "invisible" triangulation. That can be tweaked without actually cutting the quad.
In my 3dsMax -> Unreal workflow, I only cut, or turn the edges in quads if the silhouette or the baking will benefit.
Then I export as an ase file and I import it Unreal. The triangulation is madre properly and no "turning" occurs.
But again, this may vary depending on the software you use and the probably the export settings you use.
that should enforce the triangulation while allowing a clean qua based work flow.
worth a shot
a few folks posted about this in the why not to use max for normal map baking. it does happen
Some apps it is essential, by default in max if you render with an un-triangulated mesh, you will get lots of smoothing errors as max randomly sets your invisible edges for you(regardless of any invisible edge tweaking you do, it will set them as it sees fit). From my tests when you use 3Point Shader with quality mode this is no longer an issue however in max.
In maya(and likely other apps), if you render your maps without triangulating, and the triangulate before exporting you will get smoothing errors, as your mesh normals change(a quad creates different smooth normals in maya than a triangle). However, if you simply "lock normals" before triangulating, you will retain the same normal data on the quads(or even ngons) when it gets triangulated. Its important if you have a maya plugin to export to your game that you do a lock normals before triangulation.
Sometimes(not always) on mirrored parts of meshes in both max and maya, i have had to triangulate entire chunks so that the edges were identical on both sides, this goes back to the first invisible edge problem i mentioned. The odd part is this is sort of up to chance, and sometimes it works perfectly, other times it is totally broken.
So depending on exactly what you need to do, you *may* need to triangulate. The default answer a lot of people seem to say is "always triangulate" but i feel few actually understand why that is said.
So yeah, you can manually triangulate the quad as you wish, or turn the invisible edge.
QFT. I see this all the time, and just helped somebody out last night because of this very issue. Even on symmetrical meshes, Max will create asymmetrical triangulation, leading to normal mapping issues if you've mirrored your UV space prior to triangulation. It is important that you pay close attention to the turning of your edges otherwise you WILL have issues, with normal mapping, rigging, etc.
Yes, but max's scanline renderer doesn't always respect those edges when baking your normals, so while it may export the invisible edges correctly, you have to bake, and then painstakenly tweak to get it to work correctly.... This was my experience atleast, and it seemed reproducible at the time.
what??
I go and kill max now
is this only an edit poly problem or does invisible-editable-mesh edges fall under the same rule?
If get X shaped smoothing errors in max, that is a tell-tale sign that the invisible edges are mis-matched.
it might become like so after you apply symmetry and vertexes are welded and that only in a few places at the welded area. But after you turn it the way you like, it will always stay like so.
i haven't systematically traced what causes it in some places and not in others but i assume it is the location of the vertex to be welded.that is, if it is exactly on the seam or a bit "in" or "out". I have observed that if max if forced to move that vertex to weld it then it will guess but only on the "mirrored" area usually.
also i have not seen such behaviour when exporting assets from 3dsmax.
but i am mainly using fbx to export.
that the invisible edges stay in max as is was never contested
As for the max scanline invisible edge thing, I've never had them flip or go crazy. In Maya it used to re-triangulate all the time, which was a bit maddening because models where never really symmetrical. I don't know if they fixed it in newer versions or not I seem to remember someone saying they did...
On a side note it definitely *isn't* an issue when using 3ps quality mode. Nor is pre-triangulation necessary.