Home Technical Talk

Triangulate when baking normals?

polycounter lvl 14
Offline / Send Message
ENODMI polycounter lvl 14
This may be a silly question...If i build a lowpoly mesh in quads, should i triangulate the mesh before i bake the normals? I ask this because ive baked my normals for my lowpoly as quads, it looks great on the quad version, but if i triangulate the mesh the normal map starts creating small distortions along the newly added edges. Is this what would happen if i plugged my quad version into a game engine? would it triangulate the mesh and cause my quad based normal map to produce distortions?

Replies

  • Norman3D
    Offline / Send Message
    Norman3D polycounter lvl 14
    If you want to make sure there aren't any kind of problems, I would triangulate the mesh when baking and when exporting the mesh to the engine. The problem is that if you have quads, when exporting or importing in the engine, these quads will have to be triangulated. It's possible that this triangulation is randomly made, depending on the software you use.
    In case of 3dsMax, even though you are working with editable poly and quads, there is a "invisible" triangulation. That can be tweaked without actually cutting the quad.
    In my 3dsMax -> Unreal workflow, I only cut, or turn the edges in quads if the silhouette or the baking will benefit.
    Then I export as an ase file and I import it Unreal. The triangulation is madre properly and no "turning" occurs.
    But again, this may vary depending on the software you use and the probably the export settings you use.
  • r_fletch_r
    Offline / Send Message
    r_fletch_r polycounter lvl 9
    have you tried putting a turn to poly modifier into your stack with the 'polygon limit' set to 3
    that should enforce the triangulation while allowing a clean qua based work flow.

    worth a shot
  • PredatorGSR
    Offline / Send Message
    PredatorGSR polycounter lvl 14
    I've never had an issue with baking in quads and exporting to a variety of engines, using meshes created on both Max and Maya.
  • r_fletch_r
    Offline / Send Message
    r_fletch_r polycounter lvl 9
    I've never had an issue with baking in quads and exporting to a variety of engines, using meshes created on both Max and Maya.

    a few folks posted about this in the why not to use max for normal map baking. it does happen
  • EarthQuake
    Its sort of a complex problem. It depends on a few things.

    Some apps it is essential, by default in max if you render with an un-triangulated mesh, you will get lots of smoothing errors as max randomly sets your invisible edges for you(regardless of any invisible edge tweaking you do, it will set them as it sees fit). From my tests when you use 3Point Shader with quality mode this is no longer an issue however in max.

    In maya(and likely other apps), if you render your maps without triangulating, and the triangulate before exporting you will get smoothing errors, as your mesh normals change(a quad creates different smooth normals in maya than a triangle). However, if you simply "lock normals" before triangulating, you will retain the same normal data on the quads(or even ngons) when it gets triangulated. Its important if you have a maya plugin to export to your game that you do a lock normals before triangulation.

    Sometimes(not always) on mirrored parts of meshes in both max and maya, i have had to triangulate entire chunks so that the edges were identical on both sides, this goes back to the first invisible edge problem i mentioned. The odd part is this is sort of up to chance, and sometimes it works perfectly, other times it is totally broken.

    So depending on exactly what you need to do, you *may* need to triangulate. The default answer a lot of people seem to say is "always triangulate" but i feel few actually understand why that is said.
  • Norman3D
    Offline / Send Message
    Norman3D polycounter lvl 14
    I think in 3dsMax you are perfectly able to turn the invisible edges, and 3dsMax will respect those changes when triangulating the mesh on export. It is true, though, that 3dsMax sets those invisible edges randomly when the quad is created.
    So yeah, you can manually triangulate the quad as you wish, or turn the invisible edge.
  • ScoobyDoofus
    Offline / Send Message
    ScoobyDoofus polycounter lvl 19
    EarthQuake wrote: »
    Sometimes(not always) on mirrored parts of meshes in both max and maya, i have had to triangulate entire chunks so that the edges were identical on both sides, this goes back to the first invisible edge problem i mentioned. The odd part is this is sort of up to chance, and sometimes it works perfectly, other times it is totally broken.

    QFT. I see this all the time, and just helped somebody out last night because of this very issue. Even on symmetrical meshes, Max will create asymmetrical triangulation, leading to normal mapping issues if you've mirrored your UV space prior to triangulation. It is important that you pay close attention to the turning of your edges otherwise you WILL have issues, with normal mapping, rigging, etc.
  • r_fletch_r
    Offline / Send Message
    r_fletch_r polycounter lvl 9
    would it then make sense to 'turn to poly' before symmetry?
  • EarthQuake
    Norman3D wrote: »
    I think in 3dsMax you are perfectly able to turn the invisible edges, and 3dsMax will respect those changes when triangulating the mesh on export. It is true, though, that 3dsMax sets those invisible edges randomly when the quad is created.
    So yeah, you can manually triangulate the quad as you wish, or turn the invisible edge.

    Yes, but max's scanline renderer doesn't always respect those edges when baking your normals, so while it may export the invisible edges correctly, you have to bake, and then painstakenly tweak to get it to work correctly.... This was my experience atleast, and it seemed reproducible at the time.
  • ENODMI
    Offline / Send Message
    ENODMI polycounter lvl 14
    Thanks guys for all the help and info. I think ill just stick with baking the normals from a triangulated mesh, and exporting the model the same. I haven't noticed any edge flipping with the mirrored pieces when triangulating, so I think all my errors are being created by the difference between the quad based normal map. Really appreciate the help, now Im off to rebake some maps:)
  • rollin
    Offline / Send Message
    rollin polycounter
    EarthQuake wrote: »
    (...) (regardless of any invisible edge tweaking you do, it will set them as it sees fit). (...)

    what??
    I go and kill max now
    is this only an edit poly problem or does invisible-editable-mesh edges fall under the same rule?
  • EarthQuake
    I'm not sure, not a huge max guy so i'm just telling what i know from my limited experience with it. There may be something i was doing wrong or a work around or something, its probably worth someone with better knowledge of max taking a look... I was dealing mostly from meshes imported from modo as well, not sure if that would make any difference. I think Per look at it at some point and found the same conclusions however, dont quote me on that tho.

    If get X shaped smoothing errors in max, that is a tell-tale sign that the invisible edges are mis-matched.
  • yiannisk
    Offline / Send Message
    yiannisk polycounter lvl 7
    that is true. max doesn't have such an issue.

    it might become like so after you apply symmetry and vertexes are welded and that only in a few places at the welded area. But after you turn it the way you like, it will always stay like so.

    i haven't systematically traced what causes it in some places and not in others but i assume it is the location of the vertex to be welded.that is, if it is exactly on the seam or a bit "in" or "out". I have observed that if max if forced to move that vertex to weld it then it will guess but only on the "mirrored" area usually.

    also i have not seen such behaviour when exporting assets from 3dsmax.
    but i am mainly using fbx to export.
  • rollin
    Offline / Send Message
    rollin polycounter
    hey are we still at the point where we say you can only figure it out when using render to texture ?!

    that the invisible edges stay in max as is was never contested
  • Mark Dygert
    Triangulating before baking? Sounds messy. I often use a projection cage and the thought of editing a triangulated cage gives me a headache. It's bad enough they don't put loop/ring functions in the projection modifier... I suspect that whatever flaws it fixes are pretty minor compared to the time it eats up? I can see triangulating problem areas, just not the whole thing.

    As for the max scanline invisible edge thing, I've never had them flip or go crazy. In Maya it used to re-triangulate all the time, which was a bit maddening because models where never really symmetrical. I don't know if they fixed it in newer versions or not I seem to remember someone saying they did...
  • EarthQuake
    The issue wasn't that the edges would turn themself back, like reset themselfs and you could visually see this, however it seemed turning edges had no effect on the rendered output with RTT. Sometime when i have time i can do a little test case and see if this is a real issue or something i am making up.

    On a side note it definitely *isn't* an issue when using 3ps quality mode. Nor is pre-triangulation necessary.
Sign In or Register to comment.