Question about baking with triangulation

Billabong

I'm a little confused about something.

When baking a mesh, Should I triangulate (Convert Editable mesh and make edges visible)the low poly first or does it really matter. This question refers to Max and Xnormal, just in case one is different than the other.

Thanks for any help.

-B

Daelus

You don't always have to, but xNormal will triangulate before calculating maps and generally each program will triangulate a little differently. If you don't triangulate before baking, it will mean the model it calculates the maps for will have different face normals than the one you triangulate later in Max. If you don't triangulate prior, and notice strange anomalies with normal maps(single faces with weird black spots), this is what causes that.

Billabong

That was very helpful, thanks so much for explaining that Daelus

Overlord

Sorry to necro a thread, but I thought it to be better than starting a new thread when my question is similar.

When is it best to triangulate a mesh? Is it just baking and export to the game engine? Are there any other reasons to triangulate?

CheeseOnToast

No, that's pretty much it. It's sometimes a good idea to manually triangulate some non-planar quads so you can control whether or not it ends up having a "peak" or a "valley" when triangulated, but again you can usually leave it to the last minute. Models that have clean edge/face loops are easier to work with for numerous reasons, which triangulation breaks.

Mark Dygert

It's not so much of an issue in Maya because non visible edge triangulation is forced to go one way (last time I checked) you can flip them but they go right back when the mesh is re-evaluated so Maya guys normally define the edge and it sticks.

In max you can flip them however you want which can be problematic if an exporter/importers calculate non visible triangulation differently.

Non visible edges:
Maya: [/][/][/][/] (always one way)
Max: [\][/][/][\] (can be any way the user wants)

Certain tools in max will mirror hidden edges too. So it goes [\]|[/] but in maya it would go [/]|[/].
It can be helpful or it can drive you bat shit nuts in both apps, ha.

Typical max baking experience when not triangulating:
Exporter: "DOODZ! You have funky flipped edges, not cool... you're going to regret it..."
User: "Shut it"
Importer: "oh this edge isn't visible so you must not care that I force them to go from lower left to upper right..."
User: "WHY YOU NO BAKE RIGHT!!!11"

Triangulating your mesh forces every edge to be defined and won't default to some default tri-stripping method.
People normally drop a "turn to poly modifier" on top of their stack, set the number of sides to 3 and then turn it off or delete it after exporting so they aren't stuck with a triangulated nightmare.

Overlord

Thanks, I was curious if it had another purpose. Most of the time, I just turn edges to change the flow of my face loops.

EarthQuake

If you're using Maya or Max but baking in xnormal, you should triangulate before you bake, as XN may(likely will) triangulate differently than your 3d app.

If you're baking in Maya, and exporting to a different app, like Marmoset Toolbag, you want to lock normals and triangulate after you bake. Its important to lock normals otherwise your mesh normals will change when it triangulates (maya calculates normals differently for quads vs tris).

If you're baking in Max, there isn't really any need to triangulate before baking, you can do it afterwords. Max essentially always stores a triangulated version of your mesh (ie: the hidden line stuff which you can edit), so triangulating in Max before baking in Max should not do anything. If you're baking in Max and using something like 3Point Shader or Xoliul's Shader which is synced to Max's tangent space, you don't need to worry about triangulation at all.

If you're baking in Max and exporting to another app, triangulate before you export.

The biggest reason to triangulate is if another app/engine does it a different way, you'll get nasty little X shaped smoothing errors when dealing with normal maps.

Mark Dygert

EarthQuake wrote: »

The biggest reason to triangulate is if another app/engine does it a different way, you'll get nasty little X shaped smoothing errors when dealing with normal maps.

I agree with everything EQ said, he knows his stuff about normal maps, he authored most of the stickies in this thread that have gone on to teach most of polycount how to deal with this stuff.

I'll add one important note in case it wasn't clear.


If on import the hidden edges are tri-stripped to always go a certain way then in max they can go from:
Export: [\]|[/]
to
Import: [/]|[/]

If you baked with [\]|[/] (like in max or xnormal) and it imports into an engine with [/]|[/] you will get nasty errors like EQ talked about.

Exporting to FBX normally warns you about flipped hidden triangles. If you export from Max to Maya it re-tri-stripes the hidden edges (at least the it did the last time I checked Maya 2008, they might have fixed that). It would have been nice to have a "force flipped hidden face to visible" option that would solve having to triangulate all the time.

Overlord

Thanks for the info. I have Max (student license), but I don't use it. I learned on Max and used it for a few years but now I do all my modeling in Blender.

FourtyNights

EarthQuake wrote: »

If you're baking in Maya, and exporting to a different app, like Marmoset Toolbag, you want to lock normals and triangulate after you bake. Its important to lock normals otherwise your mesh normals will change when it triangulates (maya calculates normals differently for quads vs tris).

A somewhat old thread, but this is an important subject to discuss about, right folks?

So, EQ, do you mean that it affects to manually edited custom vertex normals or even default calculated vertex normals?

I'm a Blender user, and I can add a separate "Triangulate" modifier for my model which is made of quads, or mostly quads. Anyway, it keeps the exact position of the vertex normals before or after, no matter what. I can even have a couple of methods for quads and n-gons to triangulate my mesh before applying the modifier.

Even the current Blender 2.70 doesn't support custom vertex normals (yet), so I can't adjust them manually to see if triangulating is affecting to them in any way. Luckily there's a python programmer who is already working on with the upcoming support for editable vertex normals.

EDIT:

Damn, I was testing the position changes with a too simple object. With more complex objects, yes - even both face (polygon) and vertex normals have some differences in positions between the original quad and triangulated versions of the mesh, as I had those two meshes on top of each other.