Normal maps and triangulation, baking in Maya

Daew

Hey polycount,

I've been trying to figure out the "proper" workflow for triangulating a mesh before baking and if it is even is a requirement.

From my understanding one of the reasons it is good to triangulate prebake is so that your normal maps dont get messed over when another app reads the normals differently.

However I have read that if I am making my models in Maya and baking it in Maya all I have to do is lock my normals before bake. But, then I also read that locking and unlocking normals is not reliable/ clients dont want it. Both of these come from two respected polycounters. Lastly I have not had many happy experiences with locking normals, though that may be due to user error. So my question is should I be triangulating in Maya before bake?

Im a bit weary of doing it though because it becomes a pain to edit after the model has been triangulated. I know I should probably make a duplicate then traingulate but it feels like an necessary evil?

I will be learning how to bake normal maps in Xnormal just to be flexible and from what I gather, triangulating is neccessary. So I guess Im wondering what other peoples workflows are when triangulating and baking.

And finally a lot of marmoset viewer models are not triangulated, so am I missing something or is everyone just skipping that part as well. But then Im sure alot of people use Xnormal so what do ._.?

cheers!

Bek

Yes you should triangulate before you bake a (tangent) space normal map, and no you do not work with the triangulated mesh (duplicate it).

You triangulate before baking not "so your normal maps dont get messed over when another app reads the normals different" (as you say it — the normal map is being read complete fine/correctly), you triangulate so that the baker loads the mesh with the same triangulation as it will be displayed with, as triangulation affects surface shading (which therefor affects your resulting tangent-space normal map)

In xNormal you potentially could avoid triangulating before export iff (if and only if) you can get a triangulator plugin that will match your software's triangulation (maya in your case) so you won't have to worry about them being different. But it's usually just simpler to triangulate before you export, and that's more universal anyway.

Daew

hey thanks, yeh my sentence was not well worded, thanks for correcting it.

I guess its a bit petty, I dont really like the idea of having to duplicate because it leaves my scene messy. However its thats what has to be done i'll bite.

What really threw me off was seeing marmoset viewer models that were quads. Also in normal wireframe shots a lot of models are still in quads, am I missing something really simple here?

EarthQuake

The reason to lock normals in Maya is so that you don't have to triangulate before baking, provided you're baking in Maya. If you don't lock normals, and then you triangulate when you export to your game engine, the mesh normals will change and cause smoothing errors. This is only something you would do at the very end before you export. Lock normals, triangulate, export, and undo each of those steps before saving your file, you shouldn't save the file with locked normals.

If you simply triangulate in Maya before baking, you can skip the lock normals step, but then you have to make a copy of your mesh, or undo the triangulation after baking. For the most part I've managed to bake in Maya without triangulating first, but I've had a few rare occasions dealing with meshes with mirrored uvs that had different triangulation direction for the quads on one side which required manual triangulation.

The reason it is important to triangulate before baking or exporting to a different game engine/app is that the way quads are converted to triangles internally in apps can vary, which will lead to X shaped smoothing errors.

Are you missing something? Some artists simply do not understand the important of triangulating meshes for baking/export. For certain types of assets, the artifacts that triangulation will prevent will be less noticeable. For instance, with organic shapes or character art where you don't tend to have a lot of harsh angles, smoothing errors in general tend to be less noticeable. For hard surface, mechanical, armor, etc, where you have more harsh shapes and often have very reflective metal surfaces, smoothing errors can be very apparent.

oblomov

Differences in how quads are triangulated can also change the result of tangent space computations, which will result in tangent space syncing errors, again more noticeable on hard surface assets.

Daew

Thanks Earthquake for the workflow, I thought the mesh was exported with locked normals.

However I do have a problem with locking normals. I set my edges hard/soft but when I lock the normals it hardens all the edges adjacent to any existing hard edge. Hard edges are on uv splits.


And I'm sorry if it sounds like i'm going crazy over this, it's not that drastic, I'm just curious is all.

This was an example I was wondering about, I'm sure Amsterdam Hilton Hotel knows what he is doing and I know you know what you are doing. Its a hardsurface asset that hasnt been triangulated. The gun looks amazing but is it considered bad practice?

Image taken from http://www.polycount.com/forum/showthread.php?t=158840

Bek

He's shown non-triangulated models in the past for presentation purposes (looking at a triangulated wireframe can be messy). So just because you see an image with a non-tri wireframe doesn't mean they don't triangulate their models.

Daew

Yeh thats the part I'm super confused on. I would like to show models that have clean wireframes, but you cant put a normal map baked from a triangulated mesh onto a non triangulated mesh.

So how are people posing their characters with normal maps applied and their wire frames untriangulated?

They are either not triangulating, comp wizards, or im an idiot.

Bek

I can only assume they're displayed in the modeling application with the textures applied and the triangulation is already there "behind the scenes" — as in, the GPU is always being fed triangles even if what you're working with is quads. So the wireframe is only showing the working geometry, not the "real" geo, similar to how one vertex in your software might actually be 3 vertices to the GPU once hard edges and UV splits are taken into account.

CheeseOnToast

Another thing to mention is that a lot of applications use a "shortest edge" triangulation method internally, meaning that it will invisibly triangulate a quad between the two closest diagonal verts. Frequently, this means your quad meshes will be triangulated the same way in multiple programs.

EarthQuake

In Maya, yes you would lock normals before exporting if you've baked with quads, otherwise you will get smoothing errors. Again this applies specifically to baking in Maya. I don't know about the issues you've had with locking normals, I'm not really a Maya expert.

Ben uses Max, in Max, your mesh is always triangulated even when working in quads, the invisible edges are stored and can even be edited if you like. Unlike maya, changing from quads to tris has no effect, so you can safely bake with quads in Max, and export as triangles as the last step. Alternatively, you can export as triangles and bake in an app like xNormal.

As far as character art goes, again, character art content is frequently much more forgiving to these artifacts, so it's possible the problems are simply not noticeable on the work you've seen, thus not really worth worrying about. As with everything, you should do some real world tests to see if it makes a difference in your workflow, with the assets you work on.

Daew

Ah alright, thanks for the answers everyone, it clarified alot