Confused about TB3 baking > Unreal workflow

jordank95

I'm a bit confused when it comes to DirectX and OpenGL. I understand UE4 uses DirectX and -Y means DirectX, +Y Means OpenGL. 

But when you "Flip Y" in Marmoset, is that switching it from OpelGL to DirectX? 

Basically, what would be the standard workflow when baking maps in TB3, then going into Substance to texture, then into Unreal? 

EarthQuake

For Unreal you want to set your tangent space to Mikkt. You can set this in Edit-> Preferences -> Default tangent space. You should also Flip Y in the normal map output options (you may need to flip Y in the preview material too).

Make sure to triangulate your mesh before baking as well.

jordank95

EarthQuake said:

For Unreal you want to set your tangent space to Mikkt. You can set this in Edit-> Preferences -> Default tangent space. You should also Flip Y in the normal map output options (you may need to flip Y in the preview material too).

Make sure to triangulate your mesh before baking as well.

Thanks for the reply @EarthQuake. Though this answered some of my questions, I still feel a bit confused. I understand using MikkT for all programs since thats becoming the standard nowadays, but I guess I'm still confused about a couple of things.

Does Marmoset bake maps in OpenGL or DirectX mode? I don't see an option for this anywhere. I see in programs like Substance you have a button for DirectX or OpenGL. 

My workflow has been to bake maps in Marmoset TB3, then import my maps into Substance Painter in DirectX mode for texturing, then import into Unreal and flip the Green channel. 

Is this correct?

EarthQuake

First off, "OpenGL" or "Direct X" are not types of normals maps. Both OGL and DX can use either type, it just depends on how the shader is set up. It's an odd terminology that Substance uses for some reason. For instance, Toolbag 2 ran in both OpenGL (Mac) and DirectX (PC), but there was no need to use a special type of normal map depending on which build you ran.

What you're asking about is the normal map handedness, whether to use right handedness (+Y or OpenGL) or left handedness (-Y or DirectX). Handedness in Toolbag is set automatically depending on the tangent space that you pick, for instance if using Maya, it will bake with right handedness, if using Max it will bake with left.

Unreal confuses matters a bit because they use a left handed Mikktspace implementation which isn't typical. So, to switch to the handedness that Unreal expects, you simply need to enable the flip Y option in the baker.

When you send the content to Substance Painter, all you need to do is make sure that it's set to use the same handedness. It should be really obvious when it isn't: details will appear somewhat backwards and the shading will look wrong. A bit of trial and error is really all you need to do if you're still unsure.