Synched normals with one smoothing group - should I even bother?

dzibarik

So after Unity 5.3 release I have tried to do it once again - bake a mesh with one smoothing group and import it into the engine with synched normals. The normal map was converted in Xnormal from object space map baked in modo. 

It worked totally fine on test cubes but as soon as I import a more complex mesh it breaks:

It looks fine in Painter: 


Horrible in Unity


Better but still horrible in Unreal



And then I thought - why should I even bother when I can harden edges of uv islands with one click and get better smoothing with the same number of verts:



yeah, I know there may be some solution to make it look good in Unity/Unreal with one smoothing group but why should I jump through hoops while I can make it look good just like that? Is there any other benefit of everything being smoothed into one group I don't know about?

huffer

Have you triangulated the mesh, or tried baking directly in xnormal the final tangent space map?  if you have an object with a lot of soft edges...something like a rounded hull, and you can't harden them all, it is preferable to sync the baked map with the engine, you'll avoid any artifacts that may appear. For hard surface, 80 percent of the time you'll have hard edges and mjnimal normal map gradients - it will make little difference if it's not synced.

ZacD

Well, a synced normal map wont be synced with Unity and UE4, they use different tangent basis. There's really only 2 main approaches, don't use a sync workflow, but model and use smoothing groups/hard edges in a way to avoid shading errors from the tangent basis not being synced, or use a completely synced workflow. A synced workflow can save time and polygons, but isn't the only way to do things. What is your target engine? 

dzibarik

@zacd: it's converted in Xnormal from object space into a tangent space map. It's actually what Handplane does for every tangent basis out there except Mikkt. 
@huffer: yes, I've triangulated that. As for baking in Xnormal I've tried it with other meshes with the same result. I'll do some baking once again, but I doubt I've missed anything.

dzibarik

hm, what the hell, I've converted object space into a tangent space in Designer and it works alright. There are still some minor issues but it's nowhere near as bad as it was. Now what could be the reason for Xnormal to not work correctly? Also I've tested it in UE4 only, I'm sure it will break in Unity.

motionblur

Just for the record ... are you aware that you can bake synched to UNity and UE4 directly from Modo 901 upwards with Farfarer's VNorm toolkit?
I am using it at work to bake normal maps synched to Unity on a regular basis (mostly technical models) and so far did not encounter these shading errors any more. Though I also AM using smoothing groups and edge breaks. If you don't need to paint on the UVs it just looks cleaner, I think. The baker from Substance Designer/Baker also has synched normal algorithms to Unity and UE.

Conversion somehow never worked for me that easily so I gave up on it very quickly. Why convert when you can bake natively?

-> Farfarer's VNorm Toolkit for Modo

Tzur_H

with unity 5.3 they integrated mikkt as tangent basis, so you should be able to just bake using the UE4 baker and it will work for both unity and UE4.

ActionDawg

ZacD said:

Well, a synced normal map wont be synced with Unity and UE4, they use different tangent basis. There's really only 2 main approaches, don't use a sync workflow, but model and use smoothing groups/hard edges in a way to avoid shading errors from the tangent basis not being synced, or use a completely synced workflow. A synced workflow can save time and polygons, but isn't the only way to do things. What is your target engine? 

Your first sentence is confusing. How can a synced normal map not be synced? If you bake with MikkT then results should appear correct in UE4 and Unity. I have no issues with this. And if there is a problem with a given engine build it's trivial to fix at the shader level if you know what you're doing.

Synced is better hands down. It is a more correct representation of the high poly normals, allows the artist to have cleaner uvs and more optimized models, etc. If it is available then it is the only way you should work because it's the correct way.

Fingus

I haven't toyed around with 5.3 yet, but have you checked your import settings in case there is a setting there? It would make sense to me that they would make it an option in order to not break existing projects with the update.

JedTheKrampus

Until 5.3 Unity didn't have MikkTSpace support and it needed things to be baked in a different, Unity-specific tangent space for the normal maps to be synced. Also, Unity MikkTSpace maps still have a flipped green channel compared to UE4 MikkTSpace maps. In Unity you might need to change import options to make the importer generate Mikk tangents.

ActionDawg

I see it here.
https://unity3d.com/unity/beta/unity5.3.0b5

If this is incorrect then it can easily be added to your own project by finding the tangent to world transform in your CGIncludes shaders.

Alternatively you could construct your own vertex shader which would look something like my code below (Using Unity 4). Unfortunately I haven't used Unity 5 much but it should be essentially the same. You could translate this directly to the appropriate CGInclude folder shaders to get the MikkT synced normals.

////////////// Vertex Shader
			struct VertexInput {
				float4 vertex : POSITION;
				float4 texcoord : TEXCOORD0;
				float3 normal : NORMAL;
				float4 tangent : TANGENT;
			};

			struct VertexOutput {
				float4 pos : SV_POSITION;
				float4 uv0 : TEXCOORD0;
				float4 posWorld : TEXCOORD1;
				float3 normalWorld : TEXCOORD2;
				float3 tangentWorld : TEXCOORD3;
				float3 binormalWorld : TEXCOORD4;
				LIGHTING_COORDS(5,6)
			};
					
	        VertexOutput vert (VertexInput v) {
				VertexOutput o;
	            o.uv0 = v.texcoord;
				o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
				o.posWorld = mul(_Object2World, v.vertex);
				
				// Construct the vectors for transforming normal maps to world space
				o.normalWorld = normalize(mul(float4(v.normal, 0.0), _World2Object).xyz);
				o.tangentWorld = normalize(mul(_Object2World, float4(v.tangent.xyz, 0.0)).xyz);
				o.binormalWorld = normalize(cross(o.normalWorld, o.tangentWorld) * v.tangent.w);
				
	            TRANSFER_VERTEX_TO_FRAGMENT(o);
	            
	            return o;
	        }
////////////// Fragment Shader
	float4 frag(VertexOutput i) : COLOR {
		// Construct the world space normal direction
		float3 normalLocal = UnpackNormal(tex2D(_Normal, TRANSFORM_TEX(i.uv0.rg, _Normal)));
		float3x3 local2WorldTranspose = float3x3(
			i.tangentWorld,
			i.binormalWorld,
			i.normalWorld);
		float3 normalDir = normalize(mul(normalLocal, local2WorldTranspose));

dzibarik

ok, I've downloaded Modo 901 trial and baked there and it's 100% identical with converting from object space in Substance Designer:



I'm not ready to drop 400$ or so to upgrade from 801 so I'll continue converting in SD. 

Also yes, it looks like shit in Unity. I've checked settings, Mikkt tangent space is used. I've tried switching to legacy and it looks even worse (as expected). I wonder what happens in Unity. 

JedTheKrampus

Are you using a Unity builtin shader to test? Is the green channel going the right way? Are the problems attributable to compression artifacts? Is the normal map set to be used as a straight up normal map and not converted from greyscale? Can you post a screenshot of how it looks in Unity so we can help you out?

dzibarik

@Jed:

here is my settings in Unity:




Also here's a map from Xnormal but with compute binormal checked in options. That was the issue:



looks identical to SD and modo so I can use that too.  I've unchecked it for Unity btw.

Ok, so I've finally figured out the workflow in details after following it blindly for about a year but now I know it works 100% right even with such shitty topology and smoothing which can be even better if I harden a couple of edges on UV borders with zero extra vertices produced.

Still there's an issue with Unity which is negated by harding edges but still it's an issue that needs to be looked into.

I've attached the mesh and the world space map I've used for anyone who wants to play with it in Unity. 

Tzur_H

Here's what I'm getting:


results are similar, it's just a bit easier to spot the shading artifacts in unity because the shader is glossier, I was too lazy to try and match them lol.
still getting artifacts though...

dzibarik

@Tzur_H: can you attach your tangent space normal map? Such artifacts (triangles) are ok for this mesh, because it's very messy. But normal map looks like it should i.e. the same as in Unreal, so I guess it works just not in my Unity. I wonder what settings are wrong.

Tzur_H

I imported your mesh to maya, and saw a node called morph, are you working with modo?  maybe that's what causing the issue here, as far as I know modo doesn't export the normal data or something?  don't catch me on that, I've read something about normals and modo's inability to export them.. but not sure what it is.  you could try exporting using pipelineIO.
I had to soften all edges, then use your world space to convert to tangent space using xnormal, compute binormal in pixel shader was on - it needs to be on for both UE and unity.
https://www.dropbox.com/s/rh59h6iq24t7v2q/xnormal_compute_binormal.tga?dl=0

motionblur

You aren't exporting smoothing groups with your FBX, are you? Unity doesn't like them. The most basic thing is just exporting geometry, mesh normals and materials - maybe with added animations if you need them.

dzibarik

@tzur_H: I export from modo and modo exports normal data. You can export tangents and binormals in 901 but if you are going for a synched workflow you don't need them afaik.  Also binormal shader doesn't need to be on for Unity according to Farfarer. It definitely needs to be on for Unreal though. I doubt that morph cage throws Unity off, because I get the same result if I kill it. 
@motionblur: ahhhh... this must be it. I've heard about it too but completely forgot about it. I  use Unity at work, so I've unchecked it here, but at home I use Unreal and it requires smoothing groups.

Tzur_H

@dzibarik  well, I don't know. I've exported the mesh as FBX with smoothing groups and triangulation on and that's it.  And I did have binormal shader on for it.
if you figure this out conclusively, do let us know please :P
edit: ok I've tested once again with and without binormal shader and the one without does have better shading, it's a minor difference but it's there.
i've also testes with and without smoothing groups on export, can't see any difference.

a3D

If the mesh is meant to be static, afaik, there is no reason not to harden shell border edges, as long as you're still using synced tangents.
This should help reducing gradients/dithering.

motionblur

@dzibarik  - great! Hope that actually was your problem and you could solve it now

dzibarik

smoothing groups it was:



edit: one more: 

gsokol

To answer your initial question "why should I bother" 

A - More efficient, less vertices.
B - Less UV seams to deal with.
C - Higher quality.  It can be difficult to mask hard edges with rounded normals.

why should I even bother when I can harden edges of uv islands with one click and get better smoothing with the same number of verts:

This is actually incorrect.  When you harden an edge, what actually happens is the verts get doubled, with vert normals facing different angles.  It won't behave like 2 separate verts when you are modeling, but that is how the model gets drawn by the renderer.   


I'm sure there is a good chance your normal's aren't actually synced 100 percent.  Even so, few things to consider:

You are having difficulty in an area where you have several 90 degree corners right next to each other, so its a particularly challenging example  with lots of normal map gradation.  Personally, if I get trouble spots around more sharp angles, I just throw edge loops in to have more vert normal info for a cleaner, more localized normal transition.

Another thing that may not be a factor here, but still worth thinking about.  If your texture is a little lower res or your mesh is LODing, things wont line up perfectly.  The lower res texture is even more of a problem with sharp gradations in the normal map. If there are only so many pixels to handle a gradient you can sometimes get some artifacting.

In your position, I'd just roll with the hard edge, or container loops and move on.  Your adding a small number of verts, and it will technically read a little better with texture mips/mesh lods anyways.

ActionDawg

I believe he was talking about having the vertex normals split where the UVs are already split, hence "harden edges of uv islands," which is of course free.

Personally I split up my UVs from how I split my normals, because it gives me 80% of the unwrapping done at the click of a button.

ZacD

Always have synced normals.

But having one smoothing group is actually worse, because after compression hits your texture, you'll get more artifacts where there was harsher gradients. This really only matters with in worst case scenario situations like glossy materials, specular highlights, reflections, and metals. 

Roughness of .125 and .0.

Goes from 1 smoothing group to split on uv seams to split on uv seams with custom normals. This using the worst case scenario mesh that has been posted many times on polycount. Baked as a TIFF in xNormal, compressed in Photoshop to a TGA, and put into UE4 using the synced workflow. Screen space reflections were disabled and this is in the default advanced lighting map. These angles were picked to showcase artifacts around where the cylinder connects to the body on the front face. Artifacts are pretty much gone above 0.4 roughness. 






I'd only use one smoothing group if it was required by the studio you are working for, or your workflow needs it.

dzibarik

gsokol said:

To answer your initial question "why should I bother" 

A - More efficient, less vertices.
B - Less UV seams to deal with.
C - Higher quality.  It can be difficult to mask hard edges with rounded normals.

why should I even bother when I can harden edges of uv islands with one click and get better smoothing with the same number of verts:

This is actually incorrect.  When you harden an edge, what actually happens is the verts get doubled, with vert normals facing different angles.  It won't behave like 2 separate verts when you are modeling, but that is how the model gets drawn by the renderer.   

actually I was talking about hardening UV islands edges and you have already vertices doubled there. So I get those for free. Baking with one smoothing group in a synced space was a matter of principle - I wanted a worst possible case loaded in an engine and looking alright with this specific workflow. I'm glad I've finally figured it out.

@ZACD: yeah, I'll harden edges on UVs splits from now on anyway (as I have always been doing), it's just nice to know that my maps are 100% synced now.