3ds Max Hotfix 4 (Normal Maps in Viewport FIXED!)

TychoVII

So, going off the news post I proceeded to download the Hotfix 4 update for Max 2011 released by Autodesk which supposedly fixes normal maps in viewports. I wasn't completely convinced, so I did some comparisons.

At first I couldn't get it to work, but then I noticed there's an extra step you have to take according to the Hotfix's readme file:

Qualified Normal Mapping
Qualified Normal Mapping mode is a new option in the INI file. You can set it to customize how normal mapping is performed on the DirectX material and MetaSL shaders in DirectX viewports. When this mode is off (the default), the normal-mapping mechanism aligns with how Maya performs normal mapping, which causes Maya-developed assets to display more consistently in 3ds Max DirectX viewports (but not in the scanline renderer). Turning on Qualified Normal Mapping mode causes orthogonalization to be applied to tangent basis vectors in order to better align with the scanline renderer.

If a legacy shader is working well then this mode should be left turned off.

Enabling qualified normal mapping is done by adding the following lines to 3dsmax.ini:

[ViewportNormalMapping]
ViewportNormalMappingType=Qualified

To disable qualified normal mapping:
[ViewportNormalMapping]
ViewportNormalMappingType=Legacy

Below are my results:



As you can see it works!
But only after you follow the instructions to edit the 3dsmax.ini file (C:\Users\UserName\AppData\Local\Autodesk\3dsmax\2011 - 64bit\enu\3dsmax.in)

I tried it both before and after with the Xoliul shader but it didn't work since I'm sure my version is old and isn't designed to work with Autodesk's fix.

Hope people find this helpful!

Xoliul

I'm a bit disappointed, again, it seems like just a visual fix and not a fix at the core, for the baking itself. Gonna test it a bit here.

TychoVII

I'm currently doing work for a game who's engine calculates normal tangents similar to max's scanline renderer. Up until now I've had to render out an image just to see if my normals were working correctly since we're only allowed to use one smoothing group, so I'm pretty happy about the visual fix.

Xoliul

Giving us the choice between two types of tangents when baking would've been ideal, i remember one of the Autodesk reps agreed with me there. Too bad to see they didn't implement it.

So i tested with the same single smoothgroup cube case: all DX shaders have this problem apparently, it's just the worthless "DX display" that gets the better normals. I'm thinking the "DirectX Material" they mention is not the DirectX Shader material, but the DX display.
So apparently they've gone with some closed, strange approach that limits the use to just a few solutions...
Absolutely not what I was hoping for, I'm gonna email Neil Hazzard and ask him about this.

TychoVII

That's pretty disappointing. I was really hoping you'd be able to upgrade your shader by implementing their fixes because I use it all the time. I hope you get some good news from him.

TychoVII

Does the 3 Point shader high quality normals display normal tangents similarly to scanline?

vahl

yes, that's the whole point of our "quality mode" normals

TychoVII

Ahhh sweet. This is exactly what I need for work then. Thanks!

jocose

Is Xnormals tangent basis the same as Maya's? I really wish there was a chart somewhere that illustrated all this.

cw

TychoVII wrote: »

I'm currently doing work for a game who's engine calculates normal tangents similar to max's scanline renderer. Up until now I've had to render out an image just to see if my normals were working correctly since we're only allowed to use one smoothing group, so I'm pretty happy about the visual fix.

Hi guys, Shader just needs some small tweaks and all works perfectly.

I hopee this is ok Xoliul:

setup basis like this

	if(!bUseObjectNormals && bUseNormalMap) //tangentspace normalmap
	{
	//normal = (In.worldNormal * normal.z) + (normal.x * In.worldBinormal + normal.y * -In.worldTangent); //transform every vertex normal component by the tangentspace normalmap
        const float3x3 basis = float3x3( normalize( In.worldTangent ), normalize( In.worldBinormal ), normalize( In.worldNormal ) );
        normal = normalize( mul( normal, basis ) );
	}

swizzle normal pixel like this

	normal = tex2D(normalSampler, In.texCoord2.xy).rgb * 2.0 - 1.0;
		
        normal = normal.yxz;

I think that's about it! looks fine here. By default it looks bowed in, not sure really why I am not a coder as such, I just fiddled with the maths and it seems to have worked out. The shader I use at work looked perfect, so I was sure I could tinker and get yours to work too. Hope it helps.

In fact, this should work with any hlsl shader for max, just set the maths up a certain way and bob's mother's brother.