Why you should NOT trust 3ds Max's viewport normal-map display!
polycounter lvl 18
OK, sensationalist headline. But it got your attention! This was a spinoff from another thread, I thought it deserved its own.
Here are my findings based on Max vs. Maya baking using the native tools and displaying the results in the viewport using the native tools.
Summary:
Unless your game engine is set up to use the same tangent basis as 3ds Max's renderer (not it's viewport), your normal-mapped models will never look 100% correct!
I have just done some simple tests tonight, and I can confirm that Max's "baking" tangent space is the same as it's "render" tangent space (no surprise, since the scanline renderer is used to bake normals in Max), however the "render" tangent space does not match the tangent space that it uses in the viewport. This strikes me as just plain insane. It means you'll never get the same result between rendering and viewport display.
If you use a "correct" normal-map for viewport display in Max, then it will look messed up in render, and vice versa... horrible.
I baked the same objects in Maya, and they are displayed perfectly in the Maya viewport. Interestingly enough, this map also looks perfect in Max's viewport! That means that the viewports in both apps derive tangent basis in the same way. However it also means that Maya's baking renderer correctly uses this tangent space! Which makes sense, of course.
It basically seems like since the viewports (and Maya's baker and Mental Ray, by extension) seem to all use the same tangent basis derivation, this is the most "correct" one, and therefore 3ds Max's renderer and baker are "wrong". This should be a standard that all applications adhere to, I see no reason why it shouldn't be.
Screenshots as promised...
Comparison of viewport shaders (just using both apps' native display of default shaders, nothing fancy or custom - grey Phong materials with high specular):
Max's scanline renderer normal map output:
Max's Mental Ray normal map output (ignore lack of anti-aliasing, that's not what's being tested here):
Maya's normal map output:
If you're feeling intrigued you could download those normal maps and flick back and forth between them in an image viewer to see the main differences. You should be able to see them clearly enough here, though.
So, out of interest, does anyone here work in a pipeline where they bake their normal-maps using 3ds Max, and if so, do you know what your game does with them? Have you ever had any problems?
EDIT: ADDENDUM!
Here are some direct links to some of the more useful posts from later in the thread:
EDIT 2! 3POINT SHADER RELEASED!
http://www.3pointstudios.com/3pointshader_about.shtml
Here are my findings based on Max vs. Maya baking using the native tools and displaying the results in the viewport using the native tools.
Summary:
Unless your game engine is set up to use the same tangent basis as 3ds Max's renderer (not it's viewport), your normal-mapped models will never look 100% correct!
I have just done some simple tests tonight, and I can confirm that Max's "baking" tangent space is the same as it's "render" tangent space (no surprise, since the scanline renderer is used to bake normals in Max), however the "render" tangent space does not match the tangent space that it uses in the viewport. This strikes me as just plain insane. It means you'll never get the same result between rendering and viewport display.
If you use a "correct" normal-map for viewport display in Max, then it will look messed up in render, and vice versa... horrible.
I baked the same objects in Maya, and they are displayed perfectly in the Maya viewport. Interestingly enough, this map also looks perfect in Max's viewport! That means that the viewports in both apps derive tangent basis in the same way. However it also means that Maya's baking renderer correctly uses this tangent space! Which makes sense, of course.
It basically seems like since the viewports (and Maya's baker and Mental Ray, by extension) seem to all use the same tangent basis derivation, this is the most "correct" one, and therefore 3ds Max's renderer and baker are "wrong". This should be a standard that all applications adhere to, I see no reason why it shouldn't be.
Screenshots as promised...
Comparison of viewport shaders (just using both apps' native display of default shaders, nothing fancy or custom - grey Phong materials with high specular):
Max's scanline renderer normal map output:
Max's Mental Ray normal map output (ignore lack of anti-aliasing, that's not what's being tested here):
Maya's normal map output:
If you're feeling intrigued you could download those normal maps and flick back and forth between them in an image viewer to see the main differences. You should be able to see them clearly enough here, though.
So, out of interest, does anyone here work in a pipeline where they bake their normal-maps using 3ds Max, and if so, do you know what your game does with them? Have you ever had any problems?
EDIT: ADDENDUM!
Here are some direct links to some of the more useful posts from later in the thread:
- Perna's normal map display fix preview: http://boards.polycount.net/showpost.php?p=1122347&postcount=456
- Undoz and CW's edit normals workaround:
http://boards.polycount.net/showpost.php?p=1046668&postcount=189 / http://boards.polycount.net/showpost.php?p=1046842&postcount=195 - Fozi's normal map space converter
http://boards.polycount.net/showpost.php?p=1072599&postcount=274 - Osman's GUI to Fozi's tool
http://boards.polycount.net/showpost.php?p=1074986&postcount=331
EDIT 2! 3POINT SHADER RELEASED!
http://www.3pointstudios.com/3pointshader_about.shtml
Replies
Second question: Does the same thing occur when using Mental Ray in Max?
You shouldn't have to invert the green channel in Photoshop, there is a setting for that in the Render To Texture "Options" dialog. Just set it to the correct Y orientation before you bake.
Every application has a control for this (in Maya it's based on whether the mesh's "tangent space" attribute is set to Left-Handed or Right-Handed), if you're inverting the green channel in Photoshop then you're not baking it right
I have only tried the Scanline renderer so far. I will try Mental Ray on the same objects, hopefully it will give better results (although that also means more inconsistent results within Max...).
Edit: After trying Mental Ray inside Max (added image to original post), while it is slightly different to the scanline bake, it is still far from being correct, if we're treating the Maya bake as the "most correct" bake.
Good to know, never saw that option
It really does look like the green channel needs flipping, and you should try using Xoliul's max shader, since the max default doesn't display normal maps correctly.
1. Which application is using the "best" tangent basis derivation? As far as I can tell, Max's renderer tangent basis is the "odd one out" - more apps like Maya, XNormal, maybe others are consistent between each other, yet Max isn't even internally consistent!
2. Why is Max's renderer tangent basis different to it's viewport tangent basis? This essentially means that if your normals look correct in the viewport, then they will look wrong in the render. Conversely, if they look correct in a render, they will be wrong in the viewport. This is ridiculous!
3. If you can write an FX shader that uses Max's "renderer tangents" in the viewport, why doesn't Max's default internal shader and supplied FX files use this method? Both Max's "Show Hardware Map in Viewport" option, and it's supplied external shader files like "StandardFX.fx" use the default viewport method, which means any bakes you do in Max will look crappy using these shaders! Insane!
Could we get the source files to test in other apps to see if its just max what does this?
I'm also wondering if this is the case then how is it that you can make normal maps in Max and use them in unreal ed so they look correct or does unreal for example have code that can render this out properly because a coder took the time to do so.
It be nice if all normals maps were created equal
Well yeah, I was wondering that too, since I thought they baked stuff in Max?
That either means they've set up the Unreal Engine to use Max's renderer tangent derivation (meaning maps baked in any other app will look wrong!), or they're getting dodgy looking stuff in-game and dealing with it by adding geometry till the problem goes away... or... they just don't care that much
Nope, it's been like this as long as I've used Max. Definitely since max 8, and it was in max 2008 and still a problem in max 2010.
Yes, exactly - we should have some sort of standards for this (ie. how the normals/tangents/bi-normals/whatever are calculated).
how that is nowadays I don't know, but last time I checked it was still easy to create tangent maps that only look perfect in scanline.
If I recall correctly then ASE used in unreal import also doesn't encode tangent stuff, which means they indeed must do things the same way max does when importing.
this
You're also adding 5000 hard edges to your models to get a correct result in max, you could take those same models with totally smooth normals in maya and bake them with results that are *near flawless*.
Mop I think it's pretty obvious that Autodesk doesn't care that much. I mean look at the Max 2010, it's awesome..
Thing is, it seems like Max's viewport calculates tangents almost identically to Maya's baker and viewport. So at least that seems like a more consistent calculation.
Is there any reason why the calculations should be different for max's renderer?!
EQ you use Modo. How much easier is it to get proper results than in say Max.
tip of the iceberg..
I highly doubt something with many many 90 degree angles would bake well in any app without adding edge bevels or adding in hard edges.
Seems max can only overcome smoothing issues based on raytracer than realtime.
There's a way to get good maps in max (or any other engine) by setup your smoothing groups and uvs properly.
Below is a map I've done with max (realtime viewport) a while ago. (before I switched to xnormal)
Anyway thanks a lot MoP, I was using xNormal without even knowing I could get away with less smoothing errors.
(I have to revise my normalmap article now)
So, you're saying you use NO hard edges, and get perfect results in max?....
Clearly you can't be saying that, or you would be showing examples that prove everyone in this thread wrong.
This is a pretty good example of what happens in max, even using hard edges with a pretty clean mesh, no outrageous issues or anything, you get visible smoothing errors, in any other app(maya, xn) these areas should render perfectly without hard edges, but in max you get errors even with them!
some years ago I looked into that stuff when doing the exporters for luxinia, which uses the nvmeshmender to create tangentspace (I could find references to it in the maya sdk code, too).
why should the calcs be different for max renderer?
Quality! in the scanline renderer you could use a more costly interpolation than in real-time (with connectivity... could make some nice smooth normals and therefore smoother tangentspace basis too) I speculate so, since that very part is not exposed in 3dsmax sdk. They do have some sample code how they create their tangent space so.
the viewport rendering is mostly a "preview" for their renderings, I don't think that the "game ready" stuff really has a big lobby with them. Else they could have been much nicer and expose stuff to do custom tangent-space and modify baker and so on, which is common in games as you know.
Why no standard? Well it's all about optimization. You can feed the tangent space with 3 vectors, or with 2 and assume the last one is perpendicular (just need the sign for direction, ie. vector3 for normal and vector4 for tangent and sign encoded in .w) One could per-pixel normalize all the tangentspace vectors again or not... so yeah even for this simple stuff plenty options... it's a ugly situation indeed. But frankly I think not that many variants are around there anymore. Also keep in mind that say crytek had their own baking tools as they started when baking in max or maya was poor still, or to make sure that nomatter where stuff came from run through own tools... then when everyone uses their own tool, they also come up with their own ways, as there is no need to ask "other devlopers" or whatnot hehe
No, I'm saying that you CAN'T make a model with tons of 90 degree angles, then simply put soft edges on all of it and bake it perfectly in maya or xnormal, which seemed to be what you were implying. When you do that, you always get those cool warping errors in realtime apps.
As for that example, yeah I've seen that happen on some of my test bakes before. If you triangulate the low poly mesh it doesn't seem to happen at all.
CrazyButcher: Ahh, makes sense, I forgot that some methods were optimised for performance and therefore slightly different. Still, it's annoying that there's no option to change the method for either renderer or viewport, since it means inconsistency internally in the app, which I find quite crazy.
But that is the thing, in maya you can get near flawless results, even when you have what most people would consider "bad" smoothing, much much better than what you get from max. This trickles down to fix virtually every "slight" smoothing error you would ever have, not just the big stuff.
The same can be said for XN, if you bake and preview in XN, but not as accurate as maya.
I've tried triangualting the mesh in max, and you can sometimes get better results by flipping the invisible edges too, but there are still obvious problems with max's renderer.
At the end of the day, there are tweaks you can make, hard edges, you can load up extra bevels to improve the quality, but in max you're just smoothing over the symptoms, and there are serious problems underneath.
You can see why it's doing it, by comparing a render of a sphere or any curved object with how it looks in viewport. You'll see the shading looks more correct when rendered with scanline, because it does some sort of interpolating correction, and not just linearly blending between vertex normals.
The crazy thing is, that Max' built in "Best" or SM3.0 default viewport material has these corrected normals in realtime. This is a shader that was added since Max 2008 and is embedded into 3DS Max from 2009 onwards (in 2008 you can actually look at the files).
Here's a screen that illustrates things:
Left: SM 3.0 realtime "improved" normals with default max material display
Center: the regular way any shader or games displays this mesh
Right: tangent space normalmap applied, you can see the incorrect baking Max has done, unable to fix the smoothing at the back.
All of these are the exact same mesh.
Now that there's finally some attention for this issue, I'd like to expand it to trying to find out what the calculations are to correct the normals like that. I tried looking for it but got nowhere...
Also, Xnormal is being mentioned as the alternative, but I have to say that Max's supersamplers are FAR superior to Xnormal. If you turn the global map sampler to Hammersley at 1.0, you get extremely nice anti-aliasing on sharp edges in the map, something where Xnormal fails. Santiago told me months ago he planned to add supersampling to Xnormal, but I'm still waiting for it. Until then, Xnormal is not an option, the maps from Max do look better, regardless of the incorrect tangent basis.
xNormal ftw, then display in whatever the F you want. (Even though I always have to flip my Green after baking in xN).
Glad I could contribute.
At work I really never go off of what it shows in max as far as most anything is concerned as far as textures go.
But what I have learned , is that if you apply your Normal Map and display it in realtime with the DX display, and then apply an edit normals modifier, and collapse the stack with it still applied, it fixes up your LP models normals to work better with the normal map normals, fixing weird issues.
Also, as far as hard edges and normal map bakes go, Yes, they do fuck up your results usually, but it depends.
I've basically learned to do an autosmooth by 80 degrees on the entire model before baking, which usually works fine. But sometimes you still need to separate certain faces to different smoothing groups to fix issues.
How are Maya's bakes compared to Max or xNormal?
I agree. Nobody should be testing their work in max anyway. Check your work in the engine it's supposed to be displayed in.
oh and you'd be surprised how many clients ask for max/whatever app viewport screencap and don't want to hear about marmoset , etc simply because the app is what they use, not marmoset or xnormal or whatever.
But, usually for my bakes, I just look for errors, and weird gradients, and if it looks clean, I roll with it and move on, and usually it works out fine for me. I don't test in max, because our pipeline makes it fast to see it in-game, which is what counts anyways.
And then they'll have to export from max to their engine and it will probably be different again. I guess if you're a freelancer you wouldn't care about that step.
Most engines I've used have slightly different normal calculations to what you'll see in max or maya. The lighting can look very different and the shaders may be more or less complicated. Obsessing over minor differences between max and maya doesn't seem as important to me when dealing with the gap between your engine and your modeling tool.
max mental ray vs maya
and for kicks, Max Mental ray vs Max Scanline
All of these are level enhanced to show the differences more clearly. ,b ut the differences are pretty clear, mostly in the green channel.
You know, just like the very first released game to utilize normalmaps - Doom3!
Its renderbumb output is different from both Max's and Maya's, and deals with extreme smoothing much better if I understand and remember correctly.
Regarding Epic and MAx bakes, at least for UT3 : the games shows some strong cases of normalmap mirroring issues (the red trainwreck in the first level), making me think that at that time, they indeed baked stuff and moved along if the result was "good enough" (especially since its mostly rusty and gritty materials). Gears looked cleaner tho so who knows. Also maybe their tangents were stripped on import - just like the vertex normals being recalculated. Making organic segmented meshes a royal pain to do...
Tangents info is important for curved shapes and seams ... maybe the Max baker just doesnt even consider them at all?
And yeah I wouldn't be surprised to hear that the viewport/scanline differences simply comes from the teams behind these two tools never having met each other. :P
Also I personally gave up on trying to preserve quads on work meshes before or after the bake. There is just way too many factors possibly going wrong at any time in the pipeline. Tris everywhere, I'd say.
Sigh ....
when you tick the normal map checkbox, click the file path button type thing to the right of it, and in the drop down menus choose Y-
JOB DONE NEVER FLIP AGAIN
Wow, I didn't know about this, where is the setting to turn this on?
I only recently upgraded to a graphics card that supports SM3.0 (was using a GeForce 7600 before a couple of weeks ago) so I wouldn't even have had that option
Pior's got a point though, there are only two ways to ensure that your normal-maps look correct in-game (of which he mentions one):
There is also option 3, which is to do both of the above things, which is what we do at SD
Eld: I think you're worrying a bit too much about triangulation here - the point of the normal-map is you should see no triangulation at all once you have baked. If you still can, then it's because either your normal map display is wrong (as shown on the middle model in Xoliul's screenshot), or your mesh is constructed in such a way that makes it impossible for the normal-map to compensate correctly (eg. having a smooth edge around a greater than 90 degree corner).
I can see your point though, you can see the triangulation direction in that screenshot, but if the normal-map was being used correctly (as on the far left) then it should not be an issue.
No you don't. Just invert the Y chanel on the "Normal Map" options before baking.
Edit: oops, ImSlightlyBored beat me to it, sorry for the repost.
MoP, this is how you turn on SM3. I used to use it, but after a few minutes the selected edges start to flicker, so I went back to software. I wonder if that was fixed on the latest SP.
You can also turn on SSAO and shadows on there, but it's slow as fuck for some reason, slowing your viewport to 5 FPS if you zoom in in your mesh.
What I ment was that I facepalmed myself with realizing that my triangulationhunt has been the result of this whole thread, that's why I posted that image to make sure it was
now a thinker: Why does max utilize sm3 to have a better realtime mode?
and: I would love to get a graphics programmer in here, they would know exactly what's happening in detail, and why many engines, including unreal, would optimize like that, what the gains are, and if it's worth it.