Why you should NOT trust 3ds Max's viewport normal-map display!

13456719
MoP
  • Offline / Send Message
  • Joined
  • 135 Topics
  • 11.5K Replies
OK, sensationalist headline. But it got your attention! This was a spinoff from another thread, I thought it deserved its own.

Here are my findings based on Max vs. Maya baking using the native tools and displaying the results in the viewport using the native tools.

Summary:
Unless your game engine is set up to use the same tangent basis as 3ds Max's renderer (not it's viewport), your normal-mapped models will never look 100% correct!

I have just done some simple tests tonight, and I can confirm that Max's "baking" tangent space is the same as it's "render" tangent space (no surprise, since the scanline renderer is used to bake normals in Max), however the "render" tangent space does not match the tangent space that it uses in the viewport. This strikes me as just plain insane. It means you'll never get the same result between rendering and viewport display.

If you use a "correct" normal-map for viewport display in Max, then it will look messed up in render, and vice versa... horrible.

I baked the same objects in Maya, and they are displayed perfectly in the Maya viewport. Interestingly enough, this map also looks perfect in Max's viewport! That means that the viewports in both apps derive tangent basis in the same way. However it also means that Maya's baking renderer correctly uses this tangent space! Which makes sense, of course.

It basically seems like since the viewports (and Maya's baker and Mental Ray, by extension) seem to all use the same tangent basis derivation, this is the most "correct" one, and therefore 3ds Max's renderer and baker are "wrong". This should be a standard that all applications adhere to, I see no reason why it shouldn't be.

Screenshots as promised...
Comparison of viewport shaders (just using both apps' native display of default shaders, nothing fancy or custom - grey Phong materials with high specular):
plank_bakes_comparison.jpg

Max's scanline renderer normal map output:
plank_bake_max.jpg

Max's Mental Ray normal map output (ignore lack of anti-aliasing, that's not what's being tested here):
plank_bake_max_mr.jpg

Maya's normal map output:
plank_bake_maya.jpg

If you're feeling intrigued you could download those normal maps and flick back and forth between them in an image viewer to see the main differences. You should be able to see them clearly enough here, though.

So, out of interest, does anyone here work in a pipeline where they bake their normal-maps using 3ds Max, and if so, do you know what your game does with them? Have you ever had any problems?

EDIT: ADDENDUM!

Here are some direct links to some of the more useful posts from later in the thread:
EDIT 2! 3POINT SHADER RELEASED!
http://www.3pointstudios.com/3pointshader_about.shtml

Replies

  • DarthNater
    • Offline / Send Message
    • Joined
    • 16 Topics
    • 909 Replies
    Did you flip the green channel? I take my bakes into Photoshop and invert the green...

    Second question: Does the same thing occur when using Mental Ray in Max?
  • MoP
    • Offline / Send Message
    • Joined
    • 135 Topics
    • 11.5K Replies
    DarthNater wrote: »
    Did you flip the green channel? I take my bakes into Photoshop and invert the green...

    You shouldn't have to invert the green channel in Photoshop, there is a setting for that in the Render To Texture "Options" dialog. Just set it to the correct Y orientation before you bake.
    Every application has a control for this (in Maya it's based on whether the mesh's "tangent space" attribute is set to Left-Handed or Right-Handed), if you're inverting the green channel in Photoshop then you're not baking it right :)
    DarthNater wrote: »
    Second question: Does the same thing occur when using Mental Ray in Max?

    I have only tried the Scanline renderer so far. I will try Mental Ray on the same objects, hopefully it will give better results (although that also means more inconsistent results within Max...).

    Edit: After trying Mental Ray inside Max (added image to original post), while it is slightly different to the scanline bake, it is still far from being correct, if we're treating the Maya bake as the "most correct" bake.
  • Vrav
    • Offline / Send Message
    • Joined
    • 21 Topics
    • 1.2K Replies
    Gasp! Good thing I use Xnormal.
  • DarthNater
    • Offline / Send Message
    • Joined
    • 16 Topics
    • 909 Replies
    MoP wrote: »
    You shouldn't have to invert the green channel in Photoshop, there is a setting for that in the Render To Texture "Options" dialog. Just set it to the correct Y orientation before you bake.
    Every application has a control for this (in Maya it's based on whether the mesh's "tangent space" attribute is set to Left-Handed or Right-Handed), if you're inverting the green channel in Photoshop then you're not baking it right :)

    Good to know, never saw that option :) Thanks!
  • Marine
    • Offline / Send Message
    • Joined
    • 55 Topics
    • 1.6K Replies
    so what you're saying is that maya is awesome and max sucks
  • Racer445
    • Offline / Send Message
    • Joined
    • 27 Topics
    • 1.2K Replies
    I pretty much bake all my normal maps with Max. I've thrown the straight Max normals into Source and Cryengine 2 and haven't once had any problems like this. Then again, I'm also mainly doing lots of hard surface and mechanical stuff.

    It really does look like the green channel needs flipping, and you should try using Xoliul's max shader, since the max default doesn't display normal maps correctly.
  • eld
    • Offline / Send Message
    • Joined
    • 28 Topics
    • 4.3K Replies
    This is definately most interesting, but I believe most people will not think this far, and even wont go as far as to properly set their hard edges even for baking :)
  • aesir
    • Offline / Send Message
    • Joined
    • 58 Topics
    • 4.2K Replies
    I paint out annoyances in photoshop.
  • MoP
    • Offline / Send Message
    • Joined
    • 135 Topics
    • 11.5K Replies
    I guess the questions we should ask here are:

    1. Which application is using the "best" tangent basis derivation? As far as I can tell, Max's renderer tangent basis is the "odd one out" - more apps like Maya, XNormal, maybe others are consistent between each other, yet Max isn't even internally consistent!

    2. Why is Max's renderer tangent basis different to it's viewport tangent basis?
    This essentially means that if your normals look correct in the viewport, then they will look wrong in the render. Conversely, if they look correct in a render, they will be wrong in the viewport. This is ridiculous!

    3. If you can write an FX shader that uses Max's "renderer tangents" in the viewport, why doesn't Max's default internal shader and supplied FX files use this method? Both Max's "Show Hardware Map in Viewport" option, and it's supplied external shader files like "StandardFX.fx" use the default viewport method, which means any bakes you do in Max will look crappy using these shaders! Insane!
  • eld
    • Offline / Send Message
    • Joined
    • 28 Topics
    • 4.3K Replies
    aesir, I think you're missing the point of what mop is trying to explain though
  • Ark
    • Offline / Send Message
    • Joined
    • 44 Topics
    • 1.6K Replies
    Interesting... the maya bake seems to capture more colour information than both of the max bakes.

    Could we get the source files to test in other apps to see if its just max what does this?
  • eld
    • Offline / Send Message
    • Joined
    • 28 Topics
    • 4.3K Replies
    I'd love get to know exactly how a top of the line engine like unreal engine does this.
  • Sage
    • Offline / Send Message
    • Joined
    • 99 Topics
    • 1.9K Replies
    Mop I find it odd that you are posting this now, I thought you used to be a Max user then had to go to Maya. I'm wondering if this happened in older versions of Max or is it an upgrade that came with 2010. I mean they did change how you could shift drag things to extrude edges, so maybe they thought it was a good idea to update the viewport display and not do so with the scanline renderer so now they don't match.

    I'm also wondering if this is the case then how is it that you can make normal maps in Max and use them in unreal ed so they look correct or does unreal for example have code that can render this out properly because a coder took the time to do so.

    It be nice if all normals maps were created equal :D
  • MoP
    • Offline / Send Message
    • Joined
    • 135 Topics
    • 11.5K Replies
    Sage wrote: »
    I'm also wondering if this is the case then how is it that you can make normal maps in Max and use them in unreal ed so they look correct or does unreal for example have code that can render this out properly because a coder took the time to do so.
    (and eld's point, added the quote in an edit)

    Well yeah, I was wondering that too, since I thought they baked stuff in Max?
    That either means they've set up the Unreal Engine to use Max's renderer tangent derivation (meaning maps baked in any other app will look wrong!), or they're getting dodgy looking stuff in-game and dealing with it by adding geometry till the problem goes away... or... they just don't care that much :)
    Sage wrote: »
    Mop I find it odd that you are posting this now, I thought you used to be a Max user then had to go to Maya. I'm wondering if this happened in older versions of Max or is it an upgrade that came with 2010. I mean they did change how you could shift drag things to extrude edges, so maybe they thought it was a good idea to update the viewport display and not do so with the scanline renderer so now they don't match.

    Nope, it's been like this as long as I've used Max. Definitely since max 8, and it was in max 2008 and still a problem in max 2010.
    Sage wrote: »
    It be nice if all normals maps were created equal :D

    Yes, exactly - we should have some sort of standards for this (ie. how the normals/tangents/bi-normals/whatever are calculated).
  • CrazyButcher
    • Offline / Send Message
    • Joined
    • 81 Topics
    • 1.4K Replies
    uhm I think even using custon FX in max doesn't save you from artifacts. Afaik the scanline baker bakes for the scanline renderer, which uses different kind of interpolation than what realtime rendering does (not just per-vertex tangent basis). At least that's what I read. One would have to modify the baker in max to get perfect matches. I also remember people here saying that some engines managed to load stuff artifact free in games, but had to do something "special" what that is, they couldn't tell due to NDA stuff.
    how that is nowadays I don't know, but last time I checked it was still easy to create tangent maps that only look perfect in scanline.
    If I recall correctly then ASE used in unreal import also doesn't encode tangent stuff, which means they indeed must do things the same way max does when importing.
  • eld
    • Offline / Send Message
    • Joined
    • 28 Topics
    • 4.3K Replies
    MoP wrote: »
    or... they just don't care that much :)

    this :)
  • EarthQuake
    • Offline / Send Message
    • Joined
    • 245 Topics
    • 11.6K Replies
    Racer445 wrote: »
    I pretty much bake all my normal maps with Max. I've thrown the straight Max normals into Source and Cryengine 2 and haven't once had any problems like this. Then again, I'm also mainly doing lots of hard surface and mechanical stuff.

    It really does look like the green channel needs flipping, and you should try using Xoliul's max shader, since the max default doesn't display normal maps correctly.

    You're also adding 5000 hard edges to your models to get a correct result in max, you could take those same models with totally smooth normals in maya and bake them with results that are *near flawless*.
  • Sage
    • Offline / Send Message
    • Joined
    • 99 Topics
    • 1.9K Replies
    Mop how did you notice this? Did you start writing hlsl shaders or something and noticed the difference? I imagine since unreal is using hlsl for their shaders it's the same, so maybe they have a different baker. Maybe some of epic guys can say something about this. :D

    Mop I think it's pretty obvious that Autodesk doesn't care that much. I mean look at the Max 2010, it's awesome..
  • MoP
    • Offline / Send Message
    • Joined
    • 135 Topics
    • 11.5K Replies
    Sage: I noticed it first, a long time ago, when I was working purely in Max. I thought for a while that it was my lowpoly geometry causing the issues. Then after working with XNormal then Maya for a bit, I realised that they could bake the same stuff and display perfect results, yet Max's was always dodgy.

    Thing is, it seems like Max's viewport calculates tangents almost identically to Maya's baker and viewport. So at least that seems like a more consistent calculation.
    Is there any reason why the calculations should be different for max's renderer?!
  • Sage
    • Offline / Send Message
    • Joined
    • 99 Topics
    • 1.9K Replies
    EarthQuake wrote: »
    You're also adding 5000 hard edges to your models to get a correct result in max, you could take those same models with totally smooth normals in maya and bake them with results that are *near flawless*.

    EQ you use Modo. How much easier is it to get proper results than in say Max.
  • eld
    • Offline / Send Message
    • Joined
    • 28 Topics
    • 4.3K Replies
    Mop, are you implying that max should be and has always had perfect features?

    tip of the iceberg.. ;)
  • Sage
    • Offline / Send Message
    • Joined
    • 99 Topics
    • 1.9K Replies
    Mop the reason is most likely they forgot it was different and no one renders normal maps. Oh wait....
  • Racer445
    • Offline / Send Message
    • Joined
    • 27 Topics
    • 1.2K Replies
    EarthQuake wrote: »
    You're also adding 5000 hard edges to your models to get a correct result in max, you could take those same models with totally smooth normals in maya and bake them with results that are *near flawless*.

    I highly doubt something with many many 90 degree angles would bake well in any app without adding edge bevels or adding in hard edges.
  • Chai
    • Offline / Send Message
    • Joined
    • 11 Topics
    • 461 Replies
    Well I heard of this before, but very interesting to see an example cheers ...
    Seems max can only overcome smoothing issues based on raytracer than realtime.

    There's a way to get good maps in max (or any other engine) by setup your smoothing groups and uvs properly.
    Below is a map I've done with max (realtime viewport) a while ago. (before I switched to xnormal)

    Anyway thanks a lot MoP, I was using xNormal without even knowing I could get away with less smoothing errors. :D
    (I have to revise my normalmap article now)

    f3_plasmapistol_04.jpg
  • EarthQuake
    • Offline / Send Message
    • Joined
    • 245 Topics
    • 11.6K Replies
    Racer445 wrote: »
    I highly doubt something with many many 90 degree angles would bake well in any app without adding edge bevels or adding in hard edges.

    So, you're saying you use NO hard edges, and get perfect results in max?....

    Clearly you can't be saying that, or you would be showing examples that prove everyone in this thread wrong. =)

    This is a pretty good example of what happens in max, even using hard edges with a pretty clean mesh, no outrageous issues or anything, you get visible smoothing errors, in any other app(maya, xn) these areas should render perfectly without hard edges, but in max you get errors even with them!

    maxfail.jpg
  • CrazyButcher
    • Offline / Send Message
    • Joined
    • 81 Topics
    • 1.4K Replies
    MoP wrote: »
    Thing is, it seems like Max's viewport calculates tangents almost identically to Maya's baker and viewport. So at least that seems like a more consistent calculation.
    Is there any reason why the calculations should be different for max's renderer?!

    some years ago I looked into that stuff when doing the exporters for luxinia, which uses the nvmeshmender to create tangentspace (I could find references to it in the maya sdk code, too).

    why should the calcs be different for max renderer?

    Quality! in the scanline renderer you could use a more costly interpolation than in real-time (with connectivity... could make some nice smooth normals and therefore smoother tangentspace basis too) I speculate so, since that very part is not exposed in 3dsmax sdk. They do have some sample code how they create their tangent space so.

    the viewport rendering is mostly a "preview" for their renderings, I don't think that the "game ready" stuff really has a big lobby with them. Else they could have been much nicer and expose stuff to do custom tangent-space and modify baker and so on, which is common in games as you know.

    Why no standard? Well it's all about optimization. You can feed the tangent space with 3 vectors, or with 2 and assume the last one is perpendicular (just need the sign for direction, ie. vector3 for normal and vector4 for tangent and sign encoded in .w) One could per-pixel normalize all the tangentspace vectors again or not... so yeah even for this simple stuff plenty options... it's a ugly situation indeed. But frankly I think not that many variants are around there anymore. Also keep in mind that say crytek had their own baking tools as they started when baking in max or maya was poor still, or to make sure that nomatter where stuff came from run through own tools... then when everyone uses their own tool, they also come up with their own ways, as there is no need to ask "other devlopers" or whatnot hehe
  • Racer445
    • Offline / Send Message
    • Joined
    • 27 Topics
    • 1.2K Replies
    EarthQuake wrote: »
    So, you're saying you use NO hard edges, and get perfect results in max?....

    Clearly you can't be saying that, or you would be showing examples that prove everyone in this thread wrong. =)

    No, I'm saying that you CAN'T make a model with tons of 90 degree angles, then simply put soft edges on all of it and bake it perfectly in maya or xnormal, which seemed to be what you were implying. When you do that, you always get those cool warping errors in realtime apps.

    As for that example, yeah I've seen that happen on some of my test bakes before. If you triangulate the low poly mesh it doesn't seem to happen at all.
  • MoP
    • Offline / Send Message
    • Joined
    • 135 Topics
    • 11.5K Replies
    Chai: Cool, I like your normal map reference page, it's great.

    CrazyButcher: Ahh, makes sense, I forgot that some methods were optimised for performance and therefore slightly different. Still, it's annoying that there's no option to change the method for either renderer or viewport, since it means inconsistency internally in the app, which I find quite crazy.
  • EarthQuake
    • Offline / Send Message
    • Joined
    • 245 Topics
    • 11.6K Replies
    Racer445 wrote: »
    No, I'm saying that you CAN'T make a model with tons of 90 degree angles, then simply put soft edges on all of it and bake it perfectly in maya or xnormal. When you do that, you always get those cool warping errors in realtime apps.

    But that is the thing, in maya you can get near flawless results, even when you have what most people would consider "bad" smoothing, much much better than what you get from max. This trickles down to fix virtually every "slight" smoothing error you would ever have, not just the big stuff.

    The same can be said for XN, if you bake and preview in XN, but not as accurate as maya.
    As for that example, yeah I've seen that happen on some of my test bakes before. If you triangulate the low poly mesh it doesn't seem to happen at all.


    I've tried triangualting the mesh in max, and you can sometimes get better results by flipping the invisible edges too, but there are still obvious problems with max's renderer.


    At the end of the day, there are tweaks you can make, hard edges, you can load up extra bevels to improve the quality, but in max you're just smoothing over the symptoms, and there are serious problems underneath.
  • Ben Apuna
    • Offline / Send Message
    • Joined
    • 123 Topics
    • 2.5K Replies
    I've used Epic's SHTools plugin for Max to render out normal maps for Unreal in the past. The resulting maps were supposedly more "correct" than Scanline or Mental Ray bakes. So I'm guessing that's how they handled the odd baking properties of Max.
13456719

Leave a Comment