Home Technical Talk

Normal Map problems

e_x
polycounter lvl 18
Offline / Send Message
e_x polycounter lvl 18
I'm using max 7, following the tutorial Mop posted a while back. I've haven't had any problems untill now. Here is a screen of my problem:
normal_applied_thumb.jpg

As you can see, the lighting is all borked. Here is the normal map, where you can see the problem clearly.

door_lowNormalsMap.jpg

Here is a screen of my two objects, showing the side of the high poly object that produces the problem in the normal map.



Now in the screen, the lighting error doesn't appear, it only happens when I project the normal map.
both_objects_thumb.jpg

The lighting problem doesn't appear on the high poly mesh, it only shows up on the normal map when I project it.

Any ideas as to whats happening?

Replies

  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    Hmm, actually, I got a small error like this in wheel I did for the tutorial. It looked exactly like that (weird normals emanating from 2 certain points) but I thought I'd just done something silly.
    Have you tried triangulating the low-poly mesh, or converting to Editable Mesh before rendering the normal map?
    I don't know why this is happening, and I'd also like to know what's going on...

    MoP
  • e_x
    Offline / Send Message
    e_x polycounter lvl 18
    I've actually fixed it, but I'm not sure how frown.gif

    Here's what I did:

    Reset both the high and low's XForm then converted both to Editable Poly again. Then I applied a meshsmooth to the high poly object (it didn't have any Sub-D's), but I popped it off the stack before I redid the projection for the normal map. So it was either the XForm reset, or the collasping to an Editable Poly.

    I plan on doing more normal map work and will be back if I can repeat it and find a fix....
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    Might be the XForm. That fixes a lot of random things. I'll give it a shot on mine, and see if the problem goes away!

    This is what it's generating at the moment, after resetting XForms on both low and high mesh... it's doing the same thing exactly 180 degrees round the wheel, on the other side.

    normalbug.jpg

    Weird...

    So far I have determined it's nothing to do with:
    1) Resetting XForms.
    2) Pivot points of both objects having to be identical.
    3) The Lowpoly object being Editable Mesh or Editable Poly.
    4) The orientation of the objects in world space, with or without reset XForms.
    5) Whether the smoothing is native EPoly NURMS, Turbosmooth, or Meshsmooth in the stack.
    6) Exporting and importing either or both objects from 3DS format (like an "extreme" XForm reset!).
    7) The renderer. Scanline or mental ray makes no difference, nor does changing any of the renderer settings.

    The testing continues... I'm beginning to think it's something to do with UV-maps, now. I re-mapped the wheel completely, and while the error still shows up in a similar capacity, it's on a different part of the normal-map! How odd!

    I'm also noticing as I do more and more quick 512x512 normal-map renders, Max is getting slower and slower... the viewport runs fine but as soon as I change anything in the material editor, load or save a new image, it grinds to a halt for up to 20 seconds... ugh!

    MoP
  • e_x
    Offline / Send Message
    e_x polycounter lvl 18
    That is weird...

    I've noticed max 7 slowing down too, but I thought it was just my machine, maybe not.... especially loading a new map in.

    Have you ran into the problem where your mesh's edges will turn red and your normal map doesn't show up in the viewport? I've ran into that a couple of times, usually when adding and deleting lights from the scene.

    And what's up with one side of the mesh being very very dark? It seems like when you 'render to texture' all the backfaces come out dark, and won't even be lit even when you put a light on them.

    I'm going to do more testing tomorrow, see if I can find anything myself. I wish I could remember better what I did to fix my door. I know I didn't re-UV it, so I'm not sure if that will fix it either. Maybe I just lucked out. I do remember that I reset all of my materials and started over, so maybe that matters too. Could be several things....

    [ QUOTE ]
    6) Exporting and importing either or both objects from 3DS format (like an "extreme" XForm reset!).

    [/ QUOTE ]

    HA! that's funny!
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    Yeah I've heard other people had that problem with the mesh just displaying red edges (all the time though, not just when adding/deleting lights) ... I've never had that happen on mine though.
  • FAT_CAP
    Offline / Send Message
    FAT_CAP polycounter lvl 18
    hmmm - yeah I've had the red mesh thing aswell. It only happens on mine when I add certain types of lights - i.e skylights etc. It seems to work perfectly with just omnis or spotlights.

    And maX 7 is really slow for me sometimes aswell - I notice a lag when undoing operations - nothing huge but something that is definately annoying. Hopefully this sort of stuff will be sorted out for MAX 7.1 ?
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    Ah, that's it then. The Red wireframe will only show up when you put a light in the scene that the DirectX viewport manager doesn't support.
    You can use any Omni, Spot or Direct light, but as soon as you add a skylight the mesh turns to red wireframe. Just get rid of the Skylight and everything will be fine. I just tested that now.

    MoP
  • Eric Chadwick
    That weird normals problem looks to me like the normals are suddenly turning inside out. I wonder what would happen if you tesselated the low-poly model temporarily, just for the bake...?
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    Interesting idea, Eric. I just tried that, it still gets the exact same problem in the exact same place on my wheel.
    The tesselation actually also makes the initial projection cage more accurate, although it's still better to Reset and just Push it back into shape.
    The normal map it generated also has slight irregularities where the tesselated geometry was, instead of being perfectly smooth.

    OK, I just re-uv-mapped the whole thing, and actually changed where the seams and layout of the map is going.
    The error is still there, but in a different place. I also notice the error seems to be mirrored in each instance, but on a different axis.

    normalbug2.jpg

    That's the viewport display with this normal-map:
    normalbug2_map.jpg

    Compared with the original normal-map I was generating on these UV-maps:
    normalbug_map.jpg

    It seems that the normal-map renderer and viewport shader deal very well with UV-seams, though... that new map I generated has a big seam all the way round the tyre, and the map is different pixel density on each side of the seam, yet it hides it very well in the viewport display. Shame about the glaring huge error, eh? smile.gif
  • Eric Chadwick
    Are the models in the center of the world?

    I bet you'll get an answer if you post on the Discreet webboard.
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    Eric: I've tried them with identical pivot points, different pivot points, at the centre of the world, off-centre, on top and below the default grid... the error is always there, in the same place.

    I just rendered out a World-Space normal-map now, and it is perfect, not a glitch in sight. But as soon as I switch back to tangent-space, the error appears again.

    I've been looking through the Discreet forums, seems other people have this problem too, their solution so far appears to be "read the troubleshooting" and flip UV's around...

    MoP
  • e_x
    Offline / Send Message
    e_x polycounter lvl 18
    OK, I think I got it solved.

    It's a smoothing group issue. I've always read/seen that you want your low mesh to be only 1 smoothing group. This is correct, but only after you render to texture your normal map in max 7. I sort of stumbled onto this too. I was working on a new model, and rendered the normals as a test. The image looked fine but I had some tweaking to do to the low poly model, which I did. And before I re-rendered the normals, I set all the smoothing groups to one group, like you're suppose to. And BOOM, problem. I go back, do an auto-smooth and FIXED, no problem.

    I must be something with the way max 7 does its new render to texture. It seems like it uses the viewport for the projection, at least for the lighting information. So when you have only one smoothing group, it uses that lighting when rendering, for whatever reason. And I think we all know how crappy the lighting with one smoothing group can be too. I think this is also related to why one side of my model can be lit by a light in the viewport while the other can't. Its fine if I actually render it out.

    And yes, my box for my door was one smoothing group, and I fixed it by auto-smoothing it, which I didn't remember till now.

    Can anyone confirm??
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    Aha. You're onto it. It most certainly is a smoothing group problem. I just removed all smoothing from my wheel, left it faceted, and the glitch disappeared. However then it also generated a nasty normal map with fake "seams" along where the edges are in the facets. I'll have to find a happy medium ... will post screenies when I get it working!

    Right, here's what I get with Smoothing Group 1 on the left-hand side of the wheel, and SG2 on the other side:
    normalbug3.jpg

    On the left is the smoothed, un-mapped low-poly mesh. You can see the seam produced by the 2 SG's.
    On the right is the low-poly mesh with normal-map applied. When SG1 and SG2 are set as shown on the left, the normal-map looks fine, just with the seam being slightly obvious down the centre of the tyre.
    HOWEVER! As soon as SG1 is applied to the whole model, the bug becomes apparent again! So the normal-maps still have a slight problem, but the smoothing group setup makes it less obvious... all i can say is "WTF!".

    ... Big rotating hairy arses floating in space. Cockmonsters. Other vague and menacing insults thrown in Max's general direction.
    The smoothing group thing is definitely the issue, but changing the groups is like jumping out of the frying pan, into the fire, in my case. On an object like e_x's door, it's OK, because the edges are at 90 degrees so you don't notice the extra "fake seams" that no smoothing groups creates when rendering the normal-map. However in the case of my smooth tyre, it's very obvious. I've tried every combination of smoothing groups I can think of, and the error is always there, mocking me. It just comes and goes to varying degrees depending on how I set up the smoothing. But it's ALWAYS there. Someone should probably hit me before I go insane here.

    MoP
  • e_x
    Offline / Send Message
    e_x polycounter lvl 18
    WTF is right. I'm just happy you could reproduce it.

    Thanks for you help Mop grin.gif
  • Illusions
    Offline / Send Message
    Illusions polycounter lvl 18
    MoP...there is a very simple solution to your problem. Set the entire model to 1 smoothing group. I didn't see the problem at first when looking over what you guys were discussing because I thought it was common knowledge that when working with a normal map, the entire model should only have 1 smoothing group. Why is this you ask? Think of it like this:

    Your normal map is attempting to complete a job, which is to display how the model will appear when its lighted. Your smoothing groups are also attempting to do the same job as the normal map. One smoothing group can work just fine with a normal map...adding more than one causes confusion in the work force...and you get errors in rendering like that.

    So...each Element (an element by max's definition of Element), should only have 1 smoothing group assigned to it. Hope this helps...

    Edit: If it helps, think of your normal map as your smoothing groups...those other "smoothing groups"...have been replaced...given the pink slip...lol told never to clock in again...so don't have more than one on your low poly model at anytime.
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    Ah, perhaps I didn't explain properly. My first test (see my very first wheel image in this thread) was generated from a low-poly mesh with only 1 smoothing group.

    Sorry! Nice try though smile.gif
  • Illusions
    Offline / Send Message
    Illusions polycounter lvl 18
    [ QUOTE ]
    Ah, perhaps I didn't explain properly. My first test (see my very first wheel image in this thread) was generated from a low-poly mesh with only 1 smoothing group.

    Sorry! Nice try though smile.gif

    [/ QUOTE ]

    Ahhh...perhaps if you or E-X could send the .max file to me @ IllusionsCG@gmail.com, I could take a look. A picture, or in this case a file, is worth a thousand words...
  • e_x
    Offline / Send Message
    e_x polycounter lvl 18
    Sent my door to you Illusions with instruction on how to hopefully recreate the problem.
  • Illusions
    Offline / Send Message
    Illusions polycounter lvl 18
    Ok...I've narrowed it down a lot. Its not a problem with either mesh. It seems to be a problem with the UV map itself. I found this out because I decided to run the uvHelp script on MoP's uv's to see if it was a problem with distortion. What I discovered was two things...that uvHelp didn't like how MoP had the back of the wheel mirrored horizontally ( :P ), and that it had nothing to do with the way MoP mapped it. When I remapped it though, the error shifted to the bottom of the tire. So, I loaded up Photoshop and found that it looks like whatever is causing the error is creating it by smearing a few pixels in the Red channel in the places where the error appears.

    So...I still have no idea how to fix this...but perhaps this will help a bit.

    help.gif

    Edit: E_X have not recieved your door yet for viewing. I'll tell you tomorrow if I've recieved it, and if I haven't just resend it and I'll take a look smile.gif
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    Yeah, thanks for looking at that.
    Messing around with the UV-maps only seems to make the error move around, not get rid of the error at all... which is annoying.
    Smoothing group changes get rid of the error, but add other problems - bah!

    MoP
  • Illusions
    Offline / Send Message
    Illusions polycounter lvl 18
    E_X just received your door, gonna check into it. Hopefully this will give more insight...

    Edit: Alrighty, I don't know whats causing it but I know where its originating from...the low poly mesh. Somehow...theres something wrong with it. In your case lucky enough its just a cube, and is pretty easy to rebuild. I rebuilt your low poly door, re-uvmapped it, and...it rendered fine without the error. Odd...

    there has to be a better solution to this though, cause going in and correcting this by either rebuilding the low poly model, or paint-correcting the normal map in photoshop would be a real bitch on some more complex models...
  • e_x
    Offline / Send Message
    e_x polycounter lvl 18
    Rebuilt as in how?!? It's a box?!?! All I did was drag out a box and size it to the high poly one.... weird. Too bad a simple XForm reset doesn't fix it....

    Thanks for looking at it.
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    I just made a totally new low-poly model (out of a box this time, instead of a cylinder like last time), and gave it totally fresh new UV-mapping co-ordinates.
    This low-poly mesh has less segments than the other one, and is constructed in a fairly different way, geometry-wise.
    The exact same error is appearing in the exact same place.
    It's not down to geometry then...

    Ohhh! This is an interesting find... I accidentally hit Render to Texture when the hi-res mesh was Hidden in the scene - naturally it rendered it as red to show the rays hit nothing - yet the error still appears in the same place on the low-poly model being displayed in the viewport! What the hell!

    And whatever I do to the UV-maps, the error is always there, just in a different place depending on how I place the seams and rotate or mirror the UV's ... grr!

    The quest goes on...
  • Illusions
    Offline / Send Message
    Illusions polycounter lvl 18
    MoP, that seems really odd, as if you rendered an empty normal map (red miss), then you shouldn't have any error appearing at all...as the normal map should be black...and the object should then appear black as well.

    Edit: MoP, its definately the Low Poly mesh or however you're modelling it. I just redid your wheel the same way I did E_X's door and it rendered out a normal map perfectly. All I did to make the wheel was use a standard cylinder, then used a projection mapping, opened up UVW, and ran flatten mapping on it...and it came out fine.
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    Right, I think I have finally found the problem.

    The problem is... 3D Studio Max 7.

    The evidence:
    normalbug_found.jpg

    On the left, normal-map rendered and displayed on the low-polygon mesh in NVidia's Melody tool. The meshes were exported from Max in .obj format.
    On the right, the exact same normal-map from Melody, displayed on the exact same low-poly model, in Max's viewport with the DirectX shader enabled. Notice the errors and general dodgy shading.
    So, it looks like the problem is with Max in general, not any specific method. The errors show up on the flat maps when rendered from Max's normal-mapper, but do not appear on the maps rendered from Melody. Bah!

    MoP
  • Illusions
    Offline / Send Message
    Illusions polycounter lvl 18
    Yeah, I noticed other problems as well...on my smoke elemental when I attached the head mesh to the body...the normals went weird. And I don't mean welding verts or anything, just using the 'Attach' button. Like if I chose to attach the head to the body, the head would end up with weird shading. If I attached the body to the head, the body would end up with weird shading. I don't even understand how 'attaching' them could cause this. Its not a problem with the materials, as they both share the same multi/sub object material. Guess we chock this one up to a bug... :shrugs:
  • Thermidor
    Offline / Send Message
    Thermidor polycounter lvl 18
    It really does seem like the 3d programs havnt been set to cater for normal mapping yet ... max and maya both have problems as far as i can tell ...maybe some of the trouble lays in the hands of the gpu manufacturers , from what ive heard they use slightly different methods for rendering normal maps ... frown.gif

    Either way , we all have troubles displaying and using things that should just work ..
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    I just found out something even more stupid.
    When I was testing the model shown in my last post, I made a new material, turned on "DX Display of Standard Material", added a Normal Bump map in the Bump slot, and AS SOON AS I DID THAT, the shading error appeared. No bitmap loaded. Just the natural shading with the pixel shader enabled, has that "dent" of a shading bug, as you can see on the right-hand-side of the above image.
    What the hell!

    MoP
  • FatAssasin
    Offline / Send Message
    FatAssasin polycounter lvl 18
    Have you posted your findings on the Discreet webboards? I can't say I'm surprised to learn the a new feature in Max doesn't work Seems like Discreet likes to get us 90% there and then leave us hanging or having to do a ton of trouble shooting and work arounds. Hopefully this issue can get resolved in a service pack soon.
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    Yeah, I've posted on the discreet forums. One of the QE guys should be looking at the .max file I posted up there soon.
  • StrangeFate
    Offline / Send Message
    StrangeFate polycounter lvl 18
    The problem is the smoothing groups of the lowpoly mesh, they are taken into account when generating the nromalmaps.

    It's not a max problem and they wont fix it, it's just the way it works, in any app i'm afraid.
    Usuall workarounds are to either use proper smoothing groups (too high= bad results, if they look bad in a OGl viewport they will bake bad)), unweld pooints for baking were needed, or subdivide the lowpoly mesh for baking, that will reduce the size of the areas being baked wrong.
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    SF: Yeah, but when I put the exact same models through Melody, with the low-poly model being all one smoothing group, then it generates a perfect normal-map and looks great in the preview window (see screenshot earlier). So it must be something that Max and Melody at least are doing different.
    And smooth groups certainly don't have anything to do with the "dent" that's appearing as soon as I add a "Normal Bump" material to the object in Max, even without loading a file - that's GOT to be a Max bug, or at least a DirectX bug.
  • StrangeFate
    Offline / Send Message
    StrangeFate polycounter lvl 18
    The door problem is definetely the usuall smoothing group problem. It's not apparent with round objects as they need higher smoothing groups in the first place to look smooth.

    If removing the smoothing groups fixed the problem in your model then it should be the same problem somehow. Did you try subdividing the lowpoly mesh just for baking the map ?
    That affects the smoothing groups to some degree, might be enough of a change to fix the problem.
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    Yeah, Eric Chadwick suggested tesselating the low-poly mesh before baking the normal map, I tried that and it had exactly the same results as the un-subdivided mesh.
    However, oddly enough, you are right on the smoothing groups point. I tried changing the SG's (they pretty much match the UV-splits now, I wonder if that helps?) and then baked the map. It came out fine! Hooray!

    normalbug-fixed.jpg

    However this still does not explain why Melody can render a perfect normal-map from a low-poly object with all 1 smoothing group, yet Max cannot.

    Also Max's lighting is still weirded in this fashion - I make an omni light, move it to the right of the object, and everything lights fine.
    Move the light to the other side of the object, most of it remains dark, except some bits which light up to some extent. It doesn't really make sense, because if the normals were flipped, surely they would light up when the light is on the "wrong" side of them? In this case they just don't light up at all from any angle...
    Hmm!
  • FAT_CAP
    Offline / Send Message
    FAT_CAP polycounter lvl 18
    Maybe it could be because you have mirrored the geometry after modelling, and attached the two halves together?

    Maybe try the RESET XFORM - its always handy in situations like these!
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    Waaaaaay ahead of you FAT_CAP, but thanks for the suggestions. I've reset the xforms a number of times, exported and imported the objects, flipped the normals, edited the UV's and smoothing groups... other people on the discreet webboard have reported similar problems with lighting their normal maps in the viewport... seems fine when rendered though...
  • FatAssasin
    Offline / Send Message
    FatAssasin polycounter lvl 18
    I noticed the same problem the first time I tried using normal mapping in Max. It was a experiment with a simple sphere, so no mirroring. One side just didn't light up as I moved the light around. I figured I was just doing something wrong and haven't experimented much more with it.
  • StrangeFate
    Offline / Send Message
    StrangeFate polycounter lvl 18
    Can't help there, don't use max. You could try importing the model and normalmaps into a game or ORB and see if they lit properly.
    There's quite a few ways to handle normalmaps, some dont support mirrored UV's, this and that, maybe Max doesn't has the smartest implementation.
  • FatAssasin
    Offline / Send Message
    FatAssasin polycounter lvl 18
    [ QUOTE ]
    Also Max's lighting is still weirded in this fashion - I make an omni light, move it to the right of the object, and everything lights fine.
    Move the light to the other side of the object, most of it remains dark, except some bits which light up to some extent.

    [/ QUOTE ]

    Did you ever figure this out? I was playing around with Max's normal mapping and I think it has to do with the Tangent, Local XYZ, Screen, and World options in the Normal Bump Options and in the Projection Options. Make sure they match. I had the same problem when I used Local XYZ in the Projection Options, but didn't change the default Tangent in the Normal Bump Options. Changing to Local XYZ in the Normal Bump solved the problem.
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    Nope, I have both of those set to Tangent, and the lighting still acts weird.
  • Eric Chadwick
    Has this been logged in the Discreet bug database?
  • FatAssasin
    Offline / Send Message
    FatAssasin polycounter lvl 18
    Sorry that didn't help. Have you tried redoing the normal map using other methods, like Local XYZ to see if the problem is still there?
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    The normal map displays fine if the settings in the Normal Bump map are set to anything but Tangent. As soon as Tangent is set in the material editor, the lighting goes all weird.
    Eric: There's been a few complaints of this type on the Discreet forums, so I'm assuming it has been logged - where can I go to check?
  • poopinmymouth
    Offline / Send Message
    poopinmymouth polycounter lvl 19
    Ok, I have had to be tackling this problem the last few days at work. Here are my findings.

    If you render it in any method but tangent, it comes out fine, no errors. If you want to render in tangent with no errors, you have to apply smoothing groups, at least one per UV element. Then you will get no errors. The problem with this is that you now have butt fugly seams in your model, where they don't line up. So the only answer I can come up with atm, is to render the light bake in max, but do the normals on the exact same model in Melody. Melody knows how to handle xyz, as does every other single frickin normal mapping program. I don't know how this got passed Discreets testing, but I am assuming they did not have anyone who knows how normal maps are made in a production pipeline test it. So till discreet comes in line with proper workflow, everyone use melody. 8-)
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    Yes, that was the solution I am also using at the moment...
  • Eric Chadwick
    Heheh, makes me glad I'm still in 5.1. Well, kinda.

    Defect Report Submission Form
    (Found by going here and choosing Feedback from the left panel.)
  • Mojo2k
    Offline / Send Message
    Mojo2k polycounter lvl 18
    mop, back on your tire, what if you set one side to group 1, the other side to group 2, then the middle strip to 1/2 then its, not one smoothing group, but you also do not have the fake seams,


    never mind, just tried, no worky, tried several other things also, no work,
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    Yeah Mojo, I tried that, evidently you found the same thing - it doesn't help smile.gif

    I'm gonna see what happens on the current high-poly model I'm doing. Almost finished the low-poly mesh, gonna UV-map it then do normal-map renders from 3dsmax, and hope it works.

    If not, Melody is the best bet...
  • r13
    Offline / Send Message
    r13 founder
    i know this doesnt help as much, but if you can get a copy or trial of kaldera, try it. it is much more robust and understandable. i have a copy at work, and all the creature guys ask me to process their normals as it does such a better, cleaner job than max7 does natively.
Sign In or Register to comment.