Home Technical Talk

normal maps and smoothing groups

1
Ruz
polycount lvl 666
Offline / Send Message
Ruz polycount lvl 666
I ran in another problem with normal mapping ie the lack of smothness on the low poly model spoils the effect of the normal map.

Ihe head I am working on is about 1200 polys and turning edges can only make it look smooth to a certain degree.There are always going to be artifacts showing as ther are n't enough polys to descibe a smooth surface

If I add more polys to make it smooth, then there seems no point having a normal map to describe that detail.
I was looking at poops tutorial and his low poly mesh seems very low indeed.
For example if that had to be facially animated, he would have to build a lot more poly detail in .
confused i am

Replies

  • thomasp
    Offline / Send Message
    thomasp hero character
    are you talking about the roundness of the silhouette or the smoothness of the surface? because in my experience normal mapping takes total care of the surface itself - you do not have to worry about turning edges or inserting smooth groups/creases except for very extreme angles.
    only the lack of roundness of the silhouette spoils the illusion.

    personally, i like the look of that mix between lo and hires.
  • Ruz
    Offline / Send Message
    Ruz polycount lvl 666
    I mean the surface smoothness
    What seems to be happening that the detail in the normal map seems to be added to the not very smooth surface of the low res mesh, like an overlay layer in pshop for example

    I have all smoothing group set to 1 BTW.
    So I wonder what is happening here. Surely the whole point of normal mapping is not to display the normals of the low res surface so it appears smoother
    The silhouette issue doesn't bug me too much, but this does
  • ScoobyDoofus
    Offline / Send Message
    ScoobyDoofus polycounter lvl 20
    Ruz: As I recall Ben's lo-poly model was only 1200 triangles. Very low indeed.
    It was not designed for facial animation.

    Your head is 1200 tris all by itself! That should be MORE than enough for whatever your doing I'd think...

    You arent generating your normal from a 1200 poly model or something are you? Im confused.

    I hope your talking about silouette.
  • ScoobyDoofus
    Offline / Send Message
    ScoobyDoofus polycounter lvl 20
    ack. Ok, late post. Um...I figure your working on NDA stuff, but maybe you can privately show it to me, or Poop or Pior/Per somebody via ICQ or something, and they can help out out?
  • Ruz
    Offline / Send Message
    Ruz polycount lvl 666
    No ,i correctly generated the normal map from the high res model i made in zbrush.
    but when the normal map is displayed on the low res head model( 1200 poly model ) it looks shit. (It has the green channel the right way around in case you were thinking it was that)
    the detail of the normal map is showing up fine , but the underlying none smoothnes of the low res model is spoiling the effect
  • Joshua Stubbles
    Offline / Send Message
    Joshua Stubbles polycounter lvl 19
    Can you post a pick? Public or private?
    Hard to know what the issue is, if we can't see it ourselves.
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    Ruz, try using Ben Cloward's normal map shader if you're currently using the default 3dsmax one. 3dsmax's default normal map shaders make things look like ass anyway. I'm guessing it's probably just that.
  • Sean McBride
    Offline / Send Message
    Sean McBride polycounter lvl 18
    Could I get a link to that paul?
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    http://www.monitorstudios.com/bcloward/resources_shaders.html

    Putting "ben cloward normal map" in Google brings up something pretty close to that, too wink.gif

    Big props to Ben for all this stuff, it's invaluable. I use the 3-lights specular version of the shader, it's ideal, and also doesn't downsize viewport textures to 512x512 like 3dsmax's default shaders.
  • Ruz
    Offline / Send Message
    Ruz polycount lvl 666
    well the odd ting abou this is that when i render the normal map in max, I can definitely see that the surface anomolies in the low poly mesh are getting baked in to the normal map output.
    The normal map output from say zbrush is very smooth looking .. which is what max should be doing.
  • Daz
    Offline / Send Message
    Daz polycounter lvl 18
    What are you using to generate the actual normal map Ruz? What blows me away is the different outputs I'm seeing. Im finding the latest version of Orb for instance to seem much nicer than my Zbrush generated normal maps. I must be doing something wrong, because in zbrush I seem to get strange cubic artifacting in my normal maps. I'd be curious as to what most people use to generate their maps with. There's so many different ways to do this stuff. I see folks spitting out displacement from zbrush, and using other tools to generate normal maps from that. All sorts of stuff.
  • Ruz
    Offline / Send Message
    Ruz polycount lvl 666
    I have tried just baout everything and z brush seems to produce the nicest stuff of what i have tried so far.
    Melody keeps on crashing, the ati normal mapper produces similar stuff to max.

    There is a script coming out soon which enhances z brush normal map output. (comes out end of october).
    They just released a displacemnt map script thingy also
  • Daz
    Offline / Send Message
    Daz polycounter lvl 18
    Can you post pics of your issue or is it work stuff?
  • thomasp
    Offline / Send Message
    thomasp hero character
    normally i'm using max's builtin tool, however for special jobs orbs comes to the rescue. it's far slower but has much better error tolerance and helps to bake problematic areas.

    what's great is granny's (radgametools) builtin baker. easily 10, 20 times faster than orb and noticeably faster than max. yet the output seems like the best of both worlds. it's not freeware though, granny licenses are like 10k and they don't distribute this tool separately.

    it's a command line tool with integration into max and maya.
  • Eric Chadwick
    [ QUOTE ]
    when i render the normal map in max, I can definitely see that the surface anomolies in the low poly mesh are getting baked in to the normal map output.

    [/ QUOTE ]If this is happening, then no shader's going to fix it, IMHO you're gonna have to re-cast.

    I see this with heightmaps here sometimes. My remedy is to temporarily stick a Meshsmooth/Turbosmooth modifier on the low-res, just during the baking. By increasing the number of ray-cast normals, I get smoother results in the bitmap. Just gotta be careful with the UVs, Meshsmooth tends to round those off as well.
  • Ruz
    Offline / Send Message
    Ruz polycount lvl 666
    cheers Eric , nice idea.
    Actually most of the problems were caused by our low poly meshes not being low poly enough. I more or less halved the poly count of the low poly head and it looks sweet now.

    Its common sense really, but can't see the wood for trees sometimes.
    Thanks for the suggestions guys.
  • Joshua Stubbles
    Offline / Send Message
    Joshua Stubbles polycounter lvl 19
    Err, how is it common sense that your low poly models aren't low poly enough? I'm getting similar issues (with low poly surface anomolies), although my prop (a chair in this case) is only 300 polys.

    It's becoming rather annoying =\
  • Toomas
    Offline / Send Message
    Toomas polycounter lvl 18
    Are you sure your low and high poly models are not in contact with each other? And that you set the ray casting limits correctly?
  • EarthQuake
    I've ran into similar problems like this on countless occasions using 3dsmax's render to texture for normals maps. If you dont have very accurate normals on the low poly model you will get smoothing errors. The only solutions to this are to use smoothing groups or add more polys so that you have more accurate normals on your low poly mesh.

    Now if you we're doing this in doom3 you wouldnt get these errors because it re-normalizes the mesh and renders them out to avoid these errors.

    The engine im working with currently works exactly like max which is a shame. I havent really done any character work yet so this is all based on my experiences with props and such.
  • Joshua Stubbles
    Offline / Send Message
    Joshua Stubbles polycounter lvl 19
    Toomas - the low poly and high poly need to occupy the same space....so yes, they're in 'contact' with each other. The ray cast settings are also just max default.

    EQ, thanks for the input. It's a shame, but I'm glad to hear that others are having the same issues that I am.

    Have any of you guys had good results from products like Kaldera or Melody? I tried messing around with Melody last night, and I couldn't get it to process the normal maps correctly.

    Any input would be great, thanks smile.gif
  • Eric Chadwick
    Kaldera works well for us. I like the immediacy of Max's Projection modifier a bit more than Kaldera's manual cage. And Kaldera is a simpler interface, quicker to use. Haven't taken Melody for a spin in a long time.

    BTW, some Kaldera examples here...
    http://forums.cgsociety.org/showthread.php?t=274096&page=6&pp=15
  • Joshua Stubbles
    Offline / Send Message
    Joshua Stubbles polycounter lvl 19
    Thanks, Eric. I installed the demo and was messing around with their incldued sample objects. They don't seem to suffer from the problems I was having with 3dsmax, but then again, the demo won't let me use my own objects.
  • Ruz
    Offline / Send Message
    Ruz polycount lvl 666
    What helped me a quite a bit ( though i didnt explain it too well), was to make the low poly mesh very simple and clean, ie no weird angled polys and sudden changes in direction.The surface anomolies don't really show up much now and my final results are getting to quite pleasing at last

    I am now z brushing my low poly mesh straight up to a really high z brush version, with no intermediate model.

    The normal map looks much better this way.
  • Joshua Stubbles
    Offline / Send Message
    Joshua Stubbles polycounter lvl 19
    Well, you shouldn't have to constrict the way you model the low poly mesh though, right? I mean, that defies the purpose of the art. There is a distinct problem with the RTT in 3dsmax, as well as Nvidia's Melody application.

    From the Kaldera demo's I've messed with, this shading issue *appears* to be non-existant. I am however, contacting Diego right now, to see if I can use my own meshes with the demo version. If it does in fact resolve the issue, we're buying the plugin immediately.
  • Ruz
    Offline / Send Message
    Ruz polycount lvl 666
    actually its freed me up a bit rather than restricting me. i can knock out the head and zbrush head in a day now. the only verts i have to tweak are the lower poly ones which is not so much of a chore. I can only do this now my zbrushing has advanced to a decent standard

    Doesn't change the fact that something weird is happening with the ati normal mapper and the max version. the quality is not that great, but if you look poops normal map, because he used a really low poly model, there is more detail in the normal map as you would expect, as the normal map is made from the difference between the two meshes.
    That was done in max and look nice.
  • Eric Chadwick
    If you want, Vassago you can send me a file and I'll send you back some tests. If confidnetiality is an issue, don't worry, I'll delete the model after I'm done. echadwick(at)whatif-productions.com
  • ScoobyDoofus
    Offline / Send Message
    ScoobyDoofus polycounter lvl 20
    Yeah, I dont know what you guys are talking about in terms of the "flawed Max RTT" because, while a bit cumbersome, it works ok for me, and generates decent results. Then again, I barely know what IM talking about here, so...heh. Kinda new to this stuff myself.
  • Joshua Stubbles
    Offline / Send Message
    Joshua Stubbles polycounter lvl 19
    Eric, thanks for the offer. I'm waiting for a response from Diego right now. If he says it's a no-go, I'll certainly send the file to ya. I'm only doing R&D tests right now anyways, nothing special.

    Scooby, max's RTT does in fact generate some good results. For some reason though, it will sometimes output the funkified shading from the low polys, into the normal map.
    So there is actually some bug-ness in the Max RTT utility.
  • Eric Chadwick
    Well Kaldera has its problems too. Can only use the scanline renderer, and only the Area AA filter. A bad low-res model will cause problems (true anywhere tho). It's hella slow with the Antialias filter spinner set to >1 (we double-render, then scale down and re-norm). Height map clamping is trial-and-error. Can't bake Specular Level/Color properly, unless you copy it into Diffuse or something first.

    On the other hand, it's really good at what it does. smile.gif And you can batch-process stuff with some simple MAXScript.
  • EarthQuake
    Vassago: I dont think its a bug nor is it a "sometimes" thing, its pretty predictable where it will do it. Anywhere that you have a steep change in angle(Very inacurate normal on the lowpoly because of the nature of smoothing) you will get problems unless you split the edge(smoothing groups.)
  • Ruz
    Offline / Send Message
    Ruz polycount lvl 666
    Does anyone know if max 7.5 fixes any of these issues?
  • Eric Chadwick
    7.5 has nothing to do with RTT.
  • Ruz
    Offline / Send Message
    Ruz polycount lvl 666
    I had heard that either 7.5 or 8 had improved some of the bugs in the RTT. Hey our tech artist at work told me this. just asking
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    I think it's Max 8 that will fix the RTT and viewport shader with normalmap bugs.
  • Joshua Stubbles
    Offline / Send Message
    Joshua Stubbles polycounter lvl 19
    Yea, I just installed the Max8 demo. The viewport normalmap seam issues are now resolved, though I still get the above issue with funky shading.
    I'm at my wit's end with this now. Eric is trying to help me as well, but there still seems to be a problem.
  • Eric Chadwick
    Like I said, don't trust the Max viewport. It doesn't calculate the binormals. Use something like the Doom3 viewer, or better yet your own company's game engine.
  • Joshua Stubbles
    Offline / Send Message
    Joshua Stubbles polycounter lvl 19
    I can't get the Doom3 viewer to work properly, although I've tried everything. Our studio's engine doesn't yet support normal mapping, so that's also out of the question.
    I viewed them in a suedo-D3 viewer, but it still had artifacts (albeit a little less noticable).

    It seems that it's an issue with the actual generation of the normal maps, actually. It doesn't make since though, since I've seen meshes with more low poly shading issues, and they work perfectly fine.
  • Joshua Stubbles
    Offline / Send Message
    Joshua Stubbles polycounter lvl 19
    Anyone? Ideas?
    This is something I can't seem to resolve unless I break the mesh up with smoothing groups. Doing that completely fubar's the look the model, though. The normal map has a harsh edge where it shouldn't, yadda yadda.

    Any ideas on this would be most apprieciated.
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    Vassago: I dunno quite what the problem you're experiencing is - are you using the DX Display of Standard Material with a Normal Bump map type in the Bump slot?

    Here's a test I did in Max7 and the Max8 trial today.

    max8compare.jpg

    This mesh is 256 triangles, and has UV symmetry down the centre. As you can see, Max7's DX Display screws up the mirrored side, all the lighting is inverted. It was actually worse with the normal-map rendered from Max7, this normal-map has been rendered from Max8 and applied in Max7's viewport.

    However I can't see any of the artifacts you're talking about on either of these (the low-poly models are pretty low compared to the high!), they appear completely smooth (even in Max7 it does, but only on the non-mirrored side, as you should just be able to see.

    I'm using an ATI Radeon Mobility X700 for displaying these.
  • EarthQuake
    Mop its much more apparent on mechanical type stuff where you have hard angles on things
  • Joshua Stubbles
    Offline / Send Message
    Joshua Stubbles polycounter lvl 19
    MoP - using the renderbump has always worked fine for me. I never use it though, since I've been told it's not accurate to the in-game normal map visuals. The DirectX viewport shaders are what I usually use. With these, I'm getting the visual errors. I've tried viewing them in a doom3-based viewer (not the 'official' one, though) and I still saw the same shading issues.
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
    Hmm, yeah, actually I know what you mean. Usually you can avoid it either by adding a little extra geometry to the lowpoly to even out the smoothing, or changing the smooth groups to have hard edges where there's a 90-degree or greater change in direction on the mesh, then re-render the normals. That usually looks pretty good, I've found.

    Then again, I haven't checked these out in any game engine, I'm just going by what I see in the Max viewport, so maybe that error wouldn't show up in something like Doom3 or Unreal Engine 3?
  • EarthQuake
    The problem with using smoothing groups to break it up is, if you model subtile little curved edges in your high poly model spliting the edges looks bad. And yes adding extra gerometry will usually fix the problem but these are both just hacks and not really addressing the real issue IMO.

    In doom3 you would use one single SG and i would imagine it to be the same in UE3 aswell.
  • DDS
    Offline / Send Message
    DDS polycounter lvl 17
    My question is:
    How do you calculate normal maps for industrial stuff just applying 1 smoothing group to the whole mesh? Applying smoothing groups you get seams 'cause it breaks up all the faces.
    I only have this problem with normal mapping, in max, orb, ati, melody, kaldera, and anything else.
    Is this engine related more than because of the normal map generator?

    -DDS
  • Joshua Stubbles
    Offline / Send Message
    Joshua Stubbles polycounter lvl 19
    I finally resolved this issue earlier today. I'll be creating a tutorial for it soon. smile.gif
    Thanks for the help/input, all
  • MoP
    Offline / Send Message
    MoP polycounter lvl 18
  • Eric Chadwick
    Glad to hear it Vassago. Let us know what it came down to.

    DDS, sounds like you answered your own question. You're right, don't use smoothing groups on the low-resolution mesh.

    If your game engine massages the normals properly, you don't need to worry about shading problems on your low-res model.

    But if it doesn't, then you can work around it somewhat, by adding a chamfer or bevel to acutely-angled areas of the low-res surface.

    No matter what, always use a single smoothing group for the low-res model.
  • Joshua Stubbles
    Offline / Send Message
    Joshua Stubbles polycounter lvl 19
    I'll certainly try to get the tutorial done this week, since it's rather quick. Keep in mind though, this will only work on non-organic meshes.
  • DDS
    Offline / Send Message
    DDS polycounter lvl 17
    EricChadwik but then polycount goes so high frown.gif

    It's a pity.

    Vassago I'm expecting your tutorial, thanks for doing this.

    DDS.
  • CrazyButcher
    Offline / Send Message
    CrazyButcher polycounter lvl 20
    About normalmaps & engine. it really depends on the exporters and so on. Your 3d app will use its own way of generating the tangent vectors for the mesh before it "renders" the normalmap. If the exporter will actually take those and bring them in your engine, the results should be the same. However I guess most engines do their own tangent computation, which might be different from what the 3d app does. Thats the point when things very likely start to look different.
    This kind of information unfortunately seems to be a trial & error expierence, unless you know someone who actually knows how engine/3d app compute their tangents... ideally that would be written in some SDK doc as well.

    edit: should have said that the tangents are more the "key" than the normals to normalmapping. They are computed out of your normals and the texcoords. Which is why seams/mirroring of texcoords will make it hard for the app to know what you really want. However as seen in the max8 shots, there already are ways to solve the issues, mostly by breaking the mirrored vertices up again, which results into a higher vertex count in your mesh, then again nothing one should really bother these days, as fill-costs ie. pixelshading is more fancy (useually you see more pixels of your model than vertices wink.gif )
1
Sign In or Register to comment.