Home Technical Talk

Understanding averaged normals and ray projection/Who put waviness in my normal map?

2456711

Replies

  • killnpc
    Offline / Send Message
    killnpc polycounter
    ajr2764, start your own thread, you cretin.
  • Lamont
    Offline / Send Message
    Lamont polycounter lvl 15
  • DLoud
    Offline / Send Message
    DLoud polycounter lvl 15
    Good stuff here Earthquake
  • fearian
    Offline / Send Message
    fearian greentooth
    Nice one Lamont!

    And bloody nice one EQ! Cheers for sticking around and expanding on some of this stuff as well! Great thread!

    I know I've heard alot of this stuff before in bits and pieces, but I'd still not corrected my workflow properly. What you mentioned about 'bake and forget' - getting it right with the mesh and avoiding problems latter - really rung true!
  • EarthQuake
    This was some great information. I am still a rookie at modeling, high to low poly modeling in particular and I noticed that you mentioned creating a "model for your low", which I assume is a thread similar to this or perhaps a tutorial. Regardless, I'm having difficulty finding it. Could you post a link please? I would love to read it.

    Sorry, there is not a thread for this, and it is not "a model" but "Modeling for your low". What this means is that we plan and model our highpoly giving lots of consideration to how the final lowpoly will be constructed.
    Tickwomp wrote: »
    Ugh, been looking through simlar normal map threads to figure out what the hell is going on whenever I have to bake stuff; this sums it up nicely. I do have a question about this specific example though. Is it safe to say that the waviness is less in the far right cylinder because shape of the cylinder matches closer to the high poly? The top edge also looks nicer than I've ever been able to achieve; is that only from averaging the normals of the cage and having more sides?

    Well, two things.

    Averaged cage normals cause waviness, not "help" them, you may want to re-read the first posts a little there.

    Secondly, yes it is simply because there are more sides. The more accurately we can match the highpoly with our low, the less difference there is between the models when the rays look out at that skewed angle.
  • HAL
    Offline / Send Message
    HAL polycounter lvl 13
    EarthQuake wrote: »

    Oh, and another thing. Even if your lowpoly's normals look like A, your Cage/Projection mesh normals look like B. That is unless of course your bake is set up poorly(either using "offset" in max, match using "surface normals" in maya, or using the ray distance in XN instead of a proper, welded cage). If your cage is not averaged, you will get seams on all of your hard edges, as the normals will be facing perpendictular along those edges, and cause gaps in the projection.
    waviness07.jpg

    Great that was what I was looking for. Though I ran into a problem here.

    Blender is stupid and has no smooth groups, so I have to split the faces manually to 'emulate' that.

    That results in a difference in the vertex count of the low and the cage when I weld it back together for an averaged cage.

    What do I do now :( Use max for setting up smooth groups?
  • EarthQuake
    HAL wrote: »
    Great that was what I was looking for. Though I ran into a problem here.

    Blender is stupid and has no smooth groups, so I have to split the faces manually to 'emulate' that.

    That results in a difference in the vertex count of the low and the cage when I weld it back together for an averaged cage.

    What do I do now :( Use max for setting up smooth groups?

    Use Max, Maya or Xnormal(you will still have to export from Max or Maya, or any other app that supports proper custom normals without physically splitting edges, XSI, new Modo? etc). That is really a huge deal breaker if you dont have proper smoothing groups/hard edges.
  • HAL
    Offline / Send Message
    HAL polycounter lvl 13
    hmm thats what I suspected, I hoped that there would be a workaround or something but mehh ;D

    thank you for clearing this up!
  • EarthQuake
    HAL wrote: »
    hmm thats what I suspected, I hoped that there would be a workaround or something but mehh ;D

    thank you for clearing this up!

    Depending on what you're doing, Max/Maya are going to be vastly superior baking wise to blender anyway. If you're doing portfolio work you're going to want the most accurate tangent basis display, and you'll only find that in Maya, or in 3ds Max with 3ps shader quality mode(or the latest 2011 version with the QM fixes).

    If you're exporting to an engine with broken tangent space(read: most) its less of an issue.
  • HAL
    Offline / Send Message
    HAL polycounter lvl 13
    Since I am doing this mostly for myself this (read: it should look as nice as possible), you are right I need to switch tools I guess, I have been baking in xnormal so far with ray measurement :poly122:

    e:// Low to max-> smoothgroups -> then baking did it :)

    Thanks!
  • metalliandy
    Offline / Send Message
    metalliandy interpolator
    HAL wrote: »
    Great that was what I was looking for. Though I ran into a problem here.

    Blender is stupid and has no smooth groups, so I have to split the faces manually to 'emulate' that.

    That results in a difference in the vertex count of the low and the cage when I weld it back together for an averaged cage.

    What do I do now :( Use max for setting up smooth groups?

    Are you splitting the verts manually (Y hotkey)?

    Blender uses a similar hard edge system to Maya and you can export without manually splitting the verts.
    Just mark the hard edges and use the EdgeSplit Modifier with "From Marked As Sharp" on.
    On export though, both smoothing groups and hard edges will have the same effect as you are just splitting the mesh anyway.
    But, yea. Blenders baking is really bad. No cages or AA baking :/
  • HAL
    Offline / Send Message
    HAL polycounter lvl 13
    Are you splitting the verts manually (Y hotkey)?

    Blender uses a similar hard edge system to Maya and you can export without manually splitting the verts.
    Just mark the hard edges and use the EdgeSplit Modifier with "From Marked As Sharp" on.
    On export though, both smoothing groups and hard edges will have the same effect as you are just splitting the mesh anyway.
    But, yea. Blenders baking is really bad. No cages or AA baking :/

    wait that edge thingy does the same like smoothgroups? Great to know...... thank you very much!
    And no, I dont bake in blender I dumped that 2 years ago when it sucked XD
    I use xnormal for my bakes.
  • metalliandy
    Offline / Send Message
    metalliandy interpolator
    Yea, its the same thing :)
    And i stopped using Blender for baking and swapped to xNormal a while back too ^^
  • HAL
    Offline / Send Message
    HAL polycounter lvl 13
    Great news that solves my dilemma^^
  • EarthQuake
    You should be able to export that mesh, go into xnormal's 3d viewer, set up a cage so you have an averaged cage. This is a pain in the ass tho. =P

    The default/no cage method will use the lowpoly's mesh normals, where the cage will use an averaged projection mesh... You used to have to manually "weld" it in XN but i dont know if this is still the case.
  • HAL
    Offline / Send Message
    HAL polycounter lvl 13
    EarthQuake wrote: »
    You should be able to export that mesh, go into xnormal's 3d viewer, set up a cage so you have an averaged cage. This is a pain in the ass tho. =P

    The default/no cage method will use the lowpoly's mesh normals, where the cage will use an averaged projection mesh... You used to have to manually "weld" it in XN but i dont know if this is still the case.

    Yeah it really is a pain in the ass :P I probably will stick to the blender->max-> xnormal pipeline. Or I learn how to bake in max :)
    Yea, its the same thing
    And i stopped using Blender for baking and swapped to xNormal a while back too ^^

    Hm I had hoped that this would be the solution for my problem but upon trying it I noticed that it does the same what I did manually before.

    So there's still the 'vertex count differs in cage and lowpoly' message in xnormal.

    Conclusion: Exporting from blender for baking sucks balls since that averaged cage wont work due to the afore mentioned issues.

    meh.
  • EarthQuake
    HAL wrote: »
    Yeah it really is a pain in the ass :P I probably will stick to the blender->max-> xnormal pipeline. Or I learn how to bake in max :)

    Just learn how to bake in max, and use max shaders for texturing, much better options here than in blender or anywhere else, and with the latest max, or 3ps shader, you get accurate normals, which you wont get baking in XN and then sending to max/anything else.
  • HAL
    Offline / Send Message
    HAL polycounter lvl 13
    Will do, thanks both of you, I learned a lot in this thread!
  • Bal
    Offline / Send Message
    Bal polycounter lvl 17
    My problem with using Maya or Max for baking is handling very large files (=sculpted meshes). It's all fine when I'm just baking sub-d models like weapons and such, but for full characters it's just such a hassle to import 2-3gb worth of OBJs into Maya or Max and handle renders with such huge files (which xN handles just fine).

    I also find it really annoying to bake ambient occlusion in Maya, xN is so much faster and results seem better most of the time.

    Any insight into these issues?
  • EarthQuake
    Bal: use some sort of polycruncher to get rid of excess geometry?

    Also, when I bake in maya, I do my AO in XN. Maya is retardly slow(single threaded) and, as of 2008 which is what I used, cant even bake proper AO from high to low. So, I simply run XN on 3 cores when i'm baking normals in Maya on 1 core, AO takes about as long in XN as the normal map + RGB map in maya.
  • Bal
    Offline / Send Message
    Bal polycounter lvl 17
    Yeah, decimating could be a solution, but it complicates the process, and is not ideal when you have polypaint on your meshes for base textures (which I bake in xN anyways so isn't a problem, but it still means I'd need to export two sets of high poly objects, one decimated for Maya, and not the other for xN). Still I'll try with Maya again, maybe have the whole high poly in a separate .mb file (bit lighter), that I'd import as reference in my .ma for the bake...

    Where's xNormal 4 with full tangent basis choice and complexe project setups to launch multiple queued bakes? :D
  • EarthQuake
    Generally with organics, having the most accurately synced tangent basis is less of an issue. Hard surface stuff it becomes much more important. So, maybe its just not a big deal for those sort of meshes.
  • konstruct
    Offline / Send Message
    konstruct polycounter lvl 18
  • ENODMI
    Offline / Send Message
    ENODMI polycounter lvl 14
    Ha, yeah EQ, shape up or ship out.
  • Millenia
    Offline / Send Message
    Millenia polycount sponsor
    May I bear your children EarthQuake? :3
  • SpeCter
    Offline / Send Message
    SpeCter polycounter lvl 14
    May i eat them afterwards?
  • Stromberg90
    Offline / Send Message
    Stromberg90 polycounter lvl 11
    I just want to have he's brain in a jar :)

    Great thread, love reading you technical posts ;)
  • metalliandy
    Offline / Send Message
    metalliandy interpolator
    HAL wrote: »
    Yeah it really is a pain in the ass :P I probably will stick to the blender->max-> xnormal pipeline. Or I learn how to bake in max :)



    Hm I had hoped that this would be the solution for my problem but upon trying it I noticed that it does the same what I did manually before.

    So there's still the 'vertex count differs in cage and lowpoly' message in xnormal.

    Conclusion: Exporting from blender for baking sucks balls since that averaged cage wont work due to the afore mentioned issues.

    meh.
    Using an external cage file in xNormal requires that the cage has identical topology to the LP mesh, so im not sure the best way to tackle this tbh.
    Beg someone to enable the exportation of custom normals in Blender?
    Bal wrote: »
    My problem with using Maya or Max for baking is handling very large files (=sculpted meshes). It's all fine when I'm just baking sub-d models like weapons and such, but for full characters it's just such a hassle to import 2-3gb worth of OBJs into Maya or Max and handle renders with such huge files (which xN handles just fine).

    I also find it really annoying to bake ambient occlusion in Maya, xN is so much faster and results seem better most of the time.

    Any insight into these issues?
    Do the HP files have UVs? This can make for insane file sizes.
    Also triangulating your mesh before import into xNormal, allows you to use a higher polycount as it doesnt have to compute that step. :)
  • Bal
    Offline / Send Message
    Bal polycounter lvl 17
    Do the HP files have UVs? This can make for insane file sizes.
    Also triangulating your mesh before import into xNormal, allows you to use a higher polycount as it doesnt have to compute that step. :)
    Nah, no UVs, I always delete them because Zbrush often crashes on export when there are some wacked UVs. There's vertex colour though, that's pretty heavy as well. I've always exported quads from ZB, but I never have any probs loading the files in xN (I have plenty of ram), but I'll switch to tris I guess if it makes loading them faster.
  • fattkid
    Offline / Send Message
    fattkid polycounter lvl 15
    Hey EQ, thanks for the info. Really appreciated.

    I think I just learned something, but I could use some clarification. Regarding cages:

    I've never really used cages before, not really understanding the advantage to them. I have always used the Offset parameter in Max of Ray Distance in XN for my bakes.
    I have mostly done organic/character type models, so needing a cage was perhaps less obvious?

    Looking at your example of the low res cylinder with the hard edges/smoothing groups, and the accompanying cage with averaged normals, I think I realize the importance of the cage.

    The averaged cage will have a more complete and accurate result, as the rays being cast will be hitting the 90 degree hard edge at the averaged/45 degree angle as well, that the Offset/Ray Distance method will miss, due to it's using the low res meshes assigned hard edges/smoothing groups normals (that are perpendicular to the surface/hard edge) when calculating, and there will essentially be a "gap" in the ray cast between the hard edges/smoothing groups normals/ray cast calculations.

    Am I understanding this correctly?
  • EarthQuake
  • AlecMoody
    Offline / Send Message
    AlecMoody ngon master
    if you don't need to turbosmooth your cylindrical shapes, then dont't! you're not getting anything out of baking a highres cylinder to a low res cylinder, other than wobbly shapes in your normal maps edges, ...


    This is too general and not really correct. You can absolutely get better smoothing around the cylinder with a good bake.
  • Stromberg90
    Offline / Send Message
    Stromberg90 polycounter lvl 11
    Finaly this is a sticky.

    Great that you are sharing you knowledge EQ :)
  • EarthQuake
    fattkid wrote: »
    Hey EQ, thanks for the info. Really appreciated.

    I think I just learned something, but I could use some clarification. Regarding cages:

    I've never really used cages before, not really understanding the advantage to them. I have always used the Offset parameter in Max of Ray Distance in XN for my bakes.
    I have mostly done organic/character type models, so needing a cage was perhaps less obvious?

    Looking at your example of the low res cylinder with the hard edges/smoothing groups, and the accompanying cage with averaged normals, I think I realize the importance of the cage.

    The averaged cage will have a more complete and accurate result, as the rays being cast will be hitting the 90 degree hard edge at the averaged/45 degree angle as well, that the Offset/Ray Distance method will miss, due to it's using the low res meshes assigned hard edges/smoothing groups normals (that are perpendicular to the surface/hard edge) when calculating, and there will essentially be a "gap" in the ray cast between the hard edges/smoothing groups normals/ray cast calculations.

    Am I understanding this correctly?

    Yes, to break it down into very simple terms, an averaged cage is a "seamless" cage. If that helps.
  • metalliandy
    Offline / Send Message
    metalliandy interpolator
    HAL wrote: »
    Yeah it really is a pain in the ass :P I probably will stick to the blender->max-> xnormal pipeline. Or I learn how to bake in max



    Hm I had hoped that this would be the solution for my problem but upon trying it I noticed that it does the same what I did manually before.

    So there's still the 'vertex count differs in cage and lowpoly' message in xnormal.

    Conclusion: Exporting from blender for baking sucks balls since that averaged cage wont work due to the afore mentioned issues.

    meh.

    I figured out a fix for this issue yesterday. :)

    1. Add the EdgeSplit modifier to your model, after marking the sharp edges but dont apply it yet.
    2. Duplicate your mesh and scale the whole object along the normals as you would a cage.
    3. Apply the EdgeSplit to both models.
    4. Export your HP, LP and the cage mesh.
    5. Use the cage mesh as the external cage.
    6. Bake!.
    Baked with xNormal and hard edges with an external cage for each model.

    http://dl.dropbox.com/u/2057427/Polycount/Blender%20Normals.7z

    Normal_test-1.jpg

    Cube_normals.jpg


    Cylinder_normals.jpg
  • EarthQuake
    But why am I sticky? Awwwww!

    Actually, I still don't get how a cage is supposed to work on an object with smooth groups or split edges if the cage is then supposed to be completely smoothed. The topology is different and causes XN errors. Waaaaa!!


    I'm not sure how XN reads the normals when loading an external cage, however, this is an XN issue if it doesnt work correctly. =)
  • EarthQuake
    Some carry-over from another thread( http://www.polycount.com/forum/showthread.php?t=81116 )
    Not really, the roundness of the shape from a baking perspective is conveyed perfectly by the smoothing of the object, without pushing it into a shape that does not match your lowres. Your highres mesh need only be high poly enough to convey the shapes you ACTUALLY need to transfer, in this case the roundness of the wheel well need not be transferred it will be conveyed perfectly by the smoothing of the low poly irregardless.

    Smoothing it would only yield missed rays and wobbly normal maps.

    Example:

    Normalmapbakes.jpg


    EarthQuake wrote: »
    Alright, to put the matter to rest once and for all, here are a few more tests, in a realsitic(and very similar to this) situation.

    A. High res modeled with details that "match" the roundness of the low(without treads)
    B. High res modeled with details that "match" the roundness of the low(with treads)
    C. High res modeled to look good(without treads)
    D. High res modeled to look good(with treads)

    I've included the without tread versions because the outer shape gets the same result with treads, as the roundness comes from the treads themselves.
    wheeltest01.jpg

    wires:
    wheeltest03.jpg

    bakes:
    wheeltest03.jpg
    wheeltest04.jpg
    wheeltest05.jpg
    wheeltest06.jpg

    So, i'de like someone to explain to me how this "not smoothing" method makes anything better, in these examples every aspect of that method looks worse, and only helps to accentuate the jaggedness of the low res model. Even at accute angles where there should be a big advantage, this method seems to look worse.

    So, my conclusion here, is this method is of such limited use that is a huge waste of time to even bother doing. In addition to looking worse, modeling like this is a pain in the ass, and more destructive than keeping a nice clean sub-d mesh, which is easier to edit.
  • Tickwomp
    Offline / Send Message
    Tickwomp polycounter lvl 7
    Thanks for all the testing & information. I feel less retarded since reading this thread (and the other one); up til now I've been baking with hit or miss results.
  • krisCrash
    Offline / Send Message
    krisCrash polycounter lvl 9
    So what this and other normal map threads have taught me:

    This thread: On your low poly, it's typically better to spend geo on curvature than details like bevels and ridges; to not underestimage your normal map's ability to fake these.
    My own observation: And again for the low; smoothing group splits will prevent the applied normal map from doing its magic. I have taken to putting everything on 1 group and just splitting any edges I seriously want hard before exporting to engine, and only those edges that also coincide with an UV split (but not every edge that is UV split). So what I did was hacking with some coloured lights and basically.. baking my smoothing groups to the normal map. So.. baking the low poly to itself (or a meshsmoothed copy of it). I should really look into a good method for that.
    Sure you can bake to a faceted low but the normal map will have to be so precise it's not even practical, given the different natures of pixels and edges.
  • aphexx
    Offline / Send Message
    aphexx polycounter lvl 12
    that is valuable information! thank you for all the time and effort, earthquake!!
  • EarthQuake
    krisCrash wrote: »
    So what this and other normal map threads have taught me:

    This thread: On your low poly, it's typically better to spend geo on curvature than details like bevels and ridges; to not underestimage your normal map's ability to fake these.

    Yes, generally speaking, its better to use your geometry to support the larger shapes of your mesh, than the tiny little bevels. Adding bevels to the ends a cylinder for example can result in 3x the tri useage, so you might as well just fix your problems by adding more sides to it, as it will look better in the end as well.

    My own observation: And again for the low; smoothing group splits will prevent the applied normal map from doing its magic. I have taken to putting everything on 1 group and just splitting any edges I seriously want hard before exporting to engine, and only those edges that also coincide with an UV split (but not every edge that is UV split). So what I did was hacking with some coloured lights and basically.. baking my smoothing groups to the normal map. So.. baking the low poly to itself (or a meshsmoothed copy of it). I should really look into a good method for that.
    Sure you can bake to a faceted low but the normal map will have to be so precise it's not even practical, given the different natures of pixels and edges.

    I'm really confused here, baking low to low? What is the purpose of this?

    You can use hard edges/smoothing groups on your lowpoly without any problems, and with the exact same results as no hard edges if you simply follow these rules:

    A. Uv splits need to coincide with your hard edges
    B. Your cage needs to be averaged(this means, regardless of hard edges on your low).

    In addition to that, set up properly, there really are no "negatives" to using hard edges, as:
    A. It will result in the same seamless bake, when done properly
    B. When you have hard edges along uv splits, there is no jump in vertex count, these verts are doubled along your uv seams.
    C. Using hard edges can sometimes result in a cleaner normal map texture, and this means its easier to generate cavity maps from with less artifacts, as well as compressing better.


    So, because there really aren't any inherent "drawbacks" to using hard edges, provided your mesh is set up properly, there is no reason to avoid them or come up with weird-o crazy work flows to get around it. I just use a script in maya to assign hard edges to all of my uv splits, it takes 1 click of my mouse.
  • aphexx
    Offline / Send Message
    aphexx polycounter lvl 12
    currently i bake my stuff with xnormal and one smoothgroup applied.
    i place here and there support edges where the angles are heavy.
    then use the sbm file format for the baking, where i tick the "export tangent basis" field.
    everything worked fine so far.
    we used the hard edge/uv seam workflow also for some time but found it to be more easy the other way.
    are there reasons to not use this workflow?
  • EarthQuake
    Well it depends. If your renderer and baker are synced up, as far as tangent basis goes(IE: 3ps quality mode in max, the latest 2011 max, Maya's baker and viewport display). Then using a minimum of hard edges is ideal, because it just "works" without thinking too much about it. However, since you're baking in XN, and XN isnt synced to anything we can assume that isn't the case.

    So, when dealing with a broken baking workflow, you have two basic ways to deal with smoothing errors;
    1. Add a bunch of geo and bevels and things to smooth out the lowpoly normals and make smoothing errors less noticeable(they will still be there however)
    2. Use hard edges to do the same

    #1 generally uses more geometry than #2, and doesn't really provide many benefits over #2 when both are done correctly.
  • haiddasalami
    Offline / Send Message
    haiddasalami polycounter lvl 14
    Amazing stuff EQ. Learned a tonne from it.
  • aphexx
    Offline / Send Message
    aphexx polycounter lvl 12
    thanks mate for the quick answer.
    our renderer is our game engine. its inhouse so nothing well known.
    i also followed that discussion about synched baker and renderer and as i understood, it's always a problem no matter where you bake.
    is it in the end, that there will ALWAYS be errors, you just have to check what produces the less for your engine?
    or is the workflow with hard edges/uvchunks the ultimate one, which produces no errors at all?
    sorry for the noobish questions, but it appears to me that i dont understand some fundamentals
  • EarthQuake
    I have used engines that have perfectly synced up normals, and it basically just "works". So no, I wouldn't say there always will be errors, unfortunely i'm under NDA and can not talk about the specifics. However, just think of it like 3ps quality mode, 3dsmax's offline renderer(which produces the same results as 3ps QM btw) or how accurate Maya displays bakes, these are the gold standards to strive from as far as a tech standpoint. As they are "technically" perfect, and once you have this working correctly, you worry about the 1% of wierd geometry that sill causes errors, or things like running out of resolution to properly represent gradients(what I like to call resolution-based smoothing errors).

    I generally do not ever do only one thing, and I encourage you to experiment with different methods and find what best suits you, because it depends on so many circumstances.

    The hard edge workflow will never guarantee that you have no errors, esp if your tangent basis isn't synced up, I mean unless its like just a basic 12 tri cube with hard edges on all sides, then it will be perfect! But thats not a realistic test case. =) However, it has some pros to other methods, as I've mentioned above.


    The moral of the story is: Dont always do X, just because you read it on Y or because Z told you to.
  • aphexx
    Offline / Send Message
    aphexx polycounter lvl 12
    you are right. good to hear that i havent done all wrong now for ages haha.
    next i do is having a talk with our programmer about synching normals :)
    thanks again for your explanations!
  • TheMadArtist
    Offline / Send Message
    TheMadArtist polycounter lvl 12
    Just dropping in to say thanks for writing all of this up. Very useful info.
  • SpeCter
    Offline / Send Message
    SpeCter polycounter lvl 14
    I think EarthQuake deserves his own thread full of thanks.
    Not to forget Beer, as much he can handle, payed by us of course.
    If i would life near to him, he would be drunk 24/7 because of all the "thank-you-beer" he would get from me :)
  • ajr2764
    Offline / Send Message
    ajr2764 polycounter lvl 10
    EQ, I'm curious what your workflow is like as far as uv's/smoothing groups. Do you typically unwrap a mesh not considering the smoothing groups and then split all your uv's on hard edges. You mentioned a script for maya that does that..where might I find that. You think it's worth it to bake a mesh that is naturally like 90% hard edged in the expense of saving say 300 or less tri's?

    Saving some tris/smoother edges vs a huge amount of uv islands(15+)/increased texturing work.
2456711
Sign In or Register to comment.