Inflating high poly before baking normal map

Dantexagc

Hi everybody, I wanted to discuss and ask advice regarding a part of the high to low poly pipeline. My mentor told me that, as a last step before baking the normal map from my high poly to low poly, it is good to inflate/displace the high poly geometry (make it more puffy if you will) to completely cover the low poly model. The idea is to get a more round look on the shading of the final model.

While sometimes this works and I do see a difference between doing it vs not doing it, some other times it can be a complete nightmare. Many details are pinched or completely deformed when attempting to displace the geometry of the high.

I will had some captures from my latest model to illustrate this. You can see the high poly in white and low poly in red:



I find this part of the process extremely frustrating. I am aware I could have a low poly that is closer to the high poly, adding some basic chamfers so the high poly would not need to be displaced so much to cover the low poly. However, I am trying to optimize the LP as much as possible. I am posting to ask how other professionals deal with this or what is the workflow they follow. Thanks a lot for your time

Neox

That is a hard no. Wtf

Make sure your lowpoly matches your highpoly shapes, be aware what normalmaps can and can not handle and use a cage or proper ray distances for baking.

No need to fuck up your highpoly for baking.

pior

You could perhaps charge back this "mentor" 100USD per hour wasted because of this nonsensical advice  

gnoop

My guess  his advice goes to cage model  rather than low poly itself.   Or it may work  when you bake  world space normal map  to be re-baked into tangent space in a separate pass . 

 Especially when you use  some  3d party renderer that is not directly supported  in package  bake tool.    For world space normals  the actual shape of low poly is irrelevant.  Only the ray direction and ray starting  point  . When  you displace vertexes alone vertex /split /whatever  they are called in your 3d package normals.

It's useful to bake some  complex things with lots of  procedural scattered geometry  on top of general surface .  Some big things  like  landscape  etc.     That said  the thing in your  screen doesn't need hi poly model at all  and could be all done by rounding edges shaders  that automatically do rounding on hard edges .

It's not that nonsensical advise actually.      I bake things in Reality Capture that way for example.   Well, did  it a while ago at least.  Maybe they fixed their  baker  already but a while ago it would give you some weird/negative  results when your low poly goes under  hi-ress surface .

Eric Chadwick

These are worth a read

• Of Bit Depths, Banding and Normal Maps
• Skew you buddy! Making sense of skewed normal map details.
• Understanding averaged normals and ray projection/Who put waviness in my normal map?

Neox

gnoop said:

My guess  his advice goes to cage model  rather than low poly itself. 

But the instructions where about adjusting the highpoly before baking. Not changing the low poly.

Joao Sapiro

yeah that is one of the most retarded things i have read coming from a mentor to inflate the HP pre-bake...

gnoop

Neox said:

gnoop said:

My guess  his advice goes to cage model  rather than low poly itself. 

But the instructions where about adjusting the highpoly before baking. Not changing the low poly.

  I missread the OP.     I think he meant  inflating  low poly.    It's what I meant.     Doing it to Hipoly is indeed  weird.

Inflating  low poly along vertex  normals  to bake world space normal map on the other hand  is where  low poly itself become the cage .   That way it works in many renderers that are not supported  by regular texture bakers .   For  example  when you need to bake something of 800mil  where 2/3 of that is just instances scattered on displaced surface.   You can't  use such thing as unique  HR mesh   in substance Painter .   But with general  renderer   you can  bake whatever gazillion of polies  you have , do custom  AOVs etc   if such render supports special baking camera  and a few extra ray options.        Octane for example.    Clarisse FX  could bake  billions of instances  and super hires zbrush objects into low poly landscapes using same approach  .  it's pitty she died.

Neox

Just to be clear.

Nobody, ever, needed to inflate a highpoly to match it better to the lowpoly. I have never seen such a stunt in the past 25 years, 20 years of which we are using normalmaps.

If you have tools that need such a thing, get better tools. 

The goal is to capture the highpoly as good as possible in as little geometry as needed. Changing the highpoly pre bake is completely contradictory to that goal.

Dantexagc

Thank you all for your answers! To be honest this gives me more piece of mind because this approach has given me problems with simple models like this one and I cannot even imagine what would do with a more complex mesh that has more details or intricate shapes. With all the info you have given me I would call this discussion solved but if you are in the mood for further discussion keep reading!


 Just to add context, I really think my mentor meant well and I think maybe this technique can be a sort of hack to get some extra smoothness in the borders of the normal map. I mean this as just an opinion and I am very open to be wrong, I see how problematic this can be. I just know for a fact that my mentor has several years of experience in the triple A game industry and I don't believe they would mention this if they didn't see any value to it. Perhaps this can help in specific cases when trying to close gaps between HP and LP when the LP is very optimized, like the border areas in this screenshot:

(LIGHT ORANGE = LP       RED = HP)


Then again, any intricated details will be destroyed when displacing the HP, not great.

I took one of the parts of the model and baked both scenarios, in the first I baked the normal map WITHOUT adding displacement to my HP and in the second case I baked WITH displaced geometry in the HP. These are captures of the LP + normal map so you can see the the comparison:





For more context:
Bake done in Marmoset
LP is unchanged in any case and UVs for this have pretty low res since it is a prop for a larger scene using 1024px/m as texel density 

If you have any further comments or thoughts I would love to hear them, otherwise, thank your for your time!

pior

Well ... one can very well have "years of experience on treeple A" without having any experience working with others - resulting in odd workflows that would never work in production at a studio.

"Hey, we need to rebake this armor set"
"Sure thing - just remember to inflate the high just a tiny bit, otherwise it won't look good"

...

At the end of the day :

- If the highpoly needs to be rounder, make it rounder.
- There is no need for the gap between the low and the high to be filled, because this distance doesn't affect the normalmap. Only *normal* information gets written, not distance.
- If anything, one thing that can happen is that people without much experience tend to surface-snap their low to the high without giving it any second thoughts, resulting in a low that is skinnier than what it should be especially if the density is medium/coarse - hence a final asset with less beef than the reference high. But that's completely irrelevant here ...

I think it would be best for The Mentor to provide some specific, practical illustrations. After all, perhaps some things got lost in translation.

As far as I am concerned the *only* case where I could see such an edit of the high being worth it would be that of a very poorly modeled high with weak/razor sharp edges in need of being fixed as fast as possible.

Neox

If you want your edges to be smoother, make em smoother in the high poly phase.

This is an art direction, maybe tech art decision to be made early on and then everyone can follow this ruleset for consistent bakes/textures.

gnoop

I don't use hi poly mesh for such things at all.  All the edge rounding could be  baked by just using rounded corner / bevel shader or whatever your 3dpackage or renderer call it  and this tiny  "detail" could be just painted  in substance painter, 3dcoat ,  or any texture soft.      It's a bit less easy on cylinders crossing each other   but boxy shapes  doesn't need any  hi res  at all for sure.     In fact  you could do it all in Painter/ Designer even without special shader  using ancient blur  UV islands borders  technique  on boxes.

Just keep your Low poly without shading gradients to compensate using simple combination of hard split edges and custom vertex normals    and you wouldn't need it even with cylinders .    Saves you lots of time because you wouldn't need to do any sub-D modelling  for hi res.

Neox

Thats a whole different topic tho

And I'd say this is the sorta question tech art and art direction should defind before anyone gets to do actual content

pixelb

Inflating your highpoly sounds like maybe a misinterpretation of a technique I know from Heroes of the Storm.

Sculpts were given a slight inflate before the maps were baked to create thicker edge highlights. It ah, works a lot better with a dynameshed sculpt than it does with a hardsurface model.

EarthQuake

Generally, you want the shapes in the high and low poly to match as closely as possible - this means that often the low poly will either be exactly the same size or both slightly bigger and smaller than the high. For something like a cylinder, the low should intersect with the high in a vacillating pattern. Here's an example (white lines are the high and black are the low):



Mismatches between the size of the high and low can cause problems when baking, usually resulting in the baked content ending up on the wrong section of the low, which means the normal map lighting and the shadowing systems won't be aligned. 

When you sub-divide the high poly mesh, curved areas will tend to shrink a bit. So this may be something you need for with the low poly. If you copy the cage mesh and optimize it for the low without taking into consideration the shrinkage, you can get mismatches. But inflating the high poly universally is a poor way to solve this issue and will introduce other problems. As you note - "Many details are pinched or completely deformed when attempting to displace the geometry of the high." - this is a very poor practice and should be avoided.

Neither the low nor high should be inflated or enlarged to cover the other for baking purposes, that's what the cage is for.

EarthQuake

Neox said:

If you want your edges to be smoother, make em smoother in the high poly phase.

This is an art direction, maybe tech art decision to be made early on and then everyone can follow this ruleset for consistent bakes/textures.

 Yes, if you want your edges to be more round or to have thicker bevels for readability or style, this should simply be a part of how the high poly meshes are constructed, not a step you add at the end of the process.

pior

Knowing that this may have originated from HoTS gives the needed context IMHO - and I think this perfectly illustrates how something that may be relevant for a very specifc (and somewhat hacky) edge case is not to be taken as gospel, and can end up costing time and ressources.

I personally didn't take part in the development of HoTS but I can certainly imagine a rather unfortunate set of circumstances in which things need to "look cool" for review at the highpoly stage, yet ending up looking too weak once baked down and viewed at RTS distance. Of course such a situation is absurd since as pointed above by EQ this stuff should *obviously* be taken into account earlier up the pipe - but if the highpoly sculpts are considered throwaway assets and if everyone involved is fine with wasting time on such "double takes", heck, anything goes.

I still do believe that this is inherently absurd though, and a sign of a poorly optimised art pipeline and review process. 

Dantexagc

EarthQuake said:

Neox said:

If you want your edges to be smoother, make em smoother in the high poly phase.

This is an art direction, maybe tech art decision to be made early on and then everyone can follow this ruleset for consistent bakes/textures.

 Yes, if you want your edges to be more round or to have thicker bevels for readability or style, this should simply be a part of how the high poly meshes are constructed, not a step you add at the end of the process.

Hi @EarthQuake. I completely understand and agree with the points both you and @Neox are saying but you made a good point explaining the "shrinkage" that happens when subdividing a mesh, the low poly ends up being slightly larger, etc. I understand that if we want to make the edges of the model rounder we can do that in the high poly. However, if we do that, the mismatch due to the shrinkage you mentioned between the high poly and low poly will be even more severe. Does the cage eliminates this issue completely then?

P.S: thanks again to everyone that commented on this post!

Raphael_Bouch

Dantexagc said:

EarthQuake said:

Neox said:

If you want your edges to be smoother, make em smoother in the high poly phase.

This is an art direction, maybe tech art decision to be made early on and then everyone can follow this ruleset for consistent bakes/textures.

 Yes, if you want your edges to be more round or to have thicker bevels for readability or style, this should simply be a part of how the high poly meshes are constructed, not a step you add at the end of the process.

Hi @EarthQuake. I completely understand and agree with the points both you and @Neox are saying but you made a good point explaining the "shrinkage" that happens when subdividing a mesh, the low poly ends up being slightly larger, etc. I understand that if we want to make the edges of the model rounder we can do that in the high poly. However, if we do that, the mismatch due to the shrinkage you mentioned between the high poly and low poly will be even more severe. Does the cage eliminates this issue completely then?

P.S: thanks again to everyone that commented on this post!

Yes.