Hard surface edge baking - how to improve?

christopheduron

Hi all,

I'm trying to create a pistol and I just finished the model in 3dsmax with its UVs. Next up i wanted to bake the edges of the model so that the hard edges are slightly polished. I'm baking EVERYTHING with a cage (no choice here since i'm baking 90° angles).
 Also every smoothing group is split to its own UV-island:


Let me give you an example (previewing the normal maps in substance painter 2 if you want to know):

Just the plain model without normalmap applied.


Here you can see the normalmap being applied. While the edges worked out nicely (or so i think) i do have some distortions on other parts (view picture below).


Here you can see some of the distortions that are occuring. I believe this is happening because of the turbosmooth modifier i used to create the highpoly model, it changes the curve of that area slightly. 


To fix this curve problems i added a turbosmooth modifier to my lowpoly model. This method fixes distortions if you'd bake several screws or something to a surface. This would fix the screws from being skewed towards the center of the plane.
Anyway, this is the result. As you can see the distortions along the curve is fixed, but in my opinion the edges themselves aren't as smooth anymore.


The left is a picture showing the edges with the turbosmooth modifier and to the right without like in the first pictures. For sure the second picture has better edges right?


Does anyone have an idea to combine the edges of the first method with the curve of the second method? Then my bake would be perfect (i think).
Thanks for the help guys!

jaker3278

On first impressions it looks like your edges on your high poly are way to tight and this is resulting in very little smoothing info getting onto the normal map. The very front looks like its getting no normal info, could have a look at your cage and inspect that part. Could you post pics of your high poly and the normal map. 

christopheduron

yes ofcourse, i'll show you the lp/cage of the one where i fixed the curves (and there the edges aren't great). Aso, it's the point that the edges are very 'tight' if i understand what you mean. I just want a very slight polish of the edge (like it is in the reference).

The lowpoly (topology is exactly the same as the cage, so look below)


High poly


Cage with edged faces on. (the actual lowpoly has a lot less polys ofcourse, but i turbosmoothed it so that those badly rendered curves would be fixed.


Here's the actual low poly model where the normalmap is being applied on.


And finally here's ther normalmap!


If you want, i can add the lp file so you can try it out yourself? Should be easy enough to make your own highpoly since all i did was chamfer all the edges and turbosmooth it a bit.

Thanks for your time though! (i'll have to get some sleep now, so i hopefully see you tomorrow)

musashidan

Yes, the edges of your high are too tight. For edge baking the general rule is to go softer/more round than the edge is realistically. Tighter edge=less pixels=bad mipping and less readability.

For your first problem with the curve just add another edgeloops to the low to better represent the silhouette. 

christopheduron

Okay, i'll try to bake it again with softer edges.

I'll first try to bake it again with a turbosmoother lowpoly so i don't have that curve problem to begin with. If the edges still don't look good i'll see if simply adding more polygons would give a desirable effect.

Is it perhaps possible to combine both normals over eachother and then simply masking out the parts that aren't working properly? Personally i don't think this will work since everything is happening along the same edges.

christopheduron

I made the chamfer edges twice as large and i added only a few extra loops to the curve on my lowpoly model i used to bake. The actual lowpoly being used and displayed hasn't changed at all. As far as i can tell the results look great, but i'd feel more secure if youguys would take a look at the results below and share your opinion. Thanks for the help!

musashidan

Why are you subdividing the lowpoly? By doing that you're quadrupling the polycount and essentially make the bake redundant.

christopheduron

no no, the lowpoly is staying the same. I'm only subdividing it to use in the baking process. By adding a few subdivisions the UV doesn't change at all, but the subdivided lowpoly does resemble the highpoly a lot more in the curved area, thus removing the curve problems i was having at the start.

So don't worry, the polycount in the final model doesn't change a thing!

Kroma!

Your normal map is only going to be accurate on the mesh on which it was baked. If your low poly has 1 smoothing group across a 90 degree corner then your normal map is going to have a strong gradient to make it shade correctly. Once you change the geometry of your mesh in a manner that changes the shading of the low poly, the normal map will be different. If you then apply this map to a different low poly, it's going to be wrong because that map was not baked for that mesh. Take a look at the sticky threads in Tech Talk for a better explanation.

m4dcow

Kroma! said:

Your normal map is only going to be accurate on the mesh on which it was baked. If your low poly has 1 smoothing group across a 90 degree corner then your normal map is going to have a strong gradient to make it shade correctly. Once you change the geometry of your mesh in a manner that changes the shading of the low poly, the normal map will be different. If you then apply this map to a different low poly, it's going to be wrong because that map was not baked for that mesh. Take a look at the sticky threads in Tech Talk for a better explanation.

I assume he's doing the skew mesh method, and once you keep the vertex normals the same it should be fine. (I use transfer attributes in Maya, it's more reliable than eyeballing any changes)

Kroma!

m4dcow said:

Kroma! said:

Your normal map is only going to be accurate on the mesh on which it was baked. If your low poly has 1 smoothing group across a 90 degree corner then your normal map is going to have a strong gradient to make it shade correctly. Once you change the geometry of your mesh in a manner that changes the shading of the low poly, the normal map will be different. If you then apply this map to a different low poly, it's going to be wrong because that map was not baked for that mesh. Take a look at the sticky threads in Tech Talk for a better explanation.

I assume he's doing the skew mesh method, and once you keep the vertex normals the same it should be fine. (I use transfer attributes in Maya, it's more reliable than eyeballing any changes)

Yeah it's keeping the vertex normals the same that I'm trying to get at. As long as the Turbosmooth is set to smooth by smoothing groups the vertex normals won't be affected right?

musashidan

christopheduron said:

no no, the lowpoly is staying the same. I'm only subdividing it to use in the baking process. By adding a few subdivisions the UV doesn't change at all, but the subdivided lowpoly does resemble the highpoly a lot more in the curved area, thus removing the curve problems i was having at the start.

So don't worry, the polycount in the final model doesn't change a thing!

The UVs do change. They will often skew across seams. This has been a well known problem with TS for years that has a few hacks to resolve, none of which fix the issue 100%. Open sub-div was supposed to fix this smooth-uvs issue but isn't 100% either.
Having baked a tangent space NM the RGB data from the normals is encoded in the texture so changing the Geo after the bake will result in incorrect shading. The skewmesh method that @m4dcow mentioned is usually done by baking an object space NM(from the smoothed low-poly in this case) and converting it to a tangent space, which you can then apply back to the unsmoothed low-poly.

christopheduron

Kroma! said:

m4dcow said:

Kroma! said:

Your normal map is only going to be accurate on the mesh on which it was baked. If your low poly has 1 smoothing group across a 90 degree corner then your normal map is going to have a strong gradient to make it shade correctly. Once you change the geometry of your mesh in a manner that changes the shading of the low poly, the normal map will be different. If you then apply this map to a different low poly, it's going to be wrong because that map was not baked for that mesh. Take a look at the sticky threads in Tech Talk for a better explanation.

I assume he's doing the skew mesh method, and once you keep the vertex normals the same it should be fine. (I use transfer attributes in Maya, it's more reliable than eyeballing any changes)

Yeah it's keeping the vertex normals the same that I'm trying to get at. As long as the Turbosmooth is set to smooth by smoothing groups the vertex normals won't be affected right?

I hope so, that's usually what i do when i turbosmooth a lowpoly before using it to bake with.

I'll also check out that sticky to learn a bit more about baking. Thanks for the info guys.

AlecMoody

Kroma! said:

m4dcow said:

Kroma! said:

Your normal map is only going to be accurate on the mesh on which it was baked. If your low poly has 1 smoothing group across a 90 degree corner then your normal map is going to have a strong gradient to make it shade correctly. Once you change the geometry of your mesh in a manner that changes the shading of the low poly, the normal map will be different. If you then apply this map to a different low poly, it's going to be wrong because that map was not baked for that mesh. Take a look at the sticky threads in Tech Talk for a better explanation.

I assume he's doing the skew mesh method, and once you keep the vertex normals the same it should be fine. (I use transfer attributes in Maya, it's more reliable than eyeballing any changes)

Yeah it's keeping the vertex normals the same that I'm trying to get at. As long as the Turbosmooth is set to smooth by smoothing groups the vertex normals won't be affected right?

Your mesh normals change with any change in topology. The correct way to do this is to separate projections and tangent space generation into two processes. Basically, bake an object space map with your tessellated low, then convert to tangent space in handplane or xnormal with your final lowpoly.

That aside, you should be able to get a nice clean bake on this model without doing any sort of skew or proxy mesh system.