displacement map (height) baking problem in Toolbag Marmoset

RitaRo

Hi) I am having a problem baking the height map. Polygons are visible on it.
There is a screenshot of the problem here -
https://drive.google.com/file/d/1Bwr85rRxLB_fFW4zu3mAviw4MK1MFcC-/view?usp=sharing

Obscura

That is actually correct and needed. See the image below:

A displacement map / height map stores the distance between the lowpoly and the highpoly mesh surface. See how the lowpoly verts are alighed to the highpoly details but there are humps and valleys of the highpoly clipping through the lowpoly. This looks like as the edges of the lowpoly are visible in the baked map. But when it comes to actually displace the mesh, those vers needs to stay in place and the in betweens needs to shift. There are 2 types of tessellations though, one requires this type of a displacement map, and the another is what you are referring to. PN triangles pre-smooths the lowpoly mesh so it needs a smooth displacement map, but without this, those visible edges needs to be there. You have another issue though, that your value ranges are all over the place (clip white, clip black). This can be ok if you use high bitrate file format such as hdr or exr but in low range, you get no info on those pixels as the values are clipped.

Both type of displacement maps should work in offline rendering too. Its just the matter of pre smoothing the mesh or not. So in case of non real time application, just apply flat tessellation to the mesh before applying the displacement. Most game engines also supports both types.