Insanely frustrating baking issues with height in marmoset
This is probably going to be the last place I post this as I'm absolutely ready to just give up and get rid of Marmoset.
I'm brand new to the software but there is absolutely no proper straightforward answer I can find for my issue.
Whenever I try to bake the height map of my sculpt in marmoset it will generate a regular height map but with an outline of the low poly wireframe stuck on it.
It's more subtle in this image but you can still see it.
But this keeps causing huge issues when I actually try to displace the mesh and I can't find any fix for this.
I can't even find a single actual guide that specifically covers height map baking in Marmoset.
Please someone help me because the only responses I have been getting are "oh yeah thats supposed to be there" and to just adjust some sliders that end up fixing nothing.
I'm so desperate to fix this at this point
Replies
I'm not familiar with marmoset specifically but in any other software that would be the correct output for a simple height bake. it's giving you the distance between the two meshes - which is what you need when you displace it later during rendering.
Given a convex surface edges on the low poly mesh will be closer to the high poly mesh than the face they surround and you'll get dark marks at the edges
if your low poly mesh is closer in shape to the high poly you'll get less of this effect happening.
if it's causing unexpected results when displacing then you most likely have a scaling or color-space issue - I don't know enough about marmoset to offer any suggestions there but if you're using it outside of marmoset you'll want to make sure you're working with linear values and that no gamma adjustment or color space transforms have been applied to the image
This is generally expected and something you will see with all baking applications.
I just wrote a response to a similar issue on Reddit (I am not sure if you created this post or if it was someone else) but I'll paste it here as well.
---
A baked height map will record the difference between the high and low polygon meshes. Assuming the topology of the low poly mesh is coarser than the high poly, these faceted details are not only expected but essential to displace the low poly mesh to match the shape of the high poly. This style of height map is important to use if you plan to subdivide your mesh without smoothing - in Toolbag this would be the Regular sub-d mode.
If you intend to subdivide your low poly mesh with smoothing before applying displacement mapping, it's a good practice to subdivide it to the same level while baking the height map as well. For instance, if you intend to use 2 levels of Catmull-Clark sub-d for the final asset, use the same settings for the bake.
As mentioned, it looks like you need to increase the Inner and Outer distance settings in the height options. The values you will need to use here depend on a few things. The scale of your mode, how your scene units are configured, and how large the variance is between the high and poly meshes. Make sure to keep the values proportional, ie: -1. and 1. or -5.0 and 5.0, and avoid using values like -1.0 and 10.0 as this will shift the zero or no displacement point away from neutral gray (0.5).
Here's a simple example, from left to right we have:
And this is the baked height map. Note now the vertices in the low poly mesh create darker values, this is because they extend past the surface of the mesh. When the result is displaced, these areas need to be pulled in to create a smooth surface that matches the original mesh.