Errors baking in Marmoset - Chain mail

Iziaky01

Hello to eveyone!
I keep having some strange error in my maps when I bake in Marmoset (or even other baking softwares). Want to bake a Alpha Mask of a chain mail made in Zbrush with Nanomesh, on a simple plane. Seams easy and it should be, but it continues to add some other "geometry" that isn't in my highpoly.
Here the resuts:

This the setup in Marmoset:

Already tried:
- edit the cage
- changing the Lowpoly or Highpoly
- edit the Lowpoly Uvs

- making a new Marmoset File
- checking the Highpoly in another software other than Zbrush
Any advice? How is it possibile?

if you wanna try for yourself, here the files: https://we.tl/t-W8LnNqpQo8

rollin

Your dog is biting into its own tail. 
Check the ray distance

Alex_J

I don't believe you can adjust ray distance directly in toolbag. I had to search it but I see somebody with same issue and Joe Wilson didn't give a straight answer, so that may be a feature toolbag doesn't have yet.

However, getting good results is still easy. Just reverse the normals on your object. Then you can reverse them back after baking. I can't explain precisely why this works, it is just a thing I tried intuitively. I thought checking "ignore backfaces" might work but I guess I misunderstand what that means.

Also, a tip about chainmail - just grab the free chainmail smart material from substance share. It looks just as good and gives you fast non-destructive control

rollin

Alex Javor said:

(...)

However, getting good results is still easy. Just reverse the normals on your object. Then you can reverse them back after baking. I can't explain precisely why this works, it is just a thing I tried intuitively. I thought checking "ignore backfaces" might work but I guess I misunderstand what that means.

(...)

Some baker / options have "until first hit" which would apply the  (hmm, pls. correct me if I mixed them up) rear ray distance but kind of ignores the frontal ray distance. The baker just shoots the ray until anything was hit.
If you invert the normals you bake 'inside out' and therefor the baker will not accidentally hit e.g. a chain on the other side. But you could just as easily set both ray distance correctly and get a clean bake.

Obscura

Using an extra mesh in the highpoly would also fix this. You could have a mesh that takes up the volume, and put it inside. Leave some gap between the mail and the placeholder mesh.

The inverted normal works because imagine a ray start, and a ray end position. The start could be on the surface of a cage mesh, and the end is somewhere inside the meshes. Connect it with a line. Thats the trace. You can go either directions on it. The normal of the lowpoly will change this direction. Polygons intersecting the line will be sorted, and the closest one (to the ray origin) will be returned as a hit on the highpoly. Flipping the vertex normals will also change if the cage is inside the mesh or outside.

Anyways, I would also suggest just using a tiling chain texture instead of baking, as your highpoly one already has squishing so it doesn't look any better than a tiling one.

Iziaky01

Alex Javor said:

I don't believe you can adjust ray distance directly in toolbag. I had to search it but I see somebody with same issue and Joe Wilson didn't give a straight answer, so that may be a feature toolbag doesn't have yet.

However, getting good results is still easy. Just reverse the normals on your object. Then you can reverse them back after baking. I can't explain precisely why this works, it is just a thing I tried intuitively. I thought checking "ignore backfaces" might work but I guess I misunderstand what that means.

Also, a tip about chainmail - just grab the free chainmail smart material from substance share. It looks just as good and gives you fast non-destructive control

Thanks man! you're right, it worked inverting the normals of the lowpoly. Like you said, Marmoset's cage can only handle the Frontal rays (the one that goes outside), with the slider Max Offset. But it ignores the length of the Rear rays (the one that goes inside), and that was the cause for all the mess during baking.
In Substance Painter instead you can adjust both sides of the cage, hope that Marmoset will implement this feature.
Here's the result with the correct lowpoly (the one posted was a test version).

EarthQuake

The root issue here is that there are holes in the high poly mesh, so those rays fail to hit anything and keep searching. Some of them end up hitting elements on the other side of the chain mail.

To work around this you can create a simple mesh (a copy of the low poly would work well) that represents the shape of the chain mail, and shrink it slightly behind the chain mail and include it in the high poly. This way there will be something to hit. Give each area a unique material and bake a material ID map to create the alpha / opacity mask. Or set the inner mesh's material to 0% opaque and bake an opacity map as well.