problem with normal map bake

emogi3d

It is a cloth that was Decimated in Zbrush. It seems that the artifact is actually following the edges of the polys,  I try pretty much everything but I can't quite handle it! I thought it is something from the cage and try to make the mesh more densed it got better but it didn't fix everything. I'll be grateful if someone can help me with that, 10x in advance gise.

EarthQuake

Usually this kind of thing will happen when you have hard edges but no splits in the UVs. What does your low poly mesh look like without hte normal map applied?

emogi3d

Thank you, sir, for your time I will fix the mesh or the UVs as you suggested! This is how it looks, and in the second picture is the issue that bugs me right now. It happened when I rise the sampling to 16 and the resolution to 4K and the softness, unfortunately, Marmoset crashes 9 times out of 10 when I try to experimentally bake it again because the HD mesh is 16milion polys so I don't know where I'm wrong. This side towards us and the opposite one are the weird-looking ones, the two side walls of the building are looking normal.

EarthQuake

1. As to the first issue, it looks like your mesh normals on the low poly likely need to be smoothed.

2. As to crashing etc, you may be running out of video memory. What GPU do you have and how much VRAM does it have? One thing you can do is remove any unused meshes from your scene, and make sure any other art or VRAM heavy applications are closed when you go to bake. You can try decimating the high poly mesh to reduce the geometry count as well.

3. It looks like the baked normal map may have gotten applied to the high poly material, this can cause recursive bake errors as the baker tries to essentially bake the normal map back onto itself. Make sure that the high and low poly materials have unique names, and that you've cleared the normal map from the high poly material and this should go away.

emogi3d

Do you mean like smoothing groups? The video card is GTX 780M with 4GB of memory. But when I'm monitoring the PC resources I think it crashes when the physical memory is full, which is 16GB. Thank you for your excellent advice sir I'll put it to use first thing in the morning and I will post the result after that. As you may have guessed  I'm a bit new to all of this game stuff, so I really appreciate your time, effort and help!

EarthQuake

I'm happy to help!

Yeah smoothing as in smoothing groups. From the screenshot it looks like the smoothing is set somewhat arbitrarily. If that is Modo try setting the smoothing angle to 179 and exporting the low again.