I would guess you need to look into softening your edges, and then going through and hardening edges that have UV seams (as a catch all, though in general you'll only need to harden an edge at a UV seam that is around 90 degrees or more/ a dramatic plane change as it were).
Secondly, I would look into adding some padding between your UV shells, while looking into a way to organize them a bit better. It's certainly not necessary, but taking some of your 'curving' long strip UV shells, and straightening them, will do a lot to help your organization, and I think might make normal gradation a bit more uniform. Feel free to correct me if I'm wrong. In relation to organizing UVs, make sure that no UVs are stacked--unless you are going to be using more then one texture set, then make sure to assign the materials to your meshes so Substance Painter knows that.
Thirdly, you might need to look into a way to break up certain parts of your meshes so that they don't interfere or project onto other unrelated areas. You can do this through 'match by mesh name' in Substance Painter, or you can 'explode' your mesh so that your low and high from different areas aren't conflicting (hard to tell if they are from the screenshots, but would be my guess).
Even though you aren't using Marmoset Toolbag for baking, it has a great explanation of issues you might be running into, so I really recommend going through it: https://www.marmoset.co/posts/toolbag-baking-tutorial/
Good luck! You won't get it entirely right, but keep at it and prepare for your next mesh
Your uvs are your main concern. They need some serious redoing. They are a complete mess. Why spend all that time modeling and give little to no thought for the uvs? Uvs are vital for a clean bake and texturing. For a 4k bake you will need around 16 pixels padding plus set dilation to 16 in SP. This is so the loded texture mips properly as well as to avoid bleeding between shells.
You should also use more than 1 texture sheet. Even at 4k the pixels will be very few on the smaller islands. Trying to cram the entire character plus weapons/armour onto a single sheet is pushing it in a modern workflow.
I agree with you all but.... I just modeled a very simple axe with very clean UVs with very clean topology and the result was THE SAME. I guess there is a problem with my Substance Painter. What other software you recommend for baking?
Musashidan, more than 1 texture sheet? I didn't know this is allowed. So I should UVmap different parts separately?
You should, as mentionned above, smooth your edges to get rid of the polygon lines on your mesh. The second issue is that separate parts of the mesh are too close to each other and overlap during the raytracing. YOu can solve this to an extent by eitehr using a cage or reducing the front distance in the baking settings. For perfect results, you amy want to look into name matching: https://support.allegorithmic.com/documentation/display/SPDOC/Matching+by+name
I'm assuming you're using Blender, I haven't used it but I think this is the equivalent to Maya's 'Soften/ Harden Edge' and Max's 'smoothing groups': https://www.youtube.com/watch?v=sTJ-pryZFQk
You don't need to add more edge loops to get the smooth transition between faces. The edges/ vertex normals being set to face are what's causing the majority of your baking errors to come out grid like. The original points I brought up and other's have echoed will help with additional issues. Take some time to actually digest what's being recommended before writing them off.
Replies
Secondly, I would look into adding some padding between your UV shells, while looking into a way to organize them a bit better. It's certainly not necessary, but taking some of your 'curving' long strip UV shells, and straightening them, will do a lot to help your organization, and I think might make normal gradation a bit more uniform. Feel free to correct me if I'm wrong. In relation to organizing UVs, make sure that no UVs are stacked--unless you are going to be using more then one texture set, then make sure to assign the materials to your meshes so Substance Painter knows that.
Thirdly, you might need to look into a way to break up certain parts of your meshes so that they don't interfere or project onto other unrelated areas. You can do this through 'match by mesh name' in Substance Painter, or you can 'explode' your mesh so that your low and high from different areas aren't conflicting (hard to tell if they are from the screenshots, but would be my guess).
Even though you aren't using Marmoset Toolbag for baking, it has a great explanation of issues you might be running into, so I really recommend going through it: https://www.marmoset.co/posts/toolbag-baking-tutorial/
Good luck! You won't get it entirely right, but keep at it and prepare for your next mesh
You should also use more than 1 texture sheet. Even at 4k the pixels will be very few on the smaller islands. Trying to cram the entire character plus weapons/armour onto a single sheet is pushing it in a modern workflow.
Musashidan, more than 1 texture sheet? I didn't know this is allowed. So I should UVmap different parts separately?
For perfect results, you amy want to look into name matching:
https://support.allegorithmic.com/documentation/display/SPDOC/Matching+by+name
You don't need to add more edge loops to get the smooth transition between faces. The edges/ vertex normals being set to face are what's causing the majority of your baking errors to come out grid like. The original points I brought up and other's have echoed will help with additional issues. Take some time to actually digest what's being recommended before writing them off.
Thank you all for your answers. My problem is solved thanks to your advices, hugs!