Currently dealing with maya and normal baking so im in the zone.
Did you harden the edges on those 2 square bits in the middle of the door i would try and soften the edge that is in the crevice of the square bits and see if that makes any difference ( this may also require tweaking the cage (envelope) distance so get right but you may end up losing the detail on other bits so test that out first.
Other thing is try and add additional edge loop around in the crevice so you dont have sharp edge in there, the rays are hitting wrong parts of hi poly and that is the error you are seeing. So try and soften the transition angle in there and bake again, i believe that should solve the problem. Oh and i dont think you need that tooth in the upper area that you've marked you can easily collapse that edge there and get good bake seeing as the whole low poly is very low in detail.
Sry didn't get notifications on this thread and just now remembered.
So im guessing in the in the uv map those 4 square islands just above the 4 triangles are the square bits with black lines right? If so then they need to be unwrapped differently i've done a simple test and indeed this can be unwrapped without lines but i dont think the low poly does the best job conforming to the low poly. Anyways here is the image.
As you can see there are no black lines those were there due to aliasing the ultra sharp corner (more than 90 degrees) I've also added additional loop in the crevice to get the rays to project the normal map without hitting extreme angles and stretching/compressing the normals too much. Adds to the polygon count but fixes the issue. Every edge is softened hence the gradients in normal map now this can be avoided by splitting sides and hardening edges, if you want to go that route i still don't fully understand the drawbacks/benefits of that so i wont add anything on that subject there are people a lot more experienced than me around here (make sure to go through that thread about hard edges mentioned above, tons of ultra useful stuff in there) Yeah and also one more thing also mentioned above i need to second is fix the uvs and straighten the edges that should be straight wherever possible, straight polygons with crooked uvs can also lead to bad bakes and weird normals
Replies
Did you harden the edges on those 2 square bits in the middle of the door i would try and soften the edge that is in the crevice of the square bits and see if that makes any difference ( this may also require tweaking the cage (envelope) distance so get right but you may end up losing the detail on other bits so test that out first.
Other thing is try and add additional edge loop around in the crevice so you dont have sharp edge in there, the rays are hitting wrong parts of hi poly and that is the error you are seeing. So try and soften the transition angle in there and bake again, i believe that should solve the problem.
Oh and i dont think you need that tooth in the upper area that you've marked you can easily collapse that edge there and get good bake seeing as the whole low poly is very low in detail.
Hope it helps
soft/hard edge.
UV cut
take a look at these videos - https://www.youtube.com/watch?v=kGszEIT4Kww
https://www.youtube.com/watch?v=D16uE0IBuLo
So im guessing in the in the uv map those 4 square islands just above the 4 triangles are the square bits with black lines right?
If so then they need to be unwrapped differently i've done a simple test and indeed this can be unwrapped without lines but i dont think the low poly does the best job conforming to the low poly.
Anyways here is the image.
As you can see there are no black lines those were there due to aliasing the ultra sharp corner (more than 90 degrees)
I've also added additional loop in the crevice to get the rays to project the normal map without hitting extreme angles and stretching/compressing the normals too much. Adds to the polygon count but fixes the issue.
Every edge is softened hence the gradients in normal map now this can be avoided by splitting sides and hardening edges, if you want to go that route i still don't fully understand the drawbacks/benefits of that so i wont add anything on that subject there are people a lot more experienced than me around here (make sure to go through that thread about hard edges mentioned above, tons of ultra useful stuff in there)
Yeah and also one more thing also mentioned above i need to second is fix the uvs and straighten the edges that should be straight wherever possible, straight polygons with crooked uvs can also lead to bad bakes and weird normals