@Daustin_3D Nice mesh ! You could try to setup a mesh but, as Stef said, overlapping / layered objects are always a problem. Perhaps could be a better idea to explode the mesh and bake each part ( the skull , the wall, etc ) separately..
When baking in xNormal, the low poly tries to pick up the high poly detail around it. That means that the low poly leather will pick up detail from the high poly skull because they're close together. This is likely because your model is made of separate components baked at the same time. You may have to separate the high…