Xnormal and the UVS are separated, just like if they are different pieces, with hard edge... I'm starting to wondering if that hard edge is the problem
You might have to tweak both, high and low. But it's a bit hard to get the full context from the images shared. With unwanted projections from adjacent geometry, I'd look into Painters "match by name".
Nice Progress! Looking for more! :) you should take a look at the Hard Surface Challenge here on Polycount, there is many awesome hard surface concept! http://polycount.com/discussion/132146/week-40-the-weekly-hard-surface-challenge/p1
If there's a hard edge on a model, you need to split the UV map at the hard edge, or you need will get a seam. You could also get rid of the hard edge. Both options work, and they both have various pros and cons.
From my experience it's same in every ray-tracer I tried. And always been that way. Mostly coming from sky/environment light . Guess it's why they do small quads with displacement + actual bevels instead of hard edges outside of gamedev. Hard edges is a gamey thing. Actually I have seen a subtle hard edge reveling with…