It's the way computers convert something into an integer by default, 0.9999999 converted to an integer is still 0. I already mentioned it above but will do so again. With unsigned normal maps there is no 90° tangent value, only 90.35294117647059° which is 128 and 89.64705882352941° which is 127, smack dab in the middle of…
Looking at it again, I think a larger scene may help keep the surface planar.. but I would agree with your earlier consensus, that the problem is with the normal maps being unsigned and the break between 127 and 128 being basically exactly at 90° I think max gets around this by dithering and I have no idea what Xnormal…
I get what your saying that in 8bit color representation there is no true 90°, and 90° is directly in between the distinct 8 bit values thus even the smallest deviation would have the value round down. Pardon my ignorance, wouldn't it be the other way around ie: signed is -127 to 127 (255 values) versus unsigned 0 to 255…