Normal map issue

Rozzy85

I've had this problem for a long while and it's driving me crazy. They are always 1x1 pixel big no matter the output size of the map. I don't know if they're ray misses or the sampling for that pixel misses the model and hits what's behind it. It's also quite random. Sometimes it happens and sometimes it doesn't which makes it much more annoying.