Yes, the colours have to be solid for the masking to work as it should. So padding or diffusion on a single island like this leads to errors as the texture bleeds onto itself.
It looks a bit thin to me, especially on the grip. Material on the low poly is off, looks like plastic. There's also a really ugly smoothing error on the slide.