The trick with the texture is diffuse light baking, done in for example 3ds max? Or is there some raytracing function in the engine (baking done within the engine, like UDK)?
Yes, the colours have to be solid for the masking to work as it should. So padding or diffusion on a single island like this leads to errors as the texture bleeds onto itself.
Try to tone down the scratches in the diffuse and ramp them up with some contrast in the spec/gloss. I think you will be much happier with the results in the render.