Indeed, It's preferred to have a much "pale" color map, and let the normal and spec do the rest. As it is now, the color map has too much noise, plus, you should avoid having pure black on the color map, It always looks weird on the final piece, imho.