No, it is perfectly fine to used colour specular maps, a few people when starting think that they must be grey scale, but they can contain as much colour as needed, that is why a lot of people, when working on their diffuse, also use it as a temporary specular map.