That "diffuse texture" really depends on the end application of the asset. If you are going to present it with unlit shader, then you don't even need to delight it, and then its probably good as it is. If you want to use lit shading, you are better removing the diffuse and specular lighting from the scanned texture, or try…
Oh yeah that should work. I forgot about that. You'd add it to the diffuse slot of the hp material and you'd bake diffuse map. Now I tried if simply displaying the vertex colors with the vertex paint modifier would also work with the baking but it does not.