I moved over to a full 16bit workflow a while back and only use 8bit images at the very end of a pipeline if I must. It's the best way to preserve information and allows for a much larger and less destructive tweakable range in many cases too. Things like levels adjustments are much less destructive in 16bit.
It's because you are using an 8bit map. TGA (or any 8bit format) cannot store enough information to reproduce a smooth transition because it can only have 256 levels between black and white. You need to be using at least a 16bit image, such as tiff, png or even OpenEXR or raw to get anything decent when using heightmaps.