I'm guessing there wasn't ever a highpoly, you really made the normalmap in the worst possible way, and you could be better without it. Try involving a highpoly in your process, or keep the normalmap flatter than what it is now, if you didn't bake anything, use it more like a flatter but sharper bumpmap.