Is Height Map supposed to do this?

Jagulars

Above it looks like with height channel disabled. Nothing unusual.

Below it looks like when height channel is enabled (without displacement or tesselation):

This is the height map, which doesn't seem to suggest anything that would cause the changed visuals:

There is no edge of an UV island visible here either. Sometimes the height channel problems are seen between polygons, sometimes not:

Why do the visuals change here when the height channel is enabled? I thought Height Map does nothing if displacement/tesselation is not enabled.

poopipe

It falls back to parallax mapping if displacement is disabled I think