You can't do shader math in Photoshop, as far as far as I know. So there is no equivalent. You could make two layers, sand below, stones above, and put a stones height map in a layer mask for the stones layer, then edit that mask. But you can't use math to edit the transitions. The equivalent would be to use Pattern layers…