This was the issue! Clearly it's something to do with the HDRI. Maybe it's refracting darker values from the IBL or something, but I changed the image to a brighter one and got very different results. It also really shows off the details much better. Thank you both for the replies!