That would depend on the color of your diffuse. If your diffuse is black then yes. The diffuse should operate as a clamp in such situations. Any color on the cube map that isn't brighter than the diffuse, shouldn't show. I.E. reflections should never darken anything further than the diffuse color. thats why chrome =…
You can also use the diffuse specular convolution in HDR shop. If you are building the panos yourself from photo source this can be a good way to go (especially since cubemap gen will load up the cross output from HDR shop) and it also seems to be the best way to blur an HDR cubemap.