This is sort of a point I'm curious about as well because in my mind the fresnel should handle most cases of specular, and for all intents and purposes, to me 'fresnel' is the specular term. Multiplying a reflected cubemap with the fresnel term (after you've adjusted it's falloff etc) is pretty common for shiny materials,…
The way I've been using it is multiplying the final spec value by the frensel value then adding that to the diffuse * diffuse lighting. The fresnel value is usually extremely low (most non-metal surfaces are between 0.01 and 0.04 - metals can go up to about 0.5). This means that the specular result is strongest at glancing…
Are you multiplying you fresnel by an environment map? Is that environment map linear? If you have a full diffuse (with all it's lighting and intensity set up without the spec factored in) then lerping between your full specular reflectance (environment map) and full diffuse reflectance (ideally including the indirect…