Yeah AO is not a real world thing. AO is a rendering hack we use because realistic lighting is far too expensive to render in real time. The ideal use for AO is to be used in a shader that multiplies on the ambient light as Farfarer notes. In Toolbag 3, the AO slot multiplies on the ambient (image based lighting) diffuse…
You don't multiply the AO against the albedo, because then it would affect direct lights as well as ambient lights (and would make the ambient light artificially dark because you're presumably multiplying the ambient light by the AO as well, before multiplying it with the diffuse light). AO should only affect ambient…