I thought this worked ok. Copy the blue channel of your normal map on to itself in photoshop, go to 'fade' and set to multiply .
do this a few times and you get a nice detailed occlusion map.
Obviously its not a 'proper' amb occ map but is used for overlaying on to your diffuse texture.
Replies
A faster way is to just copy the blue channel and do an Auto Levels on it, then gaussian blur it by about 0.5 or something. Less effort than doing the fade/multiply thing a few times.
I also like the technique of using Stylize -> Find Edges on the full RGB normal map, desaturate the result, then Gaussian Blur it - gives a fairly decent base of fake ambient occlusion.
Jogshy's XNormal program has a cavity map feature to extract this sort of detail from a normal map, I think it's a bit more accurate than just copying the blue channel.
Why bother faking it when you 5-10 extra minutes of baking will give you a superior ambocc map?
Of course if you have the highpoly source you should be baking ambient occlusion along with your normals, it makes no sense not to do that.
However I can think of several cases where it's necessary to use one of these methods, I've done it a few times myself recently.
Another trick I actually use a lot is to just do a drop shadow in the layer styles. You can end up isolating the shadow pixels only onto it's own layer through a menu, and then just throw it onto the multiply layer option, and there you go. Of course the settings need to have the distance to 0, and you should tweak the other two sliders (depending on the depth and "bold" look you want the AO to have).
Nothing you do with a normal map will give you an AO map. A normal map records the surface slope, flat recessed areas that would have dark occlusion shadows would be perfectly white on a normal map. Using any combination of R, G, or B channels cannot change this, and it is something to keep in mind.
That said, I still use the normal map to back in lighting. I normally do some combination of what most have done but let me point out a couple things:
First, the RG and B issue. The R and G channels hold what is sort of like a directional light; if we picture a hemisphere, one half will be light and the opposite side will be dark. Using this knowledge, we can create a deeper and nicer B channel if we are not happy with the existing one. We can run a Curves on the R and G channel, that goes from 0,0 (black is black), .5,1 (mid-grey is white), 1,0 (white is black). This will make both slopes darker and will remove any directionality. Do this to your R and G channels, increase the contrast of your B channel a bit, and desaturate the whole thing. You'll just essentially get a more detailed version of your B channel.
I also take the G channel (or whatever controls the vertical lighting in your case), and bake in some top-lighting into the texture. So I have some highlights and shadows baked in from the normal map's G channel (or inverted G channel, depending). This is especially helpful in outdoor daytime scenes.
Nothing takes the place of a proper AO texture, and it would be worthwhile to hand-paint your adjusted B channel from your normal map to turn it into a better AO approximation. Using any of the above techniques will ALWAYS yield something substantially different than a true AO map. This is especially apparent in areas where the low-poly normals are pretty bent, such as along a right angle. You will get lots of issues using a desaturated normal map.
To clarify, with the normal map shading, after I overlay it, I set it to a really low opacity, so that it has some definition, but not overpowering (so that the normal map itself can still do it's work). It's just a way to quick start some depth into your diffuse. Then I will always do my drop shadow method for Ambient Occlusion as well (this is of course if I do not bake out a high res normal map, in which case I would just bake out my AO as well).
Also, with the drop shadow method, you only use it for when the texture piece should be extruding outward. If it is recessed, then you would use inner shadow, using the same/similar settings.
Thanks to Vahl for turning me on to this program.
Oh yeah, and it works great in tangent with Xnormal, since you can ask Xnormal to bake the texture from the high poly (AO bake from Faogen on autogenerated UVs) down to the UVs of the low poly. Saves a crapload of time.
I did actually say its not a subsitute for AO, just a quick method of laying some detail in to my diffuse map
Rendering AO in max doesn't always do it for me, there is a lot of detail missing. I tend to use a combo of techniques, including overlaying the blue channel of an object space map, combined with the green channel of a tangent space map+the cavity map from zbrush
Anyway,some good suggestions here.
Thanks again!