You're looking for Cavity, AO is just that, Ambient Occlusion and ZB does a pretty sub-par work on that too.
However, it looks like you're trying to use the texture from the render.
I suggest masking the AO from the Mask panel (and play with the strength from there, in the masking panel), paint the black from there and render out your AO as a standard diffuse.
you may need to do some composition of multiple shaders to get what you want straight from zbrush. ive seen some people just use the framer04 matcap and overlay that since it does the same sort of outlining.
if this is a tool and not 2.5D canvas, the best solution is export OBJ and use xNormal to bake in a square unwarped plane.
I know you want to stay in zbrush to simplify, i also like that...
but i found that export obj and have everything baked with xnormal (including polypaint) ends up being much faster and with better results.
we have to do a lot of compositions with zbrush passes and in some cases xnormal will give you a finished map without having to edit it.
try and see if the process is faster and with better quality
So I'm trying to condense my workflow almost entirely into ZBrush because it's a lot faster, but I'm a little stuck when it comes to baking the ambient occlusion. Anyways, any idea why I'm not getting all the fine detail in my AO map? My only temporary solution is to overlay an AO map generated by nDo2 from the normal map which has the details, but oddly it ignores the hard edges from the normal map.
Here's my setup and the results:
Don't worry about the tiling, I'm going to crop it out correctly later because my computer is so touchy with the crazy amount of polygons in the document.
I wanted to reply to this yesterday, but couldn't because of some goofiness with my phone.
Upon close inspection of your AO Settings in ZBrush, they don't look right to me, and that may be why you had a bad render in the first place.
Try bumping up your rays to at least 600, and your Angle to 150 ~ 160. That may help you tremendously.
Replies
However, it looks like you're trying to use the texture from the render.
I suggest masking the AO from the Mask panel (and play with the strength from there, in the masking panel), paint the black from there and render out your AO as a standard diffuse.
I know you want to stay in zbrush to simplify, i also like that...
but i found that export obj and have everything baked with xnormal (including polypaint) ends up being much faster and with better results.
we have to do a lot of compositions with zbrush passes and in some cases xnormal will give you a finished map without having to edit it.
try and see if the process is faster and with better quality
I wanted to reply to this yesterday, but couldn't because of some goofiness with my phone.
Upon close inspection of your AO Settings in ZBrush, they don't look right to me, and that may be why you had a bad render in the first place.
Try bumping up your rays to at least 600, and your Angle to 150 ~ 160. That may help you tremendously.