Annoying AO error

3Dyan

Hi there.

At times when I'm baking AO, areas which have intersecting faces are not being occluded fully. Example:



I'm baking with 3DSMax, and the areas of the mesh that are hidden act as if they are not occluded at all (expected) however they bleed out a bit and this is what's causing these white edges.

What is the best way to overcome this?

Mark Dygert

1) What are you baking With?

MentalRay AO, Scanline shadowmap, Scanline lightmap, scanline complete map?

2) Does it look different if you render it?


3) Are you baking with enough padding?

If your viewport resolution set to the default or lower and so it's displaying a lower level mip-map, allowing the white space to bleed in. The same thing will happen in most game engines and you should bake with more padding.

4) How is your viewport setup? Direct3D, OpenGL, Nitrous?

If you're using Direct3D make sure you follow step 2 of this tutorial.
If you're using nitrous use the Nitrous Texture Tweaker to increase the resolution in the viewport.

5) What does that section of the AO texture look like?

3Dyan

This happens in MR, scanline lightmap and xnormal.
This isn't a padding issue, as these meshes intersect and continue a little into each other.
This is visible in every software I use (max, toolbag, udk, etc...).

Here is what the map looks like:

It is as if the occlusion ends a pixel or two prematurely.

Bryan Cavett

Try making your material 2-sided on the object and then baking the ao/lightmap. I cant remember if that trick is for mental ray or lightracer though...

Valandar

In Max, have you tried assigning a 2-sided material to the objects before baking? I used to have that issue before I did that, and really couldn't for the life of me explain why the 2-sided material works, but it did, in both scanline and Mental ray.

Edit: Ninja'd by Bryan!!!

3Dyan

I tried that. The result is the same, only now I'm getting abrupt black edges instead of white.
Not pretty.

Bryan Cavett

I think I've seen this happen with someone at work. They were trying to project matIDs to matching matIDs and mental ray didn't anti-alias the intersection point or changing matID edge (if that makes sense). I don't think we ever found a solution.

Noors

I think if you boost the sampling quality you'll get more AA. But if the sampled texel is "inside" the mesh, i'm affraid you'll alway get a bit of bleeding on intersecting geo. We'll need some kind of offset. What came to my sick mind is to put a little negative push modifier on top of the model for baking, so the dark area start 1 or 2 pixel further.
test :

3Dyan

That's an interesting idea, I just might try that on my next asset.
Good thinking.