Low poly clips High poly - Render to Tex

cow1787

So, I don't work too much with high poly/normal mapped stuff at work, but this one project I've made a high poly model and a low poly to project for the maps.

Right now I'm having trouble rendering to texture an Ambient Occlusion from the High Poly to the Low Poly.

Here's the problem...
http://www.marklemmons.com/files/stryker_ao_02.jpg

I know the red is the cage clipping, I have no problem with that part, I can fix that... But those black triangle areas on the right (and all up on the left) are where the low poly model is clipping through the high poly model.

I have "Use Cage" checked and "Include Working Model" unchecked, yet it's still 'seeing' it?

Is there a way to hide the low poly model and yet still render from it? Either I figure this out or it's a lot of clean up for me tonight in photoshop

Paul Pepera

Try right clicking on the Low-Poly model and select Object Properties. Under Rendering Control de-select "Cast Shadows" and "Recieve Shadows" ( you can also try to de-select all of them). See if that works.

gamedev

can you post a viewport grab of what you've got? And you're sure you have no overlapping UV's and all UV's you want to bake to are in the 0,1 space? I assume then your normal map came out ok?

Chai

I believe the image you posted isn't small enough

cow1787

Haha, sorry Chai, it was a quick test render because I knew it wasn't going to come out right so I rendered it small. (why wait 20 minutes for a messed up AO?) You can still clearly see the parts that work - and the parts that don't. So in this case, size does not matter.

Eraserhead - I forgot to bring the file home with me to work on, but I'll try all that out tomorrow and post again if it works.

gamedev - I am most definitely sure my UVs are current. The Normal map was decent, just cage issues that I passed off without fixing because we may not be using normal maps anyway (red areas). They were not the same mistakes that are the black areas on the AO though.

Chai

Well it's small enough to hurt my eyes.
I'll assume you use max and mental ray ?

The triangles are usually caused from texture padding around render to texture dialogue - especially around floating geometry.

You can still render from highpoly if you use the "render hidden geometry" in render preferences - however don't use this if you're baking with finalRender, as this causes errors.

If problematic areas show fine in your normalmap, you can rule out it being bad highpoly, however sometimes the highpoly will have bad geometry and you'll see both in normalmap and AO.

Also if you have meshes seperated for baking (so they don't collide during baking, and lead to a clean bake),, make note of the max distance of AO rays (usually should be as big as the cage offset), that way you won't have the problem of meshes accidently casting AO on each other.

JohnnyRaptor

rightclick on your lowpoly mesh, properties, and uncheck Cast Shadows in General tab, and also Cast Shadows in Adv. Lighting.

The problem is caused by your lowpoly casting a shadow on the highpoly, and then this getting baked out on the texture.

ImSlightlyBored

I once did all that, and had the same problem. I solved it, I think, but just selecting all faces in edit poly mode, hitting hide selected, then rendering to texture.
http://boards.polycount.net/showthread.php?t=57449
Only had the issue with mental ray AO, normals etc baked fine.

this is assuming your using max.

cow1787

After trying all of your suggestions, ImSlightlyBored's is the one that worked, but thanks to everyone!

I figured you'd have to hide the geo somehow, just didn't know how. But an Edit Poly on top worked. Weird stuff. And yeah, I'm using Max 9 32bit - SP2 with all hotfixes by the way.