Displacement map bulges from Mudbox texture render

Stu2Prof

Hi,

When I rendered out AO, Disp and Normals Map from Mudbox I got a bulging effect on the edges of polygons which created a hot air balloon-like effect on a little 3d octopus I'd been designing. I do not know how I created this error and would be really grateful if someone could point me in the direction of a tutorial to fix my mistake.


Model in Mudbox before texture extraction

Model after texture extraction


Model after extraction with AO, Normals but Disp removed


Normals

oglu

could you post your bake settings... i assume you have to use smooth target in the bake settings...

Stu2Prof

I selected smooth target and that seems to have fixed the bulges problem. Thanks Oglu!
Bake settings:


New texture:

Stu2Prof

Hi I revisited this project today and found that I am still having problems with my normal map(also AO slightly) The issue is that around the mouth of my octopus model I am getting stretches/hard edges on my Normals where the low-poly model edges can be found. I am using the bake settings as before but that hasnt solved the problem. Is there a step that I have missed along the way or is it that I need to improve my UVs?

Sinking

I'd tick the "test both sides" option in normal maps, since it looks like your highpoly is intersecting with the lowpoly, so it cannot project properly where the highpoly is smaller than the lowpoly (because it is inside the lowpoly); ticking the "test both sides" options should test inside too. You can also change the settings to "furthest inside/furthest outside" of whatever it is called.

the reason I don't remember is because I gave up baking in Mudbox. While it is actaully pretty clean for the most part, Mudbox always had a slight offset in my normal map projection, which becomes particularly notable with geometric, e.g. round mesh bakes.

I use Xnormal for the most part for baking. I used Maya too, but it kinda annoyed me by looking totally differently in Maya and UDK. I'll retry baking normals with MayaLT.

Stu2Prof

Thanks for the suggestion Sinking but that doesnt seem to have solved the problem. I am starting to worry that this may be a graphics issue with mudbox as I cant see those seams/hard edges in the texture itself and yet they appear in the Mudbox viewport. How can I test if this is true?