Maya AO bake issue - Black lines

Brygelsmack

Hey guys. I'm making small stones for a larger object (Skyrim inspired forge) and I've run into an issue I haven't seen before.

I'm getting some weird lines on the edges of my LP. The lines follow the edge flow spot on.

Any ideas on how I can fix this?

LP with normal map.



LP with a quick diffuse slapped on to showcase the problem. In PS I have two layers of the AO map, both set to Multiply. I have two for a stronger effect. Even when I use just one layer the problem persists.



Wires.



My render settings. The normal map and the AO were baked out simultaneously, but I'm just showing the settings for the AO here.



Thanks for any advice!

CGloko

Hi,
I am having the same problem and I don't know what I do wrong. I am Max user too and I never experienced this issue when baking AO maps. Either I use Max to bake AO or I fix it in Photoshop shrinking the selection a pixel around the shells and using the filter "dilation" for photoshop that comes with xnormals.

Zipfinator

Can you post your texture? I'm guessing that the background of the AO is black and there's not enough padding for the mips to render properly.

Brygelsmack

Certainly. Here it is. I hope you meant the AO.

r_fletch_r

you just need to add some edge padding. Thats the 'fill texture seams' slider. I dont know what the unit is in maya but im guessing its pixels. set it to something like 64 or 128. you literally can never have enough padding

dand3d

There also is a filters for photoshop that can dilate your textures. I use Flaming Pear's Solidify A at work.
Also What r_fletch_r said is correct, X normal also has a edge padding feature that can be tweaked to have as much padding as you desire.

CheeseOnToast

The trouble is that Maya uses Mental Ray to do AO, which doesn't respect the same settings as you used for the normal map. It always seems to ignore any edge padding setting you have, as well as generally being a pain in the arse. I'd also recommend using Xnormal instead.

Mekhollandt

r_fletch_r is right about the Fill Texture Seams option. It's all the way at the bottom of the Mental Ray Batch Bake options box. However, for Maya, it just goes by pixels. Setting that setting to 2 should be fine for that small amount of texture space, though the larger the texture, the higher you may need to set that. I've never had to use more than 4, but I'd be surprised if you need more than 2.

A quick fix would be to just take the black that you see on the AO map outside the UV islands and do a paint bucket fill with white. Problem solved. If you don't want to do that, for whatever reason, you can do it this way:

Overlay the UVs and see that they are just outside or right on the edge of the ao-texture itself. This can be corrected in PS by making a selection of the UVs (inside with the magic selection tool), expanding the selection by one pixel (select -> Modify -> expand -> 1 pixel). Make a new layer under the AO, set the AO layer to multiply, and fill the selection on the new layer with white. Combine them, and that should fix it. You will have to do this with all of the UV islands, but it works without screwing anything up and you don't have draw around the edge. There's honestly a bunch of ways to do this, I just find this the quickest for me without having to set up a mask, etc.

Brygelsmack

No luck... Here's what I've tried:

> In mental ray Common Output I increased 'Fill texture seams" to 32, which was the highest value. Didn't work.

> I switched to Maya Common Output for rendering the AO, and I set the Fill texture seams to 128. No luck there either.

Any other settings that I can change? What's 'Filter size'? I have that set to 3.

Thanks guys for the replies. Appreciate it.

@ Mekhollandt: That sounds like a nice solution, I'll try that.

Kbrom12

You can't just make the background white in maya? I do that in max, seems like it'd be an option since its very simple but then again...its maya

edit: my buddy that uses maya has this happen occasionally, we just use the xnormal dilation filter to solve it

Brygelsmack

Sorry for bumping this... But I'm still having this issue. It seems that AO baking with Maya doesn't add any edge padding at all, no matter how what I set the "Fill texture seams" to. It's really noticable when I overlay the UVs over the normal map and the AO, the normal map is clearly getting padding and the AO is not. Has my renderer decided to stop working or is this a common issue with Maya?

Also been searching for dilation filters, tutorials and whatnot, but there's surprisingly little information on the subject. Does anyone know a good tutorial for manually adding edge padding or something? Or a tutorial for Xnormal's dilation filter?

Always getting black lines on my models is getting really annoying and it just results in me not baking AOs at all, which kinda sucks.

So any ideas what I can do to get AO baking in Maya to use edge padding?

Current render settings for AOs:




Thanks in advance.

JamesWild

Just out of curiosity wouldn't it be better to get the maps fully done and then just paint some padding manually so it matches up to the final colour?

Brygelsmack

Sorry I don't get what you mean here? Fully done? Match up to the final?

sybrix

I think JamesWild is talking about going into Photoshop and manually "painting" in edge padding? Like what Mekhollandt suggested already.

I just ran into this problem with no AO map padding and just shrunk the UV shells a tiny bit in Maya until the black lines went away, but I was only using a diffuse map.

Maybe give Xnormal a try? I rarely have a problem with bakes when I get frustrated with Maya and go running to that.

JamesWild

Yeah I mean just filling the gaps with some colour in an image editor. No point doing that yet because you're going to want colour/photos/etc. overlaid on top. Leave until the end so you only do it once and it's as continuous as possible.

lysaara

What I do is I render to a TGA with alpha, then in Photoshop cut it out and use Flaming Pear's Solidify B to fill the space. Really helps with mip problems.

I have the padding problem at home, but not at work. At home what worked for me when I'm using Batch Bake is to set the Bake Optimisation to multiple objects, even when I'm just baking a single object. Not sure why, but that worked.

Brygelsmack

Thanks for the advice, I'll try to fiddle around some more and see if I can come up with a solution.