Could someone help me with setting up AO map in Unreal...?

DustyShinigami

Hi

I'm completely at a loss with this issue, and I've tried asking in various places, but with no luck. I've tried on Reddit and the Unreal Answer Hub and I've either had no response or very little in the way of suggestions. I'm trying to get a baked AO map from Substance Painter to work in Unreal Editor for a character, and it just isn't working. I've tried looking around on Google, and what I've found suggests that the maps only work with Static Lighting...? Is this the case? Dynamic Lighting seems to generate its own AO anyway, but I've been told that an AO map is vitally important, so I shouldn't really skip it for my characters.

Whenever I connect my AO map up in the Material Editor, the map doesn't appear to make any difference from what I can tell. I'm only seeing the AO generated by the Dynamic Lighting. In fact, I've gone back to previous projects/models and noticed the same thing. I've checked the Red Channel and the AO map is there. I've tried enabling/disabling sRGB and setting it to/from Colour/Linear Colour, and I've hooked the texture from the red channel node to Ambient Occlusion. When I discovered I couldn't see any difference I then tried setting up a parameter in the hopes of increasing the strength, but that doesn't appear to be working either. :-\ Apart from disconnecting nodes and hitting apply, is there another and quicker way of checking to see if the map is having an affect on the Unreal...? I'm guessing not, but each time the shaders compile and the model goes blank/black for a few seconds, it makes it tricky to really notice if it has made a difference. As I say, I honestly can't tell. Then again, I did save two screenshots of before and after...

rexo12

Just to check, this is UE4?

Right now your shader graph suggests you're multiplying the AO map by 0, which would erase any information in the map. Why are you doing this? If you're trying to debug its contribution, try using a Power node instead (and set your parameter to >=1!)

Baked AO in UE4 is a bit strange, and frankly I genuinely haven't really noticed it's contribution. I think it impacts the specular term more than the diffuse term - dampening reflections, which might be why you aren't seeing a dramatic change between renders. It is true that Material AO is only incorporated into lightmaps - so if this is a dynamic scene that may be why it's not appearing. Try baking your lighting.

I don't think having baked AO is a hard rule, particularly if DFAO can carry it which it looks like it's doing to me.

DustyShinigami

Yeah, UE4. I think that screenshot was taken right after the graph had been set up, so nothing had been enabled. I've not used a Power Node before, so I'm not sure how to correctly set that up. ^^; I'm not sure how to go about baking the lighting either. I'm guessing it's not related to rebuilding the lighting, right...?

rexo12

Sorry, I use baking <-> rebuilding interchangeably. Yes, rebuild the lighting. And further, make sure your skylight, lights, etc are set to Static rather than Stationary or Dynamic. (That said if this is a character, you'd expect it to be dynamically lit).

The Power node is just a mathematic operation - like Multiply or Add, etc - that takes every pixel in a shader and outputs its value to the power of the 2nd parameter you provide. Like x^y for all x, y. It has the effect of increasing the contrast of the AO map, which can help debug whether or not it's actually appearing. Use it instead of the Multiply that you already have. After you compile/save the shader, you can scrub through your 'AO adjust' parameter in realtime, and see if there's anything changing.

Note that since the Material preview viewport is always dynamically lit, you will not see anything changing there. You will have to build the lighting and check the main viewport.

See the Math function reference: https://docs.unrealengine.com/4.26/en-US/RenderingAndGraphics/Materials/ExpressionReference/Math/

DustyShinigami

Hmm... It seems a bit beyond me, I'm afraid. :-\ Nothing's turning up about a Power node, and looking up a Math node is just confusing me even more. I'm not even sure how I'd go about setting it up to adjust the contrast of the map. Nodes are not my forte. Also, I've noticed (in a new project scene at least), that by putting all the lights to Static, there's great big ugly seams showing up on the model/Normal map. The texturing for this model is still a work-in-progress... ^^;

BagelHero

DustyShinigami said:

Hmm... It seems a bit beyond me, I'm afraid. :-\ Nothing's turning up about a Power node, and looking up a Math node is just confusing me even more. I'm not even sure how I'd go about setting it up to adjust the contrast of the map. Nodes are not my forte. Also, I've noticed (in a new project scene at least), that by putting all the lights to Static, there's great big ugly seams showing up on the model/Normal map. The texturing for this model is still a work-in-progress... ^^;

https://docs.unrealengine.com/4.26/en-US/RenderingAndGraphics/Materials/ExpressionReference/Math/#power
Not as difficult as it sounds! Heres the official doc page for the node, comes with a fairly easy to understand example.

As for the seam, you likely need to flip your normal map's y axis/green channel (iirc).

rexo12

The seams are likely caused by how your lightmaps are laid out. UE4 generally autogenerates them, but their resolution may be too low. This can be changed in the Static Mesh window (when you double click on the prop in the content browser). Compare this screenshot anyway with your previous ones, in lighting mode.

For the power node, try "Pow" or "^", like so:

AO map goes to Base, 'AO Adjust' goes to Exp (exponent).

However, all that said, I feel that this is looking like trying to force a square peg through a round hole. You should give some thought to how valuable this baked AO really is. Particularly as this looks like a very stylised character, i'm not sure if it's truly going to massively improve its look, especially over dynamic DFAO. Characters should be dynamically lit anyway, and so forcing static lighting just for baked AO is a bit of a hack.

DustyShinigami

Thanks for the suggestions, guys. I’ll be sure to give those a try and see what happens. 

As to the baked AO vs the Dynamic - I agree. I much prefer the Dynamic lighting and the AO it produces. However, I want to at least try and get the baked map to work. It’s just useful to know how to do it.

Mink

Yeah, I've never noticed UE4 AO node doing anything ever at all, I'm considering replacing the slot with a cavity map at this point, because I really can't tell what it does, if anything, ever.

DustyShinigami

BagelHero said:

As for the seam, you likely need to flip your normal map's y axis/green channel (iirc).

Sadly, that didn't work. I'd be surprised if it did to be honest, as both Unreal and Substance Painter use DirectX, which is what everything was baked out at. I read that in order to flip the Y axis, it would need to be changed from DirectX to OpenGL from the Project Configuration, and then re-baked out.

Kanni3d

the tangent basis is likely correct as you mentioned it was correctly synced by baking out as directx and set up properly in unreal/painter. It's a little interesting that it could be lighting/lightmap related as mentioned, since it only popped up once you made the asset static vs dynamic.

When you baked her out, did you have all stacked UV's offset by one UV grid space over?

DustyShinigami

rexo12 said:

The seams are likely caused by how your lightmaps are laid out. UE4 generally autogenerates them, but their resolution may be too low. This can be changed in the Static Mesh window (when you double click on the prop in the content browser). Compare this screenshot anyway with your previous ones, in lighting mode.

Sadly, this hasn't helped either. :-\ I've tried changing it to both 128 and 256 but the seam is still there. It's very confusing as I've noticed these seams don't show up in my 'render environment' I've set up, but a new default empty scene does, so I figured it must be something to do with the lighting. Also, if I put the lights to Moveable or Stationary, the seams disappear.

Someone on Reddit was convinced it was because I'd overlayed symmetrical UVs, such as the face, but that shouldn't be the problem, right...? I mean, it looks great in Substance Painter, which use a PBR workflow, and a previous attempt I made of this character didn't have the issue in Unreal. It made me wonder if the Smoothing Groups had to be separate for each overlayed piece that was symmetrical, but again - I didn't do that before and it came out fine on my previous model.

DustyShinigami

Kanni3d said:

When you baked her out, did you have all stacked UV's offset by one UV grid space over?

I did, yeah. Then any issues with the Normal and AO I painted out/over. Should the offset UVs be kept offset or is it fine to overlay them again?

DustyShinigami

rexo12 said:

For the power node, try "Pow" or "^", like so:

AO map goes to Base, 'AO Adjust' goes to Exp (exponent).

I can confirm this works beautifully! Thank you. It also works with Dynamic Lighting, but it's much more subtle. Also, it looks like it's just Movable for the lights that removes/heavily reduces the seams. Both Static and Stationary causes it to appear.

Kanni3d

DustyShinigami said:

Kanni3d said:

When you baked her out, did you have all stacked UV's offset by one UV grid space over?

I did, yeah. Then any issues with the Normal and AO I painted out/over. Should the offset UVs be kept offset or is it fine to overlay them again?

You can move em back post-baking without an issue. But yeah as you further investigated with the lights - it's something to do with your lightmaps, perhaps none are set up. Did you let unreal handle them, or did you create your own lightmaps in another UV channel?

AO seems to do littlle to nothing in unreal right away, but under the hood it likely causes specular reflections to dull down in areas of AO (so shadow/cavity areas don't reflect as much as others). If you really want, you could multiply your AO over your basecolor either in photoshop, or in that material editor where you used the power node. This effect is possibly what you're looking after since this is a stylized borderlands texture.

DustyShinigami

Okay. And no, I've not created my own lightmaps. Never done it before. I'm still slowly learning Unreal. Haven't learnt as much about it compared to other software like Max, Maya, Blender, ZBrush etc. I did create my own render environment based on a tutorial, but that was more to do with materials and setting up lights such as a fill, back and key lights.

I did read about combining the AO with the diffuse, but have read divided opinions about it. Plus, combined with the AO generated by the Dynamic Lighting, it might look too strong...?

Kanni3d

I did read about combining the AO with the diffuse, but have read divided opinions about it. Plus, combined with the AO generated by the Dynamic Lighting, it might look too strong...?

It's usually a no-no, and "pbr incorrect" if you are doing realism/realistic textures. However, for stylized, there's always plenty of things that get overlayed to get certain details to come through. It could look too strong, but can be controlled via that multiply node (.1 = 90% less, .5 = 50% less for example)

Creating and setting up your lightmap is really easy, but mind you - if this character is dynamic and moves, it's totally not needed since it'll be a dynamic/movable asset. If you're using unreal just to simply render, then yeah maybe get some lightmap uv's set up.

https://www.youtube.com/watch?v=fNTBzwa8arc

DustyShinigami

Okay, that makes sense. As to the lightmap, I'll look into it just as something else to learn for future projects, so thanks for that video. Although for this project, I do plan on making a simple idle animation, which I'm aiming to learn how to implement into Unreal. It should prove useful though for idle poses though.

Kanni3d

I do plan on making a simple idle animation, which I'm aiming to learn how to implement into Unreal.

If so, leave it as dynamic/movable, and you won't have to worry about the lightmaps then But yeah, for idle renders, if you really wanna bake lighting for that bit more quality, then give it some lightmap uvs.

Great work btw, Tina is one of my favorites in BL universe

DustyShinigami

Will do. Thing is, if I bake the lighting, will I need to do a completely new unwrap for Tina? Or will the current unwrap suffice? I've not checked if she's been given a default lightmap yet.

And thanks. She's my favourite too. I've even been in touch with the original artist who made her - Wes Parker - so I've been picking his brains on his workflow whilst making her. Learnt loads whilst making her too.

rexo12

DustyShinigami said:

I can confirm this works beautifully! Thank you. It also works with Dynamic Lighting, but it's much more subtle. Also, it looks like it's just Movable for the lights that removes/heavily reduces the seams. Both Static and Stationary causes it to appear.

How bizarre! Unreal Engine's material AO is a magic black box to be honest - often not worth the trouble! Glad you got it sorted out though.

Kanni3d

will I need to do a completely new unwrap for Tina? Or will the current unwrap suffice? I've not checked if she's been given a default lightmap yet.

Nope, not much work is required to create a workable lightmap uv. Your current unwrap should suffice if you simply copy over your UV0 to UV1 (UV1 will be your lightmap uv).

Make sure the lightmap uv has NO inverted and/or overlapping uvs, and that everything is packed within the 0-1 space. My only concern would be the seams down the middle of her face due to the symmetry may cause some artifacting, so I'd recommend simply grabbing both halves of her face and welding them together to be one. 

ZacD

Mink said:

Yeah, I've never noticed UE4 AO node doing anything ever at all, I'm considering replacing the slot with a cavity map at this point, because I really can't tell what it does, if anything, ever.

I believe I've seen Epic themselves suggest to use the AO output for "cavity AO". Most of the AO should be coming from the actual lighting, SSAO, and maybe DFAO. 

DustyShinigami

ZacD said:

Mink said:

Yeah, I've never noticed UE4 AO node doing anything ever at all, I'm considering replacing the slot with a cavity map at this point, because I really can't tell what it does, if anything, ever.

I believe I've seen Epic themselves suggest to use the AO output for "cavity AO". Most of the AO should be coming from the actual lighting, SSAO, and maybe DFAO. 

What's the best way to do a cavity map? I know Substance Painter does a Curvature map... Plus, Painter exports Unreal textures with standard AO, roughness and metallic all combined.

Kanni3d

What's the best way to do a cavity map?

You can "extract" technically from your curvature, since curvature is both cavity and convexity - but that could be tricky if you don't know how. Best to use something like Marmoset to individually bake out a cavity map.

Plus, Painter exports Unreal textures with standard AO, roughness and metallic all combined.

this isn't much a problem, you can just plug in your cavity map into your AO channel in substance painter for it to export as an AO channel in that packed map you mentioned. Or you can manually paste over the AO with the cavity map in the appropiate channel in photoshop.

DustyShinigami

Kanni3d said:

What's the best way to do a cavity map?

You can "extract" technically from your curvature, since curvature is both cavity and convexity - but that could be tricky if you don't know how. Best to use something like Marmoset to individually bake out a cavity map.

Plus, Painter exports Unreal textures with standard AO, roughness and metallic all combined.

this isn't much a problem, you can just plug in your cavity map into your AO channel in substance painter for it to export as an AO channel in that packed map you mentioned. Or you can manually paste over the AO with the cavity map in the appropiate channel in photoshop.

Aha. Cool. Good to know. Thanks for the tip. I did try using xNormal as that gives an option to bake a curvature map, but it keeps giving me an error about one of the meshes having an n-gon. I'm guessing it's the high poly. Although I did get Max to triangulate it. So I'm not sure if it has an issue with the high, the low, or the cage.

Kanni3d

prob the cage, i recall xnorm being super picky about it which is why i moved onto baking in painter, then marmoset.

DustyShinigami

hmm, okay. I'll have to try triangulating it. Not sure how n-gons could have got there though as they were made from the low poly in Maya.

DustyShinigami

I triangulated it, but xNormal still keeps complaining. It was going on about how the number of meshes between the low and cage were different or some such. Not sure why as there should be an equal amount. I do have separate cages for the mask and the main character, which I baked individually in Painter, but even combining everything together, xNormal still wasn't happy. I'll leave it. I'll have to look at approaching it a different way. Maybe Marmoset...? Although I don't have much experience with it. Only a bit in Toolbag for rendering.

Kanni3d

Cage should be the exact same as the target mesh due to how raycasts work from the tangents/vertices of your mesh. If you have a mismatch (cage having either higher or low amount of vertices, especially in different areas) it wouldn't behave as expect and the baker will simply prevent it from baking.

If you're adamant on using something like xnormal which requires you to make your own cage - then simply duplicate your low res, use a push modifier in Max until it encompasses past your highpoly/source data - and then export that as your cage.

On the left is an ideal scenario of the cage having exact same vert/positions as your low res, on the right is a busted scenario with a mismatching cage.

DustyShinigami

I didn't think about duplicating the low poly. I'll give that a try and see. Thanks.

DustyShinigami

Back to the seam issue - I know I shouldn't worry about how things look in 3D software compared to in engine, but that same look is present in Max as Unreal. Not sure if this is still a lighting problem, or if it's something different...? But the overlayed parts do seem to have a darker shading compared to the opposite. Hmm... This keeps happening with the Normal map by the way.

Kanni3d

could be the way it was baked, the uv setup being overlapped/symmetrical. Ensure its welded, smoothing is set up, and the overlapped sides are set one exact uv set to the right, and rebake?

DustyShinigami

Hmm. Maybe. I think I mentioned before this is a re-attempt, mainly to fix up loads and looooaaaads of issues, and to re-do the texturing, and I never encountered this with the first attempt. Not in Max or Unreal. So something is definitely amiss. And yet it looks all gravy in Substance. Very weird. I'll try checking if things are welded properly as I know that can cause weird smoothing issues.

DustyShinigami

Never mind. Looks like it's perfectly normal. With the screenshot from Max, I have put the Normal from 30 to 100 so the details stand-out more. Probably a bit overkill though, hence the seams. But I went back to my first attempt and the same thing happens with the Normal map applied. The seam is still noticeable even at 30 too

But man... Looking back at my first attempt, the texturing is awful! So rushed and lacking sooooo much detail. XD So glad I've gone back and improved so much of it, along with the high poly, and it's also taught me loads of new methods and improved workflows.

DustyShinigami

@Kanni3d Okay, I've tried duplicating the low poly and adding a Push modifier so it covers over the high poly. But no matter if I merge the sub-objects together into one object, or have everything in a group like the low poly, xNormal still keeps complaining that the cage, or different parts of it, don't have the same number of vertices as the low. I checked and they do. For one part anyway. Both have 300 vertices. o_O

It's honestly not a problem though if this can't be fixed.

Obscura

Sorry for not reading the entire thread but I think this might not have been mentioned yet. So for your own sake, the ao material input only works with stationary or static light sources by dafault. This is how it is, when you have static lighting allowed in the project settings. It will never show on its full intensity, because its determined by multiple factors. For example, the indirect lighting contribution on the given pixel. So  in an area that has more baked shadows and less baked direct lighting, it will show more prominently. And in an area lit fully directly, it will almost not show. There is a way to get it working with dynamic lights as well, but thats not a realistic  environment in my opinion. Most games goes with baked lighting. You should only tick that if you are practicing for the specific purpose of making it for dynamic lighting, WITH ue4.

Obscura

With ue5, this is a different question again. As it leans towards a slightly more realistic lighting approach. ambient occlusion is a made up thing in the reality, and real lighting doesnt really do that. AO basically means a crevice where light can hardly bounce around so you get more darkness. With lumen in place, the ao slot doesnt work again. Something to keep in mind.

Cerulean_Deity

Obscura said:

With ue5, this is a different question again. As it leans towards a slightly more realistic lighting approach. ambient occlusion is a made up thing in the reality, and real lighting doesnt really do that. AO basically means a crevice where light can hardly bounce around so you get more darkness. With lumen in place, the ao slot doesnt work again. Something to keep in mind.

Hello! I was in a vaguely similar situation as OP in that I couldn't see any effect from material AO, and came across this thread as part of learning today that AO texture maps don't render fully in dynamic lighting unless static lights are turned off in the project settings.  I disabled them earlier and now I see AO as expected, but then I saw your comment about how it's not realistic, so I'm guessing I need to revert back to the default.  With that said, and on top of you mentioning the AO slot not working with lumen in place, is it safe to say that AO texture maps won't be necessary with UE5 and lumen enabled?  I'm dealing with massive project bloat and am moving into an optimization pass and I'm wondering for the long term, if I should just scrap all my AO maps, or if they are, or will be, still used in some capacity with lumen.

svv3d

It looks like you are using a mirror UV for the head model. And the normal map for that part is wrong. Try to fix it with this solution

https://polycount.com/discussion/51088/tutorial-fixing-mirrored-normal-map-seams/p1

DustyShinigami

Oh that’s awesome. Thanks so much for sharing that. That should be super useful. When I finally get back to that project, I’ll have to try that out. :D