This is somewhat more of a theoretical question, but as the title says, could a normal map be made with vector lengths from 0-1 according to the values of an AO map rather than having a separate AO texture? I think this would have the effect of multiplying a grayscale texture onto the object, but I'm not completely sure. The normal map could be normalized to get the original, scalar 1, values.
Replies
http://wiki.polycount.com/NormalMap#Ambient_Occlusion_into_a_Normal_Map
I would think the right way would be to add a new layer and fill it with 128,128,128 and then use the inverted AO as a layer mask...?
However, it still results in incorrect lighting because it affects all the lighting not just the ambient. Better to use a separate AO.
If possible you need to adjust the shaders to re-normalize the normal where needed or extract the ao value on its own. Most lighting calculations which use the normal in combination with a dot product, have no problem with the scaled normal (lambert diffuse etc), the simple math is:
dot(L,N) * AO = dot(L, N*AO)
The issue is, that you loose important information of the normals, low AO values will reduce the available number resolution, which will introduce artifacts, especially when using 8 bit channels for the normal. Just think of an AO value of 0, which would result in a normal of (0,0,0). There's no way to reconstruct the normal from this value. Therefore low AO values will distort the normals.
This might not be obviously in some cases, i.e. low AO values will result in low lighting and the artifacts will not be obviously visible. But other effects based on the normal (i.e. SSAO or fresnel highlighning) could result in really ugly artifacts.
Yeah, I can see how there could be a resolution problem, especially if you're normalizing the map to get 'non-occluded' values for dynamic lighting. I was thinking maybe for static meshes if there were a way to prebake the normals accounting for the AO of that specific scene/lighting, but in that case you could probably just put the AO in the diffuse for the same effect.
It doesn't seem it'd be worth the loss of resolution and extra instructions to treat the normal map as a place for sneaking in an AO map, so I don't think I'll be pursuing it(I'll stick to deriving the blue channel), but it's really cool knowing it's a thing !
I think I must be doing it wrong then. I just tried it and I'm getting my down-right vectors shadowed correctly, but my up-left vectors aren't. The darken version as I understand it will shorten any vector pointing right (greater than 128 ) but not the ones pointing left (less than 128 ).
Multiply the values.
Multiply will make vectors pointing in negative x/y even longer (further from 128 ) and will brighten them.
Edit: Lol I keep getting this smiley 8)
Edit: I've seen people do similar math and transformations in photoshop before with actions, so it should be possible to set up something that would allow you to make a bent normal map right in ps provided the nm and the ao (maybe by splitting the normal map into two images and recombining in the end).
If you start off with the normal vector xyz (.71, 0, .71) and you wanted to shorten it to 50% length, you'd divide the length by 2. You could also get the same result by 'blending' it 50% towards zero - as in, add our vector to the zero vector and divide by two. Now in rgb 0..1 space it's harder to divide the length by two, but it's trivial to blend 50% towards zero. Zero in our case is rgb (128,128,128).
I still don't get the darken method, if it does work then I'm musunderstanding the process.
edit: mixed up a number
edit: oh, looks like that has been suggested multiple times already.