Using normal lengths <1 for AO type effect?

Gestalt

This is somewhat more of a theoretical question, but as the title says, could a normal map be made with vector lengths from 0-1 according to the values of an AO map rather than having a separate AO texture? I think this would have the effect of multiplying a grayscale texture onto the object, but I'm not completely sure. The normal map could be normalized to get the original, scalar 1, values.

Neox

search for bent normalmaps they should be what you want

cptSwing

Not sure this is relevant, but many engines normalize normals maps, though (ie clamp the length)?

Eric Chadwick

We have some info about this here
http://wiki.polycount.com/NormalMap#Ambient_Occlusion_into_a_Normal_Map

Kurt Russell Fan Club

I think the advice there is wrong. If you use darken with the AO then only vectors pointing straight towards you (128,128,255) or down/right will be shadowed. Lower red and green values should also be lightened towards 128.

I would think the right way would be to add a new layer and fill it with 128,128,128 and then use the inverted AO as a layer mask...?

Eric Chadwick

It works, try it. If you clamp the range to be from 128 gray to 255 white, then set it to Darken, it does the right thing.

However, it still results in incorrect lighting because it affects all the lighting not just the ambient. Better to use a separate AO.

Ashaman73

Gestalt wrote: »

This is somewhat more of a theoretical question, but as the title says, could a normal map be made with vector lengths from 0-1 according to the values of an AO map rather than having a separate AO texture? I think this would have the effect of multiplying a grayscale texture onto the object, but I'm not completely sure. The normal map could be normalized to get the original, scalar 1, values.

From a theoretical view this is not a problem, but you have to deal with some issues.

If possible you need to adjust the shaders to re-normalize the normal where needed or extract the ao value on its own. Most lighting calculations which use the normal in combination with a dot product, have no problem with the scaled normal (lambert diffuse etc), the simple math is:
dot(L,N) * AO = dot(L, N*AO)

The issue is, that you loose important information of the normals, low AO values will reduce the available number resolution, which will introduce artifacts, especially when using 8 bit channels for the normal. Just think of an AO value of 0, which would result in a normal of (0,0,0). There's no way to reconstruct the normal from this value. Therefore low AO values will distort the normals.

This might not be obviously in some cases, i.e. low AO values will result in low lighting and the artifacts will not be obviously visible. But other effects based on the normal (i.e. SSAO or fresnel highlighning) could result in really ugly artifacts.

Gestalt

Ashaman73 wrote: »

From a theoretical view this is not a problem, but you have to deal with some issues.

If possible you need to adjust the shaders to re-normalize the normal where needed or extract the ao value on its own. Most lighting calculations which use the normal in combination with a dot product, have no problem with the scaled normal (lambert diffuse etc), the simple math is:
dot(L,N) * AO = dot(L, N*AO)

The issue is, that you loose important information of the normals, low AO values will reduce the available number resolution, which will introduce artifacts, especially when using 8 bit channels for the normal. Just think of an AO value of 0, which would result in a normal of (0,0,0). There's no way to reconstruct the normal from this value. Therefore low AO values will distort the normals.

This might not be obviously in some cases, i.e. low AO values will result in low lighting and the artifacts will not be obviously visible. But other effects based on the normal (i.e. SSAO or fresnel highlighning) could result in really ugly artifacts.

Yeah, I can see how there could be a resolution problem, especially if you're normalizing the map to get 'non-occluded' values for dynamic lighting. I was thinking maybe for static meshes if there were a way to prebake the normals accounting for the AO of that specific scene/lighting, but in that case you could probably just put the AO in the diffuse for the same effect.

It doesn't seem it'd be worth the loss of resolution and extra instructions to treat the normal map as a place for sneaking in an AO map, so I don't think I'll be pursuing it(I'll stick to deriving the blue channel), but it's really cool knowing it's a thing !

Kurt Russell Fan Club

Eric Chadwick wrote: »

It works, try it. If you clamp the range to be from 128 gray to 255 white, then set it to Darken, it does the right thing.

However, it still results in incorrect lighting because it affects all the lighting not just the ambient. Better to use a separate AO.

I think I must be doing it wrong then. I just tried it and I'm getting my down-right vectors shadowed correctly, but my up-left vectors aren't. The darken version as I understand it will shorten any vector pointing right (greater than 128 ) but not the ones pointing left (less than 128 ).

Gestalt

Kurt Russell Fan Club wrote: »

I think I must be doing it wrong then. I just tried it and I'm getting my down-right vectors shadowed correctly, but my up-left vectors aren't. The darken version as I understand it will shorten any vector pointing right (greater than 128 ) but not the ones pointing left (less than 128 ).

Multiply the values.

Kurt Russell Fan Club

How's that going to work? A normal map value of 128,128,128 means a zero-length vector so when you want a normal to get shorter you want it to go closer to 128.

Multiply will make vectors pointing in negative x/y even longer (further from 128 ) and will brighten them.

Edit: Lol I keep getting this smiley 8)

Gestalt

Yeah the normal map would have to be transformed to -1 to 1 first. (so *2 -1)

Edit: I've seen people do similar math and transformations in photoshop before with actions, so it should be possible to set up something that would allow you to make a bent normal map right in ps provided the nm and the ao (maybe by splitting the normal map into two images and recombining in the end).

Kurt Russell Fan Club

Yep, but the maths I'm suggesting works and is much easier (you can do it in PS without scripts in sbout five seconds).

If you start off with the normal vector xyz (.71, 0, .71) and you wanted to shorten it to 50% length, you'd divide the length by 2. You could also get the same result by 'blending' it 50% towards zero - as in, add our vector to the zero vector and divide by two. Now in rgb 0..1 space it's harder to divide the length by two, but it's trivial to blend 50% towards zero. Zero in our case is rgb (128,128,128).

I still don't get the darken method, if it does work then I'm musunderstanding the process.

Gestalt

hrmm, on that line of thought if I understand what you're saying you could use the AO map as a mask for the opacity of a 128,128,128 layer over the original normal map (the opacity would act as a % toward 0). It might work but I haven't tried it yet.

edit: mixed up a number

Scruples

Have you tried setting the 128-255 AO to Multiply?
edit: oh, looks like that has been suggested multiple times already.