Hi, I've been writing a maxscript tool to render out directional lightmaps using 3ds Max and while I have the process down the final result isn't quite right. The script rotates the surface normals of the mesh before rendering each of the 3 lightmaps and the lightmaps themselves look correct but when combined back together in the shader the light looks like its coming from the wrong directions.
The 3 lightmaps for an unwrapped box end up coming out like this:



But the final lightmap shader outputs this:

I feel like there is a matrix transform i'm missing with the normal basis but i'm not sure.
Max doesn't exactly provide the greatest control over surface normals, tangents and bitangents so I've had to take a function from someone on cgsociety to generate the face bi/tangents manually and use those bi/tangents to transform the normal basis.
The code for skewing the normals is as follows:
- -----------------------------------------------------------------------------------------------------
- -- computeTangentSpace function taken from HalfVector on cgsociety
- -- The face position has been removed from the resulting matrices
- -- http://forums.cgsociety.org/showpost.php?p=3881243&postcount=11
- function computeTangentSpace obj =
- (
- local theMesh = snapshotAsMesh obj
-
- local tSpace = #()
-
- -- Do we have to flip faces?
- local flip = false
- local indices = #(1, 2, 3)
- if dot (cross obj.transform.row1 obj.transform.row2) obj.transform.row3 <= 0 do (
- indices[2] = 3
- indices[3] = 2
- flip = true
- )
-
- for nFace = 1 to theMesh.numFaces do (
- local face = getFace theMesh nFace
- local tface = getTVFace theMesh nFace
-
- local v1 = getVert theMesh face[indices[1]]
- local v2 = getVert theMesh face[indices[2]]
- local v3 = getVert theMesh face[indices[3]]
-
- local uv1 = getTVert theMesh tface[indices[1]]
- local uv2 = getTVert theMesh tface[indices[2]]
- local uv3 = getTVert theMesh tface[indices[3]]
-
- local dV1 = v1 - v2
- local dV2 = v1 - v3
-
- local dUV1 = uv1 - uv2
- local dUV2 = uv1 - uv3
-
- local area = dUV1.x * dUV2.y - dUV1.y * dUV2.x
- local sign = if area < 0 then -1 else 1
-
- local tangent = [0,0,1]
-
- tangent.x = dV1.x * dUV2.y - dUV1.y * dV2.x
- tangent.y = dV1.y * dUV2.y - dUV1.y * dV2.y
- tangent.z = dV1.z * dUV2.y - dUV1.y * dV2.z
-
- tangent = (normalize tangent) * sign
-
- local normal = normalize (getFaceNormal theMesh nFace)
- if flip do normal = -normal
-
- local binormal = (normalize (cross normal tangent)) * sign
-
- append tSpace (Matrix3 tangent binormal normal [0,0,0])
- )
-
- delete theMesh
-
- return tSpace
- )
- function SetNormalsModifier nodeList &editNormalsMod lightmapType =
- (
- normalBasis = [0, 0, 1]
- if (lightmapType == 1) then
- (
- return 0
- )
- else if (lightmapType == 2) then
- (
- normalBasis = [sqrt (2.0 / 3.0), 0, 1.0 / (sqrt 3.0)]
- )
- else if (lightmapType == 3) then
- (
- normalBasis = [-(1.0 / (sqrt 6.0)), 1.0f / (sqrt 2.0), 1.0f / (sqrt 3.0)]
- )
- else if (lightmapType == 4) then
- (
- normalBasis = [-(1.0 / (sqrt 6.0)), -(1.0f / (sqrt 2.0)), 1.0f / (sqrt 3.0)]
- )
-
- for j = 1 to nodeList.count do
- (
- obj = nodeList[j]
-
- -- Get the face TBN's
- meshFaceTBNs = computeTangentSpace obj
-
- normalFaceCount = editNormalsMod.getNumFaces node:obj
- normalVertexCount = editNormalsMod.getNumNormals node:obj
-
- -- Create an array of arrays to contain the vertex normals for averaging later
- normalsListArray = #()
-
- for i = 1 to normalVertexCount do
- (
- append normalsListArray #()
- )
-
- for i = 1 to normalFaceCount do
- (
- faceSelectionArray = #{i}
- normalSelection = #{}
-
- -- Get the normal indices of the selected face
- editNormalsMod.ConvertFaceSelection &faceSelectionArray &normalSelection node:obj
-
- -- Get the face TBN
- TBN = meshFaceTBNs[i]
-
- for k in normalSelection do
- (
- -- Transform the normal basis into tangent space?
- normal = normalize(normalBasis * TBN)
-
- -- Append the new normal to a list for this vertex
- append normalsListArray[k] normal
- )
- )
-
- for i = 1 to normalVertexCount do
- (
- normal = [0, 0, 0]
-
- -- Average the normals from the list for this vertex
- for k = 1 to normalsListArray[i].count do
- (
- normal += normalsListArray[i][k]
- )
-
- normal /= normalsListArray[i].count
-
- -- Set the vertex normal
- editNormalsMod.SetNormal i (normalize normal) node:obj
- )
- )
- )
The full script and a DirectX shader for you to look at is
available here.
The script is set up as a macroscript so will need to be added as a button to your UI. The shader requires lightmap coords and texture coords in channels 1 and 2, and has a checkbox to state which is which.
Thanks in advance

.
Replies
The shader function I have for calculating the lightmap output is as follows: The NormalMap argument is unbiased before it is passed to the function.
With that I get the following result in my test map:
...which is pretty bad. If I flip the normals Y value it gets somewhat better with the walls rendering correctly:
...but the floor is still wrong and I don't know why