Normalmap Confusion (isn't it always!)
polycounter lvl 6
Before I ask my question, I would just like say a big thank you Polycount Forum users for past help and that as a result I finally managed to get my first character model finished!!
Can be found here on the Steam Workshop - gm_tank
(not a shadow on some of the work seen on Polycount, but my first try)
However, although I have released the model, I am still having some issues with the normalmap that I don't understand.
I made the model in Max and then sculpted in Mudbox. I extracted the normalmaps from Mudbox (final normal created by overlaying 3-4 different extractions in photoshop). I then simply applied that normalmap directly to my model ingame.
The model is running in the Source engine (L4D2). The problem I am having is that in some places the normalmap is rendering correctly but in other places it is inverted, extruding INWARDS where it should be extruding outwards.
When extracting the normalmaps from Mudbox it gives you a choice of Maya or Max tangent space, I used Maya because from my (limited) understanding it is the same that Source uses.
What I obviously dont get is why its rendering ok in some areas and not in others... it does seem that its certain UV islands that are rendering completely inverted.
Below are some photos that show the problem...
In the first image (front shot) you can see how on the right arm, chest and head things are fine, but on the LEFT arm the normals are inverted.
In the second image you can see at the top how the normals are ok on the rear belt and tags, but just underneath (on different island) on the jeans the seams are indented rather than extruded. In the middle you can see the side belt tags are correct until it crosses over to another uv island and then it becomes indented again. At the bottom you can see the tendons on the bridge of the foot for channels rather than ridges.
The final image is the normalmap itself, I resized to 1024 from 2048 but the main details should still be obvious.
Its been a long time since I unwrapped this model so I don't remember exactly what I did, I wondered if MAYBE if I had used the mirror function then this could be the result... but honestly I don't think I did use it, and I didnt think that using the mirror function was supposed to invert normals anyway.
I really hope this can be fixed without having to make any changes to the UVs themselves, my diffuse map is over 100 layers in photoshop, many handpainted in Mudbox and making anything but the simplest changes to the uvw space would be a huge pain...
So if anyone can help, tell me why this is happening and what I can do to fix it I would be very appriciative... the only thing I can think of is re-extracting all the noramls again using Max tangent space and replacing the UV islands that are incorrect... but that feels like it might be ignoring the real problem, and I am not even sure it will work. But I dont really understand the tangent space thing, photoshop has 4 possible choices; X,Y - InvertX,Y - X,InvertY and InvertX,InvertY, but Mudbox only offers 2, Max and Maya
Can be found here on the Steam Workshop - gm_tank
(not a shadow on some of the work seen on Polycount, but my first try)
However, although I have released the model, I am still having some issues with the normalmap that I don't understand.
I made the model in Max and then sculpted in Mudbox. I extracted the normalmaps from Mudbox (final normal created by overlaying 3-4 different extractions in photoshop). I then simply applied that normalmap directly to my model ingame.
The model is running in the Source engine (L4D2). The problem I am having is that in some places the normalmap is rendering correctly but in other places it is inverted, extruding INWARDS where it should be extruding outwards.
When extracting the normalmaps from Mudbox it gives you a choice of Maya or Max tangent space, I used Maya because from my (limited) understanding it is the same that Source uses.
What I obviously dont get is why its rendering ok in some areas and not in others... it does seem that its certain UV islands that are rendering completely inverted.
Below are some photos that show the problem...
In the first image (front shot) you can see how on the right arm, chest and head things are fine, but on the LEFT arm the normals are inverted.
In the second image you can see at the top how the normals are ok on the rear belt and tags, but just underneath (on different island) on the jeans the seams are indented rather than extruded. In the middle you can see the side belt tags are correct until it crosses over to another uv island and then it becomes indented again. At the bottom you can see the tendons on the bridge of the foot for channels rather than ridges.
The final image is the normalmap itself, I resized to 1024 from 2048 but the main details should still be obvious.
Its been a long time since I unwrapped this model so I don't remember exactly what I did, I wondered if MAYBE if I had used the mirror function then this could be the result... but honestly I don't think I did use it, and I didnt think that using the mirror function was supposed to invert normals anyway.
I really hope this can be fixed without having to make any changes to the UVs themselves, my diffuse map is over 100 layers in photoshop, many handpainted in Mudbox and making anything but the simplest changes to the uvw space would be a huge pain...
So if anyone can help, tell me why this is happening and what I can do to fix it I would be very appriciative... the only thing I can think of is re-extracting all the noramls again using Max tangent space and replacing the UV islands that are incorrect... but that feels like it might be ignoring the real problem, and I am not even sure it will work. But I dont really understand the tangent space thing, photoshop has 4 possible choices; X,Y - InvertX,Y - X,InvertY and InvertX,InvertY, but Mudbox only offers 2, Max and Maya
Replies
Have you tried Reset Xform on the model? You will need to re-bake after doing so.
Have you moved all the mirrored UVs out of the 0-1 UV box? Some game importers rely on the mirrored UVs not all being coincident, to calculate a seamless tangent space.
Inverting the color channels is different from baking into a specific tangent space, that's why Photoshop has different options than Mudbox.
Have you tried baking in a different program, like Xnormal, or 3ds Max?
usually X is red, Y is green Z is blue.
In photoshop, find the area of the normal map that is not working right, select around it, then go to the channels palette. Invert the selected area on the appropriate channel, save and re-export.
Hard to say what's caused it without seeing the source models etc, but you can at least work with what you currenrly have.
I normally reset x-from quite often and normally always after unwrapping the UVWs - however its been quite a long time since I was at that stage, and I honestly dont remember clearly... I used a turbosmooth modifer on the low poly in Max to get all quads for Mudbox. Its possible I may nor have reset the x-form after turbosmoothing. Are you suggesting that I reset the x-form, import into MB as a new object and then bake into that?
No everything is in the 0-1 box on x and y.
Yeah, like I said my understanding of this is somewhat limited
I havent. I don't have xNormal. My sculpted mesh in Mudbox is over 7million Polys i think which I thought Max may have problems with. Will it be okay with that polycount or not? The thing is the only way I know to bake the normals in Max would be to use a projection cage. The problem with that is that the model is very tight under the arms and the sculpted version actually intersects with itself a bit. Maybe I can break it into separate objects, I just worry that the quality of a projection in Max would not be so accurate as a Mudbox bake - but maybe I am wrong about that.
I dont know if it helps, but I noticed another model I made in the same way Max, Mudbox, Source (CS:GO, this time) was suffering from the same problem.
Thanks Vailias!! I will definitely try this first as it certainally sounds the simplest!! I suppose its the blue channel I will have to invert - I'll play around, there's not that many possible combos, hopefully will find the right one!
I would certainlly be curious to know the cause, like I said I noticed another model I made has the same problem the other night... and I would really like to avoid the problem in the future.
But... inverting the channels won't fix it if the problematic areas are mirrored (for example, left arm and right arm using the same UV space). Inverting a mirrored part of the texture will just cause the other side to look wrong.
The reason I suggest Max or Xnormal for baking is maybe those bakers will match Source better. Don't know which baker matches Source, though it seems people have used Xnormal and had good results. Source Engine Tangent Basis?
Another thing to try... move all the mirrored UVs out of the 0-1 UV box, and re-bake. http://wiki.polycount.com/NormalMap#UV_Coordinates
Lol... of course I picked the only WRONG one!!
Maybe I must apologize here cos I think I didnt explain very well. No parts of my model are using the same UVW space, each polygon has its own unique place in the 0-1 UV square... I was trying to figure out why the problem was effecting only SOME of the UV islands and I wondered if maybe using the UV mirror function (button like M with vertical line through middle) could have had an effect - I didnt think it normally should make any difference, but you never know.
I havent read the whole link you posted to the Tangent Basis thread (its very late now) but it seems to kinda confirm what i beleived to be true, that Source uses a system to Maya - but this doesnt really explain for me why certain parts of the UVs render perfectly and others are incorrect... I would have thought it that was wrong, it would ALL look wrong - but once again I really have little understanding of what tangent space means in practical terms.
I assume the last part doesnt count if I have no UVs using the same space...
If you go back and look at the normalmap I posted at the start you can see the UV space for each part of the model. The four big pieces are the chest, back and 2 arms, the 2 medium ones are the legs and all the littler pieces are head, feet, fingers, toes, tongue etc.
Looking at it now one thing I notice is all the pieces rendering incorrectly are (I think) also the pieces lying on their "side" in the UV space - the left arm (bottom right UV space) the two legs (middle-right UV space) and the back of the shorts (far left)... but why on earth should that make any difference??! I always thought orientation in the UV space was irrelevent.
You could also try what Vailias suggested, just invert red or green or both on the flipped islands, see if one of those combos fixes it.
Islands being on their "sides" vs. not, the tangent basis is used to fix seams between these different orientations. Different issue than your large shading errors; this is about fixing seams between islands. There's a nice pic here that helps explain it.
http://wiki.polycount.com/NormalMap#Tangent_Basis
But... the baking tool and the game engine need to be synched, they need to use the same tangent basis for the seams to be solved properly. Take a look at Handplane, it was created specifically to address the mismatched-tangent-bases problem. Official handplane support thread
However whilst testing these different normalmaps I noticed something else that I hadn't seen before - on the parts of the model I had thought were rendering correctly as the lighting source was moved around from certain angles the normalmap was also extruding in the wrong direction. Below is a 4 frame .gif showing the model under different lighting angles (same normalmap for every pic) and you can clearly see that from some angles the veins look extruded and from others indented:
This lead me to suspect that maybe the rest of the normalmap wasnt as ok as I thought it had been. Anyway, what I did in the end was to take my original normalmap and rather than isolating the parts I thought were incorrect, I simply inverted the GREEN channel accross the whole normalmap. This fixed the parts that were rendering incorretly before and some testing seems to suggest it has also fixed the problem of the normals rendering incorrectly in certain lighting conditions.
I wonder if this means that in the end the problem is simply something between the way Mudbox and Source calculate noramlmapping which require the green channel be inverted for things to work correctly. Maybe it wasnt anything that I did "wrong" when making my model/UVWs after all.
Once again, thank you all very much for your time and help... thats a really great article about normalmaps you linked me to Eric, I skimmed through it and will definitely read it all carefully again a couple more times.