Shading artefacts only in Source engine (normal map related?)
polycounter lvl 9
UPDATE:
This issue has been solved and had nothing to do with textures or meshes, in this case the problem was caused by a file called sharedbones.qci, after replacing it with another version randomly found, the problem was fixed. more info in this reply: http://www.polycount.com/forum/showpost.php?p=1764090&postcount=30
Hi,
I wanted to ask about normal mapping in the Source Engine.
When I bake and texture my models I usually preview them with Xoliul2 or in tools like http://p3d.in or Marmoset Toolbag. On all this programs, I get my models to look nice, no shading problems, and normal maps look good, they have depth to them and add a lot of detail to the low poly.
The problem comes after exporting to the Source Engine (Left 4 Dead 2 particularly). When I test the model in-game, my normal map seems to be much more plain, like it was applied with 50% intensity or something. And I also get some shading artefacts.
I've actually tried multiplying the normal map (duplicate layer with blue channel leveled and overlay blending mode), that adds more depth but doesn't get rid of artefacts or makes them more noticeable.
Here's a comparison of my model in p3d.in and in Left 4 Dead 2:
And you can actually see the model oline at http://p3d.in/Xwhrg
In case it helps, here's my VMT material:
Any other relevant details i can think of:
I'm sorry for making a long post like this, but i wanted to give all details you may need.
Any help is greatly appreciated.
Thanks!
This issue has been solved and had nothing to do with textures or meshes, in this case the problem was caused by a file called sharedbones.qci, after replacing it with another version randomly found, the problem was fixed. more info in this reply: http://www.polycount.com/forum/showpost.php?p=1764090&postcount=30
Hi,
I wanted to ask about normal mapping in the Source Engine.
When I bake and texture my models I usually preview them with Xoliul2 or in tools like http://p3d.in or Marmoset Toolbag. On all this programs, I get my models to look nice, no shading problems, and normal maps look good, they have depth to them and add a lot of detail to the low poly.
The problem comes after exporting to the Source Engine (Left 4 Dead 2 particularly). When I test the model in-game, my normal map seems to be much more plain, like it was applied with 50% intensity or something. And I also get some shading artefacts.
I've actually tried multiplying the normal map (duplicate layer with blue channel leveled and overlay blending mode), that adds more depth but doesn't get rid of artefacts or makes them more noticeable.
Here's a comparison of my model in p3d.in and in Left 4 Dead 2:
And you can actually see the model oline at http://p3d.in/Xwhrg
In case it helps, here's my VMT material:
VertexLitGeneric
{
$baseTexture "models/weapons/melee/fasthawk/difuse"
$bumpmap "models/weapons/melee/fasthawk/normal"
$phong 1
$phongboost 1
$phongexponent 1
$phongfresnelranges "[.1 .6 .65]"
$normalmapalphaenvmapmask 1
$envmap env_cubemap
$envmapfresnel 1
$envmapFresnelMinMaxExp "[0 5 .4]"
$detail "models/infected/hunter/hunter_01_detail.vtf"
$detailscale "1.75"
$detailblendfactor .001
$detailblendmode 0
$detailAmount 0
Proxies
{
BloodyHands
{
resultVar $detailAmount
}
Equals
{
srcVar1 $detailAmount
resultVar $detailblendfactor
}
}
}
Any other relevant details i can think of:
- Model is done and baked with 3dsmax 2012
- Green channel is set to down (Y-)
- I compress the textures with DXT1 for difuse, DTX5 for normal map
- Normal map texture has specular in alpha channel
- VTF files are compiled from TGA with VTFEdit 1.3.3
I'm sorry for making a long post like this, but i wanted to give all details you may need.
Any help is greatly appreciated.
Thanks!
Replies
Does it have the same problems in the Model viewer and does anything look funky with your model normals or normal map there?
Also, make sure you're saving your Normal Map as 32bit so your Alpha slot can be read.
Thanks for the reply.
Unfortunately I'm quite sure the green channel is right because the model looks as intended in the previewer (you can see for yourself here http://p3d.in/Xwhrg) but if I flip it, it generates different artefacts than in the game, and looks even worse.
I'll test flipping green in-game again just to be sure but I believe I tried that with a previous model with out luck.
Hi Ace-Angel,
Thanks for replying.
I have tried all of those too and no luck, i'm starting to think maybe something is wrong when i export the mesh to Source, because i'm comparing my textures and normal maps to those in other mods and original game models and everything looks the same, except my normal maps look poor in-game.
More over, flipping the green channel in-game didn't change all that much, so it still looks to me like the normal map is not doing it's full effect, but i can't explain how would that be possible. :S
Good luck!
There is something wrong here, you can very clearly see the dents and other 'little holes' going out instead of in, the entire part of the axe in the font is curved inside because of this, something if flipped in the normals.
Flip the green or red channel manually in PS or in your baker app when baking, don't do that in the engine since most engines will do only a 'side flip' and not a full flip on your channels.
Before baking and exporting, make sure you Reset-XForm and the pivot of the mesh. Make sure all your Normals ON YOUR MESH are unified and points correctly outwards, and that you don't have them inverted or double face-rendered.
Also, if you want 1:1 bake, last I checked, Source uses XNormal's default bake setup.
I have taken a look at this thread http://www.polycount.com/forum/showthread.php?t=107420 and i was also getting the idea i should try rebaking from xnormal, i'll give it a shot.
I'm also a bit paranoic about Reset XForm, and use it quite often in 3ds max to avoid problems. I also use xViev to check for mesh errors and things like that.
I have just checked unifying normals and nothing changes so i take it they are correct in that sense.
Will try baking from XNormal next.
Edit: Tried XNormal for bake, still the same result..
https://developer.valvesoftware.com/wiki/Bump_map
It says about checking the flag that the texture is a Normal Map, and it sounds like you need to normalize it before importing? You could also try Uncompress the texture, since this usually means reconstructing the blue channel in the engine, which could lead to errors?
I always turn the "normal map" flag in VTFEdit on when i save a normal map, although tbh i don't know what really changes.
I tried uncompressing it and it looks better as expected (normal map looks better in general) but the shading artefacts persist.
If by normalize you mean the vertex normals in the mesh, i made sure those are normalized too, apparently they were in the first place becasue pressin normalize didn't change a thing.
Next i'll try compiling the normal maps file with Vtex instead of VTFEdit, if that doesn't help i guess i'll take an extensive look at that wiki starting from https://developer.valvesoftware.com/wiki/Model_Creation_Overview ...
And thanks again for all the suggestions!
For the spacular to show in the first place i have to save the normal map without alpha, since its alpha channel is being used as specular channel and is low to fit in-game.
Now i'm even more clueless since i don't know where else to look..
respawnrt: Fresnel ranges of 1 1 1 are hardly realistic for any surface. Generally you'll want something very low in the centre and higher as it goes towards the perpendicular edges, so those fresnel ranges are fine.
If anyone else wants to take a look, feel free to do so.
Farfare: I have just tried Handplane, not sure if I'm using it right but I get literally the same normal map.
What I did:
1) Went back to 3dsmax and bake object space normal with default settings and Y+
2) Go to handplane, load lowpoly FBX, object space n/m, and select Source as target engine
3) At this point I have to set Input green channel to "Identity" or it will render green and red where it should render blue on most surfaces.
4) Then I get the exactly same result as if I baked tangent directly in 3ds max. I compared in Photoshop and both looks equal except for the added padding.
--
I'm also confused by the fact the half life model viewer renders the exported and compiled model correctly without artefacts, yet it only fails in-game. Wouldn't that rule out normal map or mesh problems and point to the VMT or something like that?
Again thanks everyone for the tips, I haven't found the problem yet but I've definitely learned some interesting stuff already..
Then put a brightness/contrast adjustment layer on top of that and crank up the values.
You might see small - but important - differences in the normal maps. Even those tiny differences can make fairly noticeable flaws in the surface when applied as a normal map.
Regarding the HLMV shot; You can still see flaws in the normal map there. Especially on the flat sides of the axe head near the front. I don't think there's a difference between HLMV and the game other than the lighting in the game shows up the flaws more.
Thanks for the reply.
Yes, the model was triangulated before baking, hoping to avoid flipped edges causing problems.
I was thinking of an inverted channel too when I first saw the problem but I've tried a couple of times inverting the green channel. I guess I'll give it another round of tries inverting both green and red channels in the resulting normal map from Handplane, to be sure I rule out options.
If that doesn't fix it, I'll probably redo about everything:
- Move high poly and low poly to a clean 3dsmax scene,
- Recheck for topology with xview/stl check,
- Reset xforms and pivots and collapse modifiers,
- Redo the bake in object space,
- Run it through Handplane,
- Export mesh/normal map to source SMD & compile
(let me know if i'm missing anything)
Also sorry for repeating myself but note that the picture in http://www.polycount.com/forum/showpost.php?p=1762646&postcount=11 is from the compiled source model, taken from HLMV, and looks good from the beginning, it's only in-game where the problems surface. I have no clue how HLMV differs from the actual game engine in terms of shading, though.
Make a tangent space map with these settings, if it looks really wrong then hit the flip y/z option but not the others.
But when I applied 1 sm group to whole model, exported into source, everything looks good, cheers:
(hlmv shot)
About the smoothing groups, it might be that i didn't export them to the .obj or .fbx, but the model is smoothed for me in max.
AlecMoody: Regarding Handplane exports, i'll probably try again tomorrow with a clear head and if i don't get any good results i'll post pics showing what handplane generates with/without the flip y/z option.
Thanks again to everyone for the support.
Maybe there is an issue with the export process? What format are you exporting in and what are the boxes you ticked? If you're using FBX, I think you need to change it to Binary and change the Z/Y is Up format.
Again, take what I say with a hint of salt, since it's been ages since I last used Source, and L4D2 changed alot of stuff since then.
Thanks Ace-Angel,
I export to .SMD with a couple of plugins, there are two: one from wunderboy.org and the other from cannonfodder. I have tried both and got the same results, though.
If tomorrow I can't figure anything out and redoing the whole process doesn't help, i think I'll try luck on Gamebanana, since there's quite a source modding community there too..
By the way are there any manuals of PDF's for Handplane? I just want to make sure i'm not missing anything obvious when converting maps with it because so far i've gotten really strange results with it, and if that works as feedback/testing for the developers, all the better.
anyway here is ingame shot
I'm wondering if it might have to do with l4d2 engine because your export does look perfect.
Could you let me know how you changed SM groups exactly, did you go to the mesh, select all faces and set them to SM Group 1 or did you do it differently?
Also what exporter did you use?
I tried both wunderboy and cannonfodder and both gave me the seme result.
If you could share the max file you used to export that might be interestig.
--
Regarding Handplane, the problem was the object space map generated by 3ds max, it was completely broken. I baked it with XNormal and handplane didn't give any problems at all.
I dont have max file as I did rly save it, but I might send u .smd?
Only the viewmodel (the first personview of the weapon) has the problem. The worldmodel is ok even in-game. They are exactly the same though, same mesh, textures, and material.
I've noticed the HLMV tool does show the problem in the viewmodel, so it doesn't seems to be exclusive to the L4D2 version of the engine or anything like that.
I'm taking a look at my .qc files now (used to compile the models), removing lines, changing stuff, making the V_ one look more like the W_ one, so far no changes.
In case anyone wants to see them:
v.qc
w.qc
Edit:
After testing line per line, it has something to do with the sharedbones.qci file, an include that lists bones for the arms of the player. Apparently each weapon needs this file to be adjusted accordingly for it's animations or something like that.
My current sharedbones.qci file should be for the katana weapon in left 4 dead 2, although i'm not sure.
I'll update when i find anything else or figure out how to fix this file on my own.
Finally I got it to work by using some random version of the sharedbones.qci file I found lying around. *sigh*
And the screenshot of victory: