smoothing groups, normal maps, and UDK hate me... help please.
polycounter lvl 7
Hi all, I've searched extensively to help me fix my problem but to no avail, so I figured I'd bite the bullet and create a thread. I have a few modular sci-fi wall piece I am using for a current environment of mine, but I am having some weird smoothing group issues either in my normal maps from when I bake from HP mesh to LP. At least I think it's smoothing groups. Long story shot, my workflow is this :
Create my high poly mesh form a low poly block out by adding my control edges and changing geometry as needed to get the high poly look I need.
Subtract control edges and optimize accordingly to create my low poly mesh.
Unwrap
Set my low poly mesh to one smoothing group ad bake out the normal map.
Apply my normal map in max and make sure everything looks good.
AO, diffuse, spec, etc are then created.
Needless to say once I put my mesh into UDK I am getting some weird splotching on my mesh as well as the lighting is bending weird on flat surfaces as if they were not flat. Sry if this is confusing, I am really confused myself. If anyone needs any more info please let me know and I will accommodate to help get this problem fixed. I am overdue on my deadline for this scene because of this problem so I am trying to get it finished asap. Thanks again polycounters! Here are some screen caps to iterate on my problem.
So firstly, everything looks great in max, yayyy!!!
Then I put into UDK and I get this...
And here is one of the meshes in the mesh viewer. As you can see eve in preview light is acting weird on flat surface a if it were not flat.
I am guessing I am doing something wrong with my LP mesh before baking because I do see where in my normals where i know a wall is flat I still get color gradients and weird pinches as if it were baking from my LP with one smoothing group on. Here's one of the normal maps for the wall piece that's giving me grief. Also note that this normal map looks great in max, but I get the weirdness in UDK.
Again, any help is appreciated. cheers all!
Create my high poly mesh form a low poly block out by adding my control edges and changing geometry as needed to get the high poly look I need.
Subtract control edges and optimize accordingly to create my low poly mesh.
Unwrap
Set my low poly mesh to one smoothing group ad bake out the normal map.
Apply my normal map in max and make sure everything looks good.
AO, diffuse, spec, etc are then created.
Needless to say once I put my mesh into UDK I am getting some weird splotching on my mesh as well as the lighting is bending weird on flat surfaces as if they were not flat. Sry if this is confusing, I am really confused myself. If anyone needs any more info please let me know and I will accommodate to help get this problem fixed. I am overdue on my deadline for this scene because of this problem so I am trying to get it finished asap. Thanks again polycounters! Here are some screen caps to iterate on my problem.
So firstly, everything looks great in max, yayyy!!!
Then I put into UDK and I get this...
And here is one of the meshes in the mesh viewer. As you can see eve in preview light is acting weird on flat surface a if it were not flat.
I am guessing I am doing something wrong with my LP mesh before baking because I do see where in my normals where i know a wall is flat I still get color gradients and weird pinches as if it were baking from my LP with one smoothing group on. Here's one of the normal maps for the wall piece that's giving me grief. Also note that this normal map looks great in max, but I get the weirdness in UDK.
Again, any help is appreciated. cheers all!
Replies
What do your UVs look like for both your first and second UV channels? I've seen issues in the past where a messed up 2nd UV channel can actually cause your issue in both the generic browser and when baked as well.
Reset XForms pre-bake on the low? Made sure there aren't any extra bastard verts that aren't welded properly pre-bake?
etc. If you swear by Zeus these things aren't the issues, email me the .max file and i'll take a peak.
I think Ott is probably right about the 90° angle thing, even if you don't have 90° angles sometimes it's just too much of a change in surface direction for the normal map to handle in UDK. Try separating your mesh with some smoothing groups and re-bake the normal map.
You could also try adding an extra edge loop just next to the indent on the flat panel and then re-bake. That would reduce the amount of color gradient on the panels themselves.
If you do use separate smoothing groups you'll have to separate the UVs as well. Make sure to check out the Normal Maps: Can Somone Explain This "Black Edge" issue thread.
Or maybe:
Make sure the triangulation of your exported to UDK mesh matches the triangulation of your baking normal map mesh. If the triangulation doesn't match up then you will experience have some bad shading errors as well.
I hope that helps.
#1. Be cautious of looking at the UDK generic browser as an indicator of shadow and lighting quality. It's never going to be great.
#2. Your lumpy "sub-surface" normal map is probably what's causing a lot of the excess noise that just looks bad for a large-scale object of this size. Even smaller objects have a hard time getting much detail frequency from that sort of stuff, and is why most of us use a tiling detail normal map as part of our material instead. Those super-fine small details for a large room / wall like this would have a hard time looking good.
#3. As far as the majority of the lighting issues and odd seams, UDK can be somewhat unforgiving on large flat surfaces that suddenly change angles drastically. The "no 90 degrees" rule is a good one to follow, but a series of long giant polygons and suddenly changing to a 88 degree angle can be just as bad.
It is usually unnoticeable for small props and world objects, and after texturing is generally ignored. But for bigger surfaces like this, you can either tessellate the hell out of it and add a bunch of additional "support" edges so that the bending isn't as noticeable, or simply add more chamfers. I would generally do a conservative combination of both.
Take your low poly, and for those long channels, chamfer them again until the smoothing on the low poly is improved. The normal map will attempt to compensate as best as possible, but for larger objects like these, adding a few extra tris to compensate is acceptable.
#4. Your very far back top corner is 90 degrees sharp. Would need to clean that up.
#5. Near the bottom of the piece, you have an odd / out of place diagonal cut that can also cause issues. Do your best to keep your cuts up and down on flat surfaces like these.
#6. Lastly - a perfectly symmetrical modular piece like this is a great example of something that could be cut in half and then deco'd with mesh paint / decals / projectors....therefore improving your texture rez and allowing you to add a few more tris to smooth out these issues.
Just some thoughts!
I've learned that if I'm going to bake lightmass onto something with large smooth areas, get it looking close to right before the normal map is applied (within reason of course :P). This has a bonus of allowing you to use flat normal maps for details, as seen here http://i.imgur.com/d3wCY.png
That prop uses a flat blue normal map for most of it, the smooth edges are a mix of support edges and chamfers. I tried and tried, couldn't get it looking right with a normal map otherwise
Good luck, looking forward to this scene!
If you can see clear shading gradients on flat surfaces you need to add a smoothing group to clear it, create UV seams along the borders of the region and then create some bleed space between the new island and its neighbours. also you will need to bake with an averaged cage and not ray distances.
Alternatively you can add bevels to the sharp edges. these have the advantage of giving nice silhouette shapes and better shading at grazing angles)
I wouldnt bake the surface noise into the normal map. you can add it later in UDK, You'll get a much sharper texture this way as the noise isnt limited by the texture resolution.