UDK and Max are fighting over my normals...

GarageBay9

So I've been fighting with mapping, baking, and finally painting a modular shipping container (I'm building it so different sections of the walls can be cut out). I built a simple hi-poly, got my low res, baked in Max (because xNormal just would NOT produce anything remotely usable - warping, stacking and artifacts everywhere), and did a little bit of tweaking to clean up my normal as viewed with Xoliul's shader in the Max viewport. Thought I had it more or less where I wanted it (for the moment), so I moved on to my base diffuse texture. Grime and markings are going to be overlaid in the shader so I can switch colors, markings, and wear dynamically.

Here's the CONEX container in Max, via Xoliul, with the diffuse and normal. I know there's some issues, but most of the edges on the main framing (not the cubes on the end) are good enough for now, and give me a decent idea of the edges with the normal map. I'll make a cleanup pass later.



I realized that I had my diffuse good enough for a rough pass, and was going to start moving to UDK so I could preview the assembled triple-layer shader as I worked. Model imported fine, got my lightmap assigned (it's UVW channel 3, 1 and 2 are for the diffuse base color and side markings, 3 serves double duty as both the map channel for grime and for lightmapping).

Import normal map, plug into the shader in UDK and MRRPHHH :: gakk cough :: Wh... WHAT THE HELL IS THIS CRAP?! Somebody shanked my normals!



I'm pretty sure I accounted for -Y green in any minor hand editing I did, and I know you're 'not supposed to bake from Max' (and I wouldn't have, if xNormal had spit out something even REMOTELY decent...), but I have never had my results in Max diverge this massively from what I got in Unreal. There is some horizontal mirroring of the normals in the UV map, but my understanding was that horizontal is ok, vertical is bad.

And I was supposed to have this and several accompanying objects done by the end of Sunday.

I've fought with this thing for a couple hours now and I'm at a loss. What the hell is going on? I'm going to post the model, my textures, heck, I'll even post the .ase and UDK .pak file if that'll help. I just have to make this work. :poly122:

EDIT: .max file, textures, and source texture .psd: http://www.garagebay9.com/storage/40ft_Conex.zip

gsokol

When you import your texture into UDK - make sure that you are changing the Compression Settings to "TC_Normalmap" like so:



If you aren't already doing this that could be the problem.

GarageBay9

Pretty sure I got that part, but I'll double check.

EDIT: Yup, set to Normal compression and LOD group is WorldNormalMap. It's like something's scrambling the X and Y directions in various spots, and tossed out the Z depth for good measure...

Xoliul

I'd expect it to be some issue with your .ASE exporting. Looks like you have unified normals in Unreal.

GarageBay9

Xoliul wrote: »

I'd expect it to be some issue with your .ASE exporting. Looks like you have unified normals in Unreal.

Well, I just tested to make sure it wasn't unified, and apparently it was. However, I'm now getting different seams (some of the edge seams have changed, a few are right, others are wrong still or wrong differently), and even worse, blatantly incorrect mirroring along the tiled areas.

Ged

has this ase fix made your normals pop out like they should though? I think I may have the same problem in my udk level. Is it easy to replace an old ase with a new one in udk, I think it kept telling me the file was in use and that I couldnt delete/ replace it so I keep having to add new files when I make an edit and then place said new files in the level all over again instead of just updating whats already in there

GarageBay9

No, it hasn't popped the normals out any better. That's one of the first things I noticed - the normals for the wavy metal sheeting of the walls seem very, very flat. I thought it was the skylight and Lightmass, but now I'm not so sure. Didn't have this problem before I went from UE3 to UDK...

Xoliul

The problem you have now is the typical UE3 normal mirroring problem. This has been covered before, though I don't really have experience with it.

feanix

You can also trying boosting the normals in the Material Editor:

http://imgur.com/rpNvw.jpg
http://imgur.com/HcMYr.jpg

BlackulaDZ

I've found that using lightmass washes out your normals a decent amount; that normals multiplying trick works wonders imo

kdm3d

Heres the trick. I've found on anyhting that is mirrored, the normal map has to be uncompressed. There is something about the way UDK compresses them that breaks the normals. tc-normaluncompressed I htink it is.

ALSO, I've found that if you are using a material instance, any normal maps in the parent material have to be uncompressed, mirrored or not. other wise it breaks the mirrored one on the child level.

GarageBay9

Thanks, guys! I'll do some testing here with these suggestions and see what I get. That compression issue sounds especially promising - thanks, kdm.

GarageBay9

Ok, more frustrating discoveries in Max. Some faces appear to be transposing their vectors (swapping R and G) or getting vectoring in directions that don't have any data. For example, this. The small chunk of normal is the normal data that selected face is mapped to, with a 1 pixel buffer. There is NO data in the Red channel for that face, but the normals almost look like they're inverting along a gradient from one end of the face to the other. I'm completely stumped. This is Xoliul's shader in Max 9.

PredatorGSR

I don't know if it relates to your issue or not, but a 1 pixel buffer is usually not enough. Stuff gets downsized and compressed, and that 1 pixel buffer will blur together real quick, especially in UDK. Also, even at true res, if you bake your normal map to a 2k for example, and the uv shell doesn't take up enough space on that map, you will get seams. For example, if the edge needs 1 dark pixel and 1 light pixel for each side of the edge, and your uv shell is small enough that there is only 1 pixel, it blurs together and can't calculate the seam. This is especially bad in xNormal, I usually bake a 4k map if I want a 2k, and downsize it because the accuracy isn't good enough.

GarageBay9

I was just describing that I copied and pasted the mapped area plus one pixel around the edge, so if anybody was looking at that sample, they would know there's an extra pixel of stuff around the edge that's outside the UV coordinates for that face.

kdm3d

yeah, transposing the normal direction is an artifact of compression... I ahd the same problem. That should fix it IF its coupled with enough edge padding. Depending on the size of your object, your mip levels could be affecting that too, so make sure there is enough padding so when it mips, its covered. A good rule of thumb is to hae your LOWEST mip have 2 pixels of padding. so if your lowest mip (4 levels I believe) is a 128, you'll want to have 16 pixels padding on a 1024. You can cheat this, but its a start.

GarageBay9

...but there isn't any compression. That's with an uncompressed 24 bit TGA via Xoliul's shader in Max9. That's what's got me stumped...

EDIT: This, um, appears to be a bug with Xoliul's shader. I've switched to a different SM3 viewport shader and not only are the strange gradients and inversions gone, but the normals are displaying much more accurately.

Looks like time to upgrade Max...