Yet another normal-baking issue

leechdemon

I'm doing some Normal Map baking in Max, and I'm having an issue with the UV seams turning rainbowey.

First off, I have smoothing groups in here, and no, I'm not trying to bake across the gaps. What I was hoping to do was to bake a smoothed surface on either side of the edge, but instead what I'm getting is a line down the edge. So I took my UV shells and split them apart where the smoothing groups were. Now I'm getting a rainbow bake (looks like a sphere in tangent space) on all of my UV shell borders. If it was off the edge of the mesh I wouldn't care so much, the problem is that my UV shells are on the middle of the mesh, right where the Smoothing Group splits are. The only hint I have is that I'm getting ray misses along the borders. They're small, but I'm noticing them.

I tried turning on Global Supersampling, upping the Anti-Aliasing, increasing the scale of the mesh. I made a test bake using smoothing groups and I didn't have the rainbow issue, but I can't figure out what's different between the test and my actual mesh.

Any help?

throttlekitty

Did you try scaling the UVS? There's little room for pixels in there.

leechdemon

I did, actually; I tried scaling the larger ring so that it was about 3 or 4 times larger; it ended up with the same effect. It's not being resolution-bound, because the pixels are of the wrong color, not just a blur.

16bit

Rainbows are good, but only when you use 1 smoothing group. Also try to get it on 1 uv island. Your rainbows are caused by smoothing groups and the uv islands.

Actually maybe not, what program are you using to bake?

leechdemon

16bit wrote: »

Rainbows are good, use 1 smoothing group, try to get it on 1 uv island.

Well, the issue is happening on the body of the model, which is a car. I've yet to see a car mapped to a single UV shell, but I'm open to suggestions.

As for one smoothing group, the goal of this mesh was to have the vertex normals do the brunt of the work. Everything is mapping correctly, except for the UV seams. If I made everything one smoothing group, I still get the problem, I just get it on the outside of the smoothing group. Are we saying that all normal maps do this, and I've just never noticed?

16bit

Does your car have individual paneling? Or all one mesh?

leechdemon

It's modeled as one piece, but split along the body panels to break it into UV shells. I also split the mesh along Smoothing seams; the high poly has this as well, so I'm not worried about getting false gaps or anything. I want the smoothing the way it is in the low poly, I just wanted more detailed normals throughout the interior of the shape.

16bit

If it's all one piece you should definitely try to get it on as few uv islands as possible. Then apply smoothing groups on each of the uv islands. You can fit an entire car on 1 uv island.



This car has 2 uv islands, one for the top and another on the bottom. This might not work for you depending on how many polygons you have.

leechdemon

Well without seeing the actual UV's that doesn't really explain much. Also, my question is about baking a normal map, not about the actual Smoothing Groups. Any idea why they're baking the way they are?

To reiterate, I'm basically getting a gradient on the outside of my UV shells; not inside the shell, but the part that's getting Mipped. So my surfaces are all fine, except on the edges where they blend horribly.

EarthQuake

If you're not using the actual projection cage in max ie: using "offset" instead, you will get gaps in the projection where you have hard edges/smoothing groups.

16bit

Well smoothing groups affect the way normal maps are created.



The quirks of the normals at the uv shells could be from the edge padding.

EarthQuake

16bit wrote: »

Well smoothing groups affect the way normal maps are created.

Only if your bake is set up incorrectly should smoothing groups have an effect on your end result. If we follow these rules, the results should be the same with/without:

A. Split uv chunks where hard edges/smoothing groups are
B. Make sure projection cage is averaged

Putting everything on 1 smoothing group may cause other problems, especially if you're exporting to an engine that isn't synced to your baker, like smoothing errors.

The quirks of the normals at the uv shells could be from the edge padding.

Yes but in this case, the results are showing up on the actual model. The texture itself looks worse than it is because of padding, but the padding isn't to blame, its simply skirting out the data(which is the problem) that is on the edges.

leechdemon

@Earthquake - I'm using a cage, but I'm not doing anything fancy with, just Reset and a very small Push until I stop getting ray-misses. The shell doesn't follow the smoothing group's normal, though, it's following the normal as though the group didn't exist. Are you saying I'm getting a gap, even though the high poly has this same gap? I'm not trying to bake a curve onto a smoothing group, just a smoothing group onto itself. I can push the cage to one side or another, but I'm guessing it'll just move the problem to the opposite shell. Just googled and found http://www.polycount.com/forum/showpost.php?p=1129597&postcount=32, should I be doing that to preserve hard edges?

leechdemon

EarthQuake wrote: »

A. Split uv chunks where hard edges/smoothing groups are
B. Make sure projection cage is averaged

A - Figured that one out the hard way. Then went back and split all my edges by hand (I'm guessing there's a tool for this someplace that I'll have to find). This gave better results, but still the ring around each shell.
B - Based on your other post (in the link), it sounds like Max has it averaged by default... which you're saying I want? So then when WOULD you unweld the cage? And if that's not my solution, what should be?

EarthQuake

What does your high look like? You mention your high has a gap?

You really should never bake without a proper averaged cage, its just going to cause you problems. So if you're not doing anything weird, it shouldn't be the projection cage that is the problem.

RE: A - Make sure you've not only split the edges, but provide enough pixels in-between each island so you don't get errors. From your uv layout it looks like you've done this though.

leechdemon

I basically took me working model and applied turbosmooth to a clone, with Smoothing Groups on. So the high looks pretty much the same, except the areas on the inside of the shells are much smoother. My goal in baking was to fix THAT, not to get soft edges across my smoothing groups.

I'll post some more pictures when I get to the office.

leechdemon

Alright, 4 screenshots:

A screenshot of my High-Poly mesh that I'm baking from. It's essentially the low-poly, complete with smoothing groups, that I added a Turbosmooth to to increase the resolution of the normals. I'm not expecting the normal map to do anything but that.

Next, my low/high and the cage I'm using. I don't know anything about cages (I baked in xNormals without a cage until recently, now I'm baking in max with a cage). All I'm doing is resetting and Pushing slightly. The hard edges aren't preserved in the cage... They're still Welded. I'm not sure if this is correct or not; I'd assume that if my normals aren't being averaged, I wouldn't want the cage to be either?

Also included my UV's and the proceeding bake. The majority of the bake is good, except for on the smoothing group seams (which are currently the UV seams as well... this is how I want to do it, correct?), where you get the rainbow strip right where the padding should start.

During the bake I get ray misses on my borders, so I'm thinking this rainbow has to do with that? But the high/low poly meshes are identical, with the exception of the turbosmooth; Are these coming from the gaps created by the smoothing? If so, how do I avoid this?

leechdemon

Bump?

cw

Best if you share up your scene or part of it and someone (me?) might have a look. It doesn't appear that what you are trying to do is out of the ordinary in any way, just might be some small things you are doing strangely which I cannot spot from this distance.

Noors

Breaking the uv is needed because of mipmaping. I don't see the interest of it if you don't add a proper space between your shells (depending of you anisotropic level, but at least 4 pixels). Add more space or you'd better keep the edges welded. imo you've splitted it way too much, having inner gasps thick of few pixels only.
This is probably not related to your outline artefacts though.

leechdemon

@Noors - I broke the UV's because of the mipping issue, but I didn't realize I needed to until after I unpacked it. I was planning on repacking if needed once I got the issue fixed, but never got around to it. Some places on the UV's suffer this worse than others, but the problem isn't better or worse corresponding to those. Good advice, otherwise.

@cw - I couldn't find the file I actually baked from, so I started to set up a test using an existing file.... and the problem went away.

To prep the file for cw, I changed UV channels (I was on UV2, I moved them over to UV1), reflattened to Poly, rebuilt my Bake mesh, and reexploded my mesh. Nothing else was changed, except now my results give me a clean bake. Is there a problem with baking normals onto UV2?

Thanks for everyone's help with this... the problem seems to be fixed, but I have no idea why. Any advice I could get in avoiding this in the future would be hugely appreciated... /baffled..

cw

Try baking to uv2 for a test and see if it works this time.

Sometimes the node properties can get corrupted and cause various weird things to occur.

If it happens again with another file, repost with the file etc. and I'll have a look. Hopefully it was just a strange glitch and you'll be fine.

leechdemon

I tried baking on both channels, and the first test I did got a bad result, then every bake after got the correct ones. I even reloaded the file to see what changed, and it's coming up correct from the first bake, now.

Not sure what that's all about... I'll try and run the whole thing and see if it works correctly, or if my issue resurfaces.

cw

IT shouldn't matter, but sometimes max does something funny with the normals of the object. In order to clean them all up, you can convert to mesh, collapse, convert to poly, collapse, and then pop an edit normals on top to check it is all reset.

Otherwise I'm not sure what was up, still you have it working now so that's the main thing.