Blender -> XNormal Baking Issues

PlateCaptain

Hi everyone,

I've read a ton of threads here on this topic, and I'm still having some issues baking some simple normal maps in XNormal from Blender. I'm hoping if I post my own examples and particular workflow, I might be able to get some more specific advice.

I'm starting with the high poly on a simple test object. I have Bevel and SubSurf modifiers set, and Apply Modifiers checked in my export options.

Next, I duplicate my high-poly mesh, remove those modifiers, mark my seams anywhere there's a hard angle, and unwrap my mesh. I also set up a Triangulate modifier to be applied on export. UV Layout and object with seams marked look like this:





Next, I duplicate my low poly mesh and scale everything outward a bit for my cage. I then export all three objects using the following FBX settings. I've tried exporting as OBJ as well, with similar/identical results.



These are my settings in XNormal:



Here's the normal map that results:



And here's how it renders out with a white diffuse shader and the normal map:



I'm still seeing sharp edges where my UV seams are, and I can't seem to figure out a way to smooth those out. I've seen people suggesting using the Edge Split Modifier and/or Sharp seams and/or smooth shading, but everyone seems to have different opinions on which (if any) are necessary.

Can anyone see or suggest anything that might help me smooth those seams out?

billymcguffin

What does the High Poly look like?

PlateCaptain

High Poly:

respawnrt

Seems to me like you are not doing anything wrong, that seam is always going to be visible if you zoom in, so the solution usually is to add a chamfer (in the low poly) if you want something better. Also try the opengl viewport as well to check it out.

With that said if you add a chamfer and redo the uv you'll also want to update the hard edges to uv seams and the most simple way to do that in blender is by going into uvs / seams from island / mark sharp (this will automatically mark seams as sharp).An aditional step will be to set the shading to smooth (at this time you wont see hard edges in viewport anymore but the data is still stored when you export)

After that in order for the map to be displayed correct in the viewport(opengl), you have 2 options since blender 2.71 :
1. Add edgesplit modifier and tick sharp edges
2. Go into object data and turn autosmooth on and you wont need the edgesplit modifier to see the hard edges.

mAlkAv!An

Simple answer: use a cage for baking.

With no cage there always will be projection errors whenever the low poly has hard edges.
http://wiki.polycount.com/wiki/Normal_map#Working_with_Cages

respawnrt

Of course, but OP already used one if you read his post.

PlateCaptain

Thanks respawnrt. A couple clarification questions, if you don't mind:

1) If I add a chamfer (same as a bevel, in my understanding) to the low poly, I would end up with two mesh edges for each edge of the object. Does it matter which one I put the UV seam on? Is it better to then cut the bevel in half and put the UV seam there?

2) Do I need to set only the low-poly to smooth shading, or the high-poly as well? If the high-poly needs it, I assume it also needs the sharp edges marked...?

3) Is the Edge Split modifier necessary for the baking, or only for displaying properly in the viewport?

4) Is Autosmooth different than setting the object to smooth shading? If so, what's the difference?

Thanks again for your help. I can try your suggestions when I get home tonight, and I'll update with the results.

EarthQuake

Not a blender guy, but at some point blender would export hard edges/smooth group splits as broken (detached) edges rather than mesh normals. This would cause gaps where baking unless you export a cage mesh from blender as well. Don't know if this is still the case of if they've sorted out mesh normal exporting yet.

JedTheKrampus

Don't use edge split modifier in 2.73. Instead, you should shade your whole mesh smooth, check Auto Smooth, set the angle to 180 degrees so you never get a hard edge that you didn't ask for, and place hard edges as needed. Auto Smooth allows you to have hard edges and custom normals without splitting the edges manually, which can be important for making the cage. Basically look at it as a "please let me use hard edges" checkbox that lets you export things correctly.

Set low-poly and high-poly both to smooth shading, and make sure you have auto smooth checked for the low-poly mesh. If you're having cage problems try making the cage with the xNormal 3d viewer by adjusting the global cage extrusion parameter. You should be able to see if your cage is split where it shouldn't be if you just push the global cage extrusion up really high.

As far as chamfering goes, it depends on what renderer you're targeting but I often prefer to use a chamfer that's two polygons wide rather than just one. On a modern graphics card running a modern deferred renderer that properly stripifies meshes the performance difference is minimal, and you get less skewing error in the bake, and it's easy to figure out where the UV seam should go. If you're rendering offline with Cycles the extra thousand triangles on your mesh matters even less. Everything is situational though, and if you're targeting mobile or Dota 2 or you know your usebase runs wooden PCs or last-gen console you will still want to save those polys, so bevel safe out there.

respawnrt

EarthQuake wrote: »

Not a blender guy, but at some point blender would export hard edges/smooth group splits as broken (detached) edges rather than mesh normals. This would cause gaps where baking unless you export a cage mesh from blender as well. Don't know if this is still the case of if they've sorted out mesh normal exporting yet.

This now seems to work great, at least with fbx exporter.(tested with blender 2.73 and max2012)
Been missing some blender updates but looks like Bastien Montagne implemented Split Normals to make that possible.

And yes i do agree with JedTheKrampus that's the better way to do it in 2.73.

PlateCaptain, feel free to add me on skype ( respawnrt) and ill run you through the whole thing.

PlateCaptain

Thanks JedTheKrampus. I'll give that a shot once I get home as well.

Also, I don't have Skype, respawnrt, but thanks for the offer. I'll be sure to check back in either way.

PlateCaptain

Okay, here's what I tried:

Did a double bevel on the low poly mesh, so that I can easily put the UV seams and the Sharp Edges right in the middle. Low poly looks like this:



Both low poly and high poly are set to smooth shading, and the low poly has Auto Smooth enabled and set at 180 degrees. No Edge Split modifier, but a Triangulate modifier on both the low poly and the cage.

I duplicated the cage straight from the low poly mesh, and did nothing to it other than scaling it out a bit, but for some reason XNormal gave me the following error:



I don't understand that, since it's the exact same mesh, but slightly bigger. Anyway, I got around that by using XNormal's built in cage, adjusted using the 3D Viewer as JedTheKrampus suggested. Resulting normal looks like this:



And rendered with and without the normal map applied:



I feel like there's not much difference there at all, and it's actually worse with the normal map, because of the shading weirdness at the top of the image. The smooth shading seems to be creating the gradients in the normal map (if I understand it correctly), which I would think I don't want...?

Anything I'm doing wrong / misunderstanding / need to be doing at this point?

JedTheKrampus

Looks to me like you forgot to set your faces to be smooth shaded in Blender. That could also be why the cage had a different vertex order, because the exporter created more verts to replicate the flat shading. You have to set faces to be shaded smooth, then set your hard edges as desired and make sure Auto Smooth is on.

PlateCaptain

I'm pretty sure I did that. By Hard Edges, do you mean UV Seams, or what Blender calls Sharp Edges? Or something different?