Normal Faceting in Modo

Zack Maxwell

Unfortunately the Foundry forums are no help at all. Not a single reply in days.
I have no idea how to properly avoid faceting in Modo. There's a setting for Smoothing Angle meant to fix this, but either end of the spectrum produces its own set of problems, with no functional middle-ground.
If the Smoothing Angle is too low, I get hard lines carved into the center of my edge bevels, as seen here;


If my Smoothing Angle is too high, my bevels bow along the edges, as seen here;


A higher Smoothing Angle also causes edges to bleed together terribly, as seen here;


Anything in the middle will just produce a lesser degree of both problems; there's no avoiding it.

While I'm on it, as you can see in the screenshots my normals are also heavily pixelated despite a 1024 map size and ~3mil poly count on the high-poly.

Farfarer

Set the smoothing angle to 180 and use a morph map as a bake cage, that should help you reduce the waviness of the normal map.

Zack Maxwell

Do you mean that I should use a Morph Map to expand the high-poly for the bake, so it fully encloses the low-poly? I tried something similar already, and all it seemed to do was create significantly more edge bleeding like in the 3rd shot. The bowing was still there.

Farfarer

No, use a relative morph map on the low poly mesh, then you should get the option to use it as a cage when baking (if you're using Modo 801).

Swizzle

First off, are you using Modo 801? If you aren't, then ignore anything about morph maps. Previous versions of Modo didn't have cage baking.

If you ARE using 801, you need to follow these steps:

1. Read this help page about the baking workflow in Modo 801
2. Read this thread about normal maps and hard edges
3. Set your smoothing angle to 180
4. Select edges on your model along edges that should be hard just like hard edges on your high-poly. Under the Model tab, go to Edge and press Split. Your low-poly should now have shading that looks pretty similar to your high-poly in a lot of places
5. Separate your UV islands so that the edges of those hard splits have space between them. This is for edge padding purposes; if you don't separate your UV islands, you'll get artifacts on your normal map
6. Go to Vertex Map>Set Vertex Normals. This will keep your existing vertex normals consistent for further steps
7. Under the Model tab, go to Vertex and press Merge. Under Range, select Automatic. Press OK. This will merge the edges you just split. This is pretty important later on for the cage baking steps
8. Navigate to the Lists tab (the same place where you see a list of things like UV maps) and expand Morph Maps
9. Click New Map and create a Relative Morph Map
10. In the Model tab, press Deform and then Push. Push your low-poly model until most of it is encompassing your high-poly. Do not push it so far that parts of your low-poly start to intersect the high poly. Low-poly intersections in the same chunk of mesh are perfectly fine
11. Continue adjusting this pushed low-poly until it completely covers the high-poly
12. Assign a new texture for your normal map and make sure it's set to Normal under the Effect column
13. Right click on your texture's name and press Bake from Object to Texture
14. In the Bake Texture from Object popup, select the morph map you just created and set the Distance to some number greater than the distance between your low-poly and your high-poly (this is usually not more than a meter, but it may be higher or lower depending upon what scale you're working at)
15. Press OK to start the bake

If you're NOT using 801, the steps are similar, but still different. I'd suggest using xNormal for your bakes in that case:

1. Do steps 2-7 from the previous instructions
2. Duplicate your low-poly, preferably to a separate document to avoid mistakes
3. Separate your low-poly mesh so that any chunks of it have pretty good distance between them. Make sure to only move parts that are separate pieces, regardless of what the UVs look like. I'd suggest moving all parts in standard increments so that you can easily reposition them
4. On a separate mesh layer, move the chunks of your high-poly to match the locations of your low
5. Triangulate your low-poly by pressing Shift+T
6. Duplicate the low-poly to a new mesh layer (you should probably name this layer Cage to avoid confusion)
7. In the Model tab, press Deform and then Push. Push your low-poly model until most of it is encompassing your high-poly. Do not push it so far that parts of your low-poly start to intersect the high poly. Low-poly intersections are NOT GOOD and will lead to ugly errors. Manually adjust any areas where it looks like there will be low-poly intersections
8. Continue adjusting this pushed low-poly until it completely covers the high-poly
9. Export your low-poly and your cage as OBJ files
10. In xNormal, load your high-poly and your low-poly
11. Right click on your low-poly in xNormal and select Browse external cage file, then navigate to your cage OBJ and select it
12. Scroll to the right and use the dropdown menu under Smooth normals to select Use exported normals
13. Go to Baking options, set everything up the way you'd like, and click Generate Maps

Farfarer

You can replace steps 4 to 7 in the 801 version of Swizzle's instructions by installing this; http://community.thefoundry.co.uk/discussion/topic.aspx?f=83&t=77919

Then selecting your UV map and hitting the "Harden UV Borders" button (rightmost button, top row).

Much easier

Zack Maxwell

Ouch, far more complex than I was anticipating. Especially since I don't have 801.
Thanks for the advice, I'll get on trying that. I expect it'll take quite a while XD

-Edit- Ah, yeah... I can't use any method that requires a Cage. I have the Steam version right now, which doesn't allow me to export anything as large as a high-poly. I think it caps at like 7,000 tris.

I hope this doesn't mean I'll have to manually touch up the normal in Photoshop. That'd be a hell of a process.

Farfarer

Ah, with SE it's a bit more complex.

The cage is only for the low poly - the high poly doesn't get one. So you could still export two versions of your low poly mesh for baking it in xNormal (one low and one low with cage) - or just make the cage in something like xNormal.

But that's assuming your high poly's from something external - you'll be hard pushed to export it otherwise.


My recommendation would be to use Handplane3D to get your normal maps. That workflow would go something like this...

- Set low poly material's smoothing angle to 180.

- In the render panel, add a new render output, set it's effect to Shading Normal.

- Select the render item and set the frame size to the size of your desired texture.

- With the low poly in the foreground and the high poly in the background, ensure you have your UVs selected then from the top menu choose Render > Bake from Objects to Render Outputs.

- Save the Shading Normal output as 16bit PNG.

- Export your low poly mesh as FBX. You might want to triangulate it.

- Load up Handplane3D, feed it that FBX and PNG file and select Source output, then convert it.



That's the method I use for all my Modo baking (although I use a cage in 801), works a treat.

Zack Maxwell

Hm.. there may actually be a simpler way to solve this problem. I was able to get rid of the bowing by simply splitting the edges that had them before baking. The problem is that there are still some other issues, like the normal becoming highly pixelated at higher smoothing levels.

Zack Maxwell

Oh, yes! I'm a genius.
The bowing problem is definitely an issue with the Vertex Normals. There's no reason to split the edge completely; that creates its own set of problems anyway.
If I use Set Vertex Normals to create a Vertex Map, then I select the edge with the bowing distortion and apply a Split Normals so that the shading between the normals on either side of the distorted edge are hardened, this also removes the bowing.
This isn't a complete solution though, because much like with a full on edge split, it ruins the bevel I have by running a hard line through it.
But, I know for a fact now that the problem is the vertex normals and just need to find a more solid solution.

Farfarer

So with my Vertex Normal Toolkit, you can take the direction offset stored in a relative morph map, then apply that offset as a normal.

You can then bake using that - effectively a cage, but it doesn't have a distance seeing - and then set the vertex normals back afterwards for exporting to FBX and conversion in Handplane.

It's about the best you'll get without 801.

Zack Maxwell

Unfortunately, I don't think I have access to morph maps or advanced normal manipulation tools; they were both added in 801.

WarrenM

Use xNormal. Seriously, I love Modo, but the baking tools are questionable at best.

Of course, if you can't export your high poly this will be a problem...

Farfarer

Morph maps have been in Modo since, I think v1.

Although the normal tools perhaps not... my kit used to work with Modo SE but I dunno if it works any more.

Zack Maxwell

Farfarer wrote: »

Morph maps have been in Modo since, I think v1.

Although the normal tools perhaps not... my kit used to work with Modo SE but I dunno if it works any more.

Ohhh, your vertex normals toolkit. I didn't catch that XD
Yeah, I'll try that out.

-Edit- I'm not sure it does still work with SE; I'm not getting an error message, but the icon isn't there next to Work Plane.

Damian Nachman

Swizzle wrote: »

First off, are you using Modo 801? If you aren't, then ignore anything about morph maps. Previous versions of Modo didn't have cage baking.

If you ARE using 801, you need to follow these steps:

1. Read this help page about the baking workflow in Modo 801
2. Read this thread about normal maps and hard edges
3. Set your smoothing angle to 180
4. Select edges on your model along edges that should be hard just like hard edges on your high-poly. Under the Model tab, go to Edge and press Split. Your low-poly should now have shading that looks pretty similar to your high-poly in a lot of places
5. Separate your UV islands so that the edges of those hard splits have space between them. This is for edge padding purposes; if you don't separate your UV islands, you'll get artifacts on your normal map
6. Go to Vertex Map>Set Vertex Normals. This will keep your existing vertex normals consistent for further steps
7. Under the Model tab, go to Vertex and press Merge. Under Range, select Automatic. Press OK. This will merge the edges you just split. This is pretty important later on for the cage baking steps
8. Navigate to the Lists tab (the same place where you see a list of things like UV maps) and expand Morph Maps
9. Click New Map and create a Relative Morph Map
10. In the Model tab, press Deform and then Push. Push your low-poly model until most of it is encompassing your high-poly. Do not push it so far that parts of your low-poly start to intersect the high poly. Low-poly intersections in the same chunk of mesh are perfectly fine
11. Continue adjusting this pushed low-poly until it completely covers the high-poly
12. Assign a new texture for your normal map and make sure it's set to Normal under the Effect column
13. Right click on your texture's name and press Bake from Object to Texture
14. In the Bake Texture from Object popup, select the morph map you just created and set the Distance to some number greater than the distance between your low-poly and your high-poly (this is usually not more than a meter, but it may be higher or lower depending upon what scale you're working at)
15. Press OK to start the bake

If you're NOT using 801, the steps are similar, but still different. I'd suggest using xNormal for your bakes in that case:

1. Do steps 2-7 from the previous instructions
2. Duplicate your low-poly, preferably to a separate document to avoid mistakes
3. Separate your low-poly mesh so that any chunks of it have pretty good distance between them. Make sure to only move parts that are separate pieces, regardless of what the UVs look like. I'd suggest moving all parts in standard increments so that you can easily reposition them
4. On a separate mesh layer, move the chunks of your high-poly to match the locations of your low
5. Triangulate your low-poly by pressing Shift+T
6. Duplicate the low-poly to a new mesh layer (you should probably name this layer Cage to avoid confusion)
7. In the Model tab, press Deform and then Push. Push your low-poly model until most of it is encompassing your high-poly. Do not push it so far that parts of your low-poly start to intersect the high poly. Low-poly intersections are NOT GOOD and will lead to ugly errors. Manually adjust any areas where it looks like there will be low-poly intersections
8. Continue adjusting this pushed low-poly until it completely covers the high-poly
9. Export your low-poly and your cage as OBJ files
10. In xNormal, load your high-poly and your low-poly
11. Right click on your low-poly in xNormal and select Browse external cage file, then navigate to your cage OBJ and select it
12. Scroll to the right and use the dropdown menu under Smooth normals to select Use exported normals
13. Go to Baking options, set everything up the way you'd like, and click Generate Maps

I know it is a massive necro, but I just wanted to thank you for your awesomeness! This is the single greatest modo smoothing guide ever.
I love you m8.