Normal Faceting in Modo
interpolator
Unfortunately the Foundry forums are no help at all. Not a single reply in days.
I have no idea how to properly avoid faceting in Modo. There's a setting for Smoothing Angle meant to fix this, but either end of the spectrum produces its own set of problems, with no functional middle-ground.
If the Smoothing Angle is too low, I get hard lines carved into the center of my edge bevels, as seen here;
If my Smoothing Angle is too high, my bevels bow along the edges, as seen here;
A higher Smoothing Angle also causes edges to bleed together terribly, as seen here;
Anything in the middle will just produce a lesser degree of both problems; there's no avoiding it.
While I'm on it, as you can see in the screenshots my normals are also heavily pixelated despite a 1024 map size and ~3mil poly count on the high-poly.
I have no idea how to properly avoid faceting in Modo. There's a setting for Smoothing Angle meant to fix this, but either end of the spectrum produces its own set of problems, with no functional middle-ground.
If the Smoothing Angle is too low, I get hard lines carved into the center of my edge bevels, as seen here;
If my Smoothing Angle is too high, my bevels bow along the edges, as seen here;
A higher Smoothing Angle also causes edges to bleed together terribly, as seen here;
Anything in the middle will just produce a lesser degree of both problems; there's no avoiding it.
While I'm on it, as you can see in the screenshots my normals are also heavily pixelated despite a 1024 map size and ~3mil poly count on the high-poly.
Replies
If you ARE using 801, you need to follow these steps:
If you're NOT using 801, the steps are similar, but still different. I'd suggest using xNormal for your bakes in that case:
Then selecting your UV map and hitting the "Harden UV Borders" button (rightmost button, top row).
Much easier
Thanks for the advice, I'll get on trying that. I expect it'll take quite a while XD
-Edit- Ah, yeah... I can't use any method that requires a Cage. I have the Steam version right now, which doesn't allow me to export anything as large as a high-poly. I think it caps at like 7,000 tris.
I hope this doesn't mean I'll have to manually touch up the normal in Photoshop. That'd be a hell of a process.
The cage is only for the low poly - the high poly doesn't get one. So you could still export two versions of your low poly mesh for baking it in xNormal (one low and one low with cage) - or just make the cage in something like xNormal.
But that's assuming your high poly's from something external - you'll be hard pushed to export it otherwise.
My recommendation would be to use Handplane3D to get your normal maps. That workflow would go something like this...
- Set low poly material's smoothing angle to 180.
- In the render panel, add a new render output, set it's effect to Shading Normal.
- Select the render item and set the frame size to the size of your desired texture.
- With the low poly in the foreground and the high poly in the background, ensure you have your UVs selected then from the top menu choose Render > Bake from Objects to Render Outputs.
- Save the Shading Normal output as 16bit PNG.
- Export your low poly mesh as FBX. You might want to triangulate it.
- Load up Handplane3D, feed it that FBX and PNG file and select Source output, then convert it.
That's the method I use for all my Modo baking (although I use a cage in 801), works a treat.
The bowing problem is definitely an issue with the Vertex Normals. There's no reason to split the edge completely; that creates its own set of problems anyway.
If I use Set Vertex Normals to create a Vertex Map, then I select the edge with the bowing distortion and apply a Split Normals so that the shading between the normals on either side of the distorted edge are hardened, this also removes the bowing.
This isn't a complete solution though, because much like with a full on edge split, it ruins the bevel I have by running a hard line through it.
But, I know for a fact now that the problem is the vertex normals and just need to find a more solid solution.
You can then bake using that - effectively a cage, but it doesn't have a distance seeing - and then set the vertex normals back afterwards for exporting to FBX and conversion in Handplane.
It's about the best you'll get without 801.
Of course, if you can't export your high poly this will be a problem...
Although the normal tools perhaps not... my kit used to work with Modo SE but I dunno if it works any more.
Ohhh, your vertex normals toolkit. I didn't catch that XD
Yeah, I'll try that out.
-Edit- I'm not sure it does still work with SE; I'm not getting an error message, but the icon isn't there next to Work Plane.
I know it is a massive necro, but I just wanted to thank you for your awesomeness! This is the single greatest modo smoothing guide ever.
I love you m8.