Maya: Vertex Normal precision errors when smoothing mesh causing normal map artifacts
polycounter lvl 19
Hey guys,
First off all, sorry if this has been covered already. I know this is not the first thread concerning normal maps, but I couldn't find any thread covering this specific topic. If there is a thread for this already then please direct me there
Anyway. I had some free time at work yesterday and I decided to do some normal map baking research. As I was doing this I noticed some strange annoying artifacts on seemingly flat surfaces. I couldn't find any logic in this.
They would look like this (cranked levels to extreme in Photoshop to show the error)
This is a difference of 128,128,255 and 128,127,255. When is this a problem? At certain angles with an intense specular highlight/reflection these artifacts are apparent.
So after a bit of testing I figured out that this was caused by the highpoly mesh having differing vertex normals even on seemingly flat surfaces. You could catch this in the Component Editor. It would show values with 2 decimal precision as "0.00" and some values at "-0.00". This go my attention... why the minus sign? Must be precision errors on lower decimals. Yup... change the precision in Options > Change Precision and you can see these differing values. Even differences as low as on the 8th decimal would cause these normal map issues.
The error occurs when smoothing your base mesh when creating the highpoly mesh. I tried several cases. I tried in Maya 2013 64-bit, Maya 2012 64-bit, Maya 2012 32-bit. I also tried this on other computers and also had a look at other peoples generated normal maps. This was not just an issue I was having.
Here's a video which demonstrates all I've already mentioned here.
[ame="
http://www.youtube.com/watch?v=Xf-dtd1RSgs"]Maya: Vertex Normal precision errors when smoothing - YouTube[/ame]
So...
Can this be avoided?
If not, can this be corrected after the smoothing has been done?
I'd rather not resort to fixes in Photoshop. I'd sleep better at night knowing that my highpoly's flat surfaces are actually flat. After all, the highpoly might be used for other generation processes and this error could cause other artifacts as well(?).
I tried writing a python script to handle these values, but it wasn't all that trivial. At least not for me. Maybe someone else can do a better job? Or is attempting this futile?
First off all, sorry if this has been covered already. I know this is not the first thread concerning normal maps, but I couldn't find any thread covering this specific topic. If there is a thread for this already then please direct me there
Anyway. I had some free time at work yesterday and I decided to do some normal map baking research. As I was doing this I noticed some strange annoying artifacts on seemingly flat surfaces. I couldn't find any logic in this.
They would look like this (cranked levels to extreme in Photoshop to show the error)
This is a difference of 128,128,255 and 128,127,255. When is this a problem? At certain angles with an intense specular highlight/reflection these artifacts are apparent.
So after a bit of testing I figured out that this was caused by the highpoly mesh having differing vertex normals even on seemingly flat surfaces. You could catch this in the Component Editor. It would show values with 2 decimal precision as "0.00" and some values at "-0.00". This go my attention... why the minus sign? Must be precision errors on lower decimals. Yup... change the precision in Options > Change Precision and you can see these differing values. Even differences as low as on the 8th decimal would cause these normal map issues.
The error occurs when smoothing your base mesh when creating the highpoly mesh. I tried several cases. I tried in Maya 2013 64-bit, Maya 2012 64-bit, Maya 2012 32-bit. I also tried this on other computers and also had a look at other peoples generated normal maps. This was not just an issue I was having.
Here's a video which demonstrates all I've already mentioned here.
[ame="
http://www.youtube.com/watch?v=Xf-dtd1RSgs"]Maya: Vertex Normal precision errors when smoothing - YouTube[/ame]So...
Can this be avoided?
If not, can this be corrected after the smoothing has been done?
I'd rather not resort to fixes in Photoshop. I'd sleep better at night knowing that my highpoly's flat surfaces are actually flat. After all, the highpoly might be used for other generation processes and this error could cause other artifacts as well(?).
I tried writing a python script to handle these values, but it wasn't all that trivial. At least not for me. Maybe someone else can do a better job? Or is attempting this futile?
Replies
There are artifacts that pop up when I boost the levels like you do in Photoshop, but they are very subtle and I doubt they could ever be a problem (unlike your case) especially after compression anyways.
Your Sampling Quality when baking is set to Low, have you tried High?
Really don't understand why you're getting some much precision errors when smoothing, for me it's perfect with the exact same example model. Maybe something to do with the size of the mesh or the unit settings in Maya? (I'm in cm).
I have also tried fidding with the sampling quality with no difference on this issue. Then again, the problem derives from differing vertex normals.
I'm thinking it might be something in the workflow that skewes my vertex normals, but I cannot figure what. Like in my example video I don't see my doing anything weird tbh. My money is still on the smoothing actions producing precision errors.
I'm also using the default setting of centimeter units.
I just reproduced this in Maya 2011 as well (on a different comp for that matter). Same workflow as in my video.
Could you send me your smoothed and unsmoothed mesh maybe? I try to reproduce the steps from your video exactly as you do them, but it seems to work fine for me.
I'm only seeing similar'ish artifacts if i use a search envelope of 0% and search method "inside envelope only".
Setting it to "closest to envelope" fixes those, but maybe this is something different.
Did you guys check your component editor? Do you have differing vertex normals on flat surfaces?
First step would be the differing vertex normals on the highpoly. Can this issue be avoided? If not, can it be fixed?
If it can't be avoided nor fixed the next step might be to see if there's any way to bake which won't catch these small differences as color variations.
At the moment I'm hoping this could be solved at the first step.
EDIT:
Bal> If I send you my models we'll already have the differing vertex normals issue there. Maybe you wanna experiment with baking with these errors? Like I said, I'm hoping to avoid baking with these differing normals all together.
Here's an example LP + HP model. I checked, the HP has differing normals on the flat areas.
After unlocking and smoothing the vertex normals of both objects the artifacts dissapear completely.
There's still kind of ugly banding along the angles, not sure why, xNormal bake doesn't have them.
Oh and even with 10 decimals precision I can't see any values that aren't 0.0000000000, but some still have the minus in front, not sure what to make of that though.
You could increase the precision in the component editor to 15 which is the maximum.
Seems weird that the same Highpoly source shared between both of us is not showing the same vertex normal values. If you enter this MEL command "select pPlane1.vtx[14399]" and check it's values, does it not have these values?
Thanks for helping out btw.
Maybe better if you send me your Maya scene with the hipoly mesh before and after you smooth it. The locked vertex normals on my end are due to the obj I guess, but it's weird that unlocking them and smoothing them fixes the artifacts, if they aren't locked to start with on your side.
I'll upload my bake later if you want, not sure what's going on otherwise.
Ah... I had missed the part about unlocking normals. Never passed my mind that they were locked since they are not shown by default. I was aware of this issue before though, easy to forget when not working with OBJ all that often. When first unlocking and then softening edge it seems to solve the irregularities.
Good catch! This seems be a solution for solving this issue.
So new baking rule for me, soften the edges on your HP before bake. Might be in vain, might remove small irregularities.
Thanks Bal.
I always have a keyboard shortcut for toggling the vertex normal display, quite useful.
No I didn't save the "original" scene. I've recreated this issue so many times now there's no real point. From what I've gathered the issue occurs when smoothing/converting smooth preview.
So maybe some inaccuracy issue with the vertex normal calculations that are associated with the smoothing operation?
I've been able to recreate it on 3 different machines myself and I've seen the issue on a normal map I didn't create from a fourth machine. I've tried Maya 2013, 2012 and 2011. So it's not like it's localized to just my personal workstation. One thing all these machines have in common is that they have the same hardware, but that's about it.
I tried this at home (different hardware than before) out of curiosity, recreating new scene with these issues. Soften edge won't do anything for my object in this instance. Object was never exported as OBJ. I also tried exporting as obj, importing it, unlocking normals, soften edge. Still no dice... *sigh*
I find it strange that you cannot reproduce this.
Do you also get this result?
There's an .ma created in 2013 student version and an OBJ. Using either gives me the same errors when smoothing.
Ty.
Here's a fun test: set up a simple plane with 20 divisions, assign a default vertex color, should be a mid-grey, .5,.5,.5 for RGB. Select all verts, then all columns for a color channel, and type a new value, say 0.2, or 0.16.
At a float precision of 10, I see some pretty interesting numbers, but the same bad numbers each time.
I had first run into this years ago trying to solve two particular vertices on a head model that would not keep proper skin data, and had large tearing in-game between the neck. No amount of keying in values would work (even with weighing average turned off), and I resorted to editing the exported file by hand afterwards.
I never found an answer then, and I assume it's a performance tradeoff in the software, designed to do floating point operations on thousands of vertices in more or less real time. I also remember a time when a certain calculator software would handle certain math incorrectly because of issues on the Pentium 4 (or Pentium 2, I forget). Given that Bal sees little to no issues, I wonder if processor matters in this case?
Either way, I think this would be something to bring up with Autodesk.
So maybe there is something fucked up with transfer maps?
I noticed this awhile back when baking a weapon of mine, and got those artifacts but thought it was due to my low poly being messed up, but then I baked it in xNormal without, and continued to do so.
Edit: 2013x64 on Xeon X3370
Precision errors are from color banding, otherwise fine
Working units are set to centimeter
Positional and Tangental tolerance are set to 0.00328 and 0.1
Polyplane size is about 50 units/cm across.
It is indeed due to the vertex normal precision errors. The baking software has to draw the line somewhere at what normal angle will be 128,128,255 and what will be 128,127,255. This first point(128,128,255) is at a normal value of exactly 0. Anything above that, even the slightest lowly decimal, will bump it up to the next color value (128,127,255). One solution would be having a bigger range before it switches to this next color. That would essentially give it a bit of "fault tolerance" for values near the extremes 0.0 and 1.0.
Scene size cause this as in the object which is being smoothed is very small in relations to the working unit? The workaround if this is the case would be... working with larger models?
About the positional and tangential tolerances, turns out they only apply to nurbs surfaces.
Again I say it has to be something wrong with transfer maps, because these errors are pretty small and should end up being rounded up or down to what they are supposed to be.
1.0/256= 0.00390625, that is what 1 shade of 8 bit is, so the error needs to be greater/less than 0.001953125 to round up/down to the next shade, the errors I see are a good 100 sometimes 1000 times smaller than this.
The same geometry that has precision errors but can be exported to xNormal and baked without issue.
Did you manage a good bake with transfer maps with the mesh that has these vertex normal errors?
Now the vertex normal anomalies are definitely there, but my point is they are so small, that when rounding to an 8bit value they shouldn't have much of an effect, unless there is an issue with how transfer maps is rounding off these value.
So on another test the normal y of my model when smoothed, on a loop where all the values should be the same, most are 1.0 no precision errors, but there are quite a few that are 0.999999940395355.
Now I don't exactly know the math for how normal maps are represented, but you have 8bits to represent a value that I think is 0 to 1, and these precision errors don't come close to being able to affect the 8bit value, but I suspect there is a bug where they are rounding up or down (not sure which) explicitly almost like a ceil or floor operation.
When I look at my normal map the difference in the artifacting errors are 1 8bit value in the red or green channel (127 or 128), and we shouldn't be able to pick this up on an 8bit map, a 16 bit one maybe, but not on an 8bit map.
I get what your saying that in 8bit color representation there is no true 90°, and 90° is directly in between the distinct 8 bit values thus even the smallest deviation would have the value round down.
Pardon my ignorance, wouldn't it be the other way around ie: signed is -127 to 127 (255 values) versus unsigned 0 to 255 (256 values?? (I sense a comp sci lesson coming along).
Maybe someone should direct jogshy (xNormal author) over here, and he could hypothesize what is going on with the Maya baker, or at least explain how he handles this in xnormal.