Maya: Vertex Normal precision errors when smoothing mesh causing normal map artifacts

kodde

Hey guys,

First off all, sorry if this has been covered already. I know this is not the first thread concerning normal maps, but I couldn't find any thread covering this specific topic. If there is a thread for this already then please direct me there

Anyway. I had some free time at work yesterday and I decided to do some normal map baking research. As I was doing this I noticed some strange annoying artifacts on seemingly flat surfaces. I couldn't find any logic in this.

They would look like this (cranked levels to extreme in Photoshop to show the error)


This is a difference of 128,128,255 and 128,127,255. When is this a problem? At certain angles with an intense specular highlight/reflection these artifacts are apparent.


So after a bit of testing I figured out that this was caused by the highpoly mesh having differing vertex normals even on seemingly flat surfaces. You could catch this in the Component Editor. It would show values with 2 decimal precision as "0.00" and some values at "-0.00". This go my attention... why the minus sign? Must be precision errors on lower decimals. Yup... change the precision in Options > Change Precision and you can see these differing values. Even differences as low as on the 8th decimal would cause these normal map issues.

The error occurs when smoothing your base mesh when creating the highpoly mesh. I tried several cases. I tried in Maya 2013 64-bit, Maya 2012 64-bit, Maya 2012 32-bit. I also tried this on other computers and also had a look at other peoples generated normal maps. This was not just an issue I was having.

Here's a video which demonstrates all I've already mentioned here.
[ame="http://www.youtube.com/watch?v=Xf-dtd1RSgs"]Maya: Vertex Normal precision errors when smoothing - YouTube[/ame]

So...

Can this be avoided?
If not, can this be corrected after the smoothing has been done?

I'd rather not resort to fixes in Photoshop. I'd sleep better at night knowing that my highpoly's flat surfaces are actually flat. After all, the highpoly might be used for other generation processes and this error could cause other artifacts as well(?).

I tried writing a python script to handle these values, but it wasn't all that trivial. At least not for me. Maybe someone else can do a better job? Or is attempting this futile?

throttlekitty

I'm not sure there is a fix for this. Maya has always had small precision issues like this. Sometimes keying in absolute values doesn't work as expected, getting .9999999 instead of 1.0.

Bal

Mmm, I'm trying to reproduce this in Maya 2011x32, no luck, there are no smoothing precision errors when I check in the Component Editor.
There are artifacts that pop up when I boost the levels like you do in Photoshop, but they are very subtle and I doubt they could ever be a problem (unlike your case) especially after compression anyways.

Your Sampling Quality when baking is set to Low, have you tried High?

Really don't understand why you're getting some much precision errors when smoothing, for me it's perfect with the exact same example model. Maybe something to do with the size of the mesh or the unit settings in Maya? (I'm in cm).

kodde

Bal> Hmm... I haven't tried Maya 2011. I might just do that out of curiosity. Like I said I've tried on Maya 2013 64, Maya 2012 64 and Maya 2012 32 on the same machine. All had this problem. Then I went on to try on Maya 2013 64 on a different machine. Still same problem. Then I checked another users generated normal map... had this problem.

I have also tried fidding with the sampling quality with no difference on this issue. Then again, the problem derives from differing vertex normals.

I'm thinking it might be something in the workflow that skewes my vertex normals, but I cannot figure what. Like in my example video I don't see my doing anything weird tbh. My money is still on the smoothing actions producing precision errors.

I'm also using the default setting of centimeter units.

kodde

Bal> Did you try to reproduce it in a similar fashion as I do in the video? Did you any form of smooth? (Mesh>Smooth or Modify>Convert>Smooth preview to polygons)?

I just reproduced this in Maya 2011 as well (on a different comp for that matter). Same workflow as in my video.

Bal

kodde, I tried both Smooth and Convert Smooth Preview to Polygons (which is the one I usually use).
Could you send me your smoothed and unsmoothed mesh maybe? I try to reproduce the steps from your video exactly as you do them, but it seems to work fine for me.

rebb

What do your "Advanced Options" look like in Transfer Maps ?
I'm only seeing similar'ish artifacts if i use a search envelope of 0% and search method "inside envelope only".
Setting it to "closest to envelope" fixes those, but maybe this is something different.

kodde

Closest to envelope is used in the video I think. I actually reset the settings on the transfer maps dialog in the video, and "closest to envelope" is the default afaik.

Did you guys check your component editor? Do you have differing vertex normals on flat surfaces?

kodde

I see this as maybe several steps.

First step would be the differing vertex normals on the highpoly. Can this issue be avoided? If not, can it be fixed?

If it can't be avoided nor fixed the next step might be to see if there's any way to bake which won't catch these small differences as color variations.

At the moment I'm hoping this could be solved at the first step.

EDIT:

Bal> If I send you my models we'll already have the differing vertex normals issue there. Maybe you wanna experiment with baking with these errors? Like I said, I'm hoping to avoid baking with these differing normals all together.

Here's an example LP + HP model. I checked, the HP has differing normals on the flat areas.

Bal

I first imported your objs in Maya and baked right away with the same settings, and had some errors similar to yours.
After unlocking and smoothing the vertex normals of both objects the artifacts dissapear completely.
There's still kind of ugly banding along the angles, not sure why, xNormal bake doesn't have them.

Oh and even with 10 decimals precision I can't see any values that aren't 0.0000000000, but some still have the minus in front, not sure what to make of that though.

kodde

Bal> Interesting. Softening edge on the HP doesn't solve this on my end.

You could increase the precision in the component editor to 15 which is the maximum.

Seems weird that the same Highpoly source shared between both of us is not showing the same vertex normal values. If you enter this MEL command "select pPlane1.vtx[14399]" and check it's values, does it not have these values?



Thanks for helping out btw.

Bal

Yeah I meant 15 actually, and it's the same for me on that vert, dunno how I missed the irregularities.

Maybe better if you send me your Maya scene with the hipoly mesh before and after you smooth it. The locked vertex normals on my end are due to the obj I guess, but it's weird that unlocking them and smoothing them fixes the artifacts, if they aren't locked to start with on your side.

I'll upload my bake later if you want, not sure what's going on otherwise.

kodde

Ok, then at least we seem to get same vertex normal values.

Ah... I had missed the part about unlocking normals. Never passed my mind that they were locked since they are not shown by default. I was aware of this issue before though, easy to forget when not working with OBJ all that often. When first unlocking and then softening edge it seems to solve the irregularities.

Good catch! This seems be a solution for solving this issue.

So new baking rule for me, soften the edges on your HP before bake. Might be in vain, might remove small irregularities.

Thanks Bal.

Bal

So the vertex normals were locked in your original Maya scene? Don't really understand when during your simple process this could happen, kind of surprising.
I always have a keyboard shortcut for toggling the vertex normal display, quite useful.

kodde

Bal>

No I didn't save the "original" scene. I've recreated this issue so many times now there's no real point. From what I've gathered the issue occurs when smoothing/converting smooth preview.

So maybe some inaccuracy issue with the vertex normal calculations that are associated with the smoothing operation?

I've been able to recreate it on 3 different machines myself and I've seen the issue on a normal map I didn't create from a fourth machine. I've tried Maya 2013, 2012 and 2011. So it's not like it's localized to just my personal workstation. One thing all these machines have in common is that they have the same hardware, but that's about it.

kodde

Well... back at square one.

I tried this at home (different hardware than before) out of curiosity, recreating new scene with these issues. Soften edge won't do anything for my object in this instance. Object was never exported as OBJ. I also tried exporting as obj, importing it, unlocking normals, soften edge. Still no dice... *sigh*

I find it strange that you cannot reproduce this.

Bal

Try sending me your Maya scene maybe.

kodde

Here's an example model just before smoothing. If you select all the vertices on the large flat area you can see that none of them have any differing vertex normals. When I smooth this model it will produce differing vertex normals on that very same flat area, even though there is not a single geometrical difference even close.

Do you also get this result?

There's an .ma created in 2013 student version and an OBJ. Using either gives me the same errors when smoothing.

throttlekitty

Bal, what processor are you running?

Bal

Can't open the .ma file (it's empty, probably version problem), but using your obj gives me the same very visible artifacts (and pretty large precision errors in the component editor). Don't really have time right now but I'll test some more later to try to see why the result can vary so much.

kodde

Bal> Probably due to the file being saved in newer Maya? (2013 64-bit). You have to use the "ignore version" flag in the Open options to get around the problem. But seeing as the OBJ seemed to give similar errors for you it seems like this problem is consistent between us too. At least in this case. So...as it seems now using the smoothing operations in certain scenarios seems to produce these precision errors.

kodde

Anyone else wanna give this a go? Please reproduce my model in the video. Do you also get these precision errors in the vertex normals of flat surfaces?

Ty.

throttlekitty

Yes, I get them recreating your model. Maya 2012 x64, running on an Core2 Quad.

Here's a fun test: set up a simple plane with 20 divisions, assign a default vertex color, should be a mid-grey, .5,.5,.5 for RGB. Select all verts, then all columns for a color channel, and type a new value, say 0.2, or 0.16.

At a float precision of 10, I see some pretty interesting numbers, but the same bad numbers each time.

I had first run into this years ago trying to solve two particular vertices on a head model that would not keep proper skin data, and had large tearing in-game between the neck. No amount of keying in values would work (even with weighing average turned off), and I resorted to editing the exported file by hand afterwards.

I never found an answer then, and I assume it's a performance tradeoff in the software, designed to do floating point operations on thousands of vertices in more or less real time. I also remember a time when a certain calculator software would handle certain math incorrectly because of issues on the Pentium 4 (or Pentium 2, I forget). Given that Bal sees little to no issues, I wonder if processor matters in this case?

Either way, I think this would be something to bring up with Autodesk.

m4dcow

So I get the precision errors when creating the model the way you did, but I don't think that is the issue, because I baked the same models in xNormal with a perfectly smooth bake no weird artifacts.

So maybe there is something fucked up with transfer maps?

I noticed this awhile back when baking a weapon of mine, and got those artifacts but thought it was due to my low poly being messed up, but then I baked it in xNormal without, and continued to do so.

Scruples

I would put my monkey on the scene size causing this, the smoothing errors appear in a moire pattern probably caused by the vertex spacing being somewhat consistent.

Edit: 2013x64 on Xeon X3370

Precision errors are from color banding, otherwise fine
Working units are set to centimeter
Positional and Tangental tolerance are set to 0.00328 and 0.1
Polyplane size is about 50 units/cm across.

kodde

Here's what I think.

It is indeed due to the vertex normal precision errors. The baking software has to draw the line somewhere at what normal angle will be 128,128,255 and what will be 128,127,255. This first point(128,128,255) is at a normal value of exactly 0. Anything above that, even the slightest lowly decimal, will bump it up to the next color value (128,127,255). One solution would be having a bigger range before it switches to this next color. That would essentially give it a bit of "fault tolerance" for values near the extremes 0.0 and 1.0.

kodde

Scruples> What are the Positional and Tangental tolerance values you mention?

Scene size cause this as in the object which is being smoothed is very small in relations to the working unit? The workaround if this is the case would be... working with larger models?

Scruples

Looking at it again, I think a larger scene may help keep the surface planar.. but I would agree with your earlier consensus, that the problem is with the normal maps being unsigned and the break between 127 and 128 being basically exactly at 90° I think max gets around this by dithering and I have no idea what Xnormal does.

About the positional and tangential tolerances, turns out they only apply to nurbs surfaces.

m4dcow

I don't think object size has much to do with it, when I did the test model I had my scene setup to work with unreal units, and my plane was 1024 units, so unless you are talking about being much larger than that, I don't think it is the issue.

Again I say it has to be something wrong with transfer maps, because these errors are pretty small and should end up being rounded up or down to what they are supposed to be.

1.0/256= 0.00390625, that is what 1 shade of 8 bit is, so the error needs to be greater/less than 0.001953125 to round up/down to the next shade, the errors I see are a good 100 sometimes 1000 times smaller than this.

The same geometry that has precision errors but can be exported to xNormal and baked without issue.

kodde

Could it not be that the extremes are mapped to the absolute values and everything shade of color in between there has the transition span you mention m4dcow? I'd agree about calling this a bug though.

Did you manage a good bake with transfer maps with the mesh that has these vertex normal errors?

m4dcow

I didn't get a good bake when I replicated your model, and I noticed this issue on a hard surface weapon that I was doing a couple weeks back, and switched to xNormal to do bakes which fixed it.

Now the vertex normal anomalies are definitely there, but my point is they are so small, that when rounding to an 8bit value they shouldn't have much of an effect, unless there is an issue with how transfer maps is rounding off these value.

So on another test the normal y of my model when smoothed, on a loop where all the values should be the same, most are 1.0 no precision errors, but there are quite a few that are 0.999999940395355.

Now I don't exactly know the math for how normal maps are represented, but you have 8bits to represent a value that I think is 0 to 1, and these precision errors don't come close to being able to affect the 8bit value, but I suspect there is a bug where they are rounding up or down (not sure which) explicitly almost like a ceil or floor operation.

When I look at my normal map the difference in the artifacting errors are 1 8bit value in the red or green channel (127 or 128), and we shouldn't be able to pick this up on an 8bit map, a 16 bit one maybe, but not on an 8bit map.

Scruples

It's the way computers convert something into an integer by default, 0.9999999 converted to an integer is still 0. I already mentioned it above but will do so again. With unsigned normal maps there is no 90° tangent value, only 90.35294117647059° which is 128 and 89.64705882352941° which is 127, smack dab in the middle of them is 90° so even a tiny deviation can switch it's value.

m4dcow

Scruples wrote: »

It's the way computers convert something into an integer by default, 0.9999999 converted to an integer is still 0. I already mentioned it above but will do so again. With unsigned normal maps there is no 90° tangent value, only 90.35294117647059° which is 128 and 89.64705882352941° which is 127, smack dab in the middle of them is 90° so even a tiny deviation can switch it's value.

I get what your saying that in 8bit color representation there is no true 90°, and 90° is directly in between the distinct 8 bit values thus even the smallest deviation would have the value round down.

Pardon my ignorance, wouldn't it be the other way around ie: signed is -127 to 127 (255 values) versus unsigned 0 to 255 (256 values?? (I sense a comp sci lesson coming along).

Maybe someone should direct jogshy (xNormal author) over here, and he could hypothesize what is going on with the Maya baker, or at least explain how he handles this in xnormal.