"Dirty" normal maps.

Kligan

Hello!

I've encountered an issue while baking normal maps for my model.

I am making this grandfather clock:


Everything was fine, I made highpolys of different elements, unwrapped lowpolys, tested if everything bakes properly - all was fine.

Then I started to work on a top part of the clock:


I did everything exactly the same as before, as I've done god knows how many times... But after baking normals, I've noticed something strange. At first I got couple minor artefacts in marmoset in some corners, so I tweaked UV's a bit and it was fixed.
I decided to double-check it in UE4 and Mixer as well... just in case. And this is when troubles began.

For some reason, the normals of those curved parts are all "dirty" looking. Here's what I mean.

Marmoset. Everything looks perfect.
Mixer. You can see the streaks across surfaces.



Unreal 4. Same as mixer, you can see "splotches":


I've experimented with baking normal into different formats with different color depth. 8-bit results in dithering, obviously. 16-bit results in "splotches".

I've stumbled upon this thread: https://polycount.com/discussion/135613/weird-marmoset-normal-map-issue
EarthQuake said that this is due to lowpoly being too similar to highpoly or surfaces "too smooth".
If that's the case... then what should I do to fix this?

Now, addressing an elephant in the room - yes, this won't be noticeable at all on the final texture, because it's gonna be aged wood. 
However, this issue made me think - what if I was modelling, say, a car? With polished reflective surfaces? These splotches would've been VERY noticeable. Because they break reflections as you can see here:


Is there a fix for this?

Fabi_G

Is this part of the clock mirrored? I would offset UVs of mirrored or repeated parts by 1 to be sure.

Kligan

Fabi_G said:

Is this part of the clock mirrored? I would offset UVs of mirrored or repeated parts by 1 to be sure.

No, I didn't mirror anything. There was not enough space for this prop on 2k texture, and more than enough on 4k. So I just decided to go with 4k and lay everything out properly without tricks.

kanga

Are all the polys on the low poly in one smoothing group? I cant tell what app you are using from the screenshots. From the model shot it looks like you have different shade groups. in Blender it is select all polys and shade smooth. Try recalculating normals. Simple stuff but you never know.

rexo12

If you've got inconsistent results between your baker and renderer, it suggests that something is changing on your mesh when importing into either of those programs. Often this is differing opinions on how the mesh should be triangulated, so it might be worth pre-triangulating things. There could also be problems with the tangent-basis between the 2 programs (I believe Unreal is MikkTSpace), or something is happening to your smoothing.

Kligan

kanga said:

Are all the polys on the low poly in one smoothing group? I cant tell what app you are using from the screenshots. From the model shot it looks like you have different shade groups. in Blender it is select all polys and shade smooth. Try recalculating normals. Simple stuff but you never know.

I am using Maya. I have hard edges on the borders of UV shells (regular lines - hard, dotted lines - soft).

rexo12 said:
If you've got inconsistent results between your baker and renderer, it suggests that something is changing on your mesh when importing into either of those programs. Often this is differing opinions on how the mesh should be triangulated, so it might be worth pre-triangulating things. There could also be problems with the tangent-basis between the 2 programs (I believe Unreal is MikkTSpace), or something is happening to your smoothing.

That's actually it! I can't believe that solution was so simple all along! 
After triangulating it in Maya before exporting, everything looks correct.


In the past I was always triangulating my meshes. But it was a pain in the ass to tweak triangulated mesh. So I stopped doing that at some point, as it would appear that it has no difference.
Well, apparently, there is a difference on some rare occasion...

Kligan

Ok, something really weird is going on. And I'm completely lost.

I was working on the clock, everything was fine. Time came to texture it.
I imported it to mixer, split into parts, so that I can make sure that the wood grain goes the right way after projection. I begin to create wood texture.
At one point I needed to re-import the model after rotating some parts around, to fix the wood grain direction... Aaaaand everything went to sh*t. 


I double-checked if it was trianglulated - it was. 

I imported this same model to marmoset to check it.


All looks fine.

But here's a real kicker... if I create NEW SCENE in marmoset and import this model there...


Every surface is now completely screwed! Normals going crazy! 

What is going on?! 
I'm so lost! I swear this model is cursed somehow...

Kligan

It looks like the bake picks up direction of edge flow on the highpoly now.



Which doesn't make any sense! Those are flat surfaces! And everything was baking fine before... what the hell happened?! 
I'm pulling my hair here.... I have no idea what to do or what could've happened for this crap to start.

Kligan

There are also gradients appear, when normal map is applied to the lowpoly, where there shouldn't be any...

dimwalker

Maybe lowpoly normals changed since you exported the mesh and baked normal map from it. Import LP mesh into this scene, apply normal map to it and check if there is any difference.

Kligan

dimwalker said:

Maybe lowpoly normals changed since you exported the mesh and baked normal map from it. Import LP mesh into this scene, apply normal map to it and check if there is any difference.

I've re-exported and re-imported and re-baked like 20 times already.... It's all broken for some reason.

Kligan

Just as a test, I imported into Maya one of the bakes I did last week.


I exported it again, baked it again, imported the lowpoly to mixer, and... everything is perfect:


So, I created a cube, added some geometry, UV'd it, made highpoly, exported, baked in Marmoset:


Now, importing it to Mixer... aaand... there are gradients again where there shouldn't be any:


Imported the model and normal to unreal, and noticed "streaks" again. Checked in photoshop. Yup, it's all here:


So... The models that I've created last week bake and look fine. No weirdness or gradients - I've checked couple more to make sure. The models that I'm trying to create and bake now, all look f-ed.
Which means... I guess my Maya just completely broke and does some garbage ? 
I don't know, I'm out of other ideas. Nothing makes sense any more...

dimwalker

Upload a test cube LP and HP meshes and normal map you got from those.

Joopson

What format are you exporting meshes in? And have you looked at those export settings?
I tend to always export with OBJ, just because there are far fewer variables in the exporter.
And these are the settings I use.

Kligan

Joopson said:

What format are you exporting meshes in? And have you looked at those export settings?
I tend to always export with OBJ, just because there are far fewer variables in the exporter.
And these are the settings I use.

Usually FBX. Tried OBJ too when I was troubleshooting this issue.

dimwalker said:
Upload a test cube LP and HP meshes and normal map you got from those.

Here it is.
 https://drive.google.com/file/d/1lHWyl8aynT9AiGBNhghl7T6tZMOSzEJr

dimwalker

Mesh seems alright on first glance. There is no highpoly version though, so I can't try to bake it.

Kligan

dimwalker said:

Mesh seems alright on first glance. There is no highpoly version though, so I can't try to bake it.

Oh, apologies. Let me re-upload it with high-poly.
https://drive.google.com/file/d/1Px5EDV2QxwJkInys7RXN1yW4AAkBQOhm

I was testing stuff some more... I installed new version of maya, 2023. Completely clean, didn't move any configs.
Tried to make the same cube, highpoly, baked it in marmoset 4... and got the same result.
This gradient shouldn't be here:


Also, are 8-bit normals supposed to be THIS f-ed up?


Previously I was mostly baking 8-bit to .tga format, unless I needed high color precision.
And I could swear that normals never looked THIS dithered all over the place on flat surfaces. Or am I going completely insane already?

Kligan

OK! I think I'm getting somewhere...
I installed the ol' reliable xNormal. Baked normals there. 8-bit .tga. Applied to a cube... And everything looks perfect in both Mixer and Marmoset:



Which means... that something is broken with Marmoset's baking. 
Normals that I got from xNormal are perfectly clean:


Now I just need to figure out what happened to Marmoset?

dimwalker

Welp... only slightly suspicious thing I saw was different normals length on high poly, but normalizing it had no effect.
3dsmax 2021 baked normal as expected without any weirdness. I think you are right and it's all about the baker.
Substance designer insisted on having tons of dithering on flat surfaces no matter what I do. Heck, it adds dithering even when I use same mesh as low and high.
 Now I have something to dig and bang my head at too. This thread is contagious! 

Kligan

dimwalker said:

 Now I have something to dig and bang my head at too. This thread is contagious! 

Oh no! I'm sorry  
I've been beating my head against the wall with this for days now... my work completely halted. 

My main question now - what the hell happened? I was working with marmoset for years now and all was fine. But now, suddenly, all is far from fine.
I don't want to go back to xNormal. As good as it is for a free open source software, it's SO SLOW compared to marmoset.

I've created a separate thread in marmoset sub-forum to see if maybe anyone has any idea what is going on. I also included some side-to-side comparison screenshots there of what 8-bit and 16-bit normals look like from xnormal and marmoset.
https://polycount.com/discussion/233219/marmoset-doesnt-bake-normals-correctly-any-more#latest

dimwalker

If there was no update recently and I assume you changed nothing in settings then my best guess is that this problem was always there you just didn't notice it. I mean, you will never see it on something with less reflectivity and a lot of normal variation like dirt or small pebbles. Clock just had perfect topology + normals + material conditions to emphasize the issue.

Kligan

dimwalker said:

If there was no update recently and I assume you changed nothing in settings then my best guess is that this problem was always there you just didn't notice it. I mean, you will never see it on something with less reflectivity and a lot of normal variation like dirt or small pebbles. Clock just had perfect topology + normals + material conditions to emphasize the issue.

I would definitely notice something like this (in mixer):

Or this:

Or that:

That shading is just completely wrong. And I have no idea why.
It looks good in marmoset. 

In Unreal there are also problems with this normal map:


And this is with full precision UVs on the mesh enabled. Still lots of jaggedness. 

As for updates... AFTER I noticed that problems started to happen, I updated both maya and marmoset. I was postponing updating for a long time now, simply because I didn't want to deal with config/plugin/scripts transfer headache. 
After updating maya to 2023 and marmoset to the latest version... Nothing has changed. It still spits out garbage.

Joopson

Your uploaded normal map and meshes look totally fine on my end, within Maya and Marmoset. Not getting any artifacting at all, nor can I see any in the uploaded normal map you supplied (or ones I baked, to test). Some tiny minor noise, but nothing that shows.

The only thing I can say is, make sure of your export/import settings, and your settings within mixer and unreal for display. And also update your graphics drivers, if you haven't in a while. Also worth a shot, assign different materials to the high and low if you haven't, before baking in Marmoset. I think at one point, some problems could be caused if they shared a material.

I get you're frustrated, but I feel like exporting separate high and low OBJs with the settings I posted above would remove a lot of variables, so we can better try to troubleshoot. This thread is very hard to follow.

Kligan

Joopson said:

Your uploaded normal map and meshes look totally fine on my end, within Maya and Marmoset. Not getting any artifacting at all, nor can I see any in the uploaded normal map you supplied (or ones I baked, to test). Some tiny minor noise, but nothing that shows.

The only thing I can say is, make sure of your export/import settings, and your settings within mixer and unreal for display. And also update your graphics drivers, if you haven't in a while. Also worth a shot, assign different materials to the high and low if you haven't, before baking in Marmoset. I think at one point, some problems could be caused if they shared a material.

I get you're frustrated, but I feel like exporting separate high and low OBJs with the settings I posted above would remove a lot of variables, so we can better try to troubleshoot. This thread is very hard to follow.

Thank you for taking time to try to help
I've done all the things that you mentioned. I tried everything I could think of.

As for updating drivers... Now that I have re-installed a new version of Maya from scratch, re-installed Marmoset, and yes, I have the latest version of video drivers... the only thing left to try is to roll back the said drivers couple versions back and re-install Mixer. 
If that doesn't help either... then I have no ideas left.

Kligan

Ok, so... after trying countless things, re-installing literally all of my software for modelling and baking, re-installing video drivers (updating and rolling back to older bersions).... I still have absolutely no idea what has happened. 
The good news is - now everything sort of works? 
I mean sort of because I still have "leftower" issues in Mixer. Which I'm not sure if it's me specifically or it's a bug in the software - I'm gonna have to discuss it with mixer support.

What happens is - somehow Mixer introduces faint gradients on the normal map to SOME edges of UV shells.
It's the most noticeable in places like these:



You can clearly see the distortions.

Now... That being said... I still don't know what causes this, as I said before.
I might have a suspicion that it has something to do with my video card.
My old video card (rx 5700 xt) has died recently - it finally gave up after years of abuse, running at 100% load during 3d texturing. So I had to buy a new one. And I went for a REALLY new one - rx 7900 xtx. And it just might be that the card is "too new" and Mixer just doesn't know how to work with it properly. Because Mixer, being a free software, is being updated once in a blue moon. And is technically still in beta. 
Other than that - I have no other guesses.

Kligan

The latest development in the saga... 

I asked my friend, who has nothing to do with graphics or texturing whatsoever, to install Mixer and test a thing for me. 
I wanted to make sure that it's not my PC that's the problem.
I asked him (with step-by-step instructions) to open up the project in Mixer, import a normal map from Marmoset that I sent to him. And then export the normal from Mixer.
After I checked out the results.... There were still distortions, but in slightly different places and in a slightly different way.
Which means - Mixer is at fault. 

So I'm gonna go complain to Mixer support now....

What happened in the beginning and why all my bakes were picking up edge flow directions from highpoly on flat surfaces, resulting in f-ed up normals - still no idea. But at lest THAT bug is gone now. Even though, again, no idea what fixed it.

pxgeek

Oof, sounds like gpu/driver is the likely culprit. You mentioned in the other thread that xNormal was working correctly...another knock for amd i guess

Kligan

pxgeek said:

Oof, sounds like gpu/driver is the likely culprit. You mentioned in the other thread that xNormal was working correctly...another knock for amd i guess

My friend doesn't have AMD hardware. And after I asked him to do a test for me, Mixer still added some distortions on the normals, but in slightly different spots.
So I suppose it's just Mixer issue that I wouldn't have noticed if all my stuff didn't go bonkers out of nowhere.
I submitted it to their stuff for consideration:
https://help.quixel.com/hc/en-us/community/posts/11323373050525-Mixer-adds-gradients-to-normal-maps

And as for the original issue that brought me to this discovery... No idea. It just came out of nowhere and disappeared after several days and after I re-installed all of my software.