Trouble with normals + baking

MyMomHatesGames

Hello my name is Josh,

I am currently practicing how to model, UV, make normals, and finally texture. I chose to make a mailbox for my studies.

I first made the Low Poly of the mailbox

Then made the High poly version



Triangulated the Low Poly


Then UV mapped the low poly



After that I placed the High, Low, and Cage in xnormal
This is what showed in the 3D Viewer




I thought it looked alright, so I generated the maps



After that went back to Maya to put the normals on the Low Poly....



I changed the size of the cage, but the results is still the same. I am doing something wrong, but with my current knowledge, I have no clue what is wrong.


Thanks!

Asura

if the door and handle are separate meshes you can try "exploding" the mesh so that they will not get baked in with the reset of the mesh.

Also you can bump the cage to 2 and that should help.

From what it looks like, the cage might be intersecting /inverting on the door part.

MyMomHatesGames

Asura said:

if the door and handle are separate meshes you can try "exploding" the mesh so that they will not get baked in with the reset of the mesh.

Also you can bump the cage to 2 and that should help.

From what it looks like, the cage might be intersecting /inverting on the door part.

Hey Asura,

Thanks for the reply! and the help.

So yeah the door and the handle are separate. When you say exploding the mesh, do you mean select the individual door and handle part of cage mesh and make just those parts bigger?

also smooth the cage to 2? will xnormal read the low poly?

I will try these methods and post back some update pictures.

Thanks asura!

martygalinskas3d

i've noticed the same problem with maya recently, it doesn't shade how it used to, i brought the model and normal map into unreal and it looked fine. it might just be maya, try bringing the model into marmoset or unreal and see how it looks before you waste more time.

RaptorCWS

you do not have to set key frames to do this if you just duplicate your mesh. but this is an example of exploding a mesh for baking.

https://www.youtube.com/watch?v=bgYoXF6QmWw

Stosh

If exploding your mesh doesn't work, make sure that when you go to apply your normal map in Maya via your shader, that you are changing your 2D Bump Attributes under "Use As" to Tangent Space Normals. If you keep it on "Bump" you will get weird results because Maya is trying to compute for Bump. 

It's a small setting, but it's easily forgotten. 

Luxap

Apparently, Xnormal has the option to explode your models as well.



I do have to say that I haven't tried this myself but "Boy Sichterman" the guy that threw this tip around on facebook splits up his model into groups of non-intersecting pieces, and then offsets them in Xnormal itself.

Link to his post below
https://www.facebook.com/photo.php?fbid=1312298148810252&set=a.212073412166070.56496.100000902388744&type=3&theater

As for baking itself I usually split up the UV's on the hard edges to keep artifacts to a minimal, and try to stay away from having 90 degree angles, because these don't tend to bake well.

I'm not a master baker myself, but I do hope that these small tips help you on your way!

Tectonic

be conscientious about the normals on your low poly mesh in maya. do you have them softened? hardened? set to face? these different settings can change the output of your bake. I would set most edges to softened with the exception of planar pieces which you could try setting to face. Xnormal also has a toggle-able setting where it will make all your mesh normals set to face for you if you want, sometimes that gives best results, but sometimes you want that turned off.

MyMomHatesGames

martygalinskas3d said:

i've noticed the same problem with maya recently, it doesn't shade how it used to, i brought the model and normal map into unreal and it looked fine. it might just be maya, try bringing the model into marmoset or unreal and see how it looks before you waste more time.

Thanks Marty! I really hoped that would work but as in marmoset I put in the object with the normals. but it doesn't look right.

Stosh said:

If exploding your mesh doesn't work, make sure that when you go to apply your normal map in Maya via your shader, that you are changing your 2D Bump Attributes under "Use As" to Tangent Space Normals. If you keep it on "Bump" you will get weird results because Maya is trying to compute for Bump. 

It's a small setting, but it's easily forgotten. 

Stosh I also doubled check to see if its set wrong, but with set to Tangent it showed the same results.



As of now I will try exploding the model



Thanks guys for the help! I wish this will fix the trouble I am having !!!

BARDLER

Hey, would you mind uploading your meshes to download and look at? I could troubleshoot it right away.

MyMomHatesGames

BARDLER said:

Hey, would you mind uploading your meshes to download and look at? I could troubleshoot it right away.

Yeah Bradler

Thanks! its in FBX format if I need to change to OBJ just tell me. Again thank you in advance.

https://drive.google.com/drive/folders/0B61612bXyakMaHNBUndsVFpYZmM?usp=sharing

Kid.in.the.Dark

Hey @MyMomHatesGames,

I've had a look at your meshes and from what I can gather is that it's not so much to do with shading problems in maya, etc etc. I believe it's the way you've setup your UVs, smoothing groups etc.

Essentially, any angles that top 90 degrees you'd usually make sure those two surfaces are it's own smoothing group so the normals don't average with eachother and distort flat surfaces.

From how your normal map looked, it looks like you've just applied one smoothing group over the entire low-poly mesh which is why your flat surfaces have weird triangulated/warped surfaces since the low-poly itself looks that way.

There's a nice baking bible on Polycount which has been referred to more times than anyone could guess but it'll explain in depth about how you should be breaking up your surfaces with smoothing groups, separate UV islands, etc... instead of one fat piece which has all sides and angles welded together as a single island.

http://polycount.com/discussion/81154/understanding-averaged-normals-and-ray-projection-who-put-waviness-in-my-normal-map/p1

That is literally the only problems I can see with this model right now, as long as you get your head around sorting out your smoothing groups & uv islands, you'll be fine... in regards to the model it should be good enough to bake something out.

If you read through it and you still don't get some thing, just ask... any artist here would be willing to help you out and answer.

Keep going!

melli06

Yeah from what I can tell its mainly a problem with smoothing groups entirely. Should fix the majority of your problems.

Mathew O

I totally agree with the guys above me, the problem is coming for smoothing groups / lack of hard edges. My typical rule is to break anything over 40 degrees, nower days with substance, quixel etc you don't need huge islands.

Also, switch your tangent base to  mikks t space inside marmoset so that it's synced with xnormal

ActionDawg

If the tangent basis is synced and you have a sufficient bit depth and resolution (though these are slightly less important than syncing) then you can put things in one smoothing group all day every day.

These artifacts look unrelated.

Mathew O

somedoggy said:

If the tangent basis is synced and you have a sufficient bit depth and resolution (though these are slightly less important than syncing) then you can put things in one smoothing group all day every day.

These artifacts look unrelated.

Definitely, if you don't mind having gradients all over your normal map

ActionDawg

I don't.

Anyway more to the point, I'm doing a test bake of this. Perhaps later I'll have something to show.

Edit:

Mathew O

Ok well I'm glad you don't but let me give you 3 reasons as to why this isn't an optimal workflow.. outside of having un-synced engines.

1: LOD'ing will work far better if you aren't having to have all the geo required to keep a normal map with gradients from failing. If you plan on becoming a professional artist at some point you will need to make sure your work supports LOD'ing.

2: Mip'ing, will kill your normal. As soon as you lose texture resolution your nice gradient will become banded and become very apparent. 

3: For me personally and a lot of people will agree, it is alot easier to customise a normal map that doesn't have gradients but non subjectively... if you bake a texture that is going to be used on multiple props (which happens alot) such as bolts, grates, vents, destruction pieces, building modules etc you will find it far easier to build props to a pre-existing normal map that isn't full of gradients.

@metalliandy is to this day still the most knowledgeable person I know when it comes to baking so maybe he can steer you in the right direction @MyMomHatesGames

ActionDawg

I'm not here to talk about the pros and cons of using the technique as I understand them already and didn't share that information for good reason. I'm simply dispelling the bad advice given to the OP that averaged normals will result in an incorrect bake. Even you initially said this is the source of their problems before I said and showed it cannot be.

I didn't want to see the OP lead down a rabbit hole of useless debugging or incorrect thinking, when his issue clearly lies somewhere in the export or baking setup process and not in using one smoothing group on the Low.

Mathew O

Or we could teach him to bake well and correctly at the same time? Rather than teaching him bad habits

Edit: Refer to my previous post as to why baking with a single smoothing groups is a bad habit.

ActionDawg

Then it is a good thing you elaborated on certain edge cases where aggressive smoothing can cause trouble.

Another one you did not mention is compression artifacts can cause banding across large gradients. Or that even in a synced engine you can have problems due to UV precision (like UE4 which has a toggle for that) as well as the inherent precision of the normal encoding (like UE 4.12).

Mathew O

Edge cases? How many games are made without LODs and Mip Maps now?

I think it's good that you've proven that baking with one smoothing group is possible as long as you sync the tangent basis and don't allow for the 3 common issues I pointed out, but we can agree it probably isn't the best way to do it?

metalliandy

@MyMomHatesGames xNormal bakes will never be correct in Maya as it uses a different tangent basis to xNormal. You need to set Toolbag to Mikktspace under Tangent Space for all maps baked from xNormal and also check the Object space checkbox when using Object space maps or they will never render correctly (unchecked for tangent space maps).

You also need to add more dilation/bleed to the normal map when baking or you will get seams. 16px is a good value.


While having a single smoothing group for everything is entirely correct in theory it is not an optimal workflow at all and the results can be very poor in terms of quality (skewing, poor compression & banding due to there being insufficient resolution to be able to accurately represent the curvature of a given surface).

While the gradation of the normal map isn't directly linked to the issues Josh is having, I'm not sure we should be showing people to bake in this way until they understand what they are doing and how to compensate for issues that may arise. Shouldn't we be teaching people the best way to bake in order to avoid problems down the line?
These are all more problems than a beginner needs to deal with and by doing something reasonably simple like adding smoothing groups/hard edges around the UV islands they can all (mostly) be eliminated which allows them to focus on more critical things like topology, and getting the settings in Toolbag correct.

MyMomHatesGames

Hey guys! Thank you all for the feedback. So reading all this information, I will start back from step one and redo UV'ing for the mailbox.

I will make separate islands for things above things that are 40 degrees +, then make different smoothing groups like @Kid.in.the.Dark @Mathew O @melli06 explained.

As @somedoggy said I will try to read up on how to export in the best way and change settings in the way I bake my normals

After that I will generate normals, then see if looks good in marmoset and change settings for best quality that @metalliandy @Kid.in.the.Dark told me to change

Hopefully that will fix the problems. Again I learned so much and probably will learn so much more after this project.

Again I am very newb to all of this process and thank you guys for being so nice and patient

Mathew O

Please feel free to message me if I don't reply to any problems you might have, I feel invested in your success

MyMomHatesGames

Hey guys update.

Thank you everyone once again for all the helpful inputs. I finished fixing what is wrong and it looks way better, than before!





I re made the high poly tighter, since it was messing up my bake (thx @RaptorCWS)



I made hard and soft smoothing groups in the normal and re made the UV, with many more islands.



After that I made the normals, occlusion, and curvature maps







If there is anything that looks wrong feel free to leave a comment.

For now I am going to start working on texturing. I hope that goes smoothly

 >___<
 

Tectonic

looking much better! I'm glad you were able to sort things out.

Mathew O

Killed it! Looks great, feel free to max out the dilation on your maps once you're happy and move on to texturing, this helps when mip maps are generated to not give you any seems, although it looks like you have plenty of dilation it's always nice to see no real spare space. This is a preference thing though, not required at all. Great job @MyMomHatesGames !