problems with baking ( AKM tutorial )

Nanashi

Hello guys, 

I've been following the Ultimate Weapon Tutorial - by Tim Bergholz. And it is a very great tutorial with a lot of new techniques and tips but i'm stuck at the baking part in  substance painter and before diving into the problems and what I have tried I want to share what i did before taking the model to substance painter  for baking. 

* I'm a Maya user so when Tim  was using the smoothing groups in 3DS Max it was something new to me and after doing some research the conclusion was that Maya only have soften/ harden edge only and no such thing as smoothing groups, so I did soften all the edges and left the seems edges hard.

The highlighted edges (orange ) I made them Hard and the rest of the edges ( light blue ) I made them soft 

 is what i did correct and is it equivalent to the smoothing groups in 3Ds Max ?? and is doing this important ??

after doing that I exported the model into substance painter and started baking and here is my naming, uvs and low/high poly before baking

The lowpoly naming in the outliner and everything ends with ( _low)

The highpoly naming in the outliner and everything ends with ( _high)

The lowpoly model

Wireframe

The highpoly model

Wireframe

the loypoly/Highpoly placement 

The UVS

and now I will show you guys the baking settings and what I face when I bake, note that i'm using substance painter 2017.3.3.

when I import the low poly (FBX) these are the settings 

and when I want to bake I just followed what Mr.Tim did in the tutorial and used the same settings 

when I press bake I noticed I get this warning so i"m not sure if this have anything to do with the bake result??

After baking  the first problem I noticed that the floaters elements are not baked 

 

and its like this for the other floaters as well. 

On the other side I have this strange looking color on the screws and that piece

Also I noticed these wired artifacts on the model , its not smooth and looks very harsh 

I'm sorry if the post is long , but I put everything in detail so when you guys help me it easier for you to find the problem and solve it.

Andreicus

1) Regarding the hard/soft edges stuff...in Maya i never do that, as you said it is not like 3ds Max where you must specify all the smoothing groups, in Maya the only thing that you need to do is to mark the seams for UVs.
When you mark a seam, Maya will change the edge to "hard" automatically.

2) The meshes names are ok

3) The subsampling at 8x8 is useless when you are testing out the bake, if you get errors at 0 you will get errors at 8x8, the only changed would be that they will be more defined, lol
I usually test the bake at 2x2 and if everything is ok i up it to 4x4 or 8x8 if i'm in the mood to wait for the renering

4) "Mesh normals are invalid", this is a common problem with Maya, it is not Substance fault, it is Maya that is broken ( did you hear me Autodesk ? Good... ), it gives you no errors while modeling...but after you export the mesh you will get the error.
I think that the cause is due to the modeling of the mesh itself. When you extrude a face up or down that is pratically against the edge of another face, Maya fucked up. 
You have to absolutely fix it or you will get nasty shading errors on the mesh. Try deleting all the materials and textures on the mesh, move the light around and you will probably see some "diagonal lines" passing through the mesh somewhere and the mesh will be overall brighter, this is bad.

Now the solution...unfortunately if all the normals tools don't works like "set face normals", "set vertex normals", "unlock normals" you need to re-model the part that gives you problems while paying attention to leave some space between the edges borders of the various faces ( it is hard to explain ), this is the only solution that i found...

5) Strange looking color of the screws, read answer number 4 ^

6) Regarding the artifacts unfortunately i don't know what the problem could be, the only thing that i can tell you is to check out the silohuette of the low poly and high poly because they should match as closely as possibile especially the curved borders and check out if the UVs of the pieces where you have artifacts are straight

Nowadays i pratically don't bake anything anymore, i use a medium poly workflow that means, in my case, i model the object as i want it to be and then optimize it to a reasonable polycount without making the high poly and if i need to add details like bolts, screw or other little things i paint them directly in Substance Painter with hard surfaces alphas.
For other details like scratches, dirt etc. i use the standard Substance Painter workflow.

Nanashi

Andreicus said:

1) Regarding the hard/soft edges stuff...in Maya i never do that, as you said it is not like 3ds Max where you must specify all the smoothing groups, in Maya the only thing that you need to do is to mark the seams for UVs.
When you mark a seam, Maya will change the edge to "hard" automatically.

2) The meshes names are ok

3) The subsampling at 8x8 is useless when you are testing out the bake, if you get errors at 0 you will get errors at 8x8, the only changed would be that they will be more defined, lol
I usually test the bake at 2x2 and if everything is ok i up it to 4x4 or 8x8 if i'm in the mood to wait for the renering

4) "Mesh normals are invalid", this is a common problem with Maya, it is not Substance fault, it is Maya that is broken ( did you hear me Autodesk ? Good... ), it gives you no errors while modeling...but after you export the mesh you will get the error.
I think that the cause is due to the modeling of the mesh itself. When you extrude a face up or down that is pratically against the edge of another face, Maya fucked up. 
You have to absolutely fix it or you will get nasty shading errors on the mesh. Try deleting all the materials and textures on the mesh, move the light around and you will probably see some "diagonal lines" passing through the mesh somewhere and the mesh will be overall brighter, this is bad.

Now the solution...unfortunately if all the normals tools don't works like "set face normals", "set vertex normals", "unlock normals" you need to re-model the part that gives you problems while paying attention to leave some space between the edges borders of the various faces ( it is hard to explain ), this is the only solution that i found...

5) Strange looking color of the screws, read answer number 4 ^

6) Regarding the artifacts unfortunately i don't know what the problem could be, the only thing that i can tell you is to check out the silohuette of the low poly and high poly because they should match as closely as possibile especially the curved borders and check out if the UVs of the pieces where you have artifacts are straight

Nowadays i pratically don't bake anything anymore, i use a medium poly workflow that means, in my case, i model the object as i want it to be and then optimize it to a reasonable polycount without making the high poly and if i need to add details like bolts, screw or other little things i paint them directly in Substance Painter with hard surfaces alphas.
For other details like scratches, dirt etc. i use the standard Substance Painter workflow.

thanks a lot for your reply and the information, will defiantly try to fix the normals and check the model again and see the match up between the low and the high poly and I hope it will work , thanks again  

Thanez

I have to say, 10/10 on your OP! It's very easy to help you because of how you laid all of this out. 
I can't help you with the maya stuff as I'm a Max guy myself, but:
Hard edges and smoothing groups are pretty much the same thing, but each attack the situation from a different standpoint. Smoothing groups say which polygons should be smoothed together. Where you have a split in smoothing groups you get a 'hard edge'. Maya's workflow is the reverse, defining which edges should break the smoothing of polygons.

I noticed you put a hard edge along the top of your stock. You don't necessarily need a hard edge where you have a UV split, but you absolutely need a UV split where you have a hard edge. If your normals are >1024², you won't notice, but at lower sizes/mips that hard edge will become more and more apparent.

I haven't done any baking in substance painter, but for the floaters that are missing, check that their suffix is proper, or try and add them to the stock object. 

The errors at the end of your post look like projection issues due to a malformed cage or that you're baking without one. Need more input from you to give more halps
See: http://wiki.polycount.com/wiki/Texture_Baking#Cages

Nanashi

Thanez said:

I have to say, 10/10 on your OP! It's very easy to help you because of how you laid all of this out. 
I can't help you with the maya stuff as I'm a Max guy myself, but:
Hard edges and smoothing groups are pretty much the same thing, but each attack the situation from a different standpoint. Smoothing groups say which polygons should be smoothed together. Where you have a split in smoothing groups you get a 'hard edge'. Maya's workflow is the reverse, defining which edges should break the smoothing of polygons.

I noticed you put a hard edge along the top of your stock. You don't necessarily need a hard edge where you have a UV split, but you absolutely need a UV split where you have a hard edge. If your normals are >1024², you won't notice, but at lower sizes/mips that hard edge will become more and more apparent.

I haven't done any baking in substance painter, but for the floaters that are missing, check that their suffix is proper, or try and add them to the stock object. 

The errors at the end of your post look like projection issues due to a malformed cage or that you're baking without one. Need more input from you to give more halps
See: http://wiki.polycount.com/wiki/Texture_Baking#Cages

thanks a lot for the info and the help, I really appropriate it and indeed for the floaters it was a naming issue and I fixed it , now I have this problem where the normal map result is pixelated and I asked Andreicus for help and this is what i'm trying now but it wont work. 

for a start this is how the normal map looks ( notice how is looks pixelated or more like faded on the marked ares )



so I only took the stock for a test and this is the UV island size ( the problem might be the low texel density  according to  Andreicus ) 



this is the baking with anti-aliasing 8x8  and as you can see there is some pixels 





but then I did a test with a low poly cube and a high poly cube with different shapes on each side 

Low poly Cube 



The  UV island 



The high poly cube sides 



and this is the baking result and it looks perfect!!!



same settings, same principle applied to both models and one with a bad result and the other is perfect, i'm getting more confused 

please help  

Thanez

Firstly, A+ on actually testing shit because nothing made sense. It's a thrill to help you.
Second, painting on normals will never look as nice as a baked map with 8xAA. If you want AA on your painted details, make sure you do your work on a canvas twice the size of your output: Want 2048² textures? Work in 4096².
That said, I'm pretty sure substance painter has different resolution settings for stuff, as I found this: https://forum.allegorithmic.com/index.php?topic=7904.0 Hope it helps.
An easy way to think of texel density on UVmaps where you have many-a-pieces, is to think like this: I have a 1024x1024 map, and my stock takes up 1/4 of it. Only half of that (one side) will be visible at one time. Thus one side is only represented by 512x256 pixels. Is that going to be enough for what I want? If I paint a circle that covers 1/10 of that, it's gonna be represented by ~50x50 pixels. Is that circle gonna look like shit? How big will it be rendered on screen? How many pixels on your monitor will be rendering those 50x50 pixels? How low can I go before it starts being noticably below the fidelity I want? I should test that scenario.

Also worth saying is that what you're noticing on the normals right now won't look as harsh once you have all the other texture maps working together and you stop looking at it with a microscope. If you do things correctly, the normals will only help sell the idea that there's real depth to your texture.

Nanashi

Thanez said:

Firstly, A+ on actually testing shit because nothing made sense. It's a thrill to help you.
Second, painting on normals will never look as nice as a baked map with 8xAA. If you want AA on your painted details, make sure you do your work on a canvas twice the size of your output: Want 2048² textures? Work in 4096².
That said, I'm pretty sure substance painter has different resolution settings for stuff, as I found this: https://forum.allegorithmic.com/index.php?topic=7904.0 Hope it helps.
An easy way to think of texel density on UVmaps where you have many-a-pieces, is to think like this: I have a 1024x1024 map, and my stock takes up 1/4 of it. Only half of that (one side) will be visible at one time. Thus one side is only represented by 512x256 pixels. Is that going to be enough for what I want? If I paint a circle that covers 1/10 of that, it's gonna be represented by ~50x50 pixels. Is that circle gonna look like shit? How big will it be rendered on screen? How many pixels on your monitor will be rendering those 50x50 pixels? How low can I go before it starts being noticably below the fidelity I want? I should test that scenario.

Also worth saying is that what you're noticing on the normals right now won't look as harsh once you have all the other texture maps working together and you stop looking at it with a microscope. If you do things correctly, the normals will only help sell the idea that there's real depth to your texture.

what you said at the end is correct once I bake everything, it will all work together and it will look less noticeable so I will just keep working with that and see the final result, its not the perfect bake that I wanted but I learned a lot and my next time will be better.

thanks a lot for the help