Home Technical Talk
The BRAWL² Tournament Challenge has been announced!

It starts May 12, and ends Oct 17. Let's see what you got!

https://polycount.com/discussion/237047/the-brawl²-tournament

Baking Problem in SP (Modelled in Max)

polycounter lvl 12
Offline / Send Message
michaelarby polycounter lvl 12
Hi guys. I have a problem. Experienced Max user here, and I was making a little stick grenade, just as a portfolio piece. Trouble is, I cannot for the life of me ge it to bake properly. Shown below are the low poly and high poly, along with the object names. 
The head section of the high poly has been split into multiple objects, but all named as head_high (this I believe should be fine for the bake?). Most of the hp is just chamfering of edges. nothing too fancy. 

Using Max 2021, and an older version of SP. 

Problem is, I keep getting crappy bakes in substance, like below:

I've made sure both hi and low are exported an the origin point and overlap perfectly in max. 
I've tried resetting xform on all objects in max
I've Converted all objects to editable mesh, then back to editable poly. 
I've unified normals on every object, both high and low. 
I've tried baking Match method to 'always' instead of 'by mesh name' 
I've tried using the low poly mesh with smoothing groups applied, and no smoothing groups. 

I've baked dozens of objects in SP before, never had trouble like this. I cant think, what simple, stupid thing am I missing. Would appreciate any help. This is an older model i started a few years back and just wanted to finish it - could it just be corrupted somehow?

 Cheers

EDIT - EXTRA INFO:
File was exported as fbx, both the low and the High. 
UV Map is shown here:

I have attached my high and low poly fbx files, as well as the 3ds max file. the max file might be slightly different to the fbx as i have tinkered with it slightly since!

Replies

  • Eric Chadwick
    What do the UVs look like in Max? Are they all welded, and all facing forward?

    What does a regular uv checker grid look like on the lowpoly (with letters and numbers, something like this: http://wiki.polycount.com/wiki/File:Uvrefmap_grid_small.jpg) 

    A couple things to try:

    Make sure the verts aren't split:
    Vertex Weld modifier, tweak the threshold.

    Remove all edited normals:
    Edit Normals modifier on the low, "Normal" sub-object mode, select all, hit the Reset button.
    Collapse the stack.
    Smooth modifier, Auto Smooth.

    What file format are you exporting from Max? If FBX, have you tried a different version of the exporter? Or better yet, try OBJ instead.


  • Fabi_G
    Offline / Send Message
    Fabi_G veteran polycounter
    Hi! Could you share the meshes, so users can reproduce the issue on their end? If you don't want to share the whole object, you could isolate a part the issue can be reproduced with. A zipped archive can be directly attached to a post.
  • michaelarby
    Offline / Send Message
    michaelarby polycounter lvl 12
    Fabi_G said:
    Hi! Could you share the meshes, so users can reproduce the issue on their end? If you don't want to share the whole object, you could isolate a part the issue can be reproduced with. A zipped archive can be directly attached to a post.

    Thanks FAbi - I have attached the 3ds max file, as well as the fbx files of the high and low poly to the original post. 
  • michaelarby
    Offline / Send Message
    michaelarby polycounter lvl 12
    What do the UVs look like in Max? Are they all welded, and all facing forward?

    What does a regular uv checker grid look like on the lowpoly (with letters and numbers, something like this: http://wiki.polycount.com/wiki/File:Uvrefmap_grid_small.jpg) 

    A couple things to try:

    Make sure the verts aren't split:
    Vertex Weld modifier, tweak the threshold.

    Remove all edited normals:
    Edit Normals modifier on the low, "Normal" sub-object mode, select all, hit the Reset button.
    Collapse the stack.
    Smooth modifier, Auto Smooth.

    What file format are you exporting from Max? If FBX, have you tried a different version of the exporter? Or better yet, try OBJ instead.


    Hi eric, thanks for your response. UV's seem fine from what I can tell - I've attached my unwrap to the original post. ive tried your vertex weld modifier, and the edit normals modifier (did that on both the high and the low. but still giving a similar result. i was exporting as fbx, but this time I tried obj to see if it made any difference, but alas no!

  • Fabi_G
    Offline / Send Message
    Fabi_G veteran polycounter
    Hi @michaelarby Throwing the fbx files into Blender, it looks like both low and high poly have parts with inverted normals: 

    I flipped those and it bakes as expected in Painter. 
  • Fabi_G
    Offline / Send Message
    Fabi_G veteran polycounter
    Additionally, here are some general notes from looking at the meshes:
    • Highpoly (for example 'head_high') has some hard edges which will show in the normal map, would shade those soft
    • I think low poly UVs are a bit segmented while meshes are smooth for the most part. I would either add some hard edges where it makes sense (steep angles) and have the UV splits there, or could have more continuous UVs (think cylindrical mapping with some tweaks).
    • pin lowpoly has some hard edges without UV splits, that looks unintentional to me.
  • michaelarby
    Offline / Send Message
    michaelarby polycounter lvl 12
    Fabi_G said:
    Hi @michaelarby Throwing the fbx files into Blender, it looks like both low and high poly have parts with inverted normals: 

    I flipped those and it bakes as expected in Painter. 

    Thank you so much for checking that, and verifying thats the problem. I've not had a chance to open Max again yet, but glad I know what to look for now!
    I'm bamboozled though - I checked for flipped normals, unified them, flipped them etc and thought I had eliminated that possibility!

    Can I ask - I'm a bit lost with normals as a concept. When I flip faces, is that also flipping the normals? or are they separate? I flipped all the faces on the low poly, and had them facing outwards - in max, you cant select edges on an inward facing face, so I know they SHOULD have been the correct way. 

    Also - are there face normals and vertex normals? whats the difference? and do the modifiers we've mentioned work on one or the other? or both?

    Thanks again.
  • michaelarby
    Offline / Send Message
    michaelarby polycounter lvl 12
    Fabi_G said:
    Additionally, here are some general notes from looking at the meshes:
    • Highpoly (for example 'head_high') has some hard edges which will show in the normal map, would shade those soft
    • I think low poly UVs are a bit segmented while meshes are smooth for the most part. I would either add some hard edges where it makes sense (steep angles) and have the UV splits there, or could have more continuous UVs (think cylindrical mapping with some tweaks).
    • pin lowpoly has some hard edges without UV splits, that looks unintentional to me.

    Thank you for this! some of these are really helpful
    1 - yes, some areas, such as the square bumps in the middle of the head, i left unchamfered purely out of haste. should the high poly have any hard edges at all? is it only the low poly in-game mesh that needs hard/soft setup? are there benefits to hard/softing the high poly?

    2 - Will do - thats a great article, though the other sections have left me with even more questions now lol.

    3 - THANK YOU!!!! SO MUCH!!!! This issue has been bugging me for years and i never found a proper answer for why it happens. I've seen some hacky fixes for it but this will prevent so many problems for me going forward! Question - those hard lines on the seams in that image as well - are they caused by hard edges that should be soft on the low poly? or are they they just a result of the aliasing artifacts?

    4 - Funny - this relates to point 3 - i read somewhere that smoothing the whole low poly mesh will fix those aliasing seams. this may be incorrect. I also heard recently the 'proverb' - 'not all seams are hard edges, but all hard edges are seams'. Am i better to have hard edged areas on my low poly then?

    5 - possibly! ive tweaked and retweaked this model so many times im sure i missed something in my haste! ive actually UV packed it twice! for some reason the first (better) unwrap didnt save!

    Apologies for all the extra questions but I really appreciate your help!

  • Fabi_G
    Offline / Send Message
    Fabi_G veteran polycounter
    Hi! Hm, interesting that you checked in Max but normals are flipped in the fbx. A simple way to check, would be to re-import the exported fbx and check that everything looks as intended.

    Since its normals define in what direction a face is looking, I would say flipping faces and flipping normals describes the same outcome. I think face normals are possibly a construct for viewing, and I would expect the normal at the center of a face to be the average of the vertex normals defining it. But with such things don't believe someone on the internet better read up :D

    1 - Hard edges in source mesh to transfer information from ('highpoly'): I would say the simple rule here is, if you don't intend there to be hard edges, don't have them. Personally, I don't see a reason for having hard edges in a highpoly, since by there should be enough geometry to define the shading.

    2 - I think the main thing is to finish projects, any question/ issues that comes up you could share here. With project that will take longer or you want eyes on, you could create a dedicated thread for it. There's also a wealth of old threads you can comb through with the search function.

    3 - I would say those hard lines on the seams in that example image stems from a combination of things:
    - Lowpoly has hard edges and split UVs opposed to an all smooth mesh. You can do this to remove shading gradients from the mesh, meaning the normal map has to compensate less.
    - When you have a rounded bevel you will likely spot a seam at hard edges/ seams, since your baking some normal-data that's interpolated for each point on the surface into a pixel grid of fixed resolution where each cell can only hold limited amount of information (color depth). In practice, with more surface information on top, exported normal map being 8bit+dithered, it will often be negligible. You could work some against this by have a flat bevel instead of a rounded one, so the normal map color at the seam is more or less flat.
    - Seams showing gets exaggerated when the they are not aligned but cut the pixel grid diagonally or when the resolution is too low (not enough pixels for the edge detail).

    4 - How/if you use hard edges is of course up to you. But if you use hard edges and bake a normal map, you have to split the UVs to avoid artifacts. The benefit of using hard edges is that you can get the lowpoly shading to closer match the highpoly since by shading edges hard, you're removing shading gradients from the lowpoly. This means the normal map will have less gradients and is easier to reuse with other models that have similar shading (LODs). When you have UV splits along edges with a steep angle, it makes sense to add a hard edge in my opinion, as UV splits mean doubled vertices on GPU anyways. I must say, when making reasonable decisions, vertex count becoming an issue is unlikely. Another way to match the shading of low and high would be more geometry (support loops, bevels). When making choices, my number one priority would be to create a good looking asset.

    5 - I see, sorry to hear you lost work. Again, here I find it helpful to check the export, either re-importing or, more conveniently imo, in engine. Given Painter re-projects brush strokes, I would iterate on topology, shading and UV layout until it's good, no waterfall approach.

    Good luck with your project!
  • michaelarby
    Offline / Send Message
    michaelarby polycounter lvl 12
    Fabi_G thank you so much for taking the time to respond - thats a great help! Going to play about in Max now and try to finish this thing off!
  • Eric Chadwick
    Best way to check for flipped faces is to enable Backface Culling. In Max you do this in Object Properties. Then just tumble the viewport, it’ll become quite obvious what’s looking inside out.
  • Noren
    Offline / Send Message
    Noren interpolator
    michaelarby said: I'm bamboozled though - I checked for flipped normals, unified them, flipped them etc and thought I had eliminated that possibility!

    Can I ask - I'm a bit lost with normals as a concept. When I flip faces, is that also flipping the normals? or are they separate? I flipped all the faces on the low poly, and had them facing outwards - in max, you cant select edges on an inward facing face, so I know they SHOULD have been the correct way. 

    Also - are there face normals and vertex normals? whats the difference? and do the modifiers we've mentioned work on one or the other? or both?

    Take this with a grain of salt as I'm not a technical artist, but at least in 3ds max, you indeed have face normals and vertex normals. Maybe you can think of the face normals as the actual geometry (or purely in which direction a triangle (maybe polygon) is facing. That's dependent o the vertices, but not the vertex normals, which you can think of as the shading.  The shading can point in the opposite direction of the face without the face being flipped. While you can assign a temporary custom face normal via maxscript, that's not preserved in the mesh. Also, changing the face normal does reset the vertex normals.
    The Normal modifier and flipping faces manipulates the face normals (while resetting changes to the the vertex normals, like said) and the Edit Normals modifier works on the vertex normals.

    One way to determine which way a face is looking is the winding order. Max sticks to that as well when you create a polygon from scratch. It matters if you go clockwise or counter-clockwise. But then things get complicated fast, e.g. if you have a quad, one triangle will have to be CW and one CCW in order to be able to share an edge, so there's probably a separate index or parameter for the face orientation and different software and ex and importers might interpret those things differently or things get messed up some other way. So if you find there's a problem with your mesh after doing some flipping and unification, a good standard way to resolve it is to create a new object and attach your mesh to it (another would be Reset Xform, especially if the mesh was mirrored at some point).
  • michaelarby
    Offline / Send Message
    michaelarby polycounter lvl 12
    Best way to check for flipped faces is to enable Backface Culling. In Max you do this in Object Properties. Then just tumble the viewport, it’ll become quite obvious what’s looking inside out.
    This was really helpful! Have added this to my notes!
  • michaelarby
    Offline / Send Message
    michaelarby polycounter lvl 12
    @Noren thank you for clarifying about the normals - this whole thread has been very educational! 

    I've took all pn board and after lots of tweaking i fixed everything up, and even optimised my UVs some more.

    Im still getting some shadowing on the bake, but I think this is a result of the low poly being too low, and not matching the high close enough.  (Not at my computer right now but will attach pics to this post when i next am.)

    So now just need to do my texturing!
  • iam717
    Offline / Send Message
    iam717 sublime tool
    @michaelarby glad it was solved, i tend to give a crap about people not being "robbed", so if you could now remove your work from the post since this is solved can eliminate some predatory person/s from claiming the piece and using it for their own means and profit.  

    I'd like a script if possible where this would be done automatically upon "check marker", solved post option.  That is wishful thinking but would make things more stream lined when it comes to situations with "sharing" things for help.

    Again glad this is solved, looking forward to checking out the end results of it or where ever it is being used.
    To the solvers & community contributing users, thanks.  Just because.
Sign In or Register to comment.