Optimize model for AO baking?

bitinn

Hi,

I am back with another baking issue: I have a low poly model and one subd model, but I am not able to create a decent bake at the intersection between the cushion and the sofa itself.

I have added support edge loops to make subd model hold its shape (for example the edge of cushion). But I run into a situation where my AO either get pure white region (caused by Ignore Backface option) or pure black region (caused by small Frontal/Rear Distance).

I can disable the Ignore Backface option, but the bake is much more dirtier without it. I read this thread and tried adjusting other option, I am not sure what else I can do to make it better.

Is there a problem with my model? Would using Toolbag instead of Substance Painter help me in this case? 

Any help appreciated!






TO ADD: the result of baking from low poly is better BUT I previously had another issue with using only low poly for baking. So errr, I am not sure what I should do here:

Eric Chadwick

Since your queries are about Substance Painter, they should go in the Allegorithmic section instead of Technical Talk. You'll get better feedback there, direct from their support techs. I'll move it for you.

Also you should add the program names to your topic titles. This will also pull more relevant eyeballs.
http://polycount.com/discussion/63361/information-about-polycount-new-member-introductions/p1#use

bitinn

Eric Chadwick said:

Since your queries are about Substance Painter, they should go in the Allegorithmic section instead of Technical Talk. You'll get better feedback there, direct from their support techs. I'll move it for you.

Also you should add the program names to your topic titles. This will also pull more relevant eyeballs.
http://polycount.com/discussion/63361/information-about-polycount-new-member-introductions/p1#use

Thx, I hesitated about it, as I wasn't sure the problem is with SP...

bitinn

I gave marmoset toolbag a spin and it seems like there are no way for my model to bake well given the intersection. I could use the two-sided option but it just occludes the intersection to fully black, which is also unrealistic.

Guess I have to go back and design a square-ish pad that fully cover the seat instead...

pior

One very simple solution that works in any program consists of :

- Splitting off the cushion from the rest, along the problematic concave edge
- Moving it up (both in the lowpoly and highpoly) by a known amount of units. The resulting chair and cushion parts don't have to be two models - they can remain one model, just with spread apart parts.
- Baking
- Moving the parts back together
- Welding the edge, but keeping it as hard.

(This of course requires that the edge in question is of high enough polygon density. But that is pretty much always possible when dealing with uniquely unwrapped assets like this, since you already have a budget generous enough to allow for unique textures per asset. And of course, hard edge means that split UVs are needed, as usual).

Overall the downside is that the main part of the chair will not cast occlusion over the sides of the pillow, but I suppose this can be dealt with easily with some cleverly placed proxy objects.

EarthQuake

I see you're trying this out in Toolbag, the simplest solution is to use bake groups. Put the base of the chair in one group, and the pillow in another group. That will solve the intersection errors.

Then, turn on "Ignore Groups" to get AO to cast between the two groups.

How to do this is explained here: https://www.marmoset.co/posts/toolbag-baking-tutorial/ - go to the Toolbag Setup section, then Bake Groups.

I think Painter has some similar features to do this as well.

To prevent this problem in the future, generally an asset like this would not be two mesh chunks in the low poly. What you would do is create one watertight chunk where the pillow is integrated into the main section. This will save polygons and give you more texture space as you're not wasting either on the underside of the pillow, and make it easier to bake.

You can keep the highpoly as two objects though, there's no real requirement for your high and low to be the same number of objects. It's usually best to model the high poly as it would be in real life, separate meshes for the separate parts, but think of the lowpoly as more of a "shell" where you only model what will affect the silhouette and be visible to the player.

Generally the only reason to have the pillow modeled as a separate chunk would be if the pillow was removable or damageable in game.

bitinn

@EarthQuake

Thx for the suggestion, I am aware of "Ignore Groups" option in Toolbag as well as the "Same mesh only" AO baking in SP. But I would like a bit clarification: I believe either I get a result where cushion and chair are baked separately, hence no occlusion between them, or I get a result where cushion and chair affect each other.

Do you mean that, using "Ignore Groups" option with 2 groups, I can somehow get the best of both world? Where I get occlusion between cushion and chair, while not triggering the artifact? I believe this is not possible in SP.

I gave myself some slack on the mesh design for sure, because I believe I would need to orient/resize the pad in future.

@pior yeah I think that's pretty much my understanding, I want to preserve the AO but the gaps under cushion seem like a big challenge to occlude correctly (when I got 2 meshes instead of 1).

OH wait, do you mean split only the cushion edges near chair from the rest?

EarthQuake

@bitinn yes, Bake Groups + Ignore Groups in Toolbag should give you the best of both worlds. You can use baking groups to isolate the projection so you don't get projection errors when the baker casts on the wrong mesh (ie, from the pillow to the chair).

Then, the Ignore Groups AO option tells the baker to use groups for projection, but not for AO. In other words when the baker searches for the surface it only finds the surface in the group, but when it calculates AO, it takes the other groups into consideration.

Here's a practical example:



Here the chair, pillow and legs are split into 3 separate bake groups. This means we don't get any intersecting errors, everything is super clean.



This is what the default AO looks like, it's clean but we're not getting AO between the groups.



This is what we get if we turn on ignore groups, it's still clean (no intersection errors), but we're getting inter-group AO now too. I turned on Two-Sided there too to make sure that the full occluded intersection points between the groups show up as black (they will show up as white otherwise).

Here's the Toolbag scene, OBJ file and normal/ao map: https://www.dropbox.com/s/9wciv16sc06zhdp/chairbake.tbscene?dl=0 - please forgive the messy geometry and UVs, I only spent a few minutes on this.

pior

@bitinn :
 
EQs example above is precisely why Toolbag is imho by far the best baking solution out there - It's actually made by actual users who know about this sort of issues first-hand. This + the fact that it allows for direct manipulation of objects in the scene at any time makes it a no-brainer in my opinion. It's really that good !

Separating baking and texturing between TB3 and SP is really not much of a time loss, and probably even saves time in the long run since it allows for more flexibility compared to trying to do everything "blind" in SP.

bitinn

@EarthQuake

Thank you for putting an example together! I have some follow-up questions at the end of this reply, but they are nothing major.

@pior

Yeah a few features in Toolbag might just leave me wanting to always bake in it.

----

I don't want to turn this into a Toolbag vs Painter baking thread but since I am doing this anyway, here is my result (apologize for heavy images):

- First, I went back and separate my model into 3 groups: seat, pad and legs.

- Now I tried to bake AO using similar setup in Painter and Toolbag (adjust some settings for my personal taste).

Painter:





Toolbag:



Results ("the best" result I manage to get using a capped 256 ray count):

Painter:



Toolbag:



----

My personal reflection as a new user of Toolbag:

1) I enjoy the visual cage, it took me a while to figure the perfect frontal/rear distance for my cage in Painter.

2) Per-group cage settings are critical to reducing the unnecessary white gap.

3) "Two-sides" option, I think it's more or less a hack, but it got the job done! It wouldn't be possible in Painter.

4) "Floor occlusion" option, this is helpful for my case as I am making a more stylized asset. I would otherwise use Position map to achieve the same thing using material, but being able to see floor AO immediately certainly helps.

5) I understand many of these benefit I could workaround without Toolbag: I could put a floor in the scene for Painter to bake floor AO; I could make my high poly model fit better so per-group cage are not needed; I usually use 0.01/0.01 for cage distance anyway, so having to adjust them means I could have done a better job at modeling.

6) But hey Toolbag got the job done, even when I kinda suck, that's pretty great!

7) and I must praise whoever took their time to write proper tooltip for each options. Painter's baking option description are often, well, less than useful.

But to be fair to Painter, I did went into some troubles with Toolbag:

A) AFAIK Toolbag only recognize FBX version up-to 2015, import anything newer and nothing happens. Well, something did happen underneath, I just have to look at the console to find out

I am seeing some very obvious black spots when baking AO in Toolbag, I set Sample to 16x to make them less obvious. But I have no idea if it's the right way to fix them?



C) Painter's AO baker, when using its "cosine" ray distribution, results in much more focused occlusion; along with its "smooth" attenuation option, could look quite good in some cases. Does Toolbag want to implement such a thing?

----

Alright, this reply is getting long enough. I apologize for doing this in the Allegorithmic forum, I didn't know it would end up like this

EarthQuake

@bitinn

A. Toolbag supports FBX 2015 but not newer versions, that is correct. Make sure to set FBX 2015 in your export settings. This is something we will probably update in a later release.

B. This may be a bug, could you zip up your high and low poly meshes along with your .tbscene for me to look at? I see you're on a mac as well, if you have a PC to test this on I would be curious to know if you're seeing the same issue.

C. Toolbag does not have anything akin to "smooth attenuation" currently. By default the AO method is cosine, but there is also a uniform option which you can try (I tend to like uniform better). More controls for this is something that we may potentially add in a later version.

Also, I moved this thread to Tech Talk as it is more of a general thread than a software specific thread at this point.

bitinn

@EarthQuake

These are the models/scene file/bake result. I haven't tried it on Windows yet as my workflow is entirely on macOS. Thx for looking into it.

EDIT: I should add the black spots only appears when Two-sides option is on.

EarthQuake

I was able to reproduce this and have logged it in our bug system. I'm seeing the same result on PC, so don't bother trying to find a different computer to test it on.