Marmoset Consistently Crashes While Baking Objects with High Tris Count

Sidney Eliot

Marmoset Toolbag 4 crashes while baking at around 40-50mil tris for me (split among 120 high objects). Anything below that and it doesn't crash, but when I bake with objects that are more than approximately 40mil tris, Marmoset just closes without an error shortly after I click bake.

I monitored my hardware via the task manager (I'm on Windows 11) and nothing seemed to spike. Everything was at normal values during the crash, no RAM overflow or GPU spikes. The CPU goes to 100% when I click bake and then returns to 4% after Marmoset crashes, but that's normal.

- Sapphire Pulse AMD Radeon RX 5700 GPU (I use Adrenalin and the drivers are up to date)
- AMD Ryzen 9 5900X CPU
- 64GB RAM

I'm suprised it crashes, I would expect for it simply take long to bake which I would be fine with, but I'm not able to bake without it crashing.

I've optimized the high as much as possible and am not able to reduce the tri count by much more.

The Marmoset file is 2.5GB so I can't attach it, but if anyone would like to take a look at it, then I could upload it to some place like mega (the high fbx is 1GB).

EarthQuake

You are most likely running out of video ram. Toolbag's baker is GPU-accelerated and all mesh data must fit on the GPU. Here is some advice to stay within the limits of your GPU's video memory:

1. Close any 3D or 2D applications or any other VRAM-hungry applications that you may have open. Web browsers with many tabs open can take up quite a lot of VRAM as well.

2. Remove any unused meshes and textures from your Toolbag scene. If you have multiple bake projects in the same scene, create a unique scene for each one instead.

3. Reduce the bake texture resolution.

4. Reduce the bake texture bit-depth.

5. Decimate or reduce the sub-division level of the high poly mesh to reduce the triangle count.

6. If the high poly mesh has UVs but does not need them, delete them and reexport the mesh.

Sidney Eliot

Point 6 is a good one to know.

I'm curious, from a software architecture standpoint, is it not possible to load things into the VRAM sequentially and if the need arises for extra data storage during a bake, to use methods like temp storage.

Why would one baking group have to know of the existence of another baking group, the GPU could simply bake until its VRAM is almost full. Then it could write its baked results, of all completed baking groups into the texture file and remember what pixels on the texture it didn't write to (this will then be used later as a "mask"). Then the GPU could continue with the rest of the baking groups that it didn't do yet and after completing them it could write again into the texture, but this time using the "mask" from before to not overwrite the already baked areas.  
 
This process would be repeated every time a VRAM overflow is about to happen, thus preventing Marmoset from ever crashing no matter how intense the bake. 

(That's just a rough concept, the implementation logic would probably look quite different.)
 
Also, I'll loop back to what I said with "Why would one baking group have to know of the existence of another baking group". I say this because the only reason UV shells in a baking group need to know of other shells, is because the AO relies on cast rays interacting and bouncing off of all the objects in a baking group to properly place shadows (depending on the settings of course).

Sidney Eliot

The problem however arises again, when "ignore baking group" is enabled.

Sidney Eliot

I just think that having a poly limit for baking is something that Marmoset should try to overcome at some point, even if one does a good job in ZBrush to not overdue it with topology, there are just sometimes projects that have so many objects, and / or with small details, where one simply can't get away with sub 50mil tris on the high.

And yes one can bake one part of the character and then bake another part after that, but that is firstly extra effort, to maintain many different baking scenes and re bake all of them when one wants to change something small down the line, and secondly often not possible, if one wants cast shadows to be affected by the rest of the mesh. (Although this is getting less important with the increase in usage and importance of ray traced shadows in games. I'd actually say that the AO map itself will never go away as one can't simulate shadows in baked crevices, however the need for the "ignore baking group" option is less vital when ray traced shadows are used, and often even muddy the result and make the mesh less adaptive for different environments).

EarthQuake

The short answer is no, it is not possible to have only some of the mesh data loaded into VRAM. To be able to bake quickly, all of the geometry needs to be easily accessed from VRAM. That's what makes Toolbag's baker so fast.

Another thing you could try is going to Edit -> Preferences -> GPU -> and changing Ray Tracing from Accelerated / RTX to Generic. This will switch from the DXR ray tracing backend to the generic one, which has a lower memory footprint for the ray tracing acceleration structure. This will result in slower bakes (and worse performance with the RT renderer). So I would recommend changing it back when you're no longer baking with dense meshes.

If you're regularly baking 50 million or higher triangle counts, you should consider getting a GPU with more than 8GB of VRAM.

Sidney Eliot

Thanks for the reply. I'll try out generic RTX when I run into this issue in the future again, and yeah upgrading my GPU at some point would probably also be a good idea.

I also fully agree with you, that one of the reasons why Marmoset is so darn good, is because it's able to bake blazingly fast. One is able to do many minor adjustments to the lows topology and shading and in between the adjustments, constantly check the baked result with basically no waiting time.