Trouble setting up window transparency from Substance Painter export

Alfax

I'm relatively new to using Toolbag 4 and I can't seem to find any resources that handle transparency masks for objects that aren't purely made out transparent materials. I decided to test the transparency exports from Substance Painter by setting up an old scene on Marmoset, and unfortunately I can't get the transparency export to work at all. What I get is usually either a completely transparent glass pane, or a completely opaque one. Many older guides for Toolbag 3 refer back to Additive transparency, but all that's left in 4 is either Cutout or Dither, and the only way I've gotten my mask to work is to use Cutout in combination with a refractive transmission map.

Here you can see how the dirt and minor reflections in Substance Painter seem to be completely gone in Toolbag 4, despite all texture maps being unchanged from the SP2 export. Keep in mind this is with raytracing enabled and 8 transmission passes.

The transmission and transparency setups are what you see here. I greatly appreciate any feedback that can lead to a better import to MT4. Absolutely adore this program so far, but setting this up has been a bit more difficult than I could've anticipated.

EarthQuake

Refraction is the correct shading model to use for something like this. You'll want to create a mask and load it into the Mask Map slot in the Tansmission panel. White values = full refractive/transparent, while black values = opaque. For instance, if the metal bars were part of your glass texture, you would want those to be black in the mask. For dirt or dust, you would want that to be black or perhaps some value of gray if you want that to be slightly transparent. You can also try adding variation to the normals and/or roughness, making the glass rougher in certain spots can be an effective way to make it look dusty and old.

Cem Tezcan wrote a great breakdown on creating textures for and setting up glass materials in Toolbag 4: https://marmoset.co/posts/texturing-and-rendering-glass-with-ray-tracing-in-toolbag-4/

Alfax

Thank you so much for the quick response! And I'm truly sorry for getting back to you after such a long time. Things have been pretty hectic on my end. I checked out the tutorial you linked and it's just fascinating how flexible Marmoset can be when it comes to glass rendering, so I'm tempted to split the window and the glass into separate objects for better control over the material data.

I went ahead and created a mask for the Refraction channel, but it doesn't seem to actually do anything as is. Changing Reflectivity to Refractive Index makes it so that my Metalness map can no longer provide definition to the metal bars, so I'm not entirely sure I should keep this as a single mesh/texture set.

EarthQuake

I'm not sure I fully understand the metalness issue, but yes, generally, it's best to use separate materials/texture sets when you have something like glass. Firstly, because it will make it easier to control the material settings, but secondly because it's generally more efficient to do it this way if you're doing game art. If you have refraction or transparency on a large mesh but only a small section needs it, the renderer will need to calculate those expensive effects for more faces than it should. It's often more costly to add an alpha channel to a larger texture set too. With texture compression, adding an alpha channel will double the amount of video memory typically. So you can often create an additional, smaller texture set for just the glass or transparent area (ie: hair or similar materials), use less memory, and have a better-optimized asset.

Alfax

That makes a lot of sense! Thanks a lot for clarifying.