Vertex Colour Masks

Chrismartinartist · May 23 2025

Hello, I'm fairly new to SP, and just wanted to see if there's a better workflow for what I'm trying to do.

Currently I've been assigning Vertex Colours (in Blender via TexTools), but when I'm in SP, if I'm playing around with smart materials, generators and such, I'm finding it hard to get the effects just on the areas I want it. It seems as though the way I'm doing it the mask for the layer/smart material/generator fights or overwrites the colour selection which seems to also act as a mask. I had a play around with smart materials/generators on (what is essentially) a glorified shader ball, and found it fairly easy to get what I wanted to happen, i.e. masking out areas, using generators, filters and paint layers etc to layer up. It just seems when adding in Vertex Colours it becomes a problem to do this.

Is there a correct way to layer with the Vertex Colour technique? Or is there a better way of doing this?... The goal being to have control over texturing individual parts of a single object within a single UV.

Thank you for any advice

Fabi_G · May 23 2025

When composing a mask from multiple layers, they need to use an appropriate blend mode. If it's left at 'replace' the layer on top will replace the one below.

On more complex models I often create folders for the different material IDs with the corresponding mask.

Other than vertex color, I like to use different materials that are assigned to different parts of the highpoly

Perhaps I didn't get the question :P

poopipe · May 25 2025

deriving masks from a coloured texture is almost always a bad idea - you can do up to 4 channels if you are able to split by channel and use alpha (I don't think painter has anything built in that supports that) . for anything beyond that you're just doing a color-pick and you'll get artefacting at edges.

Either just paint the masks in painter or bake a grayscale texture for each separate mask.

The former is less bother in the long run and would be the preferred option for anything that isn't difficult to paint around.

gnoop · May 25 2025

It's so annoying neither Designer no Painter could bake or use cryptomatte exr that could store all the perfect masking in one file. I am regularly asking for the feature for decade already probably.

poopipe · May 26 2025

I messed around with the idea of using deep EXRs (which i assume a cryptomatte is or it wouldn't work properly) to manage texture sets a few years ago and while it worked fine the additional overhead of processing and previewing the data - as well as maintaining any tooling - far outweighed the tiny benefit you get from having everything in a single file.

packaging this stuff up in a single file makes sense in vfx/film-making where you have a lot of disparate data types that all need to be composited together and passed around between multiple studios but for video games where 99% of the time you put textures in a folder and never move them because version control exists there's effectively no benefit

gnoop · May 26 2025

poopipe said:

I messed around with the idea of using deep EXRs (which i assume a cryptomatte is or it wouldn't work properly) to manage texture sets a few years ago and while it worked fine the additional overhead of processing and previewing the data - as well as maintaining any tooling - far outweighed the tiny benefit you get from having everything in a single file.

packaging this stuff up in a single file makes sense in vfx/film-making where you have a lot of disparate data types that all need to be composited together and passed around between multiple studios but for video games where 99% of the time you put textures in a folder and never move them because version control exists there's effectively no benefit

As far as I know the "deep" exr stores records of multiple hits a ray-tracing ray does when it goes through a scene for a single pixel . like two intersecting clouds and a wall . I t stores samples of cloud 1, then when it goes through and hit cloud 2 and then the wall/object behind . it can record layers and layers of data for a single pixel each time something is changing on the ray way . So it's only true way to composite clouds,dust, light volumes etc . But I never use deep exrs really so not an expert .

While cryptomatte is just a mask pack. it records an ID/attribute and transparency weight in each pixel so a single RGBA is just masks of two things R-atribute id, G-alpha weight and BA for another object. and so on . So it doesn't have to be deep exr. If Substance Designer could read regular multi-layered exrs like those that store beauty layer , normal AOV etc we probably could extract masks with pixel processor in SD . Besides it would be a cool and simple way to propagate multiple mask within node flow too instead of keeping lots of connection lines.

poopipe · May 27 2025

I'm pretty confident it would need separating into separate textures in the same way that PSDs are broken up cos the texture formats available to designer only support 4 channels - but yes.

I may have mixed up my terminology - it's been a few years since I poked around with openexr - but I was under the impression that a deep exr was anything more complex than a 4 channel image with consistent bit depth and size

Vertex Colour Masks

Replies