Blendmask painting via Viewport Canvas - 3dsmax

Obscura

I am trying to paint a blendmask between 2 textures. It works, but I have a little issue. When I enable the painting on the mask, then the 2 textures disappears from my model, and it shows only the masks, which isn't really a good thing. Is there a way to paint to the mask without hiding the textures?
I use a composite material, viewport canvas, and 3dsmax.
Thanks!

Mark Dygert

Short answer: Nope.

Longer answer: Viewport canvas will only display the thing you are painting. There are some other methods using custom shaders but they might not work with newer versions.
http://www.polycount.com/forum/showthread.php?p=1672714#post1672714

I'm not really sure why autodesk hasn't updated max so this kind of masking works in the viewport with native tools.

Obscura

Thanks Mark, but actually now I found a way for doing this! It works the same as in Photoshop. I apply the two textures onto 2 layers on the layer editor, and then I can add a blendmask to the top one, which can be painted without hiding the 2 textures Then I can save it as a PSD, and then I can grab out the mask from the file in PS. Tadamm! Works like a charm. There should be a more user friendly way though. It took more than a hour to figure it out to me, and there is nothing about this on the internet.

Mark Dygert

NICE! That's awesome!

It took more than a hour to figure it out to me, and there is nothing about this on the internet.

I think everyone else (like me) just gives up.

Just so I understand, are you using a composite map in the diffuse slot? And then using viewport canvas to target the blend mask of the top layer?

Obscura

No actually there is no need for any composit material. It works with a standard material, because the magic is happening inside the viewport canvas layer editor. So you simple apply the very bottom texture to a standard material's diffuse slot, and then you choose this in the viewport canvas tool to paint on. then this will appear on the base layer. then you can add a new layer, where you can paste your second texture, and then you can add a blendmask to this layer, which can be painted now. I played with it with 4 textures/3 blend masks (on the layer editor), and then I simply exported the viewport canvas project to PSD. Then I could open it with Photoshop, and the blendmasks were still there, So I could copy-pase them into a blendmap (different masks in the different channels).

Obscura

Quote:
I think everyone else (like me) just gives up.

Yeah I also almost gave up

Obscura

Its a bit more tricky If you want your textures based on the first uv channel (overlappings) and your masks based on the second uv channel (no overlappings). The best working solution that I've found is to quickly bake your textures from uv1 to uv 2 and using these in the layer editor/standard material. This is because the Viewport canvas thing can't handle multiple uv channels at once. Later you can switch back to the raw tiled textures, and they can be applied based on uv 1 and the masks based on uv2.

Obscura

Okay so here is a bit more detailed explanation about whats going on. It is not really possible to use multiple uv channels on the different layers with viewport canvas. In my example, we use a texture that is tiled with the uvs, so its uvmap looks like this:

And it has a second uv channel, so that we can use with our masks.

This setup won't work with viewport canvas, except if we make tiled versions too from our textures.I made the tiled versions with quick render to texture from uv 1 to uv 2. Then we can use these on the viewport canvas layers, and we set the uv channel to 2 in our standard material.

Now we can make new layers, and import the baked,tiled textures.



We can add layer masks, and we will paint on these.


Result, after a few minutes of playing:

You can save this as a psd file, so it can be opened with Photoshop, and the layer masks will be still there.


You can save out your maks in Photoshop and use it in your game engine, or in 3dsmax, with a composite material, or anywhere, and you won't need the tiled (baked) versions anymore, because there, you can use multiple uv channels .

It works but its not the most user friendly...

- The composite diffuse also cant't use different uv channels for the textures and the masks, if you want to show it in the viewport, but it works on the render...

Eric Chadwick

Hmm, I'm going to have to check this out at some point. Cool solution!

Eric Chadwick

One way to get Max to show multiple UVs in the viewport is to use a Mix map, with your mask in the Mix Amount slot, and your two color maps in Color slots. Then you can tile the two bitmaps in the Material Editor instead of with the UVs, just increase the Tile spinners.

Doesn't work with Viewport Canvas though, it completely disregards the Material Editor.

Mark Dygert

Oh I see, you're using the layer editor in viewport canvas as a preview and then extracting the mask from it. NICE!

Obscura

Eric Chadwick wrote: »

Hmm, I'm going to have to check this out at some point. Cool solution!

Thanks!

Eric Chadwick wrote: »

One way to get Max to show multiple UVs in the viewport is to use a Mix map, with your mask in the Mix Amount slot, and your two color maps in Color slots. Then you can tile the two bitmaps in the Material Editor instead of with the UVs, just increase the Tile spinners.

Doesn't work with Viewport Canvas though, it completely disregards the Material Editor.

Yeah I know about Mix map, but it can handle only two layers, and I have to display 4 layers. Now I use composite with my masks, and I simply set the textures uv channel to 1 and the masks channels to 2. It works fine.

- If you are tiling inside the material editor in max, and you will use the meshes in your game engine later, then you'll have to set up these tilings again in the engine. This can take time if we are talking about many meshes, complex scenes, so personally I prefer tiling with the uvs.

Yes, I tried our many things with viewport canvas to work good with multiple uvs and layer masks, but it looks like there is no real connection between the material editor and the viewport canvas.

Mark Dygert wrote: »

Oh I see, you're using the layer editor in viewport canvas as a preview and then extracting the mask from it. NICE!

Exactly, its a proper preview, when I paint. Thanks !