I've been trying to find a way to control the effects of the displacement modifier by masking the areas I don't want it to affect. Here's an example of the masking.
I've been able to do it using vertex paint but it will only work when "Use Existing Mapping" is checked. This is fine for something like the sphere pictured but for more complex meshes with faces at extreme angles to one another you get stretching/overlap.
From what I can tell it's using the normal direction or something like that and I can't seem to override it and use spherical mapping with the vertex paint mask. Does anyone happen to know if it's possible to do so? I have found it can be done by first UV'ing the mesh and then simply using a texture mask but it's not very flexible and more tedious to get the right results compared to using vertex paint.
Replies
That's my stack and a Composite material with a VertexColor map in the mask slot. I apply vertex paint with the modifier and then specify the channel to mask with in VertexColor. I've come to the conclusion that it's not possible to do it the way I need and using existing mapping is the only option if I wish to mask it this way.
I can't think of any other possible way to mask an area you don't want to be affected by a modifier. I did find a plugin that allows you to mask modifiers using bitmaps but it's very old (from 1999) and the recompile doesn't work with my version.