An alternate masking method: use a Volume Select modifier, set it to Vertex, in the Texture Map slot put a Vertex Color map, and set the Mapping Channel to Vertex Color.
I've been trying to find a way to control the effects of the displacement modifier by masking the areas I don't want it to affect. Here's an example of the masking. https://www.dropbox.com/s/cbon8pcmu6y94t4/example.png?dl=0 I've been able to do it using vertex paint but it will only work when "Use Existing Mapping" is…
https://www.dropbox.com/s/n3stxa1zsfl006h/stack.png?dl=0 That's my stack and a Composite material with a VertexColor map in the mask slot. I apply vertex paint with the modifier and then specify the channel to mask with in VertexColor. I've come to the conclusion that it's not possible to do it the way I need and using…