If you created the Smart Mask in Designer, I think you should check the label of the material, if not, I'm not quite sure, if I'm wrong you can add some images and I'll be able to give you additionnal help Cheers
Did a quick sketch using Stefan Morrels method of using mask by alpha in Zbrush to get a good base start on creating high polygon work. Baked out an AO and Normal Map in xNormal and then played a bit in Marmoset.
He wants the violet plane to xray the egg shape. So i guess it has to be dynamic and react if you move the plane ? Real time or precalculate ? That beeing said, i have no clue on how to do this. Some mask projected ?
I think something like this would typically be done with just a normal procedural greyscale image (created in something like allegorithmic mapzone) and then in code they use it as a mask going through th levels. Kinda like photoshop's threshold adjustment.
Strike a balance, yeah you need more tessellation than normal but consider using a black and white mask to modulate the vertex blending so you don't have to use as many verts. There's a super basic example in UDK at: Material'MaterialTemplates.PerfTricks.VertexColorModulateTransition'
Cover your whole face in tattoos, you can still go to work if you wear a ski-mask and huge 70´s sunglasses (the ones with brown glasses and golden frames, otherwise you get caught and fired for sure).
I believe it is set up correct, but maybe I am missing something. Under Renderable Cameras I have the alpha channel(mask) checkbox checked,32 Bit Framebuffer, and EXR format...Anything else I'm missing?
There's a 'make 3d' option in the alpha palettte...you can also try to mask it on some geometry and use the 'extract' thingy http://www.zbrushcentral.com/showthread.php?43513-Alpha-Make-3D-Minimal-Modeling http://www.pixologic.com/docs/index.php/Mesh_Extraction
A tip in case you don't know: you can export the flow mask from world machine and use that in your splat map. Can give some nice results compared to only using height/slope detection alone.
pull/flatten the cheek bones back a little (Black), mask out the rim of the eyes a little (Green) to hold the shape while adjusting the cheek. fill out here a little (Red), soften/smooth inwards here a little (Yellow)