You need to press SHIFT+SPACE inside of 3DO with that mask highlighted in Photoshop. Otherwise 3DO won't be able to transfer the mask details to Photoshop.
not really they have their own exporter that uploads directly to the server from maya or max so there is no actual format it just transfers the geometric data not a format
It just requires a highpoly OBJ with the polypaint data, no UV's or textures needed. It will also only transfer the RGB polypaint values, not any materials in zbrush.