Even with the UDK example decals the decal does the same thing, Is it a udk Limitation? or am I doing something wrong somewhere? Need the decal to not show up where the objects material is masked. (Circled)
I don't quite understand the specifics your asking. However, I believe decals work using Emissive, not Masked. And their blend mode should be Modulate. They should still work. If you need proper setup knowledge, take a look at one of the decals materials in udk. Tear it apart until you see what's going on.