I dunno what the exact workings of this problem but how about this: make a black material, 000,000,000 no specular no gloss no nothing make a white material, 255,255,255 100% self illuminated. assign materials to objects you want layered out, render use as layer mask.