Hard to understand the setup. How about using Group masks? But ultimately this might be done better in a compositing tool, where you have a better material-based layout. After Effects comes to mind. Heck, even 3ds Max's node-based material editor might do this better.
Ok this is a super long time coming. You plan on doing something and then life slams into you. The effect works perfectly @RN just like I'd hoped. I've got about six layer groups for each direction of the pattern, and I'm applying and doing some extra tone updates after the fact for effect. The effect is being applied to…
To do this in Photoshop you need to use a smart object, multiply-blended groups, threshold adjustment layers and clip-masking. (Source image was this.) The layers needed (I actually used more for the above): (1) Group folder, always in Multiply blend mode. (2) A Threshold adjustment layer with the threshold set to what you…