PS: Painting tones affecting lower layers at different values

Stinkhorse

I'm fully expecting the answer to this will be "that's not possible".



I've got an effect I'm working on at the moment, building off a tutorial for creating a money style effect using multiple layers. The guilloche patterns are used as mask selection image that's been copied once for each time the patterns will be applied. each layer of the image has it's threshold adjusted so the darker patterns affect the darkest shadows while the lighter patterns affect the faintest shadows. This takes a lot of effort to set up, which isn't a problem for a still image, but I'm looking to create an animated gif of this where a light source adjusts the shadows causing the patterns to adjust correctly.



The problem here is that because the effect is applied via a mask across upwards of eight layers, I can't just paint on one layer and have the tone affect some layers more than others in a way that I know of. I'm currently looking at Dan Fessler's HD Index Painting In Photoshop Tutorial but I'm not sure it's going to work as I expect.

If anyone has any thoughts on this it would be appreciated!

Eric Chadwick

Hard to understand the setup. How about using Group masks?

But ultimately this might be done better in a compositing tool, where you have a better material-based layout. After Effects comes to mind. Heck, even 3ds Max's node-based material editor might do this better.

Stinkhorse

I was expecting an answer like that. Group masks are out because the light has to effect a deep shadow layer more than the faint shadow layer ironically. Here's an example from the tutorial site.

Imagine the light pulsing and pushing the linework around like shadows. That's the effect I'm after.

Eric Chadwick

I would try approaching this with a shader. Doing this like that with layers sounds like an insane amount of work. Look into cross hatch shaders.

Stinkhorse

I'm not building any of this in any program at the moment (edit: which is to say any game development package, I'm just mucking about in photoshop). And I don't know how to make shaders, but After Effects sounds like something I could stumble my way through. 

throttlekitty

So an action/script that for each frame in an animation...

1. Copies the input image in entirety or by layer groups.
2. Opens a template document with your pattern layers ready to go.
3. Pastes into each layer mask, adjusting the colors for each tonal range.
4. Profit?

Stinkhorse

Hey @throttlekittythat sounds like a compelling solution. I would build the animation using all my usual techniques, and I have the action to produce the money effect already. There's less of an immediate pay off in terms of painting equals a visual output but nothing to be done there I suppose.

Like @Eric Chadwick mentioned, a real time solution would be better, but I've been down the rabbit hole of game development before. Three months of pulling teeth and hair to determine that a visual concept wasn't going to work isn't something I'm out to ensure again.

throttlekitty

I'm still with Eric, that a shader solution would be better. I've another half-idea for a panel that has buttons for tone/hatch patterns with dark/light adjustment buttons next to each- clicking one would send you to the appropriate layer for painting. Just a small timesaver but could probably work with that immediate feedback, but still an insane workflow. ZBrush has matcaps and a layer system, but I don't know how feasible that would be.

Eric Chadwick

Do some googling for "unreal shader hatch" or "unity shader hatch" and you'll see lots of different approaches.

This paper provides some good info about how it's achieved:
http://artis.imag.fr/Members/Cyril.Soler/DEA/NonPhotoRealisticRendering/Papers/p581-praun.pdf

https://www.youtube.com/watch?v=PXh0-JPPNSY

https://www.youtube.com/watch?v=IEL32r-PRpg

Stinkhorse

@Eric Chadwick can an effect like this be applied in a 2D space? Or I suppose in a billboarded space.

The reason I'm hesitant to go back into a development space to try this is that I did the exact same thing a few years back. I tried both Unreal and Unity, and where Unreal never spit out anything functional, Unity at least displayed objects, but nothing ever moved. The visuals came out incredibly muddy, and any shaders I tried to produce were like writing out random lines on paper and hoping it spelled readable sentences in a foreign language. I never understood why something didnt work, and it was in all an incredibly demoralizing three months.



Is a development environment the only way you see this working?

Eric Chadwick

Is this a prototype for an eventual game? Or is it for a pre-created non-interactive animation? If it's not ever meant to be interactive, then I'd suggest getting into After Effects.

For example, searching for "after effects cross hatch":

https://www.youtube.com/watch?v=gsCuC6Qytm8

Stinkhorse

It's really only a proof of concept and wont ever be interactive, so yeah it looks like after effects is exactly what I'm after, thanks for the clear breakdown of the options.

RN

To do this in Photoshop you need to use a smart object, multiply-blended groups, threshold adjustment layers and clip-masking.



(Source image was this.)

The layers needed (I actually used more for the above):



(1) Group folder, always in Multiply blend mode.
(2) A Threshold adjustment layer with the threshold set to what you want to extract from the source image for that particular pattern.
(3) The source image as a Smart Object so that you can edit it in another window by the side (like on that GIF) and change it with Brightness\Contrast, Levels or Curves, save it and have it update automatically for all its duplicates (this is how you'll animate this effect, in steps). The smart object layer and the threshold layer are clip-masked to layer (4) below.
(4) The layer with the lines pattern. The lines are not faded in any way, they fill the entire layer.

Duplicate the Pattern 1 group as many times as you want, advance the threshold level of the layer (2) of each group and replace the layer (4) of each group with the different patterns\line weights.

It's the multiply blending of the group folder that makes this work, because the source image is thresholded between black and white, then clip-masked to the lines pattern and this results in some lines being flat white (so they don't do anything when multiplied) and others flat black, which darken whatever's below.

Stinkhorse

Oh my god RN, that's straight brilliant! I'm going to try implementing this immediately.

Stinkhorse

Ok this is a super long time coming. You plan on doing something and then life slams into you. The effect works perfectly @RN just like I'd hoped. I've got about six layer groups for each direction of the pattern, and I'm applying and doing some extra tone updates after the fact for effect. The effect is being applied to the entire image indiscriminately for now. The next step is to figure out how to apply the effect on a per object basis to give each thing it's own sense of material as well as to help it stand out from whatever it's next to.

RN

That's very cool.
If you plan on doing it in real-time (with Unity, for example), you'll need to use a post-processing shader.
You can pack 4 different patterns on a single RGBA image, since the only thing you care about is their transparency (the alpha channel). So with just two small seamless textures (32x32 or something like that, so they can tile and fill the entire screen), you'd have a total of 8 unique patterns.

You can then have your shader do something like this (untested):

<code>const float4 BLACK = float4( 0.0, 0.0, 0.0, 0.0);

<code>	float4 result = lerp( screenPixel, BLACK, pattern1 + pattern2 + pattern3 + pattern4 );
	return result;
}uniform sampler2D _MainTex; // Screen texture, supplied by Unity.<br>uniform sampler2D _PatternTex; // Pattern texture(s), containing a pattern transparency per channel.<br><br>[...]<br><br>float4 frag(v2f_img i) : COLOR {<br>	float4 patternSample = tex2D(_PatternTex, i.uv);<br><br>	float pattern1 = step( 0.2, patternSample.r ); // 'step' is the threshold function.<br>	float pattern2 = step( 0.4, patternSample.g );<br>	float pattern3 = step( 0.7, patternSample.b );<br>	float pattern4 = step( 0.9, patternSample.a );<br><br>	float4 screenPixel = tex2D(_MainTex, i.uv);<br>	

This should be extremely fast.

Stinkhorse

Oh man, I dipped my toe into shaders sometime... a year, a year and a half ago? I've sworn off actually trying to make stuff work in Unity or Unreal. Both of those were colossal time investments and virtually no payout. The biggest success I had with them was learning my own limitations.

Maybe someday I try my hand at it again. For now though I'll be trying to animate this with a combo of illustrator, photoshop and after effects (once I get close enough to that bridge to burn it).