Dithering Shader/Post Process Effect - need help!

Rooster128

Pixel art is amazing. One of the things I love a lot about pixel art is the dithering technique- that optical illusion of value made by various checkerboard patterns, sometimes using only two colors.

I set out to make a dithering shader in UDK, for this 3D pixel art game I want to make- I was thinking it could help me sell that pixel art look I'm going for. I'm almost done- but I need help getting it to scale! I didn't want to share this shader just yet, but I decided it was time to reach out to the polycount community for help. It beats bashing my head against the screen. I still haven't decided whether or not I want to use this as a post process effect or a shader- the application I decide to use it for might determine the best solution I'd need. But for now, assume its a shader.

So what I got so far...


This is my 'flipbook' of sorts. The size is 128x4. There are 32 4x4 tiles here, ranging from white to black. You could probably make tiles for special cases for diagonal lines- but thats a bit beyond my knowledge at the moment. Basics first.

Its important to note here- I manually have to create the UV splitting effect of the flipbook manually, in order to index the frame being shown. As far as I know, there is no way to directly index the frame of a flipbook.


In order to get the piece of the texture I want correctly, I need to divide it along the X axis. 1 / 32 = .031250.


This part is roughly straightforward if you've built a custom lighting model. I'm finding the dot product between the light vector and the surface normal, multiplying that by 32, and rounding it off to the nearest whole number. The important part here is that I'm subtracting this number by 32 and using the result to index my 'flipbook'.

Then I simply pan the texture by the index. Again, using that same decimal- 1 / 32.

If I plug all of this into my custom lighting and preview it on a cube, it looks pretty dang sweet so far. I can move the light around, and the grid pattern gets indexed correctly.

Here is where I need help
I now want to tile this segment of the texture many times over the UV space. I have had much headache in figuring out a way to do this without banding the entire flipbook texture over the surface of the model. I'm starting to wonder if its simply not possible to daisychain panners together, since it seems like whenever I do, the UV coordinates shrink, as if the coordinates are being pre-multiplied.

I'll show you what I have so far as a solution. Obviously not all the way there.


Since I wanted to repeat the same 4x4 tile over the entire UV space- or screen space- I figured I'd need to divide the space up into 4x4 segments. I assumed I'd need a modulo (remainder) node for this. Basically I divide the space by 16 (it probably should be 4 or 32, but I tried 16 since at first I just tried repeating the texture 4 times over my model- 4x4), and subtracted the modulo of 16 from it to get the coordinates rounded down.


I multiplied my texture 4 times over the entire screenspace, and piped it into another panner. I used the 'screen splitting' part of the function I explained in the paragraph above as my time for the panner. The panner I have moving at 1 / 128 - the length of the texture, so it should move over 1 pixel for every unit piped into time.

From what I understand, the coordinates for UVs and the coordinates from ScreenPosition start with 0,0 being in the upper-left corner and extend out to the right and downwards. This is good to know when repositioning the 4x4 segments, since you're essentially moving the entire texture 'back in place' to get the tiling effect.

As you see, I still get this banding effect, albeit, you can tell the 4x4 tiling is working- somewhat. Try it yourself to see what I mean.
I'm completely lost though. I've tried fiddling with the variables of this for quite some time now, and nothing seems to work right. I think the technique I'm using to do this is mostly sound, I'm just missing some important part. This is where I hope one of you can step up to help me. The fate of a pretty cool pixel art shader lies on the line!

mAlkAv!An

What you are probably looking for is the Frac node. Your material should then look something like this:
Texcoord -> Frac -> Divide by 32 -> Add Constant2Vector -> Texturesample
By adding a Constant2Vector you can determine which sub-tile to use.

Rooster128

Where would I be piping in the light index though? In the constant2vector?

tharle

any chance you can share the package to download? be much easier to debug.

rather than using a flipbook, could you use a single square texture with masks packed into the channels? you can make each combination of pattern you need out of a checker board, a corner, a side and a centre square (with lots of adding, flipping and rotating). then you can just if node through all 16 combinations and tile the original texture as you need, no problem.

not sure how it would compare cost wise and it'd be a pretty complex (if repetitive) shader network to set up but it should solve the tiling problem....

ps - i think i see what you're trying to do with dividing the uv space up into 16 - you want the texture you map to that 1/16th to tile over the whole uv space - but i dont think it works that way - i'll have a look at udk when i get home tonight and see if what you're trying to do works and if it's just a setup issue.

mAlkAv!An

Rooster128 wrote: »

Where would I be piping in the light index though? In the constant2vector?

Can't say for sure but I guess you'd have to use the panner right before it goes into the frac node.

Rooster128

I'm very close now, I can taste it. Instead of piping a panner into another panner, I found I could simply move the coordinates by piping the panner into simple math nodes to offset. Still using the Fmod network I put together, I was able to get the tile repeating over the surface of the model. Without using multiply or divide (which I thought was odd... figured I'd need to use one of those to change the scale of the UV space).

While I have it tiling now, the effect doesn't seem to emulate the lighting accurately, and I get really strange tearing in really light and dark places (which leads me to believe the amount I'm offsetting the tiles by is the culprit.

I'll post the package up here in a little bit.

mAlkAv!An

This is what I came up with:
http://abload.de/img/ditheringmat01w3ucl.png
http://abload.de/img/ditheringmat02oru6l.png

When working with fractions/floating point remainders you'll usually end up with seams/mipmap artifacts. But it's possible to bypass these by using a custom node with tex2Dgrad.
http://abload.de/img/ditheringmat03udu61.png

Rooster128

Wow- that worked surprisingly well! Thanks malkavan! Still trying to wrap my head around some of the things you did. I've never touched HLSL- but from researching I see that it can stop the clipping issues that can happen with Fmod or Frac all the same. I think maybe part of what makes ours look so different is there's multiple ways of going about doing it- but yours definitely is more concise!

I tried playing around with this to see if I could get it to work as a post-process- how would you go about doing this? I tried using a frac of ScreenPosition, divided by (ViewSize divided by 4)- doesn't seem to work.

mAlkAv!An

Did you try to use a regular TexCoord node instead of ScreenPosition.xy?

Rooster128

Nope- would that work? Maybe I was other thinking it...

Rooster128

So TexCoord did work- I found another problem too. With a post process you can't get the surface normal by using the vector (0,0,1) and dot that with the light vector to get the lighting. I've tried finding another way to pull the surface normal from any given pixel on the screen but nothing seems to be working right now.

Meanwhile, another solution I found was to use the effect to match the "average brightness" of the scene. In other words I added up the 3 channels from SceneTexture and divided by 3, and used that to index the flipbook.



Not too shabby. Thanks once again mAlkavan for your help! Let me know if you find a way to pull the surface normal out of the scene.

mAlkAv!An

I've been already wondering what kind of parameter you would use to determine which sub-tile to render.
It is possible to calculate view space normals from scene depth. Unfortunately this suffers from strong artifacts when running UDK in DX9 mode. In addition the light vector is still missing. For a single (directional) light source you might get around this by passing the light direction as a constant vector to the material.

Rooster128

So a little bit of a small update here. As a PostProcess, I think I'm fine with the dithering effect being powered by the average brightness of the scene, rather than a lighting dot product (which could return weird results on unlit materials)- it just makes more sense it would be based on the overall brightness of the scene.

There is one last thing that bugs me about this as a Post Process however, and that is apparent streaking every quarter of the screen in X and Y. You can somewhat notice it here:



This is the result of personally repurposing malkavan's code for a post process, I have yet to try it with my own Fmod method- so I'm not sure if Frac would be the culprit here.

I'm thinking it may also have to do with the fact that I divide ViewSize by 4 in order to find how many times I should tile the effect- but I'm tried playing with this value, and the scale of the tile itself, and offsetting the pattern, all to the same effect.

mAlkAv!An

Does this only happen in the viewport or also when you play in editor? Is that issue depending on the amount of tiling?
I'd guess it's some kind of floating point inaccuracy. (Maybe caused by your screen resolution which is not divisible by 4?).