Hey guys. I've been trying to figure out an issue I am having for a while and haven't come to a solution yet, so I'm turning to the lovely Polycount minds for some insight. I'll just keep the example simple for convenience sake, but essentially I have a shader that is vertex blending with an edge mask to give a nice…
Produce a normal map from your edge mask, and sample the edge mask in two places - once to define the edge, and once again to switch from the normal map to a totally flat blue. One moment while I throw together a demonstration.
Thanks guys for all the suggestions. I knew you guys would know what I was talking about. I'll play around with this later after work, but James that is pretty badass. Exactly what I was talking about judging from the screenshots
http://www.smallfiles.org/download/3219/JamesWild_Maskery.upk.html Here's a package with the material in it. I've just tied it up to a sine from time so you can see it sweeping over the edge map. If you'd like harder edges use ifs instead of blending the normal map. I'll upload some images in a moment, Imgur are the latest…
You can use a sobel filter to calculate the normal, thought you need 9 taps into the edge mask texture. When packing the texture in a clever way (ie group 2x2 texels in RGBA channel), then you would only need 4 taps.