Problem with DDX DDY normals breaking lighting when used for opacity masking

Saralie

Hi!

[TLDR: Tree's shadows break when masking out polygons that are perpendicular to the camera by multiplying the opacity mask with the Dot product of Camera Vector and Normals that have been calculated using DDX and DDY (as the tree has edited vertex normals). Shadows work when using Pixel or Vertex normals instead of the calculated ones. Not sure why this happens or how to fix it. Image of issue + graph below.]

I'm working on stylized trees in UE5 and have run into some issues with my material that I can't figure out how to solve. Another set of eyes on it would really help!
(Image of issue + graph attached below)

What I'm doing:
To hide the look of criss cross cards on the tree I want to remove any cards that are perpendicular to the camera.
As the tree has edited normals to look softer/rounder I can't use the vertex normals as that only gives me a fresnel-like effect. This is a problem that I've been pondering on and off for years, haha.
A few weeks ago I found a video by Ben Cloward that finally gave me a way of solving this, and I was so excited!
This way I can calculate "regular"/unedited normals for the polygons to use for my calculations.
I get the Dot product between the DDX DDY normals and the Camera Vector, to see which cards face the camera, and then I adjust the gradient, put it through a dither and multiply with my opacity mask.

The problem:

At first glance the method works fine for hiding the perpendicular cards, but for some reason it breaks the tree's shadows.
Viewing from the light's angle, the tree look fine and has both cast shadows and self shadowing.
Looking at the tree from the shadow side, you see the light shining right through, casting shadows from the branches but nothing from the leaves (they have different materials).
The cast shadow on the ground (and trunk) also disappears when the camera gets too close.

I've identified that the issue lies with using the DDX DDY normals for the Dot product. Using Vertex or Pixel normals do not cause this issue at all.
But I have no clue why that causes this issue? Or how I could possibly solv it?
I would love to be able to hide the perpendicular cards as it hugely improves the look of the foliage, as well as my workflow!
Is there a way to fix this or do I have to give up on the idea? Or is it a different way for me to achieve the end result I'm looking for?

The tutorial I followed:
https://youtu.be/1HTMoCaY0xw?t=347

Thankful for all help!

poopipe

does it work if you mask using the built in fresnel function?  using derivatives for this feels a bit open to weird shit 

Saralie

poopipe said:

does it work if you mask using the built in fresnel function?  using derivatives for this feels a bit open to weird shit 

Using Fresnel does not cause the issue, but also doesn't give me the desired result. I want to remove cards that face the camera, but as the normals of the tree have been edited the fresnel will only detect the edges of the tree (similar to fresnel on a sphere), so that method won't work with the vertex nromal information I have.

Here's an image for clarification:
Left is how the tree's world normals look with the edited Vertex nromals from the FBX. 
Right is each card getting individual normal direction (like if the normals hadn't been edited), calculated as I showed in the last image.

I need to get each card's geometry direction, regardless of normals, to calculate if they face the camera or not. But doing it like I have causes the weird issues...

Saralie

Ok I just realize I was incorrect about the DDX DDY normals being the problem. Everything seems so cursed and inconsistent that it's hard to tell what works and what doesn't x)
Even with vertex or pixel normals the shadows break. So it seems to be something with using the Dot product with the Camera Vector?