Any way to achieve these kind of trees in unreal engine?

Goldenvale

I've been developing my game in Unity so far but because it lacks too many features to continue I decided to port it to Unreal Engine. Everything is working out so far but I have no idea how to port my tree shader.

My trees currently look like this in Unity: 

It uses a shader that dithers certain tangents that are facing the camera at an angle. It also doesn't cast shadows on itself but does cast one on the ground. This can't be done in deferred rendering mode but I don't know if unreal engine can exclude it.

Does anyone know a way to achieve this in unreal engine?

I've managed to do this in unreal engine but it's far from good:

Obscura

This may help you:
http://wiki.polycount.com/wiki/Vertex_normal

Basically you need to edit the normals to have them standing outwards.
As for the shadowing, you can probably disable self shadowing in the static mesh editor.

Goldenvale

Obscura said:

This may help you:
http://wiki.polycount.com/wiki/Vertex_normal

Basically you need to edit the normals to have them standing outwards.
As for the shadowing, you can probably disable self shadowing in the static mesh editor.

It's the same tree like in the first picture, the normals are already modified. And I don't see any option to disable self shadowing in the mesh editor. Thanks for the suggestion.

Obscura

Ok if they are already modified then you need to import the normals. Maybe by default, it calculates it, instead of reading from the fbx file.

Goldenvale

The problem isn't the normals because it would look very different if it was. I just can't figure out how to dither the planes at a certain angle and disable self shadowing.

Obscura

ok then do whatever you want

Rurouni Strife

Goldenvale, you're looking for billboarding. It's done differently in Unreal-I want to think it's in an asset blueprint? I did it once and it was a bit odd, if I remember right in Unity it's a shader. I wouldn't doubt that there is a shader option as well. 

Something along those lines. 

https://docs.unrealengine.com/latest/INT/Resources/Showcases/Stylized/Materials/index.html
http://www.gamedev.net/page/resources/_/technical/graphics-programming-and-theory/billboarded-foliage-in-unreal-engine-4-r4246
https://www.youtube.com/watch?v=hH-_-wPwkgw

Vailias

Its not billboards, and the normals ARE transferred, you can see that from how the leaves are being shadowed.  Its the shading model. 

Take a look here https://wiki.unrealengine.com/Two-Sided_Foliage_Material

This helps with foliage by allowing for a more transmissive material and should help the light wrap around more spherically like you're expecting.

Goldenvale

Vailias said:

Its not billboards, and the normals ARE transferred, you can see that from how the leaves are being shadowed.  Its the shading model. 

Take a look here https://wiki.unrealengine.com/Two-Sided_Foliage_Material

This helps with foliage by allowing for a more transmissive material and should help the light wrap around more spherically like you're expecting.

I've tried that but you can still see the planes that are at a steep angle. This is what i'm trying to get rid of.


It's even more noticeable with shadows turned on.


I think the only way to fix this is to hide the planes when they're at a certain angle relative to the camera. 

Praglik

You are absolutely right, this is the way to do it. Try plugging a Dot product between a Camera Vector and a VertexNormalWS, then multiply your opacity with it.

Vailias

That should work, but you'll need to have a -1 after the dot product to get rid of the edge on stuff. 
You could also use a fresnel node. https://docs.unrealengine.com/latest/INT/Engine/Rendering/Materials/HowTo/Fresnel/index.html

Since that's doing the same thing but with additional functionality you'll likely need anyway.

It won't dither by default, so if you're using an alpha masked material there will likely be more setup needed, but if you're using a translucent material the polygons will fade out as they get more tangent to the camera vector.

Goldenvale

Praglik said:

You are absolutely right, this is the way to do it. Try plugging a Dot product between a Camera Vector and a VertexNormalWS, then multiply your opacity with it.

Vailias said:

That should work, but you'll need to have a -1 after the dot product to get rid of the edge on stuff. 
You could also use a fresnel node. https://docs.unrealengine.com/latest/INT/Engine/Rendering/Materials/HowTo/Fresnel/index.html

Since that's doing the same thing but with additional functionality you'll likely need anyway.

It won't dither by default, so if you're using an alpha masked material there will likely be more setup needed, but if you're using a translucent material the polygons will fade out as they get more tangent to the camera vector.

Thanks guys. I've tried both methods but the first one makes the tree look like this:



And the fresnel method has exactly the same effect:



Do you think the tangents need to be calculated somehow? 

Vailias

Don't invert the fresnel. You want the opacity to change near the center when you're looking at the leaf planes more straight on. 

Also the math as described works off of the normals of the object. I overlooked you'd modified them, so the fresnel effect would behave differently than expected. 

Do you have a link or the source for the Unity shader?  That might make it easier to replicate. 

Another angle I can think of is to bake out a world space normal map of the tree with its normals set hard to each face, then use that as the normal input for the fresnel node, but don't change the existing mesh. That way you could know the actual geometric face normal (which is what you want for hiding the geo at certain angles), while preserving the modified vertex normals for proper foliage shading.

Goldenvale

Vailias said:

Don't invert the fresnel. You want the opacity to change near the center when you're looking at the leaf planes more straight on. 

Also the math as described works off of the normals of the object. I overlooked you'd modified them, so the fresnel effect would behave differently than expected. 

Do you have a link or the source for the Unity shader?  That might make it easier to replicate. 

Another angle I can think of is to bake out a world space normal map of the tree with its normals set hard to each face, then use that as the normal input for the fresnel node, but don't change the existing mesh. That way you could know the actual geometric face normal (which is what you want for hiding the geo at certain angles), while preserving the modified vertex normals for proper foliage shading.

That's a very good idea. The shader is now using the mesh with all the normals pointing out.


Baking a map with its normals set hard to each face would work. In my Unity shader I use a script to calculate them but how do I bake a map with the original normals?

Vailias

Well save that second mesh there as its own thing, just be sure the UV's are the same.

I'm not sure what tools you have available, but if you have substance designer, it has a built in world space normal baker. I believe Xnormal does also.  That will be the simplest way. 

There's also the render to texture options in 3ds max. You'll need two copies of the object (one to bake from and one to bake with). There's a normal map option in there, which I believe has a world space, or object space option. ( I don't have it available right now) 

In the UE4 shader network you MAY need to use a transform node to translate the object normals to world space (or the world view vector to object space.. equivalent operation really), to get the effect to work correctly.