Tree lighting and Normals problems in UE4

drliamj

Hey there!
So I asked about this over on the Speedtree forums, and that was really helpful, although I'd still like some more advice in general on this!
So now I'm here asking you amazing folks about leaf normals, and how to get em right, and basically how to know how to adjust them per tree.
Basically, here is a screenshot of my trees in UE4. Not pretty. I I actually think they're good looking trees and materials, they just light like arse!

Here are those two trees in the modeller: 
Not too bad right? I mean I think they need some tweaking in terms of knocking back the leaf cards density, letting some more space in the actual texture, and that would certainly help with the lighting (but then again, the amount of space in the texture looks right in UE4), but, the problem is that they just don't light uniformly! I've used leaf cross meshes, just two leaf generators rotated at different angles to form an x. Pretty common with game foliage! But the at intersection of the cards I get this just weird shadow, the underside lights totally wrong, and rather than creating the illusion of that sort of 'spherical cluster' you just get what I've got in the above shot!
I've messed with all kinds of normal settings and I can't seem to get it right!


Here's a close up of that awful lighting:

The underside lights differently to the front side. Looks very bad at the intersection
So then I went into 3DS MAX to create a custon cross mesh, where I edited the normals to fix this issue. Here is the result:

But that did nothing to help. So now I'm at a loss as to what to do! I've tried like everything I can think of! I've seen this cross leaf used so many times, and usually they light nice and consistently and don't have this issue. 
Hopefully someone here can lend me some of their expertise on the subject!  Thanks in advance
Edit: It seems using more deformed planes fixes most issues. Although that makes the poly-count go crazy... And this above issue still persists just less noticeably.  

poopipe

Speedtree has (or had) a couple of sliders for bending the canopy normals .  I'd start by finding that. 

Mark Dygert

Yep speedtree has the normal editing options built in. it helps a lot, but normal editing alone can only minimize the problem it can't solve it fully.

In Unreal when you import, make sure that the Normal Import Method is set to "Import Normals and tangents". I think it defaults to "compute normals" which will reset any custom normal editing that you've done.



You might want to check your FBX export settings and make sure that you have "Tangets and Binormals" checked on, if you are trying to export from 3dsmax with custom normals. I don't think that is checked on by default.

drliamj

Yeah, With the Speedtree import into ue4 there is no option for changing normals so I think that's all good by default. The normals are set up all nicely actually. Sonya from Speedtree 'fixed' my tree but it still doesn't light nicely... 
So maybe I should have less intersections with my leaf cards? That's where the ugliness is coming from so hopefully that will solve it. Gonna try that today. 

drliamj

Ideally I would like to achieve lighting like this, where the back side look unified to the front. Love the foliage lighting in the Witcher 3. and they are speedtrees so any advice on getting this sort of lighting in UE4? 
  

drliamj

Okay so I've solved it. For anyone else having the issue, it was pretty simple, having less flat planes works wonders, also editing the sub surface textures to have much more brightness and saturation helps to reduce underside darkening. For the most part I just didn't realise how much of the issue was using a flat plane!

What do you guys think? 

Larry

Do you have screen space ambient occlusion? Is your tree a dynamic object? is the lighting baked int he last picture?

o2car

Nice!

(For others looking for solutions: A simple trick we use a lot is to transfer normals from a sphere or half sphere.)

drliamj

o2car said:

Nice!

(For others looking for solutions: A simple trick we use a lot is to transfer normals from a sphere or half sphere.)

Heard of this technique a lot. I wish there were options for doing that inside speedtree. As it is, I find the vertex normal options in speedtree a little underwhelming, and exporting trees as meshes and having it all work properly is really annoying! 

poopipe

Speedtree 7 has options for bending canopy normals for this exact purpose. I imagine it's still there in v8

drliamj

poopipe said:

Speedtree 7 has options for bending canopy normals for this exact purpose. I imagine it's still there in v8

Hey yeah you can bend normals, but not really to the same extent as transferring them from a sphere which looks much better, correct me if I'm wrong! 

drliamj

Okay I have something I like now! 
 

poopipe

drliamj said:

poopipe said:

Speedtree 7 has options for bending canopy normals for this exact purpose. I imagine it's still there in v8

Hey yeah you can bend normals, but not really to the same extent as transferring them from a sphere which looks much better, correct me if I'm wrong! 

Your problem there is that if you want to keep a live speedtree mesh you can't do things like that afaik. 

you could theoretically encode some normal information into vertex colour and/or do something clever with a shader - which is probably what they've done in the witcher 

drliamj

poopipe said:

drliamj said:

poopipe said:

Speedtree 7 has options for bending canopy normals for this exact purpose. I imagine it's still there in v8

Hey yeah you can bend normals, but not really to the same extent as transferring them from a sphere which looks much better, correct me if I'm wrong! 

Your problem there is that if you want to keep a live speedtree mesh you can't do things like that afaik. 

you could theoretically encode some normal information into vertex colour and/or do something clever with a shader - which is probably what they've done in the witcher 

Interesting, what do you mean exactly about vertex colours? 

drliamj

But like seriously WTF is going on here. I'm using all the right material and shader settings in UE4, and have followed the advice given to me by speedtree and still lighting looks so bad. Like, no matter what the normals are set to. There is no way of having those nice spherical normals so now I'm just wondering how anyone actually makes a usable speedtree? 
I've tried lots of subsurface, reducing leaves, reducing card density, lowering the opacity in UE4 ETC and nothing helps here. At this point I may as well return to hand done trees because at least I can make those look right : (
Any insight from anyone who uses speedtree would be so greatly appreciated! 

melviso

I don't know if u would find this useful:
https://www.blend4web.com/en/community/article/131/

This shows how normal editing can help low poly trees look better.

Blubberblase

Sometimes the easiest way for uniform lighting on a tree is to untick the tangent space normal map checkbox in your material. This will pretty much break your shading but can look quite nice. You will have to tweak your specular and subsurface quite a bit though, usually you should start without specularity in this case and just tweak it to low values like 0.1.

tangent normal map ticked:

tangent normal map unticked:

The tree was created in blender and has no normal editing applied to it. It is quite high poly though which is why the difference isn't as big. The bigger your leafcards are the better it will look with this method. I highly suggest using vertex colors for some shadows on big leaf cards or it can look quite flat.

You can sometimes even get away with using standard shading mode without the extra cost of subsurface because the specularity basically fakes it. This is with standard 0.5 specularity, with some tweaking it could work under some circumstances.

Hope this helps.

drliamj

Hey thanks for that, that is really helpful. I'll mess around with that! 
I feel like the default specularity is always WAY too full on for a tree, unless its got those really glossy looking leaves, even still, I always set the values a little lower. 
Also I think I was limiting myself to too low a poly count. I was aiming for about 8k but now I'm sort of thinking around 12-16k is more reasonable for a tree. Anyone have any experience with poly counts and current gen trees? 

poopipe

drliamj said:

poopipe said:

drliamj said:

poopipe said:

Speedtree 7 has options for bending canopy normals for this exact purpose. I imagine it's still there in v8

Hey yeah you can bend normals, but not really to the same extent as transferring them from a sphere which looks much better, correct me if I'm wrong! 

Your problem there is that if you want to keep a live speedtree mesh you can't do things like that afaik. 

you could theoretically encode some normal information into vertex colour and/or do something clever with a shader - which is probably what they've done in the witcher 

Interesting, what do you mean exactly about vertex colours? 

Same as a normal map,  encode a direction into an rgb value and then do something useful with it in your ue material.

I have no idea on the mechanics of this as I've not tried it with speedtree

Blubberblase

drliamj said:

Hey thanks for that, that is really helpful. I'll mess around with that! 
I feel like the default specularity is always WAY too full on for a tree, unless its got those really glossy looking leaves, even still, I always set the values a little lower. 
Also I think I was limiting myself to too low a poly count. I was aiming for about 8k but now I'm sort of thinking around 12-16k is more reasonable for a tree. Anyone have any experience with poly counts and current gen trees? 

You can get away with way more than 16k polys, it of course depends on your hardware target. You can easily use around 100k polys for your LOD0 and just create aggresive LODs. Might even get you better performance due to low overdraw.

You can even go to one leaf card/leaf, i tested a tree around 2,5million tris, it worked fine but the LOD creation is a nightmare to get right, i would not recommend it!

The tree i posted above, while just a test tree, has around 250k polys on LOD0, i then deleted branches and thinned out the leaf cards for every LOD getting it down to around 30k for the last LOD before Billboard. By using a distance blend in your opacity mask you can disable the mask for the LODs and still get a full tree without any overdraw and with much lower polycount. I can use around a hundred in a test scene without any fps issues on a 750ti. For a small scene this should always be fine, for an open world scene you should aim towards lower poly amounts of course.

I would recommend to go as high as neccesary for the look you want and then play around with the LODs and some shader magic. You will learn a lot from simple tests like that.

drliamj

poopipe said:

drliamj said:

poopipe said:

drliamj said:

poopipe said:

Speedtree 7 has options for bending canopy normals for this exact purpose. I imagine it's still there in v8

Hey yeah you can bend normals, but not really to the same extent as transferring them from a sphere which looks much better, correct me if I'm wrong! 

Your problem there is that if you want to keep a live speedtree mesh you can't do things like that afaik. 

you could theoretically encode some normal information into vertex colour and/or do something clever with a shader - which is probably what they've done in the witcher 

Interesting, what do you mean exactly about vertex colours? 

Same as a normal map,  encode a direction into an rgb value and then do something useful with it in your ue material.

I have no idea on the mechanics of this as I've not tried it with speedtree

Hey thanks for all your help! Really think I'm getting somewhere now. I've found a couple of good lighting settings in speedtree and messed with the shader a bit. I would still like to steal the normals from a half sphere but that can't be done in speedtree. Basically speedtree pushes the normals out from the parent, in most cases that being the branch that they're attached to, or from the centre of the leaf mesh itself. I would like to see speedtree being able to do a little bit more with normals as the most common foliage normal tricks can't be done in speedtree (Sphere/half sphere normals). Kinda baffles me actually. Seems like the lower polygon you go the less useful the normal settings are, Especially seeing the results of speedtree in games, the devs really must have spent a lot of time in the shader to make them look so good! 
Anyway, I've been spending even more time on my leaf cards and making sure they were in that sweet spot in terms of density. I've only been messing with really simple tricks in the material, tints and subsurface multipliers, ETC. However I would really like to learn more about using vertex colours to help with the foliage. Only problem is I can't find much online about actually doing it! I've only used vertex colours on my modular buildings, as part of a height lerp to paint on texture variation and edge wear. Might I trouble you for a little more info or an article you've read or something on the subject? Anyway thanks so much for your help!
(Oh and I haven't said yet but that tree is dope!) 
Edit: The only stuff I'm finding on foliage and vertex colours is about wind which is not what I'm after. 

drliamj

So here is a shot of what I've got so far. Thinking this is going to look damn fine in a proper scene. I also imagine that once I rebuild 4.19 with VXGI, HBAO. Nvidia volumetric lighting ETC (I usually use a game works build but haven't gotten around to it for this update yet) it will help with some of the over occluded areas. LOD0 I am keeping around 12-16K, with the final LOD at around 2K. This is an open world around the size of Oblivion or Skyrim, so I hope I'm not overdoing it for these trees! 
What do you think? 
Edit: I'm still using tangent space normals BTW.
 

drliamj

I swear there is something about the textures and how they're showing up when lit. Not sure how to describe it, or even if I can put my finger on it. Kinda like they look jittery, just sort of lack clarity. For comparisons sake here is the trees unlit. IDK if you can see what I mean from this and the above pic, but in the lit versions the detail in the texture seems to get a bit lost. Whereas in the unlit screen I can see all the needles and colours a lot clearer. I feel like I'm really getting there now It just needs that final push over the finish line. That or I've been looking at these fucking trees for almost 36 hours now without pause and have totally lost my mind. Really gotta manage my OCD better :P  Looking back at the start of the thread I can't believe I even made such bad trees lol!
Edit: And here's a quick close up of what I mean. Sorry for bombarding this thread with walls of text, just trying to learn as much as possible, and as I'm constantly working, I'm constantly posting! :P

I strangely really like them unlit haha.

Obscura

Your unlit pictures doesn't have anti aliasing

drliamj

Obscura said:

Your unlit pictures doesn't have anti aliasing

Lol, true, I really need to rebuild with game works, having GI and better AO and AA will help a lot. 

Obscura

AA works in lit mode, you don't need to rebuild anything for that

drliamj

Obscura said:

AA works in lit mode, you don't need to rebuild anything for that

No I know but I think the AA right now is over blurring the leaves. Generally happens with TAA with no super sampling which the Nvidia AA has.
This is super noticeable with the wind so nicer AA should help is what I'm saying
After reading this I think what I was talking about on the lit leaves is ghosting from the AA with the wind. 
Edit: Yep AA ghosting was really bad. Turned off AA to test, and that is the culprit! I already have my settings on cinematic so I'm thinking the best way to improve the AA is with NVidias tech, which is what I was getting at before! Hehe I'm not THAT nooby.
Edit 2: Turning up the Distance Field res and switching to two sided distance fields helps too with creating nice patches of light and shade on the trunk, and now instead of being a blurry mess you can see the leaf shadows clearly! 

Obscura

The wind blurring is because things with world position offset or displacement don't output motion vectors by default, which TAA needs to make correct result. There is a project setting to enable motion vector output from vertex deformation. You can also tweak the TAA settings to get better results.

drliamj

Obscura said:

The wind blurring is because things with world position offset or displacement don't output motion vectors by default, which TAA needs to make correct result. There is a project setting to enable motion vector output from vertex deformation. You can also tweak the TAA settings to get better results.

Ahhhh, thats super awesomely helpful thanks! Still gonna do a game works build because I love HBAO and VXGI, and epics dynamic GI is not production ready yet. Also Nvidias AA has more options to tweak. I've always wondered why things like that aren't default in the engine because they're kinda standard in games now. Anyway its really easy to build into the engine, its just that visual studio completely eats my hardrive which is why I haven't done it for 4.19 yet. I do have a 4.17 gameworks build but Speedtree 8 only works with UE4.19 and up.
Anyway back on subject. I feel like I'm on the right track now! Thanks so much I do really appreciate you taking the time to help!

drliamj

Sorry, where exactly is that setting I can't find it in project settings, and I'm going through "all settings" so it should show up!
Do you refer to the setting under engine, optimization: Accurate Velocities from Vertex Deformation? 

Obscura

Yes its that one.

drliamj

I am just gonna move on for now and worry about this later haha ;(