[UE4] First Nature Environment - WIP

DanielPeres

Hey there, I am about to use my free time from school, until the beginning of the next semester, to start a new personal project. This project will be my first project inside UE4, my first full PBR environment and also my first nature project (vegetation and things like that).

My goal is to make a small/medium environment based in the Cloud Forests (cloudy/rainy forests from places like South America contries like Panama, etc). I have been doing some research to get some detail pictures of the vegetation I am going to model. 

I am really looking for sincere critique here, so that I can improve my skills and push them even further. Also, some tips and tutorials about this matter (vegetation, UE4, vegetation in UE4, etc) would be great!! 

This is my moodboard so far, note that I have more references in my PC, this is just the base where I will get the main concept. Also, if you recognize any species that I don't have the name or the names are in the wrong flora please let me know

DanielPeres

Hey guys. I started doing some tests in the foliage matter since it is the first time I am doing it. I did some king of water fern and I was baking it so that I can start the textures and importing to UE4.

To test my bakes I usually use MT2 because it is faster, but the problem is that I am having some kind of trouble with the bakes. I am using 3Ds Max Render to Texture, with Hammersley at 1 and I am rendering 2k textures. The problem is that the textures seem a little pixelized to me, both the bake it self and the model with it applied and I am not sure why, since 2k should be quite high enough to guaratee some good quality I suppose.

2K Normal map Bake:


Render inside MT2 with 2K Bakes:


Render inside MT2 with 4K Bakes:



I have been trying to change the settings everywhere but I don't seem to find the source of my problem. The only way I can get a better result is with a 4K bake but that's way too much for vegetation, at least it think so in this case.

n4uj

Hi Nukakola7,

I'm assuming your highpoly is high enough, so that the problem doesn't come from there. Maybe this can help you:

-Bake the textures in 4k and then reduce them in photoshop to 2k (or 1k...). That should ease the problem a bit.
-You are using 4 different leaves in the same texture, but to be honest, they look very similar, and nobody will really notice at the end of the day if you are using just one model for all the leaves. You can add variation in many other ways (on the low poly meshes, stretching a bit the UVs, multiplying the albedo by a tint based in the world position, using different albedos but the same normal...). So if you only use one leave in the texture, that leave will have much more detail.
-2k is a quite heavy res for a leave as well, if you are going to add tons of foliage maybe it is just too much. However that depends on you. In my environment I'm using 512 and 1024.
-When I use RTT in 3ds max I usually choose a preset called 3dsmax.scanline.radiosity.high. It is very quick and also gives good results!

Best regards,

n4uj.

DanielPeres

Thank you for the tips N4UJ. Actually I had tought about making that "compression" after render because I saw a tutorial talking about that, that's why I was rendering 2k textures to later make them 512/1024, maybe I can do the same with the 4k, was just trying to avoid baking such big textures.

Yeah I noticed now that they look quite the same because the only difference between them is the noise modifier. I will have some tweaks in the Low poly to make them look different, I was also thinking about having more than one alpha mask so that I can have different damage in the leaves from plant to plant and now that you mention it, having more albedos would also be awesome. But before that, I will try to add some variation to the High Poly itself!

Also, I'll give it a try to the 3dsmax.scanline.radiosity.high

Jack M.

Maybe someone who has more experience in foliage could explain better but in this case what you have going on is you've made a texture atlas for your fern. Since it's split into 4 parts each leaf/branch has 1/4 the dedicated space of the entire texture. So with that in mind if you bake out a 2048x2048 texture, you're giving each leaf/branch approximately 512x512 texture space to begin with. On a raw bake that's not much resolution. You'll want to bake at at least double of what you have if you want to settle on 512x512 for each leaf/branch.

Another thing you may want to try is anti-aliasing. That may get rid of your issue.

One other thing that would be good to try is baking in another program such as xnormal to see if the problem persists.

You might check to make sure mip mapping is disabled in Marmoset. That can really downres things and if you're trying to get a 1:1 comparison of raw model and textures it can be a real problem.

You might also try saving in a different file format. Usually I like .tga 32 bit for textures with an alpha.

DanielPeres

Ohh man, you are correct about the proportions Jack, those are the details that me (a noob) makes a lot ahah. I'll try to use more texture space and see if I get better results. Also I will try all your other suggestions I just have a question about the anti-aliasing. Are you talking about rendering it bigger and make it smalled later or baking it with anti-aliasing? If yes, how do I do that, I am not sure and I have been looking for that all over 3Ds Max RTT 

DanielPeres

Do you guys think it is wise to use only 2 branches instead of 4 and then break up the similarities in the low poly model? Or what other options do I have to be able to get some variation and still a good texture space even with a 1k map or something like that

n4uj

That depends completelly on the look you want to give!! Having different alphas for a same albedo can work, it can be harder to tweak though. You can also add colour variation inside of the engine depending on the position of the mesh. You can overlay a detail normal map on top of the default one ... There are a lot of solution that might or might not work depending on what your are looking for. So try it and see how it looks!

When you are working on big environments it is a good practise to think on it as a whole as well, try to not get stuck in one little thing for too long. Sometimes we spent too much time on a issue that at the end of the day it is just the 0.5% of the whole enviroment, and that might encourage you to not finish the project because it is taking soooooo looooooooooooooooooonnggggg xD So, for instance, try to see how the fern looks with the rest of the scene, because you are not going to be able to get the perfect fern until you do that.
Try to test the stuff inside of UE4 as soon as possible, because foliage there looks fairly different from marmoset toolbag 2.
Good luck man I'm looking forward to see more stuff ;D
n4uj

Cay

I think if you are going for different versions of the same you will need to make more changes to them, right now they still seem to be the same leaf copied a few times. As in changing the silhouette by adjusting the spacing between leafs, damaged leafs and such.
In any case you could shift those leafs onto either the left or the right hand side and still have 50% of your texture space left for something else.

DanielPeres

n4uj said:

That depends completelly on the look you want to give!! Having different alphas for a same albedo can work, it can be harder to tweak though. You can also add colour variation inside of the engine depending on the position of the mesh. You can overlay a detail normal map on top of the default one ... There are a lot of solution that might or might not work depending on what your are looking for. So try it and see how it looks!

When you are working on big environments it is a good practise to think on it as a whole as well, try to not get stuck in one little thing for too long. Sometimes we spent too much time on a issue that at the end of the day it is just the 0.5% of the whole enviroment, and that might encourage you to not finish the project because it is taking soooooo looooooooooooooooooonnggggg xD So, for instance, try to see how the fern looks with the rest of the scene, because you are not going to be able to get the perfect fern until you do that.
Try to test the stuff inside of UE4 as soon as possible, because foliage there looks fairly different from marmoset toolbag 2.
Good luck man I'm looking forward to see more stuff ;D
n4uj

I will be trying doing some tweaks in the textures and low poly to get more variation. Also you are correct, I will surely move on faster because I can't be loosing this much time with a simple fern since I have a lot more to do in this environment. Also, after "building" the fern itself this little details won't be noticed that much and after having all the other props in the environment it might blend in even better and I might do the final tweaks after tha 

Cay said:

I think if you are going for different versions of the same you will need to make more changes to them, right now they still seem to be the same leaf copied a few times. As in changing the silhouette by adjusting the spacing between leafs, damaged leafs and such.
In any case you could shift those leafs onto either the left or the right hand side and still have 50% of your texture space left for something else.

Yes I tried to break the HP silhouette a little doing what you said, moving and rotating the leafs. I am thinking about adding random damage to the leafs using textures (alpha maps mainly). Also, I will be using substance (designer or painter still have to decide) to work on the textures and make it easy to randomize the textures inside UE4, so this might help me a bit.

I will post some of my progress about this little fern soon

DanielPeres

Hey there, I stopped the project for a couple of days because I wasn't able to work on it but now I am back for a little more progress. So I decided to do te foliage later since now I have the basics of it.

To do so I started working in the base floor texture, which is a leaf floor. To get the result that I want I guess ZBrush (avoiding substance designer for now) is my only way to go because I need a random but tileable texture (if you know any other way to do it, like another software or plugin for Max/Maya that would be awesome) from which I can extract a normal map, displacement, color map and also alpha masks for each of the five types of leafs.

So by now I have the following work done:



I am liking the result by know, since I already tried some renders in MT2 and they look nice. The problem is that I want to add much more leafs so that you cant almost see the dirt and this is where my problem starts. I need to offset the multiple layers (on the left of the image) together (as you do with the "~" key) but I just can't get a way to do it because whenever I try to do it, only the selected layer will move. Anyone experienced with ZBrush can give me a little tip in this matter? Taking in count that merging the layers is not a good option because that way I won't be able to render the various alphas.

DanielPeres

BUMP