5.11 world partition, landscape HLODs and crashes..

poopipe

I may be confused, I may be wrong and I would be very happy to find out that is the case

but..

Even if we pretend there's no Out of Memory crashes in the HLOD builder it still behaves very inconsistently and will frequently lead to GPU crashes on launching your level

I take a relatively small (4k*4k) landscape

I create an instance HLOD layer with tile size of 256m and streaming distance of 1024m, build HLODs and all is well

I add a parent merged HLOD layer with tile size of 256m and streaming distance of 2048m, build HLODs and it shits the bed. This reliably occurs during the generation of the merged HLODs

This is the most common repeatable error : Assertion failed: CellBounds.Min.Z <= CellBounds.Max.Z - I assume this is the result of something not getting built previously. I don't think it's a lack of memory - i don't go past 32gb ram or 8gb of GPU memory during the process so I have plenty spare

My question is.. has anyone on here successfully built a decent sized landscape using world partition and landscape HLODs? if so did you come up with cunning tricks or can you offer any suggestions on diagnosing the source of the problem?

For my purposes I need a minimum of 8kmx4km and I would really like to take advantage of world partition since it will be fairly heavily populated with meshes and will be a pain in the arse to manage using world composition

ta

To head off the obvious response to the above, This warning from the docs is bullshit, both water and landscapes have been supported since 5.1

poopipe

okay - so I got it to build HLODs, this is as much a post for future me as it is for anyone else

This is a level with two separate 4032x4032 landscapes placed side by side (indie gaea has an export size limit) , the build took around 45 minutes on a 5950x with 64gb, a 3080 and it was all running off fast PCIE4 nvme disks - the machine was very busy (and full) for the whole time .

2 HLOD layers set up as described in the above post

so - how did I fix it?

It all appears to be hypersensitive to the contents of your disk and whether the editor/hlod builder think they're allowed to overwrite/modify/delete things or not - I feel like the problem has been exacerbated by the use of version control but then again, my project would have been utterly wrecked if I wasn't using it.

Hopefully Epic will come up with a more robust way of handling one-file-per-actor and version control in future versions cos atm you're basically screwed if any process fails halfway through.

My top tips for this sort of thing are as follows...

If it shits the bed in the first few minutes it's probably because there's files on the disk it doesn't like

if you can't delete or overwrite existing HLODs, commit the files and then try it

Close the editor between deleting and building

if it's still being a dick, clear out all the cached stuff and suck up the consequent rebuild times.

poopipe

Well that didn't work ..

World partition appears - in it's current state - to be completely unusable at any sort of scale.

I have solved the unsolvable problems by dropping world partition and going back to world composition. It's still a massive pile of crap but at least you can make tools to fix the problems that WC causes

said tools include...

EUB_unfuckLandscapeLODNormals

EUW_unfuckLandscapeLODMaterials

If anyone has a need for such tools I'll be happy to share

Yarogor

Well there is a bad news about world composition, it seems that nanite meshes really rely on World Partition in terms of frustum culling. To see it you can freeze rendering and see that nanite meshes ignores it )

I had a LOT of meshes on my level and when I've moved to WP - performance have significantly increased.

About HLODs building - I can agree- it`s a pure russian roulette. There is SO many things that can go wrong.

In my experience:

Perforce issues - all files must be not only committed - but be submitted as well.

Memory issues - first of all create HUGE page file for your virtual memory and here is why:

when Merged/Approx HLODs are building with default settings they`re creating not only super high dense meshes (which is a bug) but create them with super heavy 8K textures as well. And when it do this it takes ALL of your VRAM and then - RAM, and when it empty too - your page file.

HLOD Layer Setting - do not use any automatic settings in "Texture sizing type". And decrease voxelization resolution too

poopipe

I've just started littering my landscape with nanite meshes - I'll take a look, have a little cry and then probably just turn them all back into normal meshes.

I'll get back to this stuff a bit later on so the advice is much appreciated.

When you say huge pagefile, how big are we talking? I'll buy another disk to put it on if I have to :D (it'll be a lot cheaper than getting more RAM)

Yarogor

oh well, in my case even 1 Tb wasn't enough. So if you can - use merged HLOD, but Approx is much better for trees (Merged makes them looks like LOD3 :) )

So lower settings as you could )

Yarogor

Ok, now it works

1) Go to https://docs.unrealengine.com/5.0/en-US/world-partition---hierarchical-level-of-detail-in-unreal-engine/

2) Browse to Generating HLODs section

3) Use command interpreter as described

4) UE must be closed

5) Profit

So seems like HLOD building broken only in-engine.

Yarogor

BUT even this solution not working with big forest. Large amount of trees is Approx HLODs Kryptonite

poopipe

I guess we can cling to the faint hope they get a bit closer to making it production ready in 5.2 - I'm not holding my breath since they've clearly put the development roadmap in the hands of the marketing department

work has ceased for the time being - my water pump died . probably should have ordered another disk while i was shopping for a replacement..

Yarogor

Well, there is already a lot of HLOD related fixes in their GIT, so they will be in next release, So yeah, there is a hope after all. HLODs are important part of their new pipeline so there have to be some improvements in 5.2

poopipe

soo ...
I experimented with moving my project to 5.2 and world partition today. 


good news : 
World partition conversion inside the editor appears to be viable. With my partially populated 8x8km map it didn't go past 53gb Ram usage and took < 40 minutes to complete. 

bad news : 
the world partition editor is so shit that it's nigh on impossible to select things and apply settings to counteract the appalling editor and PIE performance you get after the conversion


further bad news : HLOD building is still broken to the extent where it's not worth bothering.  on the plus side it crashes more gracefully and much earlier in the process.

conclusion : 
back to world composition

poopipe

an addendum since there was a patch for 5.2 and more information has emerged on reddit and the Unreal forums 

HLOD building works if you disable virtual textures and set your landscape HLOD layer to use the Flatten rather than Flatten_VT template material. 

a note: Default HLOD layer settings will cause your landscape proxy HLODs to be built at LOD0 resolution - this is not good, you will run out of GPU memory and it'll crash

This is obviously still garbage but it's something close to a workaround. 

poopipe

5.3 update : 
still unworkable but at least more of the crashes give you actionable information

lod selection for landscape tiles is still garbage
texture sizes for generated materials seem to be 8k regardless of what settings you select which leads to GPU memory crashes during the build (there are a couple of engine changes on the unreal forums that purport to work around this problem) 
it does appear to actually build hierarchical lods (in the sense that it will merge groups of tiles into one mesh, the selections it makes are a bit mental but it's a start I suppose)
it still runs like shit when you do get it working.  carry on with world composition

iam717

curiosity has hit me, since idk how things work, i click tools and make what i believe to be decent cloned representations of 2d concepts.

However to the point if one can "MOD" a game can one "MOD" a game engine to prevent these crashes? cause that sounds horrible to deal with.  (constant crashes, i gave the engine a shot to learn it after how long it loads and how many times it crashes i gave up.)  Oh yea idc about their Eula, just lets say someone wanted to do it for "themselves."

poopipe

In unreal's case you have the source code and you can mess with it with no legal issues. If you have the ability then you can fix anything.
Epic lean pretty heavily on the community to fix engine issues and you're free to submit fixes to their github for consideration

This thread is moany as hell but it's mostly here so I don't forget what I've found while trying to update my project at each release 

poopipe

well, I figured I'd give it another try since there's been an update. 
it's still knackered on 5.3.2 but things have changed


you can build landscape HLODs provided you don't try to build  too big a level (4km^2 for me)  provided you disable virtual textures.
you cannot build all the HLODs  - at best I've managed to get 2km^2 of my landscape to actually generate
it will always use the highest LOD for your HLOD mesh
Basically  - you deal with the fact you have to keep your landscape as a live landscape object (which is very fucking expensive in a large world)

ok - so, having accepted that landscapes are not supported by world partition we move onto static meshes
HLODs will mostly build - what it decides to build and why it decides to build it in the way it does is a mystery but it will complete builds reliably

Randomly selected levels will simply never stream HLODs in PIE. it does not matter how you set the level up. Generally speaking it will work in a packaged build but obviously that's not very quick. 
I've checked this repeatedly in single player and on dedicated servers (listen servers aren't supported)  - fiddling with settings appears to make fuck all difference.  

The fact they have 'next gen landscape' in the future plans list on the unreal trello thingy suggests very strongly that Epic have no intention of actually fixing this shit. 

I'll try nanite landscapes as an alternative just to see but last time I did that it didn't end well. 

So, yeah...
we're about 2 years into ue5 and the best advice i can give for anyone intending to make a level more than a couple of hundred metres across and put it in an actual video game is still to stick with world composition and accept that you're locked out of most UE5 features 

gg Epic

poopipe

so yeah, 
Nanite landscape might be the answer - at least until the 'next gen landscape' thing turns up (which will just be nanite landscape in a dress).  
I've not assessed performance with actual content in a cooked build yet  but it appears to be significantly better than a landscape with no HLODs and in terms of workflow it's just a button press to build your data. 

andrey_aleks

Hi guys! Thank you for sharing your thoughts and attempts! It helped me a lot recently.

I also would like to share some of the tips about the HLODs since this page is quite popular. Some of the points were already mentioned in this thread, so I just collected them once again in one message. I hope it will help somebody.
Actual at the moment of UE 5.3.2.

1. Make sure you have committed or reverted the last baked HLODs in your revision control system (SVN, Perforce and etc.) before you start building new HLODs. For some reason sometimes you can’t build new HLODs / delete previous HLODs until you submitted / completely reverted them.
2. HLODs require a lot of memory while baking. Increase your virtual memory size at least to 150GB. It will also help you with the huge PCGs generation since they also have some memory issues. I set it to 512gb and generally I didn’t have any out-of-memory crashes. How to do it - https://answers.microsoft.com/en-us/windows/forum/all/how-do-i-increase-virtual-ram-in-windows-10/4e98f34b-9bf7-4b45-b6c3-a6c9ba326294
3. How grids and layers work from a user’s perspective:
   1. Grid’s Loading Range - when we go further this distance, all the actors that are assigned to this grid are unloaded (if they have “Is Spatially Loaded” enabled) and the default HLOD layer (or the overrode per actor HLOD Layer) is enabled.
   2. HLODs’ Loading Range - when we go further this distance, the HLODs are unloaded (if the layer has “Is Spatially Loaded” enabled) and the Parent Layer (which is also HLOD) is enabled.
4. Split your level actors into 2 grids / HLODs systems - foliage actors and everything else. Create a second grid called “Foliage”, increase its Loading Range and set all the foliage actors to it by manually typing “Foliage” to their “Runtime Grid” parameter.
5. For the foliage use the “Approximated Mesh” layer type - looks like it is supposed to work with Nanite trees. For the usual buildings and props use this HLODs sequence: Instancing → Merged/Simplified Mesh.
6. Make sure you don’t have non-nanite foliage to be built in Approximated HLODs, otherwise you’ll have shadow artifacts in HLODs.
7. After you have built the foliage HLODs, pin the folder with them (RMB - Pin), open the HLODs’ mesh → material and disable Roughness, Metallic, Specular usings, set Roughness value = 1. It removes the glossiness from HLODs. For some reason Approximated layer type doesn’t use the settings you had prepared in the HLOD Layer file, it messes up the roughness, metallic, specular anyway. At least in my case.
8. Disable all the small meshes from the HLODs generation - you won’t see them from a distance anyway. It’s related to the small PCG meshes, different interior stuff, exterior boxes/stones and etc. You can create a simple Editor Utility Widget tool that disables “Include in HLOD” for the small static mesh components/actors. You can use the node “Get Component Bounds” and its output value “Sphere Radius” for that.
9. If you built your HLODs and some actors are not included in HLODs, you can try:
   1. Check if the actors have the parameter “Include Actor in HLODs” enabled.
   2. Check if the actors’ components have “Include component in HLODs” enabled.
   3. Move these actors up a bit and then return them back. Save, commit the changes and try to bake HLODs again.
   4. Move all the actors (landscape included, you can export/import heightmaps and layers masks for that) to a new empty open world map and try to build HLODs there.
   5. Check if you have a minimap volume on your level. If you don’t have it - place a world partition mini map actor to your level, scale it so it covers the whole map. Click Build → Build World Partition Editor Minimap. Some people said it could help.
10. Uncheck “Include actor in HLODs” for the landscape. Also disable the parameter “Is Spatially Loaded” for it. Looks like Nanite Landscape is the best way to optimize huge landscape for now.

poopipe

nice  - there's a few things in there i've not tried 


5..      you're HLODDing nanite foliage?  i didn't realise that was a thing

andrey_aleks

poopipe said:

5..      you're HLODDing nanite foliage?  i didn't realise that was a thing

Yes, it was the way in our case. Though I understand that in some other cases it's better to leave it as it is and just cull all the foliage at a very far distance