UE5 workflow ideas - texturing high poly meshes

Ged

I'm testing out some ideas for workflow in UE5. Heres my first test so far. I'm using baked inverted cavity (curvature) and AO on the zbrush highpoly to drive materials in unreal engine with world aligned textures. No low poly and no uv maps which is fun! Very exciting to have UE5 to play with. This runs at about a almost constant 40fps on my 1060gtx with lumen and nanite doing the hard work.

I cant seem to get normal maps working at the moment. I did notice a mikkt space error when it loaded the mesh in and converted it to nanite so perhaps there is something I could do before export? the normal maps work on unreal engine cubes and spheres etc just not on my mesh.

Please don't judge the art too much! I'm more just exploring workflows and the tech here, so the art is just mostly thrown together in zbrush with some quixel megascan textures. I made custom materials and also material functions in ue5 to do the dust/rust and edgewear. This model is around 1.4 million triangles and I had a load of them in the scene with no problems.

Ged

Just to clarify I dont think this is definitely working as a workflow for game art, its just one of many options. I also havent managed to quite get the art to export and import in the kind of quality I require - zbrush model looks better in zbrush than in ue5 (edges are faceted in some places or have odd triangulation), the bakes in zbrush arent very high quality, especially the inverted cavity map. I still have some ideas to improve things though.

Ged

I managed to get the world space tiling normals working, I needed to set the world space to true on the world aligned normals node in the shader and turn off tangeants in the details panel. It looks pretty good considering Im not using a low poly or uvs.

Ged

Im actually only getting about 24fps on my pc with 1060gtx when the scene is fullscreen, I was just looking at the editor window before which is of course much smaller than the full screen pixel amount.

Kapoff

Good to see that someone is already doing tests with VC as a "texture input"

I was thinking of  trying to push traditional "texture maps" into different VC channels, since most  DCC apps support that,  but didn't get around  to  actually test it yet, since it ain't really a priority and more of a curiosity thing. 
Vas basically thinking of  using  diferent channels in the same was as we do in a clasical PBR workflow,  but I'm not sure wheter UE will actually swallow and cew through multiple VC chans, and am also unsure of the format to use in order to export multiple VC  channels  from  lets say max or blender, into unreal. 
 There's also the mehs painting options inside Unreal intelf that I wanted to try to use in the same way, but once gain, just idea for now. 

 Maybe  those ideas will find better use in your hands Ged. 

Edit:  Dynamesh for better "texel" uniformity  and then a decimation with preserve polypaint toggled might save on the ressources.
And we need new names for this like textures and texel it this workflow is acually viable.

Alusfier

I am wondering, does nanite get rid of all of those co-planar faces

Ged

Kapoff thats really cool! I had similar thoughts about using the vertex color channels to pack information but I dont know of any way to do that in zbrush. My other concern would be how to author those maps, eg what if I want an occlusion map in the red channel and a roughness in the green and metallic in the blue, is there any software that lets you paint that sort of thing on a highpoly model straight to vertex colour.

I personally think what would be cool is if unreal engine and quixel could make a smart material system that all works inside of UE5, using distance fields and edge detection etc so all you need to export is your highpoly and an id mask in the form of vertex colours or a texture. The rest of the texturing (like what we do now in substance or quixel mixer) could all be done inside unreal engine on nanite meshes.

Alusfier I don't think nanite does anything to the actual mesh, when you are up close enough it is just the raw mesh that you exported you are seeing as far as I understand it so there are a fair amount of messy bits of geometry going on. 

Ged

At the moment Im just exporting polygroups as materials as seen in the image below. I tried exporting rgba vertex colours as an id mask and that worked too but was limited to 4 colours, a tech artist could probably make some way of finding all your colour ids and outputting them as masks. I'm ok with splitting based on materials at the moment and using material functions for shared blends between materials.

Unfortunately mesh painting in unreal engine 5 is not currently supported on nanite mesh instances.

Ged

Added a more simple modular section to give some areas of rest to the scene Also a shot to show how well it holds up when close.

Ged

Since nanite doesnt support displacement maps currently, I decided to try a different workflow for creating nanite terrain. I export my displacement out of quixel mixer, import it into blender on a subdivision surface that I can sculpt in blender. UV it quickly and apply the displacement there, then seperate that mesh into a grid of meshes ready to be exported as fbx and imported into ue5 as nanite. It works really well so far and the performance doesnt seem to take a hit! I have 4 nanite meshes making up this terrain.

Ged

some more tests trying to get displacement in blender from 2 different textures, painted the weights with vertex groups and vertex colour and then exported to ue5. There are a lot of really small artifacts on the mesh, I'm not sure how much of that is just my mistakes. Another issue is that at a distance the splits between the pieces of nanite mesh that make up the terrain begin to become more obvious.

Kapoff

Ged said:

Kapoff thats really cool! I had similar thoughts about using the vertex color channels to pack information but I dont know of any way to do that in zbrush. My other concern would be how to author those maps, eg what if I want an occlusion map in the red channel and a roughness in the green and metallic in the blue, is there any software that lets you paint that sort of thing on a highpoly model straight to vertex colour.

I personally think what would be cool is if unreal engine and quixel could make a smart material system that all works inside of UE5, using distance fields and edge detection etc so all you need to export is your highpoly and an id mask in the form of vertex colours or a texture. The rest of the texturing (like what we do now in substance or quixel mixer) could all be done inside unreal engine on nanite meshes.

I know that blender definitely lets you paint vertex color directly, and it also supports multiple vertex color channels, here's a quick sanity check to make sure I'm not spreading BS on the forums. 
Just subdivided a cube to get sove verts real quick, I guess it qualifies as programer art at this point. 
 Basically, if you setup your blend mode to add and paint in full  R, G or B, you can paint those easely it.  And I think there might be an addon or two to paint stuff with layers  kinda like painter or 3DCoat, but I'm sure it's nowhere near the level of usability you get out of the box in any modern texturing suite. 
 I was thinking of something more akin to  simple masking like that as a first test, to see is any of the modern export formats support exporting and importing multiple VC channels in UE, and worry bout the prettiness of the thing latter. 

I suppose  you could pretty much setup a a complex node based material with texture inputs and then bake it  in down into the VC. I think theres a way to do that in pretty much any 3D package nowadays. 
Worst case, you paint your stuff all in zbrush, then export multiple highres  objects, and combine all the VC into one mesh with multiple channels by transfering the color data into a new channel each time. 
Could probably do the same with quick uvis and  textures authored in painter , then baking down the textures to the VC.  
 Again, all theoretical at this point. Just throwing logs into the fire, to see what burns bright.

Ged

That's cool Kapoff I've used vertex colours in maya and blender before, didn't realise you could have multiple vertex colour channels in blender. Do those import to unreal engine? Might be worth testing out on something simple yeah. I was hoping to do all my vertex colour work within zbrush as it keeps the workflow much simpler and I wanted to be able to make quick changes in zbrush and then see them in UE5 soon after.

I did have a conversation with Simon Verstraete. He's done some experiments with Houdini and UE5. This one here is cool! https://www.artstation.com/artwork/XnQGaw

Anyway Simon told me he has already tried packing roughness and metalness into the model and the results just were not high fidelity enough compared with using actual textures. From what I understand, the way nanite optimises in clusters dynamically means the result isn't very clean.

Ged

Finished this environment off and made a movie queue render and put the info including material node graphs on artstation here: https://www.artstation.com/artwork/B1dD86

https://www.youtube.com/watch?v=6d8C4TLq70Y