Texture Swatch based workflow - Possible for tiling detail maps?

Shrike

So we are trying out a swatch based workflow, where you have 32 or whatever materials on a tiny swatch texture, with corresponding albedo, metalness, roughness

Then we use a mask for edge wear basically, and this works pretty good, but we are struggling with finding a way to apply tiling detail maps (normal, maybe albedo) for these swatches and are not sure how to go about for these or if this is even possible. Putting a normal on a swatch texture is not the issue, but tiling this small window of the swatch. Maybe with triplanar mapping somehow?

Does anyone have any idea or achieved something similar?

Eric Chadwick

Some things here
http://wiki.polycount.com/wiki/MultiTexture#Variation_Blending

Obscura

Are you referring to a texture atlas?

Shrike

I'm referring to a "Swatches" atlas workflow, where you have a small texture with a grid of x texture properties on it, and then use the UV2 of your mesh to get the texture information by placing the UV islands onto this swatch - so you place the grip of your gun on the rubber swatch (top right number 2 in the atlas per example) in the Swatch atlas and it then gets the rubber albedo, and the others from corresponding atlases



This works for albedo, gloss, metalness as these only require flat color to work and the swatch resolution does not matter

But now I would like to try if I can put a tiling low res detail albedo and detail normal on a corresponding swatch sheet,
but of course tiling wont work as the other swatches are around each other - and I am searching if there is a possible trick to get around that.

@Eric, I checked but didnt find anything of the sorts - unless Im missing something
I know about detail maps but this is a bit more crazy, I basically would have to snap-shot one piece from the atlas and make that tiling in isolation - not sure if thats even possible

Maybe If the swatches are in one 1D strip and you could have tiling in one direction and do something with that in some way? 

Obscura

I just never heard an atlas bein called "swatch". Do you want different tiling values fot all tiles for the detail textures? How many tiles do you have? Do you want it to be dynamic? So for example, for a 4x4 atlas, you would have 16 tiling parameters?

Obscura

So what you need  is a material setup that allows you to tile the tiles without the neighbour ones coming in. Is that correct?

poopipe

You'll need to slice up the detail swatch in the shader by messing with uvs.  It's entirely possible.

You'd be better off using texture arrays rather than an atlas though if the engine supports it- much less fuckery involved

Shrike

Yes that's correct @Obscura

I would only need 1 tiling value for all these as we could just keep texel density consistent (to decent degree) for the UV

(An atlas is what I take as an array of textures on one sheet from uniquely UVd meshes merely combined but sure this is also a kind of atlas, Photoshop term I guess > Color Swatches - also "Atlas workflow" would be confusing - that would just be making assets in batches in my mind at least)

Texture arrays hmm, does this not have the heavy lookup cost from the many textures or is that batched? That is not supported by shader graph or such in Unity I think however but Ill have a look. Btw are texture lookup costs proportional to texture size or is there a fixed cost in addition? 

poopipe

*caveat... 
I dont use unity at work so no idea what their implementation is like

however

Unity has support for texture arrays

AFAIK the lookup cost is significantly less than loading a list of non arrayed textures - I think its kind of dealt with before it becomes a problem. 

Under dx11 I believe you must ensure that all packed maps are the same resolution and  its not possible to dynamically load/unload textures from the array at runtime.
This can lead  to a fairly significant static memory cost but you only end up paying for a single material so usually balances out favourably. 

Under dx12 I believe its possible to have mismatched texture dimensions and to perform some loading/unloading at runtime.

This will all be dependant on engine support so  you'll need to discuss it with your programmers to be sure 

Shrike

I looked into the texture arrays and this really looks amazing, we'll likely go with that, thanks!
Now I just gotta look up some good masking workflow

pior

Hey there Shrike ! Another interesting topic, good stuff

First off from playing with this sort of approach myself I would say that there is an interesting variation to the swatch approach consisting of relying on vertical gradients as opposed to just a grid of swatches. This lets you vary value intensity by shifting UVs up and down, which can be very impactful since value read is such an important part of picture making. Like so : 



Now regarding the tiling of surface details as you mention :

"But now I would like to try if I can put a tiling low res detail albedo and detail normal on a corresponding swatch sheet"

If I may ask : why assume that the detail maps (color and normals) would have to be placed inside the square swatches to begin with ? You could simply deal with the tiliing of these components through a secondary UV channel and ... that's it !

Pushing things further you'd likely end up with 3 UV sets : set 1 for color/material swatch assignement, set 2 for vertical mapping of gradients (of course 1 and 2 could be combined is using the vertical gradient stripe approach), an optional set 3 for "regular pelt unwrap" in case uniquely mapped textures are ever needed on hero items, but also to control the tiling of detail color maps/normalmaps ; and maybe also an optional set 4 if an engine requires one for lightmap baking. And then vertex colors could also be leveraged to store AO/cavity too ...

(of course the above would require to have a different material per ... material (!), as the tiling normalmap would need to be a full square texture of its own. But ! Imagining that one would need different sets of tiling details/normals for each material is just an assumption too. Maybe all materials could very well be described using a single tiling normalmap with a few generic dings and scratches, and the tiling diffuse and roughness could very well be generic stylized noise/paint strokes. Hence a single material and a single set of textures for everything. The tiling elements and surface damage are really just tertiary/most minor components of the overall look after all).

Eric Chadwick

The old Unreal wiki explains one of these multi-tile setups
https://docs.unrealengine.com/udk/Three/TerrainAdvancedTextures.html#Texture Packing: Implementing large texture tile sets



A similar multi-uv method
https://polycount.com/discussion/comment/1588220/#Comment_1588220
(see earlier wiki link for additional examples)

Shrike

@ Eric - Cool! - I noticed the UDK docs have better shader examples than newer ones

@ Pior, 
I was also considering a horizontal strip, where we would put the albedo on top, and then metalness, gloss etc always vertically below and work with UV offset to sample those, this would save you different sheets for metalness etc

Yeah values are important but I think we will have a real palette with fitting values in the materials

Adding another UV channel however would double the vertice count and Im not a fan of that performance loss

What we tried now with the Texture arrays is:
- Make base materials, put them into one array (one array for albedo, one for normal with gloss and metalness in blue and alpha)
- Make wear materials, in a secondary array as above, all with 128x128 or so tiling maps
- Use one RGB mask (r,g,b,yellow,teal,pink, black - depending on blending order this accounts to 7 masks) - to get 7 switchable materials (we also tried grayscale splits which would account to many more masks at shader complexity and painting confusion cost)
These masks are then used in a predefined fashion (base material 1-3, highlight metal, plastics, optional 1, optional 2 or such) 
- Use one RGB mask for a wear mask, global roughness, global camo and secondary color mask (vertex colors could also be used, but any masking workflow is fine)

The wear mask is blending between the upper base material and the secondary wear materials (like dark rubber, and then bright worn rubber)

- The advantage is that we only need the basic UV channel (+ lightmap / trim uv) as its just used with masks 
- You will mainly need to paint a wear mask to differentiate between top and low layer material and get realistic looking imperfections and with full tiling detail resolution
- You can build up a material library and use them throughout and swap them easily
- With the texture array workflow you can just input new texture arrays thematically (like swapping palette for a russian prop to a police prop)
but with the extra freedom of being able to change materials with just number changes (like police palette has 2-3 variations for their Highlight metal material)
- Compared to the UV workflow, after you have set the mask, you can just change a number in your material and swap out materials in runtime extremely conveniently, even if you have an in engine UV editor, this would be way faster and is very cool to do 
- If you do it like us and many things with the same material, like sci-fi environments, you can then just scroll through materials and get a different style for the level in seconds, or just replace all chrome with copper
Edit: I guess you can also make that work in the swatch workflow with UV offsets
- Compared to the UV workflow you get tiling details for each layer

Disadvantages so far are:
- You need to paint the masks in relative abstraction (although could be solved with a example shader in painting apps - this only works if you keep a pattern tho)
- if the masks are too low res, you need to be careful with baked details, like screws, although those wouldn't work at all with the UV based workflow
- Of course do you need a lot more memory than the swatches, which could be just 4 pixel per material, but should be no problem depending on your usage
- Switching materials in the texture array is more effort than changing a swatches texture
- Arrays only work above shader model 3.5



This array stuff seems extremely powerful and Id really recommend looking into those

gnoop

Tried this  UDK approach in Blender and the seams are pretty strong if the texture quadrants differ from each other  color wise.  In distant mip levels especially.    Wonder have modern support in Dx11/12  fixed it somehow?

poopipe

You have to pad if you do that.  Or disable mip maps and filtering

Eric Chadwick

Perhaps you could adapt this to work in Blender? Note that if going out to a game engine, you'll need to do the same there too (best to only do it there).
http://oliverm-h.blogspot.com/2013/04/how-to-create-ue3udk-texture-atlases.html?m=1

That link is from here
http://wiki.polycount.com/wiki/Texture_atlas#Tutorials

gnoop

Thanks Eric.    Shrinking  UV  slightly  like it's in point 2  from  that link doesn't help much in Blender. I bet it's point 1 "Custom"  HLSL shader that does something with mips.   I am not sure I have understood what's exactly.          

Disabling filtering and mipmaps is not an option imo 

Obscura

Gnoop - Take a look at this. The reason for the seams is that the neightbours are blending in because of the texture filtering. The texture AND the shader needs to account for this. You need 1px border with the same texture, and then inside the shader, remove the border by shrinking the sampled area by 1px. For more mips, you need wider border. This way, it will be correctly interpolated:
https://shaderbits.com/blog/tiling-within-subuv-or-volume-textures

- Note that this doesn't only work with volume textures, but works with any type of atlas (volume texture is an atlas of slices basically) and you would run into issues, regardless if its a 2d or 3d texture. Flipbooks with details reaching the edge can also suffer from this (as its shown in the post).

gnoop

Thanks Obscura

I think I understand the shrinking idea.   But when I try to recreate it in Blender I still see the seam, not as strong as without shrinking but it's still there.    Same in our engine.    Looks like Unreal does some extra balck magic   Something with mips I couldn't figure out what exactly.

 

Obscura

So I did some tests and it turns out that this only work if you provide manual derivatives. I think the frac messes this up but I'm not sure (when no frac is used, the line isn't there, but then you couldn't isolate a tile). Providing the derivatives as the texture was simple tiled works. By the time I write this post, I figured that this is probably slightly off because a tile only takes quarter of the texture size so the providing the 4x4 tiled uvs is not fully correct. I can also confirm that adding wider border doesn't get rid of the line when automatic mip level is used.



Another thing that I can confirm is that the wider border may still be needed, because viewing from a grazing angle still shows the neighbour tiles a little bit. But its hard to spot when viewing it from far. I only used 1 px padding:



Textures used:

Without padding:

With padding:

Obscura

I did more tests, and it seems like for such extreme angles, you would need a very large border, at which point, you waste a lot of texture space. So I would not recommend doing that. This whole thing also takes a lot of effort starting with writing a specific shader, and continues with authoring textures this way. I'm not sure if it worth the effort these days when even mobile devices are very capable. I also concluded that Ryan in his blogpost only eliminated the issue with texture filtering on full res, but it would come back if the texture would get mipped by a lot. 

Obscura

@Shrike - This is the material setup that you would need but there are several caveats:
- You can't use mips
- The 1px border is still needed, if you want to have any texture filtering
- ???


So its better to have individual textures.

Obscura

Actually the UDK documentation example points out all of this right in the beginning, and doesn't provide a real solution either. So it looks like we are hitting the dead ends of packing tiling textures into an atlas.

gnoop

Thanks a lot Obscura  for investigating it. 

Shrike

yes this approach seems too complex, id rather get 1-2 detail maps manually in for the most important sufaces or use the texture array approach which seems more flexible and powerful, not to forget the save of an extra UV 

give it a try if you haven't

gnoop

Shrike said:

yes this approach seems too complex, id rather get 1-2 detail maps manually in for the most important sufaces or use the texture array approach which seems more flexible and powerful, not to forget the save of an extra UV 

give it a try if you haven't

Are the texture arrays you are talking about somewhat free of seams issue?

  Honestly I am not sure I understand how they work deep down    besides some vague mentioning of  using W or Z   in UV values  as ID for parts?  .     Can I recreate it in a node network  or it's a feature too deep  to gpu hardware?

Obscura

Unfortunately Unreal does not support texture arrays yet. The class exists but its not exposed. In 24, It was exposed, but creating a texture array asset didn't create anything. It even gave an error message saying that it was not created. In 25, its hidden now. There are several traces of future implementation though, when you create a material function, you can choose to have an input that is texture array type. I would be very happy if they would finally do it, because actually I have a certain use case for it.

Edit - I just found a post on their forum. It has a lot of bullshiting, but one guy found that using "r.AllowTexture2DArrayCreation 1" in the console enables them...Wtf...

If you use it, make sure to put it into an ini file so you don't have to re-enable it every time you open the editor. On the other hand, the previous forum post says that it does not work in a material so I guess we will have to try and see:

https://forums.unrealengine.com/unreal-engine/feedback-for-epic/116810-texture-2d-array-support/page4

Obscura

It actually works with nodes.... Use the W component as index:

I would assume they create a gigantic volume texture from the list of textures, because of the way it works. Like putting all texture on top of each other and select which one is needed by changing the z coordinate. I may be wrong though. Also, I don't get why is it not exposed, why are they saying that it doesn't work when actually it does.

Obscura

Works in code too...

Prime8

gnoop said:

Thanks Obscura

I think I understand the shrinking idea.   But when I try to recreate it in Blender I still see the seam, not as strong as without shrinking but it's still there.    Same in our engine.    Looks like Unreal does some extra balck magic   Something with mips I couldn't figure out what exactly.

 

If I'm not mistaken, this is caused by the filtering of the viewport, try to render it with Cycles.

poopipe

gnoop said:

Shrike said:

yes this approach seems too complex, id rather get 1-2 detail maps manually in for the most important sufaces or use the texture array approach which seems more flexible and powerful, not to forget the save of an extra UV 

give it a try if you haven't

Are the texture arrays you are talking about somewhat free of seams issue?

  Honestly I am not sure I understand how they work deep down    besides some vague mentioning of  using W or Z   in UV values  as ID for parts?  .     Can I recreate it in a node network  or it's a feature too deep  to gpu hardware?

Texture arrays are big lists of separate texture files that are treated like a single map at the time they're drawn- you could think of them as being a little like UDIMs in that you access a specific texture by index. 

You get the one draw call benefit of using an atlas without the limitations on tiling and the seams.
The downsides are added complexity in terms of setting it all up at the engine level and the fact you have to load all the textures in the array at once (dx12 fixes this I think) 

In terms of authoring, on our own engine it's completely transparent to the artists, we simply flag a set of textures as being an array  and they are compiled appropriately. 

Shrike

Edit: nvm

the array works by tricking the mip map function into storing more textures, very smart idea actually

gnoop

 Thanks guys , turned out our engine does full support of texture arrays too without any extra tricks

ps, regarding recreating  in Blender ,  Cycles does eliminate  that seam at close distance but they still are in mips

Obscura

I was able to make a 512^3 volume texture using this method. I also made a sampler that takes 2 samples of the full array and interpolates between them to get smooth Z. Performance seems to be fine. I'll post the results in a separated thread later as it doesn't relate to the original topic, but I thought this might be interesting for some people using texture arrays.