How Does Vertex Colours Work For Animations In UE4/Unity

Zoddo

I want to make wind animation for vegetation using Vertex Colours for game engines.

I've seen some great documentation on how to do it in UE4 with Max (pivot painter).

But I want to do this with Blender, and also export to Unity.

For Unity there's an asset called "Pivot Brush" with which you can vertex paint on meshes in-editor.

But I don't want to rely on beta third party assets.

So how can I paint vertex colours and export that information to UE4/Unity?

Do I have to bake those vertex colours on as a texture?

Do I have to Link/Rig leaves/Twigs/Branches?

I have never done this, and surprisingly can't find much information on how to do this besides UE4 with Max.

Can someone atleast point me into the right direction, as I am new to this?

rollin

You have a vertex paint mode in blender
And no: baking them to texture would defeat the purpose of vertex colors. They are stored as part of the mesh data and e.g. the fbx file format does help to transport them to your engine of choice

Mark Dygert

For anyone wondering how you handle this in 3dsmax and unreal... I'll paste this from a older post of mine.

Unreal and 3dsmax totally support that workflow. I use it all the time, so do a lot of other artists.

In Unreal you use the vertex color node instead of a texture sample. You can apply that directly to the base color and you get the raw vertex colors that are baked into the mesh. OR... you can mix it with other nodes to get different effects or use it as a mask to blend two textures together, it's really up to you how you want to use them, its a really flexible system.

Unreal documentation about Vertex color Materials:
https://docs.unrealengine.com/en-US/Engine/UI/LevelEditor/Modes/MeshPaintMode/VertexColor/MaterialSetup/Color/index.html

In 3dsmax you don't have to bake it to texture maps, I mean you can but that is usually not what you do, you do bake it into a vertex data channel, using the vertex paint modifier.

1) Typically you want to set it to "Diffuse Only" so you get just the vertex colors, but you do have some options for lighting. Note the option to color by vert or by entire face, it can be handy depending on if you need hard edges or soft. Play with it, you'll get to know it.

The reason for that is...
If you have it set to "lighting + Diffuse" (on by default) you get whatever lighting is also in the scene, which is usually default viewport light behind the viewport camera and that only lights the things the viewport is currently looking at and changes based on where your viewport camera is looking. That is generally undesirable and inconsistent results.

If you have lights in the scene it uses those instead and is no longer using the viewport light, yeah!

Which can be handy for things like...
Using vertex color to create fake ambient occlusion for things like bushes and trees. 

You bake the AO into the vert colors so you don't have to bake it into the texture, which is usually a handful of cards scattered around the foliage object, those cards share the same UV space and if you baked AO into the texture you would get AO from every card in the model mapped to that one set of cards.

To bake AO into a texture you have to provide unique space for every card. That can really waste a lot of texture space and it's all repeating details until the AO gets applied. You can also use a second uv channel and another texture with just the AO baked in but that is expensive so we typically bake that stuff into the vertex channel where you don't need that extra work and extra textures.

This is something speedtree does automatically and it's great but you can do it on your own in 3dsmax or whatever.

2)  To view and paint vertex colors click "Edit..."

3) There are several display modes that show you the vertex colors, mouse over their tooltips explain what they are.

4) Paint, erase and color picker options as well as brush size and opacity settings. Note the use of layers below, those can be helpful.

5) You have to click "Assign" or the colors aren't baked into the vertex color channel. Just because you see it on the mesh in the viewport it doesn't mean it's actually saved. If you collapse the modifier stack that also applies vertex colors, but you lose any layers you had going.

3dsmax vertex modifier documentation:
https://knowledge.autodesk.com/support/3ds-max/learn-explore/caas/CloudHelp/cloudhelp/2016/ENU/3DSMax/files/GUID-EADD121F-482F-4387-8D6C-0548533E4F56-htm.html

Zoddo

rollin said:

You have a vertex paint mode in blender
And no: baking them to texture would defeat the purpose of vertex colors. They are stored as part of the mesh data and e.g. the fbx file format does help to transport them to your engine of choice

ty, atleast now i know i have to use fbx and not obj and to directly paint them