Is it possible to animate vertex colors in 3ds max and will these export into UDK or Unity? I'm thinking of possibly doing some cool dynamic vertex lighting effects, like if I baked ambient occlusion into a characters vertices at each keyframe.
Even if you can do that, I'm not sure that the UDK can handle animated vertex color. Yes you can edit the vertex color, but every frame I don't think so. The only thing that you can do is animating the material with an the effect that use to affect the geometry through the vertex colors.
I think you would need to export each animation frame as a morph target, as a separate mesh. Which would be super expensive. But you might be able to do some programmatic animation effects. Like color cycling the existing colors, or brightness/contrast/hue/saturation/etc. Like, you could fade out the vertex color over…
Not that I am aware of with UDK. Why would you want to do this, anyway? I can think of many different ways to do better vertex ambient occlusion than to animate them in 3ds Max.
Realtime animation systems dont operate per vertex, and havent since the mid 90s. If you wanted to store information per vertex per "frame" you would have to create your own animation system (for example morph targets), which is completely possible in Unity, but would not be ideal for a mobile device at a reasonable speed…
You'd be better to use the vertex colours as an index that points to a row on a texture. The next row on the texture would be the next set of colours that you wanted animated - pretty much like how we used to colour animated sprites back in the day. It means you'd only need one mesh, not several, no morphs - just do it all…