Here's an example. The blend textures are loaded in the Landscape Tool, not the material. They are not stored as vertex color though, they are actual pixel data.
The height map in their example image is only tiled horizontally, it's meant for mapping on just a single strip of terrain. Instead, you want a traditional height map, like this. Also, the more contrast the sharper the blend.
I haven't used it since the November build, but here are some of my notes from back then. Be aware this might be wrong now, as the Landscape system was under heavy development at the time. * The height map creates the terrain shape. * The alpha map creates a mask for the layer. There is a different alpha map for each…
Here are some examples. http://www.youtube.com/playlist?list=PL2388E7B93BFF4575&feature=plcp Did you try some searches on the UDK forum? http://forums.epicgames.com/search.php