Not sure if you solved this completely, but are you baking and retaining the same vertex normals when viewing your model? To me it looks like you might have baked with averaged/smoothed vertex normals, then applied this result to your lowpoly which has hardened/flat vertex normals.
They use face weighted vertex normals. Ther's a thread floating around about custom vertex normals on the first or second page. Should have some useful scripts for you. edit: http://polycount.com/discussion/154664/a-short-explanation-about-custom-vertex-normals-tutorial/p1
My idea is to use the vertex color to lerp between the sand ripples and smooth areas. I started by painting vertex colors in blender to match the windward side of the dunes I used shader forge in Unity to create a custom shader that will select between two detail normals based on the vertex color:
Playing with some vertex blending techniques based on height and masks in UE4.
It uses 3 material sets, one vertex channel for all ground blending and it uses alpha vertex channel for puddles. Material sets were made in substance Designer. https://www.youtube.com/watch?v=rauSgIPlXSc&feature=youtu.be
It's been a while since I used Max, but does ProOptimizer take Vertex colour or weight as an input? Because there are definitely ways to bake a bitmap to vertex colour, and vertex colour to weight seems like a nobrainer as well. If that doesn't work, Houdini's practically made for stuff like what you're describing.
I have a 256gb OCZ Vertex 3, from when they came out almost 3 years ago. I haven't had any issue with them, but I know there have been some issues with the vertex 3, and much more with the vertex 2 so I couldn't really recommend.
You could use it like another vertex color, probably the most useful thing since vertex colors are very useful, I'm a little unsure why anyone would need 5 vertex color "channels" though, unless you want to blend 6 textures on a terrain or something crazy.
Ah. The vertex normals are twisting to such an extreme that you've entered a black hole! You need to split the vertex normals along the edges of the blade, by adding hard edges there. Also called smoothing groups. You're making me hard. Making sense of hard edges, uvs, normal maps and vertex counts
I assume its because if a broken vertex table. Under tools in 3dsmax there should be somewhere the vertex channel listener. Alternatively run this maxscript in the maxscript listener:ChannelInfo.dialog(); maybe it can give you some info on what might be wrong because I highly suspect a wrong table for the UV, smoothing or…
As an alternative, you could try using Vertex Color as your selection mask. I've done this for animating foliage, making the outer leaves move while the stems do not. You can use a Vol. Select modifier to turn the vertex color into a selection, like this. Just load a Vertex Color map in the texture slot.