If for input you have only this 2 textures you should select the black color on the rock texture and use this selection as a mask. You are unable to do any magic tricks here.
In case you need to use Max later... I would pass a sub-object selection up to the Displace modifier. I'd also use soft selection if it needed a transition area.
W+right click > Axis > Custom > Set to edge > Select one edge It's possible that it's called "Set to component". What this does is that it aligns your manipulator to the selected edge/face/vertex.
Tint the shadows with red in the AO map for starters (create a Selective Color -adjustment layer in Photoshop, select "Black" from the drop down menu and set red to something like +15).
Hello, if anybody in the future wants some customization in shi*tty and limited export setup from SpeedTree: You need to go {YOURS_SPEEDTREE_DIR}/texture_packing Then you can select Unreal's UE4.fx and UE4.xml and duplicate that (rename also to fit yours packaging method name). Open it in some code editor (e.g. VS Code or…
In Maya, you can check some topology like you want using selection constraints ( it allows you to select by triangles, ngons, and a whole list of other options. Really useful.)
In Houdini I can paint selection and just force this selected polygons to have cusped normals. Or do this procedurally. I think in 3dsmax you could use Edit Normals Modifier.
there is also the layer system, you can use the number keys or select a layer from the bottom toolbar, and use hte m key to change what layer the currently selected objects are on.
Should be able to right click > Open with > Choose default program... > Select Maya 2014 exe and make sure you tick "Always use the selected program to open this kind of file".
Appreciate the help, seems the functionality is slightly different, I found an option to select by color which picked up the UV layer lines that were in the layer and let me stroke the selection.