One more update. I've baked a normal in Zbrush again with a flipped G and put that with the Max normal in Photoshop. Now I'm using the edges (where the seams are) of the Max bake and the details of the Zbrush bake. It seems like the problem was how Zbrush bakes the edges so they don't "match" that well.
1) pyramid with divisions. 2a) select each row of edges and in turn, do "add divisions to edge" (in maya), adding more verts as you go down each row. 2b) triangulate 3) select and poke all faces. 4) not shown, move the center vertices out by their normal.
In worst case, you can move back those bad verts to a good position. But there must be some other ways. Like you can try the loops tools. It has a function to straighten or relax edges. There is the make planar button as well, but I'm not sure if you can use that on edges.
I believe this is something to do with normals, like almost every irritating issue in game art! I think... if you import your model into maya and select all the edges and set them to smooth instead of hard and then export, it should fix your problem. (Mesh display - soften edge).
The UV shell > SG can actually sometimes break, I had several cases were a cylinder was cut on one edge (a low 8 sided one too) and while the harsher angles broke, the edge didn't for some reason, I had to do it manually. Check to see if that is the case.
looks decent. but reminds me alot of the dudes from the halo series alot of your edges look very soft and round to me. at least, up close, on the left pic. i would harden those up. sharper edges. post bigger pics for more detailed crits though.
You should break your uv's whenever you have a hard edge. An edge is hard because the vertex normals are splitted, which correspond to the next to last plane, in light blue. On the last plane, the normals are averaged : there's only one smoothing group. You don't need to break the uv's.
You can probably half that polygon count and still get a similar result. Also if that picture works as your reference you can probably spend a little more time defining the edges of your rock to get that sharp look, right now it looks a little soft around the edges.
Yeah, this one has been annoying me for a while too. Especially since I actually feel like chamfered edges like on the right are much, much better looking than "regular" subdvision modeling using support edges and turbosmooth. Closer to the output of parametric nurbs modeling actually.
Here a link that can be very useful for you I think :D Still by EarthQuake ! http://www.polycount.com/forum/showthread.php?t=107196 Your edges are too narrow. The bake almost do nothing on them , you should widen them and you'll have a nice bake for the edges :D