The solution, as pointed out by haikai, is that Ignore Mirrored Faces must be checked under Transfer Maps > maya common output.
thanks to everyone that helped.
since i modelled half the geometry while instancing it, combining the two sides of the mesh mirrors the UVs, as best is my understanding. this creates the strange depth in the normal map

took the High poly from zbrush, transferred the normal map to the low poly with default settings using Maya, this is what happens. Do you see the problem? then i notice the edges have artifacts, but that is secondary. any ideas how to fix it?
it looks like the normal map is being generated from the front of the models to the back as the depth of the normal.
Replies
Maybe your low poly has flipped normals?
I installed xnormal and will try it, but i have done this before, once or twice, a long time ago, and it worked.
i'm getting a lot more responses here than in the cgsociety forums. why?
perhaps i simply have overlapping target geometry which is cuasing the strange jaggy artifacts in the normal map.
and perhaps the resolution and sampling rate must be increased for the transfer to avoid further jaggyness. (i used preview quality, since i was testing.)
i added lights... as opposed to using the maya default lighting, and it seems to work as it should.
I will try upping the resolution for the map, fixing overlapping target geo, and see if that fixes the slight artifacts in the normal map.
still though, the normal map itself looks strange to me.
It's usually a good idea to test from another baker to see if the results differ, the safest one is Xnormal, which I use alot for final bakes, or in cases of Max and Maya bakes, as an comparison bake to make sure everything is working correctly.
Can you give us a screenshot of your settings and scene setup?
You should try to bake with all the different pieces moved apart to give the casting some breathing space
Could i move all the low poly pieces up the Y axis, for example, so they are not touching at all? or do you mean to move the high poly away from the low poly version?
i should mention that both low poly and high poly versions of the mesh are in separate pieces. they are not all combined. the low poly pieces are not one object, and the high poly version is not combined into one object either. i have the undies separate from the legs, and the torso and arms separate from the mask and the cape.
why do i have a feeling thats cuasing problems...
This is the xnormal version. now that strange depth of the first normal map is gone. but the jaggyness is amplified.
This is the scene setup. Just a few point lights from when trying to get rid of the strange blackness in the render from the first wacky normal map.
The high poly pieces from zbrush are hidden here but are directly on top.
I have wireframe on. notice the overlapping of the low poly pieces in the viewport. Is that causing the jaggedness?
These are the settings for the transfer maps.
Then combine the pieces, and bake out the AO map. In quality production, you need 2 AO maps, one which is a solo AO, and the other is a combined one, so you can clean and have nice polished AO maps.
Other then that, in Xnormal, it usually a good idea to give a padding of 8, bake at double resolution and downscale it to get rid of any harsh edges. Sorry, don't use Maya alot, so my feedback in that regard is limited.
First of all, I wouldn't use JPEG. Second, you might want to play with the search envelope percentages on the low res meshes. Show the envelopes and increase them so that they envelop your high res meshes. If you have overlapping UVs within the same 0-1 space then make sure you have Ignore Mirrored Faces checked on (otherwise offset the mirrored UVs).
i was modeling with half the model instanced, and after i combined both sides of each mesh, i didnt check the uvs. i never even thought of it.
so that gets rid of the major problem. jpegs where used just for uploading purposes, they told us to use targa at school. forgot why though.
all these suggestions give me plenty to play with.