copy layer 2 times on one layer edge detect (in photoshop a custom filter with atrix [0 -2 0,-2 8 -2,0 -2 0] blur the result a bit and blur the ohter copylayer the way the edges arent that hard then use the edgedetection as mask for the blurred layer
view menu > layers dialog right click on the first layer and select duplicate. not sure, but I think that even if the layer is hidden and locked it still has to be kept in memory, so i usually just duplicate a few times, or if i'm decimating then ill duplicate incrementally as I go.
Ouch. Glad you figured it out though. Ah so it was a bunch of animation layers on each Biped? I haven't used the new Biped stuff yet. But it sounds like this is a similar issue to having a ton of layers in a 1024^2 PSD file, each layer balloons the size.