What I do to creat a custom mask is (and this assumes there is no dynamask there already), click on the mask to get dynamask, then pick one that has tight edges, then go into custom paint mode, and your edges will be there, then I untick the curvature map, and all i'm left with is what I painted.