You should work with full colors as much as possible (RGB 255,0,0 or 255,255,0, etc), then you can try tweaking hardness/tolerance, last resort is to add a small Blur on top of the color selection, then a Levels to make the now blurred mask tighter, this will smoothen the jagged edge.