ID masks need to be very sharp to handle the transition between materials correctly. You can improve the results though by adjusting the "mask from ID color" options and decrease the hardness We may add an antialiasing option later on.
Just wanted to update in case anyone else has this issue. I adjusted hardness and tolerance, then ran some filters over the mask. Usually a blur, followed by a histogram scan, and levels if it needed it. I think I had found some one else had taken a similar approach but wasn't wrapping my head around it at that time. Thanks for responding @Jerc. That put me in the right direction.
Replies
We may add an antialiasing option later on.