I'm pulling my hair for this, not sure how to explain so here's a gif of my problem.

This model haves UV's, I got rgb set up, the material is flat color.
Whenever I pick a darker color and then try to color over it with the ligther shade (in here I color picked the lighter shade color up and as you can see it slightly leaves a shadow color
that is not the correct color. It leaves this weird mark behind like color residue and I have the color 100% opacity.
I just want the color I colorpicked up there not that mark!!
I looked around but never seen anyone with this problem so not sure what to do anymore. If anyone can help it would be awesome.