it's a "matted" fibrous texture aka felt with a rubber seam you could probably find something similar that is "flat", scouring pad perhaps which might be better to deal with.... or something procedural to generate the material tennis ball felt
Even if you clean the canvas manually (ctrl+n)? And what about the flat render, is it red too? If flat render is ok, then you can apply flat material manually.