The pixelated artifacts are the lightmap compression. You need to either increase the lightmap resolution on everything or turn off the DXT compression on the lightmaps.
Is there a rule of thumb how many pixels of padding are the best for a corresponding resolution? Is 8px for 4k enough? Or should it be 16px or even 32px?