Just needed to use two "Add" nodes. One Add with a Tex Coord and my panning distortion texture plugged in. Then plug the result of that into another Add node with the same TexCoord plugged into the other port... then that into the UV coords.
Yeah I tried that but it didn't work for me. I had to use an add that plugged the TexCoord into the panning distorted texture and then use the same TexCoord and another Add and plug that into the Coordinate of the Panner for the diffuse texture.