Art-Machine is talking about rotating to arbitrary angles. You would need to convert the RGB values to vectors, rotate the vectors around Z according to the desired rotation, convert those back to RGB, and then rotate the actual selection in Photoshop as if it were a standard rotation operation for an arbitrary angle.