I have Maya 2016 and added your OBJ to my Maya, smoothing is fine ... nothing wrong .. I own MAYA Full, so not sure, may be a bug with Maya 2018, or something else going on..
EDIT: After fighting with this for two daysI have a working solution. Problem occurs only when I try to smooth mesh that is very far from 0,0,0 of the scene. When I moved mesh to the center of scene, everything worked perfectly. So this mesh that I have attached will smooth nicely because it is centered in the middle of…
I tried Maya Catmul-Clark and Open Sub-Div Adaptive. Effects are different but still similar to first one :/ As for video card problem I do not understand how? I exported this mesh and imported it on other computers in company and effects was similar. I got this problem also on other meshes, to be honest I can't make any…