Yup, that's 100% it. In some bakers it is called "bent normal map" but it is mistake caused by some very confused artists years ago :) The vertical channel of an Object Space normal map (green in the majority of bakers) is definitely what you need here.