I'm pretty sure you need to create a new image map as an output, and set its "effect" to displacement. Shading tree -> add layer -> image map -> new image -> change effect to displacement -> right click on the image map and towards the bottom you got baking options.
Hmmm - looks like in 2021 on MODO 14.2 this does not work anymore... I imported a mountain mesh (100k faces), selected it and switched to the Paint tab (never used that before) and clicked "Geometry to brush", projection along Z axis.... Modo bakes for a while and returns a completely grey image, when baking Vector…