Home Technical Talk

Maya: AO to Vertex Colors?

polycounter lvl 10
Offline / Send Message
Deadly Nightshade polycounter lvl 10
How do you bake Ambient Occlusion to vertex colors in Maya nowadays?
Mental Ray is no longer supported, developed or even shipped with Maya. Arnold lacks this functionality, and TURTLE is yet another thing in the long list of stuff that Autodesk acquired (to increase stock value) but then just abandoned.
xNormal has this support, via GPU and command line - but I would prefer to do this inside Maya.

I've done a few tests with TURTLE, and I get it working but there are always odd artifacts.
There are also some strange behaviors:
-UV's affect the result - even though we are baking directly to vertices
-Geometry with radial symmetry get different AO bake results on different sides - even though the sides are identical.
-Objects hidden in the scene affect the bake (baking it brighter).

I've tried both sample modes: the per-vertex and the triangle blend. There's maybe some other flags in the command that you can play around with but tbh the documentation is shit and doesn't seem to have been updated since like, 2012.



  • helmdubo
    Offline / Send Message
    helmdubo polycounter lvl 11
    Hello! You can try to use Vertex Chameleon plugin. Have vertex bake (mental ray), bake lighting to vertices also manual set.

  • Deadly Nightshade
    Offline / Send Message
    Deadly Nightshade polycounter lvl 10
    Thanks, this is a good suggestion but it introduces dependancies (MentalRay and a third-party plug-in).
    I managed to get TURTLE to render out good results but it required quite a lot of reverse-engineering.
    I went through the MEL-callbacks via the Script Editor, looked up the MEL-scripts and found the final command that is sent to TURTLE for the render. Unfortunately the command is bugged in Maya.cmds and PyMEL (target-flag is not parsed) so I had to do a pymel.core.mel.eval() on a command-string which means that anytime I wanna edit the command flags I have to do a string.replace()

    A bit annoying but at least this way I was able to control the rendering settings. Turns out that the TURTLE docs are a bit crap but I eventually found out that turning of bilinear filtering and playing around with the vertex bias gave me really good results.

    So yeah... I ended up using the legacy renderer TURTLE using some Tech-Art science-hacks. End results looks good!

  • reignbeox
    Offline / Send Message
    reignbeox polycounter lvl 4
    So yeah... I ended up using the legacy renderer TURTLE using some Tech-Art science-hacks. End results looks good!

    I am looking into optimization and saw your post here, checked out your portfolio and saw the tool you made.
    Is there any place that you uploaded it?

    Am trying to bake AO into the vertex color but the conventional way with Mental Ray and Maya 2018 doesn't work right now as Autodesk doesn't give out mirrors anymore.
  • poopipe
    Online / Send Message
    poopipe godlike master sticky
    Bake it to texture and import as an attribute map? 
Sign In or Register to comment.