Home Technical Talk

Is there a Maya solution for baking alpha planes to texture

poopipe
grand marshal polycounter
Offline / Send Message
poopipe grand marshal polycounter
eg.  I make some alpha leaves,  put them on some geometry and now I want to bake loads of these down to a plane 
Ideally  this would be a solution that works in maya 2018 ..

any suggestions? The main thing is that the bake needs to support alpha blended transparency


thanks







Replies

  • throttlekitty
    Offline / Send Message
    throttlekitty ngon master
    https://knowledge.autodesk.com/support/maya/learn-explore/caas/CloudHelp/cloudhelp/2018/ENU/Maya-Modeling/files/GUID-5580B297-5045-4251-9596-0F468D73A12B-htm.html

    Something like making that; actual geometry out of your cutouts, or more like scattering your alphas on the texture like a stamp? Usually we'd use a scatter brush in photoshop or something to do that.
  • Eric Chadwick
    I would just go old school and render it out using a top down camera. I've done this loads in Max for the same reason, can also capture normal maps from the leaves & branches, either by using normalmap lighting or a shader.

    Based on your post history, you probably know all this already, but in case it helps someone else, http://wiki.polycount.com/wiki/Texture_Baking#Transparency
  • poopipe
    Offline / Send Message
    poopipe grand marshal polycounter
    That  would work but sadly what I didn't clarify was that the target 'plane' isn't necessarily a plane.

    I have a feeling this might be doable with Arnold as you dont need to go through transfer maps so ill check that later.
    Fwiw I'm pretty sure I've had this working in max through rtt - it might have been with mental ray rather than the scanline renderer though. 
  • Eric Chadwick
    Maybe you could convert the mesh from UV space into geom space, just for the camera-based bake.
  • poopipe
    Offline / Send Message
    poopipe grand marshal polycounter
    Great minds .. 
    I suspect that's where illI end up. 

    If anyone from simplygon  is reading this - Ill say nice things about you on the Internet if this makes it into v9
  • sprunghunt
    Offline / Send Message
    sprunghunt polycounter
    use something like xnormal or bake in substance designer
  • poopipe
    Offline / Send Message
    poopipe grand marshal polycounter
    Neither will bake alpha blended transparent maps to other meshes afaik.
  • Eric Chadwick
  • radiancef0rge
    Offline / Send Message
    radiancef0rge ngon master
    this is actually a legitimately hard problem to solve. try mental ray with deep occlusion if using maya, and a 'complete' map if using max. 
    we ended up writing our own tool to do this. 
  • RN
    Offline / Send Message
    RN sublime tool
    You don't need the baker to support alpha, you can use some component swizzling: bake a couple of passes (one opaque RGB pass, and one opaque alpha pass where you route the alpha to all RGB channels to get a grayscale version of it), and then composite it back in Photoshop by pasting the alpha pass as a mask for the RGB pass, giving it transparency again.
  • poopipe
    Offline / Send Message
    poopipe grand marshal polycounter
    As long as it's not just me :) 

    There's a couple of ideas to try here and it's not out of the question to write a tool if necessary. 


  • m4dcow
    Offline / Send Message
    m4dcow interpolator
    I remember doing this awhile back with mental ray and using some sort of surface shader trickery rendering to a square aspect ratio camera. Not to sure how to do it nowadays with Arnold though it seems easy to do the albedo pass with one of the built in AOVs but I couldn't figure out the ambient occlusion that takes into account the alpha, or the normal which needs the alpha and the normal map of the cards.

    Workflow wise I started modeling leaves out and just having straight up geometry to bake with UVs corresponding to some leaf texture, but this isn't practical for more detailed leaves.

    Now with megatextures and their atlases having proper normal/height/ao etc... building your own custom stuff using alpha card geometry and rendering that out makes sense.

    TLDR Did something similar back in the day, but can't figure out how to do it anymore, but bumping in case someone else knows how to do it.
  • poopipe
    Offline / Send Message
    poopipe grand marshal polycounter
    RN said:
    You don't need the baker to support alpha, you can use some component swizzling: bake a couple of passes (one opaque RGB pass, and one opaque alpha pass where you route the alpha to all RGB channels to get a grayscale version of it), and then composite it back in Photoshop by pasting the alpha pass as a mask for the RGB pass, giving it transparency again.



    I think you're missing the actual problem. I need the baker to be able to account for alpha in a texture that's applied to the source mesh. 

    The card leaves example is the simplest - you make some leaf textures with alpha,  arrange them into a tree shape and then bake that to a mesh that has a closed volume.  (imagine baking the bushes in fornite down to a sphere)

    if the baker doesn't understand alpha, then you don't get correct colour, shadowing, occlusion and so on because the upper layers of the source bmesh obscure the lower ones. 

    Like a number of others here Im confident I had this soved with mental ray a few  years ago but it's not an option now because you only get Arnold (which I'll look into) 


  • m4dcow
    Offline / Send Message
    m4dcow interpolator
    So I took a pass at it and haven't really figured it out, but I wanted to share a few discoveries anyway.
    I'm attempting this in Arnold since that is built into Maya these days.
    First thing you need to do is set the attributes of the meshes you want to use opacity, under the arnold section, uncheck opaque. If opaque is checked, it doesn't matter what settings or maps you have plugged into your graph if that opaque checkbox isn't unchecked opacity won't work.

    My graph network looks like this

    • With my graph, I created 2 aiStandardSurface shaders.
    • The first one I called AOMaterial and created an aiUtility node and an aiWriteColor node.
    • In the aiUtility node I set the shade mode to ambocc and color mode to color.
    • In the aiWriteColor node attributes I created a new name called AOAlpha. This creates a custom AOV which I'll touch on later.
    • I attached those graphs as shown and also connected a normal and an opacity texture
    • On the second material I connected the opacity and basecolor texture.
    First i apply my AOMaterial shader to my geometry, and for the render mode AOV in render settings I select "AOAlpha" which I had named before in the aiWriteColor node. This is sort of a custom render pass, Arnold has a few built in but none for ambient occlusion, and while there is an Arnold ambientOcclusion shader it had the opacity inputs removed "by design" in Maya 2018. If you hit render you should get an AO map with alpha enabled and extra detail since the normal map was plugged in.

    Next i set the "Mode" to N which is my normal map You would have to add it to the active AOV list before it shows up in the mode dropdown). When you render you should get a normal image, now I couldn't figure out how to get this working properly, I suspect it is rendering some sort of worldspace normal and also backfaces are affected too, I don't know enough to delve into fixing that so maybe someone else does.

    Next I assign my AlbedoMaterial shader to the meshes and render out using diffuse_albedo (there is a plain albedo but just ended up rendering white for me). When saving the file out I saved with color management embedded because without it the file came out really dark. I reckon one could use that same slot for their other maps like curvature that they might want to transfer over.


  • marks
    Offline / Send Message
    marks greentooth
    To quote a common software development axiom : "Why doesn't it just work?" 
    3D is so weird sometimes. This doesn't even work properly in max, at least the last version I tried when you RTT any partially transparent pixel gets rendered black
  • RN
    Offline / Send Message
    RN sublime tool
    poopipe said:
    if the baker doesn't understand alpha, then you don't get correct colour, shadowing, occlusion and so on because the upper layers of the source bmesh obscure the lower ones.
    I get it now, thanks for explaining.
    I can't say about Arnold as I don't use it, but with Blender Cycles it def seems to work, baking AO and shadows takes transparency into consideration if you use the texture alpha (or something else) as a mix factor to mix your opaque shader with a fully transparent shader:





    And the transparent diffuse:
  • poopipe
    Offline / Send Message
    poopipe grand marshal polycounter
    Yet another reason we should all ditch maya for blender :wink:


    Thanks so much for that m4dcow.  I'll give it a shot as soon as I get over my current problem.  
  • poopipe
    Offline / Send Message
    poopipe grand marshal polycounter
    I can confirm that this is working for me (now i've finally got back to looking at it)  so I wanted to say thanks. 

    I'm looking at ways to get the normals behaving correctly on an arbitrary mesh but I fear it might not be possible within maya's architecture as I can't see a way of extracting tangent information from the target mesh and passing that into the source mesh's shader.

    I'm not giving up just yet mind.. 
  • dreamee
    Offline / Send Message
    dreamee polycounter lvl 4
    Thank you m4dcow, it's very helpful for me, so i keep going on your discoveries to solve the Normal and the Backface problem.

    - For the Normal : We need to render with a Ortho Camera.
    1. Face Down the camera to match the World Space Axis with a plane's tangent normal (Camera set to Ortho)
    Then we change the Shade Mode in aiUtility to Flat , Color Mode to Normal (AOV still be our "AOAlpha"), and hit render we got this (the brown yellow one is the backface,we'll fix it later) , you need to turn the gamma off in the render window.

    In case you want the camera Face another Axis , for example : 

    Then we need to change the connection between aiUtility & aiWriteColor like this(out Red to in Blue;out Green to in Red;out Blue to in Green)

    - For the Backface : We use aiTwoSided node , Front input of the node we use the above Output Color , the Back input will be another Material's Output Color  with Normal Map input Inverted Z , the Graph like this :

    Then we hit Render and the Backface Normal's fixed :


    That's it . Hope this can help someone , and sorry about my English.
    Thanks m4dcow again.
  • gnoop
    Offline / Send Message
    gnoop sublime tool
    RN said:
    poopipe said:
    if the baker doesn't understand alpha, then you don't get correct colour, shadowing, occlusion and so on because the upper layers of the source bmesh obscure the lower ones.
    I get it now, thanks for explaining.
    I can't say about Arnold as I don't use it, but with Blender Cycles it def seems to work, baking AO and shadows takes transparency into consideration if you use the texture alpha (or something else) as a mix factor to mix your opaque shader with a fully transparent shader:

    Blender does bakes AO and shadows that way, everything that's on the target surface itself. But there is no way it could see beyond. So with "Selected to Active"  it doesn't work and thus it's pretty useless in Blender. Your empty normal pass for D part is showing it clearly.

    Arnold can bake alpha planes  to low res geometry, in MAx 2019 at least, using "complete map"  in RTT window as radiancef0rge mentioned .   It could only bake properly to a target low-res surface not intersecting those alpha "hires" source.  So you have to bake into rather "cage" mesh   while it doesn't support any cage ray direction functionality and does it always along normals anyway. Since it bakes  world space normals only it doesn't matter,  you can always convert the bake into tangent space in Xnormal.  

    The issue I still have no idea how to solve in Arnold  is  inverted normal map colors on backsides of alpha planes.    

    I use Clarisse. It has a special "normals" visualizing material that doesn't have that issue  while similar to Arnold with "no intersect" rule and everything else.
    Also Modo Indie can bake alpha planes without  issues and workarounds  at all

  • m4dcow
    Offline / Send Message
    m4dcow interpolator
    gnoop said:

    The issue I still have no idea how to solve in Arnold  is  inverted normal map colors on backsides of alpha planes.    

    I use Clarisse. It has a special "normals" visualizing material that doesn't have that issue  while similar to Arnold with "no intersect" rule and everything else.
    Also Modo Indie can bake alpha planes without  issues and workarounds  at all

    Isn't precisely that what @dreamee has solved with the 2 sided node?
  • gnoop
    Offline / Send Message
    gnoop sublime tool
    m4dcow said:
    Isn't precisely that what @dreamee has solved with the 2 sided node?
    Yeah, actually it works  in Max too  with two sided material.     Thanks for pointing it out for me.   I  just got lost in those AO materials and channels shuffles. 
       In Max it's  just Utility node that gives you  world normal colors initially   then  Matte material   instead of AO one where  you could input opacity alphas and finally same two sided material.

    So Armold could do it perfectly ok In MAx too

    ps, Too bad  Arnold in Max can't bake UDIM  like Clarisse and Modo
  • gnoop
    Offline / Send Message
    gnoop sublime tool
    Looks like Arnold has another problem although . Its baking rays go indefinitely and looks like there is no way to limit it as in Clarisse or Modo.

    Gpu mode seems  not working either in RTT baking.   I get  " UV camera is unsupported on GPU".   Or maybe it's my old GF980ti
  • poopipe
    Offline / Send Message
    poopipe grand marshal polycounter
    I'm not familiar with the arnold implementation in max but if it's anything like Maya it'll be massively half arsed and there'll be a load of functionality not exposed through the UI 
  • gnoop
    Offline / Send Message
    gnoop sublime tool
    Yeah,   it's like archeological excavations. There should a special discipline :  software studies ,   to dig into code "cultural layers"  of Autodesk soft  :)          
Sign In or Register to comment.