Home Technical Talk

High to low ao bake maya 2009 pixel errors.

polycount sponsor
Offline / Send Message
malcolm polycount sponsor
Okay so now I'm trying to bake the high res ao down to the low res. Everything is working fine except I am getting these one pixel white and black dots all over the model. I've tried upping the rays and the resolution of the texture but it doesn't fix the problem. It almost seems like the transfer maps doesn't like the models slightly clipping into each other. I've tried setting all the relevant visability tags on the low res mesh just in case but no luck. Sorry for the low res bake in the screenshot but this stuff is painfully slow to troubleshoot and rebake.

aodots.jpg

Replies

  • malcolm
    Offline / Send Message
    malcolm polycount sponsor
    Okay I've confirmed it, the low res intersecting with the high res is causing the problem. WTF, who wrote this tool it's awful. It appears the envelope doesn't do anything when occlusion transfer is concerned. The only fix I can think of is to dupe the geo and move all the verts out where the intersections are. I tried transform component thinking I had out smarted it but making the model bigger all over where it is not needed messes up the occlusion transfer. Is there any way to apply the normal map to my low res and use the low res geo plus the normal map to bake the ao map?
  • Marine
    Offline / Send Message
    Marine polycounter lvl 19
    why not bake your ao in xnormal?
  • malcolm
    Offline / Send Message
    malcolm polycount sponsor
    Was trying to do it all with as few apps as possible, but will definitely go that route if I can't get this working.
  • Mark Dygert
    With the low poly selected go down the stack to element mode and hide all, then with it still selected go back up to the projection modifier and bake.
  • slipsius
    vig... you`re thinking max, he's using maya
  • Mark Dygert
    oh yea didn't pick up on that, use xNormal.
  • bugo
    Offline / Send Message
    bugo polycounter lvl 17
    mental ray for maya baking results are not the best counting on speed, its true, AO is very powerful and very well customized, but heavy, so if you want something with quality, you need time and lots of rays. Depending on what you want to do, I would consider baking the AO mesh in the high, and transfering the color to the Low. But that's 2x uv work (high and low). So, yeah, or Turtle or Xnormal. I would say its faster going with Turtle, but Xnormal would give you a better result.

    (Btw Malcolm, I see you posting tons of questions about Maya lately, but into your portfolio I see great environment modeling, were you using Max before?)
  • ErichWK
    Offline / Send Message
    ErichWK polycounter lvl 12
    Vig wrote: »
    oh yea didn't pick up on that, use xNormal.

    That made me laugh out loud real hard.


    Yes. Use Xnormals. It works too well not to use it!
  • malcolm
    Offline / Send Message
    malcolm polycount sponsor
    bugo no not at all I'm a maya guy. Haven't had the opportunity to work on a game where we've baked anything from high to low so I'm doing a project at home for fun.. I was really hoping the maya workflow would be working at this point but it appears autodesk hasn't fixed shit when it comes to their transfer maps tool since the last time I messed around with it.
  • Lamont
    Offline / Send Message
    Lamont polycounter lvl 15
    I don't even mess with baking anything like normals/AO in Maya anymore: Speed, stability and quality. If I need to go faster, I just open up FaoGen.
  • malcolm
    Offline / Send Message
    malcolm polycount sponsor
    I just opened xnormal. Holy fuck, this thing is so overwelming and the help file doesn't actually tell you shit except how complicated and how many billions of settings there are. Still trying to figure out where the bake ao button is.
  • ZacD
    Offline / Send Message
    ZacD ngon master
    I think xnormal needs to be set up and simple like crazy bump
  • malcolm
    Offline / Send Message
    malcolm polycount sponsor
    Cool so I found a nice step by step flash tutorial only to find out xnormal doesn't actually do raytrace ao it's one of those gpu based ao things and even worse it generates vertex ao and bakes that to a texture. It says you can bake to a texture but you need to unwrap your high res mesh. Not what I was looking for really. I wish this maya shit just worked. So yeah still no solution for this workflow out of maya.
  • Xoliul
    Offline / Send Message
    Xoliul polycounter lvl 16
    What? Since when does Xnormal require highres meshes to be unwrapped ? I never used AO baking in Xnormal since Max works easy and fast, but I imagine Santy wouldn't put that sort of things into Xnormal...
  • Lamont
    Offline / Send Message
    Lamont polycounter lvl 15
    malcolm wrote: »
    Cool so I found a nice step by step flash tutorial only to find out xnormal doesn't actually do raytrace ao it's one of those gpu based ao things and even worse it generates vertex ao and bakes that to a texture.
    I think that's just for the cage option.
    malcolm wrote: »
    It says you can bake to a texture but you need to unwrap your high res mesh.
    No you don't.
    1. Add High res.
    2. Add low res.
    3. If needed, use the tool to calculate max front and rear ray trace. Then go back to step two and adjust those numbers on the low res.
    4. Select the output you want (AO, normal, RAd Norm, Depth) and size.
    5. Hit the button.
  • malcolm
    Offline / Send Message
    malcolm polycount sponsor
    Yeah sorry it's the simple ao tool which requires uv's on the high res mesh. So, I baked my ao and besides all the errors in it is there any option to increase the spread of the ao?
  • Marine
    Offline / Send Message
    Marine polycounter lvl 19
    click the ... icon next to ao in the maps box
    this http://donaldphan.com/tutorials/xnormal/xnormal_occ.html is helpful for setting it up too, bit out of date, but the majority is good stuff
  • malcolm
    Offline / Send Message
    malcolm polycount sponsor
    Thanks I'll take a look at that and see if I can't get the quality I'm after.
  • EarthQuake
    Anytime i need to bake normals in maya, i have to use XN for ao. This is pretty quick to do as the XN UI is very simple once you understand it, just load the high into the high field, low into low, click the correct map and hit render =D There are of course extra settings here, but that is the basics.

    One thing to note with XN, if you have any hard edges on your mesh from maya, make sure you set "smooth normals" to "Average normals"(or smooth all edges in maya before exporting obj). This will ensure XN is using the same averaged projection for ao, otherwise you will get stuff that doesn't line up correctly and seams/missed detail on edges.

    From what i've been told you cant bake high->low in maya, you have to uv your high, bake AO onto the high, then use transfer maps to transfer that texture as a diffuse map..... I'm surprised you managed to get it to work as well as you have in your shot!

    [edit] oh also, pop into task manager, right click on xnormal in the processes list and do set priortiy -> below normal, or else your AO rendering may cause your entire system to run pretty slow(then you can do other work while its baking etc)
  • EarthQuake
    ErichWK wrote: »
    That made me laugh out loud real hard.


    Yes. Use Xnormals. It works too well not to use it!

    Not sure if this is sarcasm or what? Usually i dont care for the "Just use max lol!" type responses but in this case it is actually helpful/a real solution.
  • malcolm
    Offline / Send Message
    malcolm polycount sponsor
    Hey EarthQuake I've been giving xnormal a try for the last couple hours couldn't come up with anything good. I set number of rays to 1024 and spread to 179 and I got a pretty low res ao bake. Am I missing somthing? Specifically I did not see any detail on any of the rebar. This is the render out of maya ideally I'd like to bake this to the low res.

    aobakeiwant.jpg
  • pior
    Offline / Send Message
    pior grand marshal polycounter
    From what i've been told you cant bake high->low in maya, you have to uv your high, bake AO onto the high, then use transfer maps to transfer that texture as a diffuse map..... I'm surprised you managed to get it to work as well as you have in your shot!

    Hmm just for clarification : that's incorrect, Maya can bake high to low without UVs on the high, there is a render-to-texture equivalent in there. The interface itself is actually pretty nice. Still a pain to use anyways hehe.

    There is something possible to : baking AO on the high at the vertex level, then baking that. Still nothing as good as XnNormal anyways.
  • malcolm
    Offline / Send Message
    malcolm polycount sponsor
    pior already tried baking vert ao as a test for quality maya crashed there's too many verts in my model.
  • EarthQuake
    pior wrote: »
    Hmm just for clarification : that's incorrect, Maya can bake high to low without UVs on the high, there is a render-to-texture equivalent in there. The interface itself is actually pretty nice. Still a pain to use anyways hehe.

    There is something possible to : baking AO on the high at the vertex level, then baking that. Still nothing as good as XnNormal anyways.

    The functionality/UI is there of course, but actually being able to *do it* is another story, as seen by malcom's first example and numerous other posts on same subject. I would LOVE to know how to do it correctly, and by that i mean without any errors from the high/low intersecting or whatever it is that causes all these artifacts. Everyone who i've talked to who knows anything about maya pretty much tells me there is no way to simply bake AO high to low in maya without extra tools(turtle etc).
    malcolm wrote: »
    Hey EarthQuake I've been giving xnormal a try for the last couple hours couldn't come up with anything good. I set number of rays to 1024 and spread to 179 and I got a pretty low res ao bake. Am I missing somthing? Specifically I did not see any detail on any of the rebar. This is the render out of maya ideally I'd like to bake this to the low res.

    aobakeiwant.jpg

    This actually looks like a pretty good AO bake to me and exactly what you should expect to see from plain ao.... I think we've had this issue before, but you're doing the broken pillar eat3d tutorial yes? The example "AO" in that tutorial isnt a simple AO bake, but something more complex, i think he uses a ground/ceiling plane or something like that.... We had a thread about this a while ago and i think that was the conclusion, a guy was having similar problems because he couldn't reproduce that example, but in reality no app will give you that result without doing something extra.
  • willy-wilson
    Offline / Send Message
    willy-wilson polycounter lvl 8
    are you useing the simple ao?
  • malcolm
    Offline / Send Message
    malcolm polycount sponsor
    No I'm using the regular ao.
  • malcolm
    Offline / Send Message
    malcolm polycount sponsor
    Ha ha! Fuck, finally! Almost got something working. This is directly out of maya transfer maps tool high to low ao bake. A couple errors to figure out and I should actually be able to use maya to complete this tutorial I'm doing.

    aoalmostworking.jpg
  • EarthQuake
    Cool! can you give an overview on what you had to do to make this actually work?
  • malcolm
    Offline / Send Message
    malcolm polycount sponsor
    Sure thing I just need to figure out why I'm getting a couple of errors and see if I can fix them.
  • bugo
    Offline / Send Message
    bugo polycounter lvl 17
    Right, Malcolm, true, Maya is not easy at first eyes for baking, you might will get pissed sometimes, but I tell you MentalRay has one of the best AO I've ever seen. But if you want easy solution and still wants inside Maya, Turtle is the right path.
  • malcolm
    Offline / Send Message
    malcolm polycount sponsor
    Yo EarthQuake, here's the settings I've used and a brief description of what you have to do to get it working. Hopefully this saves someone some time, I lost 2 days of my life to this and totally stalled my mudbox tutorial in the process.

    aobakesettings.jpg
  • Whargoul
    Offline / Send Message
    Whargoul polycounter lvl 18
    I try to get people to always bake out an AO map for the high rez. Why? You do it once and it's always available for later, with colour transfer. You spend your time making one high quality bake and get the errors out of the way. You spend your time on the transfer, and just deal with transfer problems, rather than transfer & AO problems at the same time. Now if you reuse the high rez for other parts, it comes with AO on it. Yes, it can be a pain to UV it - but they can be pretty shitty. If you know ahead of time it's not hard to auto-uv and layout the maps. Sometimes even a full on autoUV is good enough (make sure you do it on the proxy geo, not the subdivided geo though!).
  • EarthQuake
    You know i just realized you're using 2009 here and not 2008, which i believe was the source of my problems. Sadly not something i can fix.
  • malcolm
    Offline / Send Message
    malcolm polycount sponsor
    Cody before going through all this pain I tried your suggestion of the colour transfer technique but my model is too high poly to unwrap in maya, crashes as soon as I touch it. I did manage to get an auto unwrap on it, baked and tranferred the diffuse to the low res. Looked like shit seams everywhere not usable quality at all. In the future I'd definately go with uv's on the high res before it gets into mudbox but I'm following along with a tutorial that didn't do it that way. Trial and error just like everything else in 3d. In theory I could save a displacement map from mud and unwrap the original base mesh then apply the displacement map to get uv's on it. I think I've gone to far at this point and lots of little tweaks have been done to the model so I'd probably have to do a lot of tweaking, I'll look into it if this method still fails I just noticed my latest bake has issues in it.
Sign In or Register to comment.