Okay so now I'm trying to bake the high res ao down to the low res. Everything is working fine except I am getting these one pixel white and black dots all over the model. I've tried upping the rays and the resolution of the texture but it doesn't fix the problem. It almost seems like the transfer maps doesn't like the models slightly clipping into each other. I've tried setting all the relevant visability tags on the low res mesh just in case but no luck. Sorry for the low res bake in the screenshot but this stuff is painfully slow to troubleshoot and rebake.
Okay I've confirmed it, the low res intersecting with the high res is causing the problem. WTF, who wrote this tool it's awful. It appears the envelope doesn't do anything when occlusion transfer is concerned. The only fix I can think of is to dupe the geo and move all the verts out where the intersections are. I tried transform component thinking I had out smarted it but making the model bigger all over where it is not needed messes up the occlusion transfer. Is there any way to apply the normal map to my low res and use the low res geo plus the normal map to bake the ao map?
mental ray for maya baking results are not the best counting on speed, its true, AO is very powerful and very well customized, but heavy, so if you want something with quality, you need time and lots of rays. Depending on what you want to do, I would consider baking the AO mesh in the high, and transfering the color to the Low. But that's 2x uv work (high and low). So, yeah, or Turtle or Xnormal. I would say its faster going with Turtle, but Xnormal would give you a better result.
(Btw Malcolm, I see you posting tons of questions about Maya lately, but into your portfolio I see great environment modeling, were you using Max before?)
bugo no not at all I'm a maya guy. Haven't had the opportunity to work on a game where we've baked anything from high to low so I'm doing a project at home for fun.. I was really hoping the maya workflow would be working at this point but it appears autodesk hasn't fixed shit when it comes to their transfer maps tool since the last time I messed around with it.
I just opened xnormal. Holy fuck, this thing is so overwelming and the help file doesn't actually tell you shit except how complicated and how many billions of settings there are. Still trying to figure out where the bake ao button is.
Cool so I found a nice step by step flash tutorial only to find out xnormal doesn't actually do raytrace ao it's one of those gpu based ao things and even worse it generates vertex ao and bakes that to a texture. It says you can bake to a texture but you need to unwrap your high res mesh. Not what I was looking for really. I wish this maya shit just worked. So yeah still no solution for this workflow out of maya.
What? Since when does Xnormal require highres meshes to be unwrapped ? I never used AO baking in Xnormal since Max works easy and fast, but I imagine Santy wouldn't put that sort of things into Xnormal...
Cool so I found a nice step by step flash tutorial only to find out xnormal doesn't actually do raytrace ao it's one of those gpu based ao things and even worse it generates vertex ao and bakes that to a texture.
Yeah sorry it's the simple ao tool which requires uv's on the high res mesh. So, I baked my ao and besides all the errors in it is there any option to increase the spread of the ao?
Anytime i need to bake normals in maya, i have to use XN for ao. This is pretty quick to do as the XN UI is very simple once you understand it, just load the high into the high field, low into low, click the correct map and hit render =D There are of course extra settings here, but that is the basics.
One thing to note with XN, if you have any hard edges on your mesh from maya, make sure you set "smooth normals" to "Average normals"(or smooth all edges in maya before exporting obj). This will ensure XN is using the same averaged projection for ao, otherwise you will get stuff that doesn't line up correctly and seams/missed detail on edges.
From what i've been told you cant bake high->low in maya, you have to uv your high, bake AO onto the high, then use transfer maps to transfer that texture as a diffuse map..... I'm surprised you managed to get it to work as well as you have in your shot!
[edit] oh also, pop into task manager, right click on xnormal in the processes list and do set priortiy -> below normal, or else your AO rendering may cause your entire system to run pretty slow(then you can do other work while its baking etc)
Yes. Use Xnormals. It works too well not to use it!
Not sure if this is sarcasm or what? Usually i dont care for the "Just use max lol!" type responses but in this case it is actually helpful/a real solution.
Hey EarthQuake I've been giving xnormal a try for the last couple hours couldn't come up with anything good. I set number of rays to 1024 and spread to 179 and I got a pretty low res ao bake. Am I missing somthing? Specifically I did not see any detail on any of the rebar. This is the render out of maya ideally I'd like to bake this to the low res.
From what i've been told you cant bake high->low in maya, you have to uv your high, bake AO onto the high, then use transfer maps to transfer that texture as a diffuse map..... I'm surprised you managed to get it to work as well as you have in your shot!
Hmm just for clarification : that's incorrect, Maya can bake high to low without UVs on the high, there is a render-to-texture equivalent in there. The interface itself is actually pretty nice. Still a pain to use anyways hehe.
There is something possible to : baking AO on the high at the vertex level, then baking that. Still nothing as good as XnNormal anyways.
Hmm just for clarification : that's incorrect, Maya can bake high to low without UVs on the high, there is a render-to-texture equivalent in there. The interface itself is actually pretty nice. Still a pain to use anyways hehe.
There is something possible to : baking AO on the high at the vertex level, then baking that. Still nothing as good as XnNormal anyways.
The functionality/UI is there of course, but actually being able to *do it* is another story, as seen by malcom's first example and numerous other posts on same subject. I would LOVE to know how to do it correctly, and by that i mean without any errors from the high/low intersecting or whatever it is that causes all these artifacts. Everyone who i've talked to who knows anything about maya pretty much tells me there is no way to simply bake AO high to low in maya without extra tools(turtle etc).
Hey EarthQuake I've been giving xnormal a try for the last couple hours couldn't come up with anything good. I set number of rays to 1024 and spread to 179 and I got a pretty low res ao bake. Am I missing somthing? Specifically I did not see any detail on any of the rebar. This is the render out of maya ideally I'd like to bake this to the low res.
This actually looks like a pretty good AO bake to me and exactly what you should expect to see from plain ao.... I think we've had this issue before, but you're doing the broken pillar eat3d tutorial yes? The example "AO" in that tutorial isnt a simple AO bake, but something more complex, i think he uses a ground/ceiling plane or something like that.... We had a thread about this a while ago and i think that was the conclusion, a guy was having similar problems because he couldn't reproduce that example, but in reality no app will give you that result without doing something extra.
Ha ha! Fuck, finally! Almost got something working. This is directly out of maya transfer maps tool high to low ao bake. A couple errors to figure out and I should actually be able to use maya to complete this tutorial I'm doing.
Right, Malcolm, true, Maya is not easy at first eyes for baking, you might will get pissed sometimes, but I tell you MentalRay has one of the best AO I've ever seen. But if you want easy solution and still wants inside Maya, Turtle is the right path.
Yo EarthQuake, here's the settings I've used and a brief description of what you have to do to get it working. Hopefully this saves someone some time, I lost 2 days of my life to this and totally stalled my mudbox tutorial in the process.
I try to get people to always bake out an AO map for the high rez. Why? You do it once and it's always available for later, with colour transfer. You spend your time making one high quality bake and get the errors out of the way. You spend your time on the transfer, and just deal with transfer problems, rather than transfer & AO problems at the same time. Now if you reuse the high rez for other parts, it comes with AO on it. Yes, it can be a pain to UV it - but they can be pretty shitty. If you know ahead of time it's not hard to auto-uv and layout the maps. Sometimes even a full on autoUV is good enough (make sure you do it on the proxy geo, not the subdivided geo though!).
Cody before going through all this pain I tried your suggestion of the colour transfer technique but my model is too high poly to unwrap in maya, crashes as soon as I touch it. I did manage to get an auto unwrap on it, baked and tranferred the diffuse to the low res. Looked like shit seams everywhere not usable quality at all. In the future I'd definately go with uv's on the high res before it gets into mudbox but I'm following along with a tutorial that didn't do it that way. Trial and error just like everything else in 3d. In theory I could save a displacement map from mud and unwrap the original base mesh then apply the displacement map to get uv's on it. I think I've gone to far at this point and lots of little tweaks have been done to the model so I'd probably have to do a lot of tweaking, I'll look into it if this method still fails I just noticed my latest bake has issues in it.
Replies
(Btw Malcolm, I see you posting tons of questions about Maya lately, but into your portfolio I see great environment modeling, were you using Max before?)
That made me laugh out loud real hard.
Yes. Use Xnormals. It works too well not to use it!
this http://donaldphan.com/tutorials/xnormal/xnormal_occ.html is helpful for setting it up too, bit out of date, but the majority is good stuff
One thing to note with XN, if you have any hard edges on your mesh from maya, make sure you set "smooth normals" to "Average normals"(or smooth all edges in maya before exporting obj). This will ensure XN is using the same averaged projection for ao, otherwise you will get stuff that doesn't line up correctly and seams/missed detail on edges.
From what i've been told you cant bake high->low in maya, you have to uv your high, bake AO onto the high, then use transfer maps to transfer that texture as a diffuse map..... I'm surprised you managed to get it to work as well as you have in your shot!
[edit] oh also, pop into task manager, right click on xnormal in the processes list and do set priortiy -> below normal, or else your AO rendering may cause your entire system to run pretty slow(then you can do other work while its baking etc)
Not sure if this is sarcasm or what? Usually i dont care for the "Just use max lol!" type responses but in this case it is actually helpful/a real solution.
Hmm just for clarification : that's incorrect, Maya can bake high to low without UVs on the high, there is a render-to-texture equivalent in there. The interface itself is actually pretty nice. Still a pain to use anyways hehe.
There is something possible to : baking AO on the high at the vertex level, then baking that. Still nothing as good as XnNormal anyways.
The functionality/UI is there of course, but actually being able to *do it* is another story, as seen by malcom's first example and numerous other posts on same subject. I would LOVE to know how to do it correctly, and by that i mean without any errors from the high/low intersecting or whatever it is that causes all these artifacts. Everyone who i've talked to who knows anything about maya pretty much tells me there is no way to simply bake AO high to low in maya without extra tools(turtle etc).
This actually looks like a pretty good AO bake to me and exactly what you should expect to see from plain ao.... I think we've had this issue before, but you're doing the broken pillar eat3d tutorial yes? The example "AO" in that tutorial isnt a simple AO bake, but something more complex, i think he uses a ground/ceiling plane or something like that.... We had a thread about this a while ago and i think that was the conclusion, a guy was having similar problems because he couldn't reproduce that example, but in reality no app will give you that result without doing something extra.