To me, assigning white materials with different IDs to the different piece (in the same way on the hp and on the lp), checking in "hit only matching material IDs", turning on light tracer works totally fine and i get 100% clean bake without exploding.I dont know if Xnormal supports this thing, but I just use max because with this option, it allows to bake without artifacts and its way faster than rendering from both meshes,plus dont need to transfer meshes between 2 programs.Later I can adjust the values in photoshop.This is my opinion, and worked perfectly in every time, every case.
I don't know about Max, but those AO errors could be more or less reduced/removed by using the anti-aliasing setting in xNormal. Maybe 3DS Max has something similar ?
I don't know about Max, but those AO errors could be more or less reduced/removed by using the anti-aliasing setting in xNormal. Maybe 3DS Max has something similar ?
Pretty sure I was using the mitchell-netravali AA filter in max. Perhaps there's a better filter or some setting I need to change.
Use super sampling, or render 2x and size down after the fact (same thing).
There are a couple things I typically do when baking lowpoly-only AO:
In Xnormal, I use the simple AO tool, this can be problematic though, as the vertex normals of your lowpoly mesh tend to contribute to the result. Usually I will tesselate(but not smooth) the lowpoly a couple times first and soften the vertex normals. This can give you some faceting on round shapes, so I usually make a copy of the low AO, guassian blur the hell out of it, and the use a layer mask to blend in the blurred part over the faceted areas. Its a bit of a pain in the ass and I still get some seam issues and the like sometimes.
Lately I've been using knald to generate low AO (load a blue(127,127,255) colored normal map so you don't get the NM-based AO too), which seams to be a bit higher quality, but vertex normals do tend to have some affect as well, so you might want to do the blurring thing again.
But I'll say it again, when you view this from the intended distance/angle, and especially when the asset is textured, these issues will be much less apparent if visible at all.
PS: I think max uses a white color for totally occluded areas in the AO, I'm not sure if you can avoid this, maybe force double sided materials would fix it. This tends to be problematic as it will give you white highlights around intersecting areas with lower resolution. Though merging your lowpoly as much as possible vs having many intersecting mesh elements will help there.
Use super sampling, or render 2x and size down after the fact (same thing).
There are a couple things I typically do when baking lowpoly-only AO:
In Xnormal, I use the simple AO tool, this can be problematic though, as the vertex normals of your lowpoly mesh tend to contribute to the result. Usually I will tesselate(but not smooth) the lowpoly a couple times first and soften the vertex normals. This can give you some faceting on round shapes, so I usually make a copy of the low AO, guassian blur the hell out of it, and the use a layer mask to blend in the blurred part over the faceted areas. Its a bit of a pain in the ass and I still get some seam issues and the like sometimes.
Lately I've been using knald to generate low AO (load a blue(127,127,255) colored normal map so you don't get the NM-based AO too), which seams to be a bit higher quality, but vertex normals do tend to have some affect as well, so you might want to do the blurring thing again.
But I'll say it again, when you view this from the intended distance/angle, and especially when the asset is textured, these issues will be much less apparent if visible at all.
PS: I think max uses a white color for totally occluded areas in the AO, I'm not sure if you can avoid this, maybe force double sided materials would fix it. This tends to be problematic as it will give you white highlights around intersecting areas with lower resolution. Though merging your lowpoly as much as possible vs having many intersecting mesh elements will help there.
Awesome, some really helpful stuff here. I'll try it all. I feel like 100% occlusion thing might be my problem. I'll give each of the workflows you described and see which suits me.
Again thanks for the help everyone. This seems to have turned into a "help ghaz get a better bake" thread, and yet everyone is so willing to help. Such a great community.
But I'll say it again, when you view this from the intended distance/angle, and especially when the asset is textured, these issues will be much less apparent if visible at all.
This is true, I did throw on a basic green texture with the AO overlayed with ~65% opacity and it was much less noticeable. However in some places it did still stand out, and I'm happy to learn some techniques that I can incorporate into my workflow to get a better result.
Still having issues with the low poly AO (and it is quite obvious on the model). I tried baking it in max after subdividing by smoothing group (after smoothing the model with a very low auto-smooth angle). This actually seemed to make the result worse.
The problem is that the AO bake makes the model look like it has shading errors, I guess due to the low poly normals. This is fixed by using a properly smoothed low poly model, but this brings the problem of causing seams as before.
tried baking it in max after subdividing by smoothing group (after smoothing the model with a very low auto-smooth angle). This actually seemed to make the result worse.
Re-read Earthquake's last post. He tesselates the model without smoothing it, then softens (resets) the vertex normals. You can do this with the Tessellate modifier, setting the Tension to zero, and the Iterations to 2 or 3. Then add an Edit Normals modifier, select all, and press Unify (I think, don't have Max in front of me here). Bake the AO, then delete the modifiers.
Re-read Earthquake's last post. He tesselates the model without smoothing it, then softens (resets) the vertex normals. You can do this with the Tessellate modifier, setting the Tension to zero, and the Iterations to 2 or 3. Then add an Edit Normals modifier, select all, and press Unify (I think, don't have Max in front of me here). Bake the AO, then delete the modifiers.
Yep, that sounds about right.
Also, try Knald for low AO (load a flat 127,127,255 blue normal map and your lowpoly obj).
Ah thanks guys, didn't know that was the way to do it. I'll give it a try. I was using knald but loading a 4K map so it outputs a 4K AO doesn't seem to load a large amount of the time. Also it's not fond of loading my tessellated model.
@kinetic No idea dude, never heard of that. I do this all over my model to triangulate it and there aren't issues elsewhere.
Are you baking normals and AO in two different packages? I had a similar issue when i did normals in max, And AO in xnormal. My problem was using a differen't cage in Xnormal than in max. Thinking it didn't affect how AO baked much. But it affected it a fair amount. My issues weren't quite this apparent though. Have you tried baking the AO in an alternative app, Just for the sake of know if its the model/cage or if its the renderer/render settings?
I'm baking my normal map in xnormal, and I've tried baking the AO in both knald and max. I would try baking with xnormal's simple AO tool but I get this error every time (with the corresponding file type for the error).
I'm also not using a cage because I'm just baking a low poly AO here, there is no need for a cage. There was no normal map on the tessellated model (minus the flat one needed for baking in knald). This is with 2 levels of tessellation, a third level won't load into knald (over a million polys if I recall). Knald is giving me a much better result than max though.
If you do the tesselated low bake thing, you need to make a copy of your bake in photoshop, gaussian blur it a bit, and then make a mask to blend the blurred areas over the faceted artifacts. Sounds like a pain but it only takes a few minutes.
An AO map is just a simple grayscale image, you can paint over it a bit if you need to.
If you do the tesselated low bake thing, you need to make a copy of your bake in photoshop, gaussian blur it a bit, and then make a mask to blend the blurred areas over the faceted artifacts. Sounds like a pain but it only takes a few minutes.
An AO map is just a simple grayscale image, you can paint over it a bit if you need to.
but... I'm afraid In the past painting over AO has never ended well for me, but I'll give it a try. Thanks for being patient with me everyone!
Painting it out seems to be working quite nicely, other than it the blurring causes seams to appear. Seems like it will require me to carefully paint near edges to try and avoid this happening (if possible). I wonder if something like mudbox would allow me to blur it as a whole object to eliminate this...
EDIT: Using mudboxing to paint away the seams is working nicely!
Use super sampling, or render 2x and size down after the fact (same thing).
There are a couple things I typically do when baking lowpoly-only AO:
In Xnormal, I use the simple AO tool, this can be problematic though, as the vertex normals of your lowpoly mesh tend to contribute to the result. Usually I will tesselate(but not smooth) the lowpoly a couple times first and soften the vertex normals. This can give you some faceting on round shapes, so I usually make a copy of the low AO, guassian blur the hell out of it, and the use a layer mask to blend in the blurred part over the faceted areas. Its a bit of a pain in the ass and I still get some seam issues and the like sometimes.
Lately I've been using knald to generate low AO (load a blue(127,127,255) colored normal map so you don't get the NM-based AO too), which seams to be a bit higher quality, but vertex normals do tend to have some affect as well, so you might want to do the blurring thing again.
But I'll say it again, when you view this from the intended distance/angle, and especially when the asset is textured, these issues will be much less apparent if visible at all.
PS: I think max uses a white color for totally occluded areas in the AO, I'm not sure if you can avoid this, maybe force double sided materials would fix it. This tends to be problematic as it will give you white highlights around intersecting areas with lower resolution. Though merging your lowpoly as much as possible vs having many intersecting mesh elements will help there.
The painting thing is semi working. The blurring is causing seams because it blurs away from UV edges. Quite a few areas on my mesh have the faceting issue. It seems easier to just bake the standard low poly mesh in knald and paint out the seams due to the hard smoothing. This isn't really an ideal way though to be honest.
And yeah, I know I could be seen as taking this too far as I've been told that it won't show up on my final model. But people noticed the shading issues when looking at my model. And when you have a nice rounded edge thanks to your normal map, a sudden change in AO really detracts from it.
I found out xnormal had been updated which fixed the simple AO generator. I was going to try baking AO to my high poly's vertex colours and then bake that to my low poly. In theory this would allow me to bake an unexploded HP AO. Xnormal doesn't seem to be baking anything to the vertex colours though (checked this in xnormal, max and tried baking it to my lowpoly).
Just did a quick test, can't promise it will work, it looks like it should, but can't say for certain.
2 boxes, put in 1 smoothing group, and set to unlit/flat so only the AO shows .
notice how with scanline there are these shadows on the edges wich is caused by the smoothing (that seems logical, maybe it's something else)(don't mind the seams, must have forgot padding or something )
if it isn't obvious, save the image and zoom out.
while in the ao with mental ray everything is nice and white.
Only downside to MR is that to get a nice bake (that isn't grainy) you will really have to up all of the relevant settings, and the render is probably gonna take a while.
(if you haven't done it, when adding a map you want the "Ambient occlusion (MR) one.
so maybe have a go at baking ao with mental ray and see if that's better.
Thanks for taking the time! It's a good idea but alas, there issue is still there (and doesn't produce a good bake either, really splotchy and almost random unlike knald)
I managed to get the vertex colour method working. Seems to be producing good results. It's not perfect but I will composite it with a knald bake to fix the errors. Will post it when I'm done.
Replies
Well, yeah I guess. Just wondered if there was something I could do to avoid it to save myself some time.
Pretty sure I was using the mitchell-netravali AA filter in max. Perhaps there's a better filter or some setting I need to change.
There are a couple things I typically do when baking lowpoly-only AO:
In Xnormal, I use the simple AO tool, this can be problematic though, as the vertex normals of your lowpoly mesh tend to contribute to the result. Usually I will tesselate(but not smooth) the lowpoly a couple times first and soften the vertex normals. This can give you some faceting on round shapes, so I usually make a copy of the low AO, guassian blur the hell out of it, and the use a layer mask to blend in the blurred part over the faceted areas. Its a bit of a pain in the ass and I still get some seam issues and the like sometimes.
Lately I've been using knald to generate low AO (load a blue(127,127,255) colored normal map so you don't get the NM-based AO too), which seams to be a bit higher quality, but vertex normals do tend to have some affect as well, so you might want to do the blurring thing again.
But I'll say it again, when you view this from the intended distance/angle, and especially when the asset is textured, these issues will be much less apparent if visible at all.
PS: I think max uses a white color for totally occluded areas in the AO, I'm not sure if you can avoid this, maybe force double sided materials would fix it. This tends to be problematic as it will give you white highlights around intersecting areas with lower resolution. Though merging your lowpoly as much as possible vs having many intersecting mesh elements will help there.
Again thanks for the help everyone. This seems to have turned into a "help ghaz get a better bake" thread, and yet everyone is so willing to help. Such a great community.
This is true, I did throw on a basic green texture with the AO overlayed with ~65% opacity and it was much less noticeable. However in some places it did still stand out, and I'm happy to learn some techniques that I can incorporate into my workflow to get a better result.
The problem is that the AO bake makes the model look like it has shading errors, I guess due to the low poly normals. This is fixed by using a properly smoothed low poly model, but this brings the problem of causing seams as before.
Anybody have any advice for me?
Re-read Earthquake's last post. He tesselates the model without smoothing it, then softens (resets) the vertex normals. You can do this with the Tessellate modifier, setting the Tension to zero, and the Iterations to 2 or 3. Then add an Edit Normals modifier, select all, and press Unify (I think, don't have Max in front of me here). Bake the AO, then delete the modifiers.
Yep, that sounds about right.
Also, try Knald for low AO (load a flat 127,127,255 blue normal map and your lowpoly obj).
@kinetic No idea dude, never heard of that. I do this all over my model to triangulate it and there aren't issues elsewhere.
I'm also not using a cage because I'm just baking a low poly AO here, there is no need for a cage. There was no normal map on the tessellated model (minus the flat one needed for baking in knald). This is with 2 levels of tessellation, a third level won't load into knald (over a million polys if I recall). Knald is giving me a much better result than max though.
An AO map is just a simple grayscale image, you can paint over it a bit if you need to.
but... I'm afraid
EDIT: Using mudboxing to paint away the seams is working nicely!
Thats a nice tip. Thank you!
And yeah, I know I could be seen as taking this too far as I've been told that it won't show up on my final model. But people noticed the shading issues when looking at my model. And when you have a nice rounded edge thanks to your normal map, a sudden change in AO really detracts from it.
I found out xnormal had been updated which fixed the simple AO generator. I was going to try baking AO to my high poly's vertex colours and then bake that to my low poly. In theory this would allow me to bake an unexploded HP AO. Xnormal doesn't seem to be baking anything to the vertex colours though (checked this in xnormal, max and tried baking it to my lowpoly).
Sigh, I just want a nice looking AO map
2 boxes, put in 1 smoothing group, and set to unlit/flat so only the AO shows .
notice how with scanline there are these shadows on the edges wich is caused by the smoothing (that seems logical, maybe it's something else)(don't mind the seams, must have forgot padding or something
if it isn't obvious, save the image and zoom out.
while in the ao with mental ray everything is nice and white.
Only downside to MR is that to get a nice bake (that isn't grainy) you will really have to up all of the relevant settings, and the render is probably gonna take a while.
(if you haven't done it, when adding a map you want the "Ambient occlusion (MR) one.
so maybe have a go at baking ao with mental ray and see if that's better.