Vertex Lighting Bake - Thoughts?

Murdoc

Alright, I'm coming off using Unreal 3 for a while now where it didn't have any vertex lighting, so I never really had to deal with it before; just had to worry about the crappy way U3 imports the normals(or mayas way, can't remember which)

Anyway, new game, new engine, new software(max) and they want to stick a AO bake on all the models which is driving me up the wall. It really adjusts the way it looks in game and my modeling style is not working with this stuff because once applied I get big dark spots and crap all over the place.

Just wondering what are some of your thoughts on it? Is it really necessary for next gen tech, I assume the shader is working on a higher level at this point baking this stuff in is redundant, not to mention the eventual post effects like AO will kind of cover all this stuff without giving me a headache.

Maybe I'm too rigid, I just feel it's unnecessary at this point.

Eric Chadwick

Depends on the engine. Vertex color is very cheap rendering-wise, so it's an easy feature to implement. It's a little tougher on the artist, you have to subdivide your meshes so there are enough vertices in the right places, but it really isn't too hard. Max lets you easily bake lighting into vertex color (check out the Vertex Paint modifier).

Murdoc

Yeah I know its easy to do, but the way you model is now different (as you mentioned subdividing)

Not to mention as I see it as interfering with the lighting already going on; I'm just wondering what are teh advantages here? It's cheap? but it's cheap tha tlooks pretty crappy(at least in my case)

Eric Chadwick

Resistance Fall of Man had a cool implementation of vertex color.
http://www.cybergooch.com/tutorials/pages/lighting_rfom4.htm
They baked lighting into bitmaps that were actually normalmaps themselves, so the tiled enviro normalmaps would get directional lighting from the lightmaps. Pretty cool. The vertex color came into play to add color to the lighting, since the lightmaps could only store lighting direction, not actual color.

For regular vertex-color lightmapping/AO, one way to help you figure out where to subdivide your mesh is to bake lighting into a temporary bitmap, assign that to the mesh, then use it as a guide to start subdividing. You can also use Vertex Paint to bake that bitmap into vertex color, to see how you're doing.

System

I don't know why you are getting big dark spots but I can tell you that a combination of AO and normal map will add that extra depth to any model.
Second to that the lights in the map will help by hiding detail at certain angles of your assets ensuring uniformity is not so apparent. Def the right way to go about creating a realistic env, just need to tweak a few settings here and there and be subtle to pull it off well enough to be believable.

Eric Chadwick

I assume the dark spots are because they want to store AO in the vertices, not in a bitmap. Different ball of wax.

Murdoc

Ok. from the responses I think our system is so far off base I'm not making myself clear, lol.

Resistence baked their lighting solution into a map, this isn't really what I'm talking about as it's dealing with how they handled lights.

We're using dynamic lighting solution, meaning very little is going to be baked in, which means no light map ala resistence. However, before importing our models we're doing a AO calculation and baking that info into the vertices(which is the assiness I was describing)

But my thoughts as never dealing with that is, what's the point when using dynamic lighting, more advanced shaders and AO post effects? It's not calculating the AO based on the normal map then baking it into the vertices, just the model themselves.

My point was I feel it's extra useless information that is getting in the way of quality in this particular case, but just wanted to know other opinions on where and how other people thought it was useful.

Eric Chadwick

Vertex-based AO can be good if the tech calculates it either on-the-fly, or else pre-calculates the AO for particular poses of the character (arms up vs. arms down, etc.) and interpolates those vertex colors as the character moves from pose to pose. That can actually look pretty good, if the character is decently subdivided (not 500 tris!).

If neither of those, then best bet is for you to simply bake nice full-on AO into your diffuse bitmaps, and not use vertex AO.

Murdoc

Thanks for the examples Eric.

What about environment stuff(which is where I am finding the issues) it feels unnecessary, but is this an older technique used in past generation games? (just trying to figure out where this studio got it in their head to use it for everything)

Eric Chadwick

Vertex color is simply more efficient than adding lightmap bitmaps. The bitmaps have to be loaded into video memory, takes time (very little, but loading/unloading them can cause slowdowns/glitches as you move around) and they eat up space that you want to use for higher-res textures.

You usually can't bake AO into enviro diffuse maps because they're tiled like crazy, so the idea is why not bake into the verts instead? Gotta add more edge loops, but the increased # of verts is usually not a problem.

Depending on the hardware/engine, vert color can also cause less render passes than lightmaps.

Murdoc

Yeah, by my opionion is it isn't lighting I'm backing it's just the AO and post effects kind of cover that already without the hassle of baking vertex colour.

The baking we are doing doesn't solve the problem of light maps, because we're not baking our lighting solution (IE shadows)

I guess it's a really particular use for very particular solutions, so my conversation is kind of pointless, lol

Ah well, thanks for the insight.

Eric Chadwick

Yeah, should have mentioned it's totally up to you and your team. Talk with your programmers, communication is the best course no matter what. Good luck man.

cman2k

At work we use a similar solution, and it works really well.

You seem to have a concern with baked vertex lighting interfering or fighting against a dynamic lighting solution? I'm not certain how your system is setup, but ours works great together, and i'll tell you a little about why.


First off, Dynamic Lighting & Dynamic Shadowing are different. Calculating AO in realtime is usually not a solution, which is why most engines now a days bake a bunch of shadows in one way or another.

Baking Vertex AO into a model doesn't mean that those shadows are permanent. Vertex AO can be placed in a "Vertex Illumination" Channel and when in-game, effected by dynamic lighting. This means they don't fight each other, because a bright light shone on a model will cause it's Vertex-based AO to disappear.....It means the shadows are baked but still dynamic in how they are drawn.

It's likely that you are using a shadow-solution that is relatively lightweight, so using vertex-baked AO is an easy way to give your scene a lot more depth with a very cheap cost (usually less than a 10% memory increase, it's a per-vertex cost).


Regardless, when setup right I think dynamic lighting & vertex-baked AO can work great together, it's just a matter of getting used to modeling with it in mind.

Daz

I think your concern is a little strange personally, but it's hard to tell without knowing what your engine is capable of, and how sophisticated the lighting model is. Personally I think dynamic shadowing and vertex based AO can work really well together, even If those verts aren't handled in a vert illumination channel at all. Just leave 'em there, and things can work out surprisingly well.

I don't particularly think that Oblivion is a good looking game, but its worth noting that many people did, and it shipped without any real shadow solution of any kind. Unless you count a hack alpha on the ground underneath trees. All buildings had was a vert bake.

Anyway, relying purely on dynamic shadowing, If that's what your engine is doing for a lighting solution, won't cut it alone. It just won't get you the visual fidelity that lightmaps will. Which is where the extra needed step of baked AO comes in. The instant large areas of your scene are swathed in dynamic shadow, they'll look completely fullbright and utterly lacking form unless you have some AO baked in there underneath.

In a nutshell: baked light maps look awesome, but of course can't fly for a day/night cycle. Dynamic shadows won't look as good as lightmaps, but they can look petty darm good if backed up by some vertex baked ambient occlusion.

Sage

Even in offline rendering I have noticed that the models look better with some of of AO baked into them especially if you want to save on rendering time.

Alex

pior

Well from an artistic point of view, an AO bake not only simulates the self-occluding look ; it also suggests a hint of dirt gathered in the concave areas.

So even if in theory, dark AO information should diseappear in the areas where light is directed to and should never be multiplied with the castshadow either as this would cause 'double shadowing', in practice it gives some depth because cracks accumulate dirt and dirt is often darker than the surface it accumulates onto.

I guess such bakes just need to be subtle especially on characters. Plus, you can always edit them with vertex painting solutions. Most likely to make them a bit fancier than black or grey.

Mark Dygert

Totally off subject and out of left field...
AO baked into your spec map also helps to keep the spec contained and keeps nooks and crannies from shinning, it helps keep the dirt pior is talking about, looking like dirt.

Jonathan

In Crysis, you use vertex-colored meshes for detailed bending plants, it's quite cool.

Rob Galanakis

Well yes you can use vertex colors for anything, really (I use them for all sorts of pretty strange stuff in some of my shaders and tools)- not sure what that has to do with AO though.

Jonathan

Sorry for the semi-off-topic remark.

Regarding baking vertex lighting, if you're having issues wit it, you can also light-map meshes with a second UV channel, which is very common with Unreal Engine 3 games.
http://book.hourences.com/tutorialsue3lightmap.htm

There's pros and cons to both.

But as for vertex lighting, here's a tutorial on that, which includes UE2 and UE3.
http://book.hourences.com/tutorialsvertex.htm

Hourence has some great stuff.

Murdoc

WellI know the benefits of AO baking, I'm not disputing that, its just I calculate the AO... looks alright, bake it into the vertex, turns to shit.

I guess it's a matter of adding more verts, and such, but still having troubles, I model things with seperate elements layered on top of each other rather then one continous mesh, maybe it's just the way I am baking it, but it doesn't seem to be working out so well.

I figured game engines were past this shit of merging and attaching meshes together to be continuous, I find it completely inefficent work flow but now it's necessary for a AO vertex light bake; which I still figured was unneccessary due to newgen shaders and post effects.

* Pior: Though after reading your comments what you said makes sense, however what I am seeing on my side does not, so I'll have to look into this. My dark splotches seem to multiply(get darker) in shadow and remain dark in light.... this is why I figured it's fighting it.