Has this been discussed yet? Dynamically changing normal maps?

Junkie_XL · Jul 2009

Is this something to worry about? Dynamically blending normal maps changing in sync with character animation? Clothing wrinkles change of course given the pose of a character. I was wondering if this has been discussed at all or if there are siggraph papers on it?

I'd hate to give some clever programmer any ideas as I feel the pipeline for next-gen game artists already gives them more than enough to worry about, but I'm sure someone is already working on this. I was just sitting here sculpting and the thought popped into my head.

onionhead_o · Jul 2009

wat a great idea. I hope someone who knows how it works shed some light on this matter. it would be valuable knowledge.

CrazyButcher · Jul 2009

it has been done before, if I remember well Eric worked on tech using it. And I am pretty positive for other stuff I've seen it mentioned as well (uncharted: drakes fortune character gdc slides or so)

Ruz · Jul 2009

well f someone is developing a 'system' for it, it will inevitiably be a an artist driven thing under the guise of being a 'system'

ie more work for the artist:)

TheSplash · Jul 2009

I think Crysis also does this with their facial animation for brow wrinkles etc.

http://wiki.crymod.com/index.php/AssetCreation_WrinkleMapCreation

Mark Dygert · Jul 2009

really common, more common then you might think. Kind of expensive for real time so they're normally limited to just a few.

Look up "wrinkle maps".

malcolm · Jul 2009

This has been done for years on ea sports games. I think if you look back to nba live 2006, or 2007 on xbox 360 can't remember what year this started? You will see blending normal maps on shorts and jerseys. I never liked the effect I thought it was a waste of texture ram and fell apart if you stopped and looked at it for a second.

Rick Stirling · Jul 2009

it's very common, but as Vig says it's expensive - you are going to be loading 2 or 3 normal maps and blending between them with either canned or dynamic responses.

For something like the wrinkles on the back of Drakes shirt or the clothing on the players in Fifa you'll be monitoring the changes in rotation between a few bones and blending the maps.

I'm convinced that Snake had wrinkle maps on his face in MGS4. As his mouth moved to an 'E' shape extra wrinkles would blend in.

MoP · Jul 2009

Yeah, MGS4 definitely had wrinkle normal maps for the facial animation stuff.

Mark Dygert · Jul 2009

I don't think anyone mentioned it, but the book Stop Starring by Jason Ospa covers using wrinkle maps for facial animation as well as some nice making and blending techniques for Maya. With a little work they're easily ported to Max and other apps.

Vailias · Jul 2009

UT3 has this for its cinematics. Yes they are pre-rendered, but they were pre-rendered in UT3.
The materials are still able to be seen though. Check out the CH_Characters_FaceFX package if you have UT3.

Ben Apuna · Jul 2009

This mel script for maya was posted a while back not sure by who, when, or where.

http://www-viz.tamu.edu/courses/tutorials/gary/index.html

I also don't use Maya anymore so I have no idea what versions of maya it will work with.

I don't think it's realtime either so it's probably useless for games, but it may give some insight into the process.

pior · Jul 2009

Drake's Fortune does that too, right?

Ghostscape · Jul 2009

pior wrote: »

Drake's Fortune does that too, right?

Yeah his shirt blends between a left crease and right crease, IIRC.

retleks · Jul 2009

Here's Rich Diamant's site, at the bottom is the PDF people are probably looking for. Goes over the pipeline for this in Drake's Fortune.

http://www.rd3d.com

ScoobyDoofus · Jul 2009

I'm pretty certain Mass Effect used wrinkle maps.

Doom 3 even used this in a few areas as well.

Junkie_XL · Jul 2009

malcolm wrote: »

This has been done for years on ea sports games. I think if you look back to nba live 2006, or 2007 on xbox 360 can't remember what year this started? You will see blending normal maps on shorts and jerseys. I never liked the effect I thought it was a waste of texture ram and fell apart if you stopped and looked at it for a second.

That's too bad to hear. Yeah this is more of my focus. Not A-E-I-O-U-Squint facial poses. Facial animation can be scripted to either be posed a certain why on or off. The body is so much more dynamic and I was wondering how this could be overcome.

There are so many different poses the human body can do in regards to clothing wrinkles. You'd have to blend between 10's if not 100's of normal maps because the limbs can all be doing different things independent of each other. It certainly doesn't seem cost effective at all the more I think about it.

throttlekitty · Jul 2009

One could probably use the joint angles/cones as a cue for when to switch on certain wrinkles. But yeah, the amount of maps needed would be quite an army for the average piece of cloth.

JordanW · Jul 2009

the new fight night uses this to great effect, you see tendons and muscles pop, it's really nice looking.

Rick Stirling · Jul 2009

Junkie_XL wrote: »

That's too bad to hear. Yeah this is more of my focus. Not A-E-I-O-U-Squint facial poses. Facial animation can be scripted to either be posed a certain why on or off. The body is so much more dynamic and I was wondering how this could be overcome.

There are so many different poses the human body can do in regards to clothing wrinkles. You'd have to blend between 10's if not 100's of normal maps because the limbs can all be doing different things independent of each other. It certainly doesn't seem cost effective at all the more I think about it.

You don't change the entire map - you use masks to blend in regions independently.

Mark Dygert · Jul 2009

Junkie_XL wrote: »

That's too bad to hear. Yeah this is more of my focus. Not A-E-I-O-U-Squint facial poses.

Very basic visemes are A, E, Fv, L, MBP, O, S, Th, UWQ, not the vowels.

Junkie_XL wrote: »

Facial animation can be scripted to either be posed a certain why on or off. The body is so much more dynamic and I was wondering how this could be overcome.

This is why there is a huge disparity between what gets done in film and what gets done in games. It's not so much technical constraints that are holding back facial animation, its lack of focus and importance. God of War for the PSP had some amazing realtime in engine facial animation and lip sync. Why is that? Possibly because they gave two craps about the process instead of saying:
"Meh just 'script it' they're going to blow them up anyway".

And like Rick pointed out, normally only a few maps possibly as little as one for each texture sheet with effected masks being animated based on bone angles. Here's an example composite map in 3dsmax for regular facial animation, using png's for masks.

Errr now what were we talking about...

Rick Stirling · Jul 2009

vig - where did that image come from? One of your tests? I noticed it's using a bump map instead of a normal map for the details, so i'm assuming a base normal map and then a detail bump map?

Mark Dygert · Jul 2009

Yea for that test I used bump maps, which is what we use at work (yea not my call).

Drop a composite map in the bump slot, drop normal bumps in the left composite map slot, masks in the right. Base in the lowest level. Same can be done for the diffuse, so wrinkle AO can be applied also. I animate the opacity which is wired into a control board that drives bones and morphs.

The trick is to first set a key for it then it shows up in wire parameters and reaction manager. Max culls tracks unless you animate them sometimes as I'm sure you're keen to already (but others may not).

I have no clue how to make a realtime shader that does the same thing, should not be too hard for someone into that kind of stuff.

Getting the masks to not bleed is a bit tricky but not impossible, just something to be aware of. I've seen the masking also done with some kind of RGB vertex colors instead of separate bitmap masks or maybe it was separate masks in RGB channels in a bitmap... probably half a dozen of ways to do it?

Rick Stirling · Jul 2009

Aye, doing the masks with vertex colours works and it's cheaper than hauling other mask maps around (though the masks can be a smaller resolution and you can use a channel per mask zone getting 4 mask zones in a single texture), but it's much harder to control and needs to be setup on a per character basis (whereas mask images have the possibility of being reused).

arshlevon · Jul 2009

we do this, i wrote the maya shader in mental mill that the animators use to visualize the wrinkles.. our tech artists set it up to wrinkle automatically based on how the face is posed, we were suppose to do a siggraph master class this year, but autodesk canned the master classes at siggraph because of the economy.. might try and dig up the shader if anyone wants to play with it..

Art-Machine · Jul 2009

Gears Of War did it already with character faces. One look at a head's shader will reveal a complex normal map blended over several wrinkle layers that can be faded in and out of the final normal map. Not exactly the use you've suggested, but the technique has been done for a while now.

Also in prerendered CG, this has been in use even longer, again mostly in flesh

3dsmaxuser · Dec 2009

Vig wrote: »

Very basic visemes are A, E, Fv, L, MBP, O, S, Th, UWQ, not the vowels.

This is why there is a huge disparity between what gets done in film and what gets done in games. It's not so much technical constraints that are holding back facial animation, its lack of focus and importance. God of War for the PSP had some amazing realtime in engine facial animation and lip sync. Why is that? Possibly because they gave two craps about the process instead of saying:
"Meh just 'script it' they're going to blow them up anyway".

And like Rick pointed out, normally only a few maps possibly as little as one for each texture sheet with effected masks being animated based on bone angles. Here's an example composite map in 3dsmax for regular facial animation, using png's for masks.

Errr now what were we talking about...

My name is Dries, and I reaaaally need help with this kind of thing.

I take it that the thread was talking about this:
[ame]

http://www.youtube.com/watch?v=sLeksjN4N0c[/ame]

I am doing my first 'big' animation and I am using 3Dsmax2009 with finalRender (use to use mentalRay, was bad)

Would you be kind enough to help me out, I could explain all the details of my scene in a email,
If you are willing to help, please mail me back

- Dries

Has this been discussed yet? Dynamically changing normal maps?

Replies