TF2 and TOON realtime viewport shader (3ds max)

rollin

(ok I´m creating a separate therad to make it easier to find the thread and to keep stuff together)

while I´m working on some custom models for tf2 i thought it would be faster to see the tf2-shader already in the max viewport..
this shader is more like a fake of the original tf2 shader but it´s already helping to get an idea what's going on

grab it if you want

thx again to Lumonix for shaderFX.. a very helpfull tool

(tested with max8 and max 2008 )

---

Team Fortress 2 shader v0.32

This shader is based on the real tf2-shader math.
That does not mean it's exactly the same but the visual result gets very close.

The Update to v0.32 brings you
- vertex color support and
- a fixed bug in the ramp shading

You can find the Download page - HERE -

+ picture of material rollout

---

Toon / Comic Shader v0.60

changes since v0.58
- completely reworked the code by hand going away from shaderFX. So I can clean up the code and I have more control over it.
- improved the sss-fake
- improved the rim lighting, it's now softer and easier to handle
- added cubemap blurring per value or per gloss map and masking through specular map
- you need a max version capable of displaying realtime hard shadows in the viewport (max 2009 or 2010, not sure)
- opacity can be read from diffuse color value
- allows up to 2 shadow casting lights


changes since v0.57
- fixed normal depth setting
- changed and fixed rimlight calculation. It's now working more the common way. You can offset the rimlight light source a bit (maybe have to increase the range) and exclude the normalmap from the rimlight calculation
- also added a simple sss fake. Have to get deeper into this but for the beginning it does a good job in fighting against black-skin-shadows

changes since v0.56
- zbrush material technique

changes since v0.55

- fixed colored specular map turned monochrome
- re arranged UI

changes since v0.53

- LitSphere support
- all major texture slots can be disabled/enabled
- normalmap red and green channel can be inverted
- added technique for 1 light/pass support - choose this if you're using the shader with the default scene light
- rim light can be calculated from normalmap or surface normals
- diffuse to ambient "leaking" can be influenced
- global opacity can be changed

changes since v0.51

- changed from shader model 2.0 to 3.0
- better rim light handling
- better ramp shading handling
- modify the nm blue channel
- some minor bugfixing



You can find the Download page - HERE -

+ picture of material rollout ( v0.58 )

---

the RampTextures

And here are some ramp textures for you to use DOWNLOAD


TF2 example:



the LitSpheres

Example LitSpheres DOWNLOAD

Example 4:

Bal

Hmm, under max 2009, my model becomes invisible when I apply the shader on it, even with a normal map and ramp texture loaded, and lights in the scene.
Will try messing around some more with it, I know you said you hadn't tested it in 2009.

rollin

I´ll grab the max2009demo and see whats going on

additional.. if you have shaderFX you can get the project file, open it -> save 3dsmax shader

you should get a shader that works with your version of max

shaderFX Project File

JohnnyRaptor

is this for 2010?

rollin

I´ve only tested it with max8 / max2008

if someone has another max and shaderFX AND the shader is not working in this version of max, than it would be cool if he could provide me with the *.FX file from his shaderFX

Jason Young

Works for me on 2010 64 bit(vista).

DEElekgolo

I don't thing you really need a ramp texture for this. If you want to get the unlit, cartoony style that tf2 has then look into a rendering method called Half lambert.

Bal

Yeah, Xoliul's max shader does half lambert, haven't tried to see if you can easily recreate TF2 look with it.

rollin

JMYoung: thx for the info!!


DEElekgolo: tf2 uses a fallof function to create the ramp-effect.. it´s not the "default" half-lambert..

but why I´m using the ramp-texture here is bc I've modified my previous toonshader (which is based on a ramp-texture) to get a tf2 shader preview in max viewport fast

this shader does only FAKE the tf2 shader behaviour.. but that's all I need while texturing

Xoliul

Bal is right, you can get the TF2 look with my shader as well. It has Half Lambert, ambient cube, rimlight (3 kinds actually) and spec and gloss control with maps. There's just no ramped shading, but I light add it. It might be a little harder to achieve it in my shader just beacuse there are so many settings.
But if you want Rollin, tudy my code to see how i did all these things, it's intended for people like you to learn from as well.

rollin

bc this question came up

if you`re wondering what cubemap format the shader is using:

take one of the maps that ship with 3ds max (you'll find them under ./max8/maps)

or create your own one with CubemapGen from ATI


you can create one out of single images like ben cloward describes here


or simply convert a cubemap cross (3:4 format!!)
you can find free ones here

Neox

DEElekgolo wrote: »

I don't thing you really need a ramp texture for this. If you want to get the unlit, cartoony style that tf2 has then look into a rendering method called Half lambert.

you definitely need if you go for the TF2 look

http://www.valvesoftware.com/publications/2007/NPAR07_IllustrativeRenderingInTeamFortress2.pdf

the half lambert is a part of the shading, as is the ramp
half lambert doesn't give you harder shading edges (like blurred cellshading) only makes the shading brighter and flatter

rollin

I would go even further and would say

full lambert is a fall-of / ramp function
half lambert is another one
tf2-"lambert" is another one too

I don´t know how they've implemented the ramp technically but theoretically I would say they've use simply a new fall-of function (tf2-"lambert") while ep1 just uses a normal half lambert fall-of/ramp

Neox

they just plug the lambert in the uvs of the ramp, read the paper its all written there how they are using the ramp

CheeseOnToast

I can't speak in techno-artist language, but having used the Source engine extensivly, I can tell you that you need a ramp texture (known as a lightwarp or phongwarp texture in Source) for sure. A smooth gradient from left to right that goes from black to 50% grey will give you identical lighting to not having the ramp at all. Anything above 50% grey will be overbright. The general-purpose ramp texture that is used on most things can be found in the pyro character textures folder if I remember right. They use a harsher, more stepped one for weapons. Even if you use a toon-shader style ramp texture, it's a good plan to blur the transitions a little otherwiswe the model tends to look too shiny/chrome-like rather than toony. Hope that makes sense.

rollin

?! can´t realy find that point you're talking about (uvw-slot) ;p

but thx for your kick in the ass to read the paper...

should start to work on v0.3 now and make the shader a bit less fake#

edit: and thx CheeseOnToast.. this helps indeed!

Xoliul

Lol, you're coding the shader, but didn't read the paper until now ?

rollin

oh I´ve looked in it but more aimed on the single shader "layers" as on the technical mathematical base


and there was no reason to do it..

when I´m doing it now than just to try to REALY simulate the tf2 shader if possible

not that it would be truly necessary

rollin

---

obsolete

rollin

ah sorry.. there was a little bug in the v0.3 download..

should be fixed now

the download link is the same but I´v updated the file

|*BILLY$CLINT*|

Neox wrote: »

you definitely need if you go for the TF2 look

http://www.valvesoftware.com/publications/2007/NPAR07_IllustrativeRenderingInTeamFortress2.pdf

the half lambert is a part of the shading, as is the ramp
half lambert doesn't give you harder shading edges (like blurred cellshading) only makes the shading brighter and flatter

Ah I was looking for this paper...Thanks a MILLION!!!

ThatDon

Awesome improvement rollin, thanks for sharing! I'm hoping to put this badboy to use real soon

rollin

I've updated both shaders

v0.31
i´ve marked all settings which are no real tf2-shader values with an underscore
the default values will get you a very close tf2-like result

added spec, rim and exponent boost values

added a texture slot to mask out the ambient cube



v0.21
added a texture slot to mask out the ambient cube

rollin

there is a new Update online

- the shader v0.2x is now implemented in my toon shader (v0.51)
- added vertex color support for both shader
- fixed some bugs

I've updated the first post
check it out

rollin

Updated the first post!

With: new update for the toon shader (now v0.53)

changes since v0.51

- changed from shader model 2.0 to 3.0
- better rim light handling
- better ramp shading handling
- modify the nm blue channel
- some minor bugfixing

I hope the model-gets-invisible problem is now gone.. but I'm very happy for every positive or negative report !!

rollin

hey ho!

I've finished version 0.55 which supports some important features and bugfixes.

changes since v0.53

- LitSphere support
- all major texture slots can be disabled/enabled
- normalmap red and green channel can be inverted
- added technique for 1 light/pass support - choose this if you're using the shader with the default scene light
- rim light can be calculated from normalmap or surface normals
- diffuse to ambient "leaking" can be influenced
- global opacity can be changed

DOWNLOAD Page

the new material rollout

rollin

small update to v0.56

- fixed colored specular map turned monochrome

- re arranged UI

hyrumark

Thank you for sharing this, great shader!

onionhead_o

the alpha map function does not work for me. im using max2009. wut i mean by that is the mesh looks like theres no alpha map applied to it at all. i even checked the use opacity. and nothing happens.

kodde

Your shaders seems sweet rollin. Damn shame that brand of 3D package hinders me from trying it out T_T

Hazardous

Dude.... NICE! Cant wait to give it a pop!

rollin

onionhead_o wrote: »

the alpha map function does not work for me. im using max2009. wut i mean by that is the mesh looks like theres no alpha map applied to it at all. i even checked the use opacity. and nothing happens.

I can't test the shader in max2009.

But in max2008 it does work.

So I'm bit sorry, but I can't guaranty it's working in all max version

onionhead_o

rollin wrote: »

I can't test the shader in max2009.

But in max2008 it does work.

So I'm bit sorry, but I can't guaranty it's working in all max version

no worries man. thanks for the reply tho

rollin

ok while working on my last model I did some work on my shader too.

I'm releasing version 0.58 now while version 0.57 is used by GoMax

new from v0.57 is a new technique that allows to see only the litsphere on the object. Nothing special.. just to get Zbrush materials into max viewport

new with 0.58
see image below
- fixed normal depth setting
- changed and fixed rimlight calculation. It's now working more the common way. You can offset the rimlight light source a bit (maybe have to increase the range) and exclude the normalmap from the rimlight calculation
- also added a simple sss fake. Have to get deeper into this but for the beginning it does a good job in fighting against black-skin-shadows


Download page is - HERE - again.

StickFigs

In max 2010 the texture works when I render the scene but it looks wrong in the active viewport.

rollin

hi, are you referring to realtime rendering? bc I don't understand what you are comparing here. could you describe the problem a bit more or show a screenshot?

StickFigs

rollin wrote: »

hi, are you referring to realtime rendering? bc I don't understand what you are comparing here. could you describe the problem a bit more or show a screenshot?

I thought this would allow me to see what the final project would look like in my 3d viewport window, basically making the 3d viewport window look like modelviewer.

rollin

so you're talking about the tf2 shader, do you? and you are comparing it with the hl2 sdk modelviewer?!


if so, the default settings should already give you a quite similar result

I don't know what you're doing.. if you use the toon shader to texture tf2 stuff it will not work (at least not with the default settings).. thats why there are two different shaders

rollin

I got my hand on max2010 and I really love the new functions for the viewport.

Also tested the toon shader in it and for just a few minutes of playing around I really like the result

Michael Knubben

Rollin: sounds to me like he's talking about the realtime portion not working properly, which could be due to Max's bake? The fact it works in a render made me think that, anyway.

Either way, Stickfigs, without images we can't help you.

Galago

This is great, really helpful for texturing. On a semi-related note, is there some sort of way to duplicate the TF2 shading style when rendering?

rollin

could we pleaaaaase use the definitions "offline rendernig" and "realtime rendering" just rendering does say nothing


you don't render an fx-shader offline.. you can use a script like this to make a high resolution screenshot though

Galago

rollin wrote: »

could we pleaaaaase use the definitions "offline rendernig" and "realtime rendering" just rendering does say nothing


you don't render an fx-shader offline.. you can use a script like this to make a high resolution screenshot though

Sorry, I'm rather new to this. Thanks.

Buck

If only someone made it so you its not only real time but would render out same way too..

Charlottje

Wow this is a great shader!
I tried testing it with a car I modeled, but I can't seem to get the toon-y effect. If I set my rampcontrast to anything above 0, then everything just gets a lot darker.
Anything from 0 to negative values, and everything gets saturated and a lot lighter.

Am I doing something wrong?
I made screenshots:


Ramp Off
Ramp On

Xoliul

Hey Charlotte, it's because ramped shading requires a texture to work.
If you're not afraid of the technical side behind it :
the shading calculation returns a value between 0 and 1. Instead of multiplying this to darken the unlit areas, this value is used as a UV-coordinate for a gradient ramp texture. Since a gradient only goes from left to right, there is no need for a height coordinate, you can just use a constant value for this. Once the gradient is sampled from the texture, this value is multiplied instead of the 0-1 of the original Phong calculation. Nifty huh?

What you are seeing in your case, is 3DS Max returning black, because no gradient ramp texture is loaded. Just load one into the correct slot, I think the shader comes with a few of them.

rollin

heyho

updated my main shader to 0.60

changes since v0.58
- completely reworked the code by hand going away from shaderFX. So I can clean up the code and I have more control over it.
- improved the sss-fake
- improved the rim lighting, it's now softer and easier to handle
- added cubemap blurring per value or per gloss map and masking through specular map
- you need a max version capable of displaying realtime hard shadows in the viewport (max 2009 or 2010, not sure)
- opacity can be read from diffuse color value
- allows up to 2 shadow casting lights



DownLoad Page


this here for example is done with the shader: unedited screengrab from max 2010