Home Technical Talk

Material editors vs. Custom shader code

polycounter lvl 11
Offline / Send Message
Bigjohn polycounter lvl 11
So I'm mostly just curious as to how other studios handle this.

The tradeoff is between a node-based material editor, or having all the shaders created by a programmer. The advantage of an editor is the artistic flexibility that comes with it, the disadvantage is that it shoots out inefficient shaders.

How is this handled at your studio? Just one or the other? Or both maybe, where you use a material editor to make the material, then have a programmer optimize the shader once it's done?

Replies

  • Ace-Angel
    Offline / Send Message
    Ace-Angel polycounter lvl 12
    Take what I say with a pinch of salt. Engines themselves actually optimize the code pretty well, granted ofcourse that the guys who made the engine knew what to do to make the magic happen.

    The entire 'nodes are inefficient' issue only exist because many who wrote said infrastructure didn't know what they were doing, and because they tend to leave behind alot of 'rubbish code' without a proper intermediate conveyer, to make the switch to shader types available (CGFX, HLSL, DXAS, etc), but thankfully (and hopefully) this will be thrown out of the window soon enough once ALL the engines actually optimize the stuff well under the hood.

    As for the programmer and how they can save the day for nodes...well, I'm sure someone who actually worked in said area can answer you better.
  • JamesWild
    Offline / Send Message
    JamesWild polycounter lvl 8
    The HLSL put out by UDK is really messy looking. Pages and pages of conditional stuff most of which is thrown out by the preprocessor, and loads of unnecessary variables all over the place as nodes save their result in a variable every time. (or at least did last time I looked)

    However, the compiler is intelligent enough to turn it into very efficient shader code so other than a longish compile time (which appears to be cached as a binary? does some processing when moving to a machine with a different GPU) it doesn't even matter.

    The bulk of the shader code which actually does the lighting and so on is kept out of sight which is a mixed blessing but I would not be surprised if this is hand written in GPU assembler or kept as a compiled binary and just linked to every material's code during the compile process.

    I'd say you probably want most of the surfaces in your world to not have anything more complex than basic lighting and texture samples simply to save on fillrate cost, so again it doesn't matter too hugely what the few custom bits of shader code do.
  • Bigjohn
    Offline / Send Message
    Bigjohn polycounter lvl 11
    What brought this on is that we're facing this situation in my studio with our own custom engine. So I'm trying to gather some info. So far it sounds like the proposition that the shaders that come out of a material editor is inefficient is overblown. Or at least, that it doesn't have to be that way.
  • JamesWild
    Offline / Send Message
    JamesWild polycounter lvl 8
    To implement it well will require a lot of programmer hours. Bad implementation might be missing features, produce inefficient shader code, be hard to use, buggy, but worst of all it might bump the loading times up considerably. If your team is working on a game that actually requires lots of custom shaders or you plan on using the engine for many games, it might be worthwhile, but in a lot of cases it would not be.
  • commander_keen
    Offline / Send Message
    commander_keen polycounter lvl 18
    Im not sure about HLSL and GLSL directly buy CG does a LOT of optimization. This makes programmatically created shaders that may look unoptimized probably just as efficient as a hand written one.
    myFloat4 *= (herp/derp)*(herp/derp)*(herp/derp);
    
    should compile to something like
    float4 herpdivderp = (herp/derp);
    myFloat4 *= herpdivderp*herpdivderp*herpdivderp;
    

    Also afaik all processing on the gpu is done on data sets of 4, so these 2 lines should be just as fast:
    myFloat4.x = otherFloat4.z/someFloat;
    myFloat4.xyzw = otherFloat4.zzzw/float4(someFloat,someFloat,someFloat,someFloat);
    
  • Brendan
    Offline / Send Message
    Brendan polycounter lvl 8
    One of the advantages of the node editors is the frankly absurd iteration speed possible, something that just coding just won't match. The speed of my additions and updates is almost beyond imagination.

    The flipside is that multiple passes, and other non-standard surface shading, is generally poorly supported (especially with Unity's Strumpy).

    Ideally, if a couple of your guys in the team know how to code shaders, then it should be alright to let the artists roam free and happy with the node editor, then get the codemonkeys to do a final pass.
  • Ace-Angel
    Offline / Send Message
    Ace-Angel polycounter lvl 12
    I remember not too long ago a bunch of peeps on some tech forums where talking about creating a unified shader system via Python (I think?), with a user interface but coded.

    So basically, instead of having nodes, you would have editor under which you could put up words and what they all mean and have the engine load it externally as such, such as;

    {Lambert + Diffuse Fresnel

    Lambert = Normal (fDot) Light
    {Diffuse Fresnel = Input some fancy code here.

    So you would have the writer do all the work behind the scene, and the person, with a little knowledge of basic HLSL and CG, could get the same benefits of everything else.

    Sadly, I can't find the topic, nor remember where I read about this, but the idea was very cool indeed, especially if you get tired of clicking all day long on nodes, or aren't too well versed in shader lingo to declare each time a parameter in the right place (especially when you have to expand the Normals).
  • Kurt Russell Fan Club
    Offline / Send Message
    Kurt Russell Fan Club polycounter lvl 9
    We do it with education and profiling.

    Education meaning we talk to anyone who's going to touch materials and give an overview of what's going to slow down the system (so in Unreal material terms, what instruction count and material dependency mean, and how they're calculated). Then we let them go crazy.

    Profiling comes in afterwards, when you're trying to get the game running at frame rate. Since 10% of the game is going to cause 90% of the slowdown, we run the game on PC and consoles and look at profiling tools to work out what's running slowest and causing bottlenecks. Fix that stuff and then ship it. :)
  • Bigjohn
    Offline / Send Message
    Bigjohn polycounter lvl 11
    I'm also wondering, do any of you guys use some kind of 3rd-party material editor that can plug into a proprietary engine? Does that even exist?
  • marks
    Offline / Send Message
    marks greentooth
    shaders are all handled by code (with input from art team) at my studio
  • Ace-Angel
    Offline / Send Message
    Ace-Angel polycounter lvl 12
    3rd party? In my case, I make my shaders from scratch in ShaderFX in Max, export to written text and forward it to my buddies for analysis, and if there is a section of the code they don't understand, they can preview it in Max or ask me.

    As for 'pure' universal translator which is easy to use and can be plugged in an engine? Don't think so.
  • Bigjohn
    Offline / Send Message
    Bigjohn polycounter lvl 11
    I thought ShaderFX was abandoned. I don't see a download link anywhere, and when you go to their site it says the store is closed.

    Edit:
    Yeah, there's a message on their site saying the guy joined Autodesk, so there's no more ShaderFX.
  • Ace-Angel
    Offline / Send Message
    Ace-Angel polycounter lvl 12
    Blimey, you're right, ugh this sucks man.
  • sprunghunt
    Offline / Send Message
    sprunghunt polycounter
    Ace-Angel wrote: »
    Blimey, you're right, ugh this sucks man.

    and it looks like mental images has stopped distributing mental mill too!

    https://www.mentalimages.com/products/mental-mill.html
  • Ace-Angel
    Offline / Send Message
    Ace-Angel polycounter lvl 12
    Seriously, WTF is going on here? Is Max so broken in the 3rd party shader department that they need to hire everyone to fix said issues?
  • Bigjohn
    Offline / Send Message
    Bigjohn polycounter lvl 11
    So is this possible in any way? Or am I barking up the wrong tree at this point? Is there any independent node-based material editor that can shoot out a shader that could be used in our engine (once our programmers mess with it)? Can UDK export an HLSL/CGFX shader?
  • roosterMAP
    Offline / Send Message
    roosterMAP polycounter lvl 12
    is there a tut to compile ur own hlsl shaders into a udk material? iv had to translate code into nodes... but i never saw a compiler.
  • Jason Young
    Offline / Send Message
    Jason Young polycounter lvl 14
    Render Monkey maybe?
  • haiddasalami
    Offline / Send Message
    haiddasalami polycounter lvl 14
    JMYoung wrote: »
    Render Monkey maybe?

    Render Monkey has sadly gone out of development. You can still find it though FXComposer is probably your best bet when it comes to creating HLSL/CGFX shaders or good ol notepad++.

    From what I gather when I looked into shaders (this was when I was first researching tech art) some places have a material editor where a tech artist can prototype something and pass it off to the graphics engineer who will profile/covert etc. Naughty Dog showed theirs off or I think it was Sony who made it and first party studios have access to it. Looked good from the slides.
  • Ace-Angel
    Offline / Send Message
    Ace-Angel polycounter lvl 12
    I personally would choke a chicken to get a UDK shader converter to plain old HLSL...
  • sprunghunt
    Offline / Send Message
    sprunghunt polycounter
    Ace-Angel wrote: »
    I personally would choke a chicken to get a UDK shader converter to plain old HLSL...

    not quite what you want but have you looked at the source window in the material editor?

    window menu - turn on checkbox marked "source"
  • Ace-Angel
    Offline / Send Message
    Ace-Angel polycounter lvl 12
    Yep I actually did that not too long ago, but I'm couldn't make head/tails out of it since it doesn't parse out everything with proper sense, especially since it has extra headers to declare the default lambert and phong terms with an IF term incase your CustomLighting fails and need a backup term to fall on.
  • CrazyButcher
    Offline / Send Message
    CrazyButcher polycounter lvl 18
    true minor corrections:

    * there is no such thing as shaders written in GPU assembler. There is something like am immediate representation which is what DirectX's HLSL compiler spits out, that is passed to the driver and then turned to true GPU assembler (as that is very vendor/hw specific there is no "standard").

    * vector operations != scalar operations. The float4*float4 as fast as float*float is not globally true. NVIDIA has a scalar architecture for quite some time and AMD recently moved to one. Typically the compiler is good enough in dead code elimination. So if you do float4 op once but later say use only .x, then only that scalar will be calculated.
    How good compilers are depends very much on the platform, your mileage will vary greatly on mobile platforms, while PC (especially DX) or consoles should be very tuned.

    Personally I use my own scripted IDE for shader work, which passes files to offline compilers (GLSL,HLSL)
    http://www.luxinia.de/index.php/Estrela/Shader
  • Bigjohn
    Offline / Send Message
    Bigjohn polycounter lvl 11
    Ace-Angel wrote: »
    I personally would choke a chicken to get a UDK shader converter to plain old HLSL...

    Yeah, I wish this was possible too. Then I could have people just make their materials in UDK, and then port them over.
  • shaderfx
    Offline / Send Message
    shaderfx polycounter lvl 9
    Hi all,

    ShaderFX was not abandoned.
    It has been acquired by Autodesk and its development has moved there.
    If you want to be on a future beta, drop me an email: kees.rijnen at autodesk.

    As for the question about node editor versus hand-code.
    It is mostly a personal preference.

    From a code inefficiency point of view. if you would know what you are doing with hand-written code to optimize performance, you could do this too in a good node-editor (for the most part).

    The compiler is also very good at optimizing.

    Some people will never want to use a node editor, and some will never want to write code by hand.

    As for code readability, I have spend a lot of time working on this in the new ShaderFX and it is doable to output code that is similar to what you would write by hand. But it does take a good amount of programmer time to get that to happen.

    Unreal does have many #ifdef in the code and it makes it very hard to do anything with the code afterwards, but I don't think it was their goal for you to print out the code and hand edit it or to take their code elsewhere. So they did what worked for them. And it is an awesome editor.

    For Mental Mill I always thought the code was very bad to do anything with and I thought that was a major fail for that editor because if you want wide adoption of your 3rd party, shader node editor people should be able to read/edit the code.

    A node editor may never make code look exactly like hand written code, but it can do a much better job then Mental Mill did, imo.
  • Bigjohn
    Offline / Send Message
    Bigjohn polycounter lvl 11
    I'm mostly concerned about my programmers. They're concerned about the performance impact of node-generated code. But if I can give them something that's like 90% of the way there, that's definitely workable.

    About the Autodesk thing, you're not talking about Slate are you? At least I hope not. Or do you just mean that ShaderFX will be part of some next version of Max?
  • shaderfx
    Offline / Send Message
    shaderfx polycounter lvl 9
    No not slate.

    I can't say when/what/if anything gets released in any future version of 3dsMax. Sorry.
    Best I can offer is to put the name of anybody interested on a "future beta list".

    As for node editor code performance. I think you can get closer then 90% performance wise versus hand written code.

    It all depends on your level of understanding of real-time shaders.

    If you know what to do and what not to do with hand written code, you can apply the same knowledge with a node editor.

    If you don't and start mixing 16 4k textures "just because you can", then yes, your code will not very fast :)
  • Minato
    Offline / Send Message
    Minato polycounter lvl 5
    I'm bumping this since i found a few things really interesting in here, especially the bits about ShaderFX(by shaderfx), since i've been recently thinking a lot about the need of a visual shader editor for Max, you know st like hypershade, and the fact that MetaSL is just too limited compared to, again st like hypershade, and it kinda was a relief to read that Autodesk has (probably) finally decided to address this, and what could be better for this than ShaderFX... One can only hope...

    Thanks btw Kees.
Sign In or Register to comment.