[ TIP ] Handpaint normal maps with less effort
polycounter lvl 9
Hey there everybody !
[ editor's note : read the previous line in a creepy tone of voice, like you're wearing a trenchcoat and talking to a group of kids]
So i stumbled onto this little trick that makes your life easier when handpainting normal maps:
- the "no foreplay" version : photoshop allows you to isolate a single channel per layer. That's usefull because whatever you paint on that layer only affects the channel you've isolated. I use this trick to separate the red and green channels of the normal map. What's also cool about this is that you can also use the blur and smudge tools when editing "single channel layer" (very usefull when you're correcting baking errors). So how do you do that fancy channel isolation thing ? :
- the "birthday sexy time" version :
I'm curious if anyone uses this trick and what else do they use it for.
Later dudes !
[ editor's note : read the previous line in a creepy tone of voice, like you're wearing a trenchcoat and talking to a group of kids]
So i stumbled onto this little trick that makes your life easier when handpainting normal maps:
- the "no foreplay" version : photoshop allows you to isolate a single channel per layer. That's usefull because whatever you paint on that layer only affects the channel you've isolated. I use this trick to separate the red and green channels of the normal map. What's also cool about this is that you can also use the blur and smudge tools when editing "single channel layer" (very usefull when you're correcting baking errors). So how do you do that fancy channel isolation thing ? :
- the "birthday sexy time" version :
I'm curious if anyone uses this trick and what else do they use it for.
Later dudes !
Replies
Can you make custom blend modes in Photoshop? Might be able to build one that allows you to preview the lighting.
Not that I'm trying to shit on your effort mate, It's good of you to put the effort in and share.
The reasons I'd be down on this method:
*Only works easily for right angles, curves will never shade correctly
*This isn't normalised so depending on the engine the shading will be off.
*Would be much easier to do with Crazybump, nDo, or Nvidia filter.
it can be a super speedy way to do --certain-- things if you know what you're doing ...and it gets you a *pixel perfect* match for a diffuse(unlike a lot of the grayscale height-to-normal conversion tools, which create a halo). This makes it a perfect trick to use with a hand-painted diffuse.
Additionally, you can do some neat style tricks with hand painted normals. However, I think that the OP's approach is a bit inconvenient...if you're gonna hand paint normals, I say hand PAINT them..like Jameswild and Ivanzu suggests.
However, I prefer a facet palette, as opposed to a smooth sphere. Here's one of the ones I've used in the past:
I've got tighter resolution ones lying around somewhere, but that facet palette does 75% of the job.
This was how we did a lot of the normal maps for Project Copernicus--it allows for SUPER fast texturework with a workflow that all our talented painters could quickly adopt.
Here's one example of it at work: on the carved wood in this interior.
and a few more: http://1.bp.blogspot.com/-roU73smdi7U/UN4N-85HnTI/AAAAAAAAAGk/1anYQFl3Q7M/s1600/Screen11.png
It would take the painters an hour or two to mentally adjust to 'painting angles' the first time, but being able to leverage their skills really did wonders.
.
.
.
EDIT:
All that said, I'd:
1--NEVER use this on characters or anything with more organic, complex undulation. It's really a straight, for-walls-and-atlas-textures, environment art trick...with painted diffuse maps there's a little wiggle room in terms of lighting/normal map accuracy, but getting too crazy introduces too many errors.
2--ONLY use when a lighting inaccuracy is tolerable. (i.e., heavily stylized games). Hand in hand with that, I'd NEVER use this on a realistic game. The margin for lighting errors is too small and the gamer's expectations are too high for realism.
3--always recognize this as a tool to be used when you're scrambling for time/resources..or when you've got a dozen texture painters with varying tech skills and you're looking for something that won't wash out their diffuse work.
Anyway, as a complement to hand painted diffuse environment textures, it's a SPEEEDY solution. (apologies for the thread hijack, btw)
I've actually got a kinda old video tutorial lying around if anyone's curious about how to quickly paint stuff like this.
*Only works easily for right angles, curves will never shade correctly --
You mean right angles in-scene, or on the texture? If on the texture then, yes, this is one of the reasons the trick works better for stylized games.
*This isn't normalised so depending on the engine the shading will be off.
You definitely see issues here, and incorporating some strength/bias tools into the shaders so the artist can dial up/down the strength helped us combat the issue. Artist-driven bias can be dangerous, but we were pretty well calibrated back at 38.
*Would be much easier to do with Crazybump, nDo, or Nvidia filter.
Haven't tried nDo (and it wasnt out yet when we were developing our pipeline), but I've heard it's awesome. With Crazybump and the Nvidia filter you get some pretty bad haloing..which is more obvious on hand painted stuff...and the results are still based on a heightmap of some kind, which still has to be generated (and can be particularly tricky to 'automagically derive' with a flattened hand-painted diffuse). The painting method tended to be just...prettier...believe it or not.
For the record i did use to(and still do mostly) paint normal info while picking colors from a baked sphere(allways on top of something generated, never from scratch). I think the example i gave ( because of it's simplicty ) was misleading as to the "end game" aplication of this method.
I do agree that painting anything curvy or organic can be very difficult with the "color picking" method bc you have to worry about a ton of hues, and should be done only in time crunches..... but i have seen very impressive work being done with the "painting per channel" method (think basorelief type stuff that would take alot of time to make as highres in zbrush and bake).
The advantage of the "per channel" method is that you only worry about values, so even someone with an ok grasp on shading(in the traditional medium sense) could paint reasonably complex shapes with this method. I've stated in the tutorial why i didn't like the workflow for this method and that's what i've tried to improve upon. I guess i did do a poor job of explaining things (mostly by leaving out explanations out of my hatred for walls of text)
Oh and i just realized that if you add a hue/sat adjustment layer that desaturates everything on top of the red and green layers you would get the benefit of seeing only values..well live and learn
lol.
What racer is trying to say is that tangent space normal maps aren't just bump maps. They are a system to replace lowpoly normals with those from a highpoly model. That means not just adding small details onto surfaces, but replacing the shading behavior with that of a high poly model.
You can do this kind of hand painted stuff on flat, non smoothed surfaces, but personally I can model most of these things faster than someone can hand paint a complicated tiling map. Bakes also have the advantage of being able to make volume based accessory maps from the highpoly geo.
Yeah, it's really a non-character-only trick, for walls, floors, etc. As for speed, it's not as slow as it looks once you wrap your head around it--that fitted stone image I posted only took half a workday or so to paint--and I wasn't one of the painters who had been practicing it all the time.
BUT the biggest reason you might want to do these hand painted shenanigans, and I forgot to mention it above, is when you already HAVE a diffuse and you need to create a normal map to fit it. Trying to sculpt/model to a boss-approved texture like that is a nightmare...you'd be better off starting from scratch and getting your normal map first-a la the traditional methods...But do that and you have a 'sculpted' look, not a 'painted' one, no matter how good a painter you are.
But yeah, couldn't agree more: it's got limited applications
i was hoping he was having a knowledge-gasm, but if you insist that he is trying to say things that everybody allready knows i appreciate that he's being brief about it...unless,ofcourse, you've read too much into his two syllables
edit: i hope you do understand that this thread is about a trick for handpainting normals, not about handpainting normals vs highres and bake
i agree, never thought or said otherwise, with the ammendment that it can be done on flat surfaces be they smoothed or not ( unless i missunderstood what you mean by smoothed )
i'd put a really big emphasis on "most" if you want to keep that statement believable. There are plenty of situations where handpainting is faster, otherwise nobody would use it (mind you most people do not start from scratch when they handpaint normals).
it's not an invalid method if it looks ok, and you can go ahead and do this--as long as you understand why it's completely incorrect.
Half a day could get you a very nice sculpt of said floor.
edit: er...make that FIVE years ago. daaang
yes but your whole texture should be based on 1 map, it should be your diffuse, spec, gloss and normals working together, and currently the best way to get all of that is a sculpt to genrate some really great maps to start with, and mathematically correct normalmap that you cant get by hand.
also i have actually sculpted details to fit a existing diffuse before, not that hard.
sculpting also sets you up with good content to start the spec, diffuse and gloss with, since you got AO, cavity and curvature and anything else you can dream up that is worth extracting from the highpoly
I would say if you are having to hand paint normals, your workflow will ALWAYS be slower, and lower quality. Baking the map is going to give you better results every time, no matter what you may think you know about them.
Were my zbrush skills leaving something to be desired? Definitely.
But I've still not seen a single zbrush sculpt--from ANY zbrusher--that looks as painterly as what the boss-people wanted--which was really just some simple light/shading information that didn't compete with the painted look we were going for...
This looks like a case of putting up chamfers, edge loops and all kinds of extra verts to compensate for the crazy shading issue certain mesh might give you, which in 90* of the studios, you won't be allowed. If they were so hell bent of getting rid of every extra edge in a game like BF3, I don't see how something like this in an MMO would fly.
I get it, this technique works, Jessica Dinh did it themselves here some time ago on PC, it was a nice fun little romp, but how much extra work is going in the mesh itself is what bothers me.
Also, you guys already mentioned this technique isn't accurate and goes more hand in hand with stylized artwork, yet for newbies, if they stumble upon this thread, they will assume it's a required technique to must learn, especially in all the noise text so far which said text will get lost.
I'm pretty knowing how to sculpt is more important then knowing how to paint normals, since many in here seem on guard about that issue.
Lastly, I would like to know why some of you guys are having 'issues' with generating said normal's from a diffuse, last I checked, unless you have a highly compressed and lossy format, CB, NDO nor NV should giving your halo's or artifacts.
I guess I get touchy about it having come off of a project that was relying on it (an MMO with a TON of texture reuse, btw)...to dismiss it outright is really weird to me, since our industry is still, at least currently, an industry of tricks.
If you have a 3x3 pixel image, with a single white pixel surrounded by black ones, the nvidia filter has no idea what to do with that--you get a halo of angles of a single pixel all the way around the image. At least you used to. Now I've got to go check. At the very least, it won't match the diffuse below the way you probably want.
Anyway, this isn't a big deal when you're talking noisy photoreal 2048s, but when you're talking about a 32x32 texture...with very obvious brushstrokes..it's pretty ugly looking.
Yeah...this is the way I would have probably done it if I'd been starting from scratch. It's been a long time, so the details are fuzzy..but that style card was the kicker one for the bosses, if I recall correctly.
Anyway, I actually could talk all day about this stuff..I love these sorts of oddball workflows...but I feel like I'm stealing the thread too much. To sum up my opinion:
Is this useful? Yes.
Is it proper? No.
Is it fun/fast? Yes.
Is it a good way to get a team of concept artists, illustrators, and old school diffuse-painters to make normal maps quickly in a way they understand? Yep!
While I agree in very general terms that painting normals like this isn't really a workflow that I would suggest people do, he does bring up a few valid points.
Generating a bump map from a height map will never match as accurately as a hand painted normal map custom fitted to your diffuse, or idealy a baked normal map that matches your ao and diffuse maps.
Check it out:
bump map converted to normal with nvidia plugin. The nature of converting a bump map to a normal map requires a certain "padding" as the height content is re-sampled into normal vectors. You need a minimum amount of pixels to define a change in direction, so you will see the normal map grow and shrink with certain types of content(like the circles here) in relation to the bump/diffuse map. IE: the halo type effects he is referring to.
Now how much this would realistically effect your assets... Thats up to debate, but it is technically correct.
Can you honestly say that sculpting a brick wall to match a photo ( like this one http://www.publicdomainpictures.net/pictures/20000/velka/brick-wall-2961297319229Y2r.jpg ) is faster than :
- generating base normals from the diffuse
- handpainting to correct errors due to the lighting in the photo
- handpainting to add slant and protrusion variation ?
This thread was intended to be a place to share tricks for handpainting normal maps, not for "this is the best way" arguments
Also, I'm curious as to how well a post-app normal vs. hand-painted fares in terms of interpretation. I say take this deeper, and maybe even create a section of polycount-wiki dedicated to it (as I previously said, edge chamfers, hand vs. post, etc).
just wanted to quote that for emphasis
I worked on an MMO where we found the 100% modeled & baked normal map workflow was taking way too long - so we started encorporated more crazybump & an in house tool that was pretty much like nDo.
Yeah, I tried my best to communicate to the various texture painters I was supporting at 38 that this was a VERY particular workflow to the project, and that we adopted it due to style, pipeline order-of-operations, and, similar to Justin's situation, speed. (it's faster and easier than it looks at first glance...honest!).
As for geo accommodations, ace_angel, we actually didn't do all that much...some extra geo for hard-versus-soft edges (which are normally *also* a normal-map nono) but our budgets didn't allow for much more in the way of verts OR texture memory.
Anyway, I'm with you Ace. I think it's worth chatting about this stuff... If enough people are interested I'd be happy to make a transfer to the wiki (at least with the stuff I've contributed here). Fast iteration environment art can frequently involve blends of old school and new school techniques like this.
Must have missed this earlier...but:
Well...you CAN guesstimate these colors...if accuracy is less important than style...and you keep it simple...and you're not making something toooo fitted to specific existing geometry...like, a two way tiling terrain texture, for example.
With all that considered, guessing is pretty easy...but, again:
1: We avoided mixing baked normal maps with a painted ones, which would be a one way ticket to crazytown (since you'd have to paint 'relative' to an existing pixel's 'angle').
2: While there was some wiggle room, you'd run into real problems trying to do anything too undulate/doubleback-y. These are the problems Alec/Racer/etc were talking about, I believe.
The pipeline really worked best with very 'relief' driven textures: tiling maps, terrain textures, walls, etc....stuff to be applied to mostly flat things. It really boiled down to being a replacement for crazybump/thenvidiafilter with sharper/painterly-er/stylized-y-er results.
As for normalization on textures, so long as each sheet was self-consistent, we could adjust in game (which, yes, introduces more inaccuracies, but angle/strength dials in a material are super common..and a much smaller crime, imho). I was worried a lot about this self-consistency issue, but the painters got surprisingly good at this...a keen eye could identify lighting/normalmap errors where the wrong color had been picked, but you really had to hunt for them.
Being quasi-tech-minded, it took me a long time not to flinch when I thought about this pipeline. It does allow for a lot of artist-introduced errors...but it worked pretty well for our needs.
So that said, for anyone browsing across this thread in the future, what the naysayers are saying is being said for good reason.
Oh! And nDo didn't exist yet when we started down this road...
Later during dev we did look into nDo (v1..or was it beta...?). I've yet to tinker with nDo2, so I couldn't tell you if it outweighs the results we ended up with this method, but I've heard nothing but awesomeness about the tool.
Although in both cases, they should even out. Painting by hand manually, will require you to eyeball and sample the color, while with nDo2 automatically, you will need to simply wrestle with the UI sometimes and all of it's options.
Both should take you roughly the same time, warts and all.
i've actually tried to get results like that but couldn't in a timely fashion due to the sculpt mode being a realtime preview of what the heightmap-to-normal generation will spit out. if ndo2 can do shapes like that and my understanding of what the sculpt mode does is totally wrong, then yay
please do correct me!
And because the diffuse maps looked so painterly it meant we could fall back on them alone initially (or for uber low spec machines) and paint the normals up only when/where we needed, as we had time. So that was a plus. I mean, you could do that with crazybump normalmaps too, I suppose.
I've gotten a few requests for a tutorial on how to paint up things this way...so maybe I'll draft something up for the wiki. (with the caveats emphasized, of course)