Sculpt to Hand-painted workflow ONCE AND FOR ALL!!!

Jakob Gavelli

Heya! 
I've been struggling with this workflow for ages, going from a high-poly sculpt and ending up with what looks like a hand-painted texture. Biggest inspirations being Faf and Orb.  

So this thread is basically me trying to figure this shit out once and for all. My prior attempts all ended up with me just hand-painting over 90% of the bakes. There simply has to be a better way than that!
Latest failed try : http://polycount.com/discussion/152423/iron-horde-handpainted-banner/p1

I've read the Vertex magazines over and over, seen a bunch of breakdowns from Blubber Busters, Pior and PixelB on how to composite the bakes etc, but something just isn't turning out right. 

I guess there's no getting around having to have a good sculpt to begin with, but here's what I'm using for this exercise. I shamelessly ripped the idea of the metal greentooth + rock floor from PixelB. 



I'd like to avoid using MatCaps in order to be able to replicate the results on a baked mesh and not be limited to tileable textures with the workflow. I'd also like to introduce painting onto the texture as late into the process as possible. 

Any help on the subject would be greatly appreciated, I've been pulling my hair out for so long!  
If anyone wants to have a go at compositing these maps it would be awesome, I'll upload the original maps if needed.

EDIT : Attached new files to this post. 

pixelb

Oh I am SO in on this. Upload the files and I'll see what I can do with them.

I'd like to avoid using MatCaps in order to be able to replicate the results on a baked mesh and not be limited to tileable textures with the workflow. 

Hadn't thought of that but it's a good point.  Still, you can sort of bake matcaps to models. Pior does it with his dota stuff and I do it sometimes to get metallic reflections on tricky surfaces.

Jakob Gavelli

Still, you can sort of bake matcaps to models. Pior does it with his dota stuff and I do it sometimes to get metallic reflections on tricky surfaces.

Oh, snap. I must've tried more than a year ago but ran into so many walls when trying to bake matcaps. Like having a UVed highpoly and stuff? I don't even know where to start with that. I remember seeing something about projecting the matcap onto the sculpt, or something along those lines. Any details on how to actually do that would be great ^^

I uploaded the maps! I also threw in a heightmap for good measure, aswell as a seperate download for the .psd incase anyone wants to see how bad I am at organizing those!

VanLogan

Jakob, I think this is what you are looking for to bake matcaps http://www.zbrushcentral.com/showthread.php?170886-MatCap-Baker!-unofficial-Information-Installation

Fwap

Hey dude, so i gave this a burl. I ended up just using your PSD as it was relatively close.
Just a few things i noticed from your initial attempt, the cavity map you where using wasn't doing the texture much justice, so i ran your normal through nDo and got a diffuse map, and replaced your cavity with that.
Then really all i did was bump the values in your gradient maps, then did a little bit of adjustment layers and finally a really crumy hand-paint pass (my wacom driver isn't working so i don't have pressure sensitivity, so i made do with what i could) Not sure if this is going to help you at all, but here's the PSD if you wanna have a gander: https://drive.google.com/file/d/0BzSs96v1FEPFNDZkcG5KUzZfTzQ/view?usp=sharing

pixelb

That's a nice one, FWAP.

VanLogan said:

Jakob, I think this is what you are looking for to bake matcaps http://www.zbrushcentral.com/showthread.php?170886-MatCap-Baker!-unofficial-Information-Installation

That's really interesting, but it's not what I was thinking of. I was thinking more like https://youtu.be/HNaxMpDpOiw
But the way I bake matcaps is different still. I'll post something on it later tonight.

Jakob Gavelli

@Vanlogan : Looks advanced ^^ I think I've seen that one before.
@FWAP : Thanks for joining me on this quest! I deliberately turned down the cavity alot, I don't think it looks very handpainted to have strong cavity. Unless it's for material definition on metallic surfaces maybe. Thanks for the file, I've looked through it and I like the tweaks to the values! I have a nasty habit of working with too dark values.
@Pixelb : That's the one! I'll have to take a look at that, maybe learn how to do proper MatCaps aswell.. I'm looking forward to see what you do with the maps!

I can't but start feeling that maybe the sculpt is the problem. No matter how I twist and turn it looks very sculpty in the end.
I started from scratch and gave it another go. Here's something I found out about how to composite the top/side light from the normal map.
If I use a group with Soft Light, and use a 50% black Lighten layer to remove the dark parts of the map it helps alot. Here's a picture of the old vs the new way of doing topdown lighting.


Maybe it would be helpful to take a handpainted rock/metal texture and try to replicate its values 1 to 1. 

tomenjerry

Hey! Glad I found this! I've been trying to figure this out for a long long time. I studied faf's tutorial in vertex, but (sorry faf) after the tutorial a lot of things were still unclear to me. I think that the texture quality tepends havilly on the quality of the sculpt. If you look at the vertex tutorial by fanny, and compare the before and after, it seems like not much has changed (maybe that's the result of low res images). In the same tutorial she uses a lot of diferent layers to define color, but I think gradient maps are a much better way to add color, but that's personal pref. of course. It to make ages to figure out how to make descent looking rock alphas, but I must say, they help a whole lot. Reusing alphas that looks pretty similar really get's rid of the randomness. If you look at the topright corner of your sculpt you can still see some strokes ( from a falten brush, I believe). Maybe getting rid of that, and sharpening your rock layers will help further. I included some pictures of my attempts at making (I'm thinking of this new term) faf-style rock layer alphas. Since I started using trim dynamic, and especially trim smooth border, it was a lot easier to get clean looking lines (she might forgot to mention that brush in the tutosial, because I can't make clean edges with the flattening brush). Wow this message turned out really long, srry XD.



I think this'll turn out to be fun thread. Hope people like fanny will find this thread some day XD. This texturing thing is one of the hardest things (I found) to find. 

Happy texture painting!!

pixelb

So. Here's how I bake my matcaps. 

I set up the scene in Blender with my lowpoly model with the settings below. I assign it an unlit material and add the normal map to the material. Then I add the mapcap sphere as a second texture and set it so it's mapped to "normals". I rotate the model so it's facing upwards, and then I bake a full render from it. No high-poly required!

The material setup:

Note, this only works for models that have a distinct "front" side to them. However you could, if you were feeling ambitious, bake matcap projections from several angles and composite them in photoshop.

This method also can also work to produce bakes similar to what's described in the link from @VanLogan. You would just apply the material to a flat plane instead of the lowpoly model. I tried it out, but the results are a lot mushier.

Still works great for tiling textures though. Check it out!

( P.S. I have no idea how to do this in Max/Maya, so don't ask.)

Jakob Gavelli

@tomenjerry  : You're probably right. Here I was, just rushing a sculpt to have something to test workflows on. I've come to realize that there's no shortcut, the sculpt has to hold up on it's own aswell. I made a bunch of new alphas and redid the top two bricks ( Didn't have time to touch the others I'm afraid. )
@PixelB : Wow, that looks really cool! No idea how to do that in Maya either, haha. How did you get the matcap into Blender? Or does blender use a similar system to Zbrush when it comes to matcaps? 
If anything I think I'll try using some matcaps from screengrabs in Zbrush, and later use the projection-technique for 3d meshes, If I can't get a good workflow going without matcaps that is

So I took my own advice and picked a texture from Faf and just tried to recreate the values and color using my bakes. Without any colorpicking or cheating. ^^ I simply tried to emulate the style and come as close and I could.  I can now see that maybe I didn't do a great job of nailing the values but, hey, it's progress! I also came to the realization that the two bottom old bricks will never look good, no matter what composite magic I do. So far so good!



I'm attaching the new bakes aswell!

Jakob Gavelli

I did another try with references from pure classic handpainted textures and tried to mimic the look a bit closer.
I boiled it down to broader strokes and less cavity and edges on the rough stones. I did this by removing alot of the cavity from the rock, and using the Paint Daubs filter on the cavity maps to get illusion of broad painted strokes.

Once again we see the difference between the top two stone slabs and the bottom ones. Good vs bad sculpt.

pixelb

I wrote a response to this last week and polycount ate it, twice. Something to do with posting attachments?

Anyway, @Jacob, your third attempt is killer. Matching the reference was a really good call. #4 is good as well, but you could give the directional light a stronger color and work more color variation into the stone.

Your second attempt, and especially the first, were too low-contast. Be careful when you use gradient maps, they're flattening out the values of your greyscale texture. You can see exactly how much if you go into proof mode and turn the gradient maps on and off. Setting them to color blending mode instead of normal mode helps here.

Here's my attempt. (Download psd)

-I used a diffuse map from nDo instead of the standard AO because I wanted crisper details, but normally I don't use diffuse maps because I like having my cavity and AO separate.

-When I'm setting up my gradient maps I use color mode instead of standard because I don't want my values to move around. Even so, color layers can sometimes modify the underlying values so I'm toggling in and out of poof mode while I work to check.

-Looking at it now, the metal could be much more metal-ly. I didn't use a matcap bake but one could have helped here.

-I think my first attempt was stronger. It had less depth but more reflected light.

How did you get the matcap into Blender? Or does blender use a similar system to Zbrush when it comes to matcaps?

Blender does have a matcap render mode, but that's not what I'm using when I bake. I get my matcap spheres by screengrabbing a pic of a subdivided sphere with matcap applied from either Blender or ZBrush, and then clean it up in photoshop and plug into my Blender material.

stkopp

pixelb said:

So. Here's how I bake my matcaps. 

I set up the scene in Blender with my lowpoly model with the settings below. I assign it an unlit material and add the normal map to the material. Then I add the mapcap sphere as a second texture and set it so it's mapped to "normals". I rotate the model so it's facing upwards, and then I bake a full render from it. No high-poly required!

The material setup:

Note, this only works for models that have a distinct "front" side to them. However you could, if you were feeling ambitious, bake matcap projections from several angles and composite them in photoshop.

This method also can also work to produce bakes similar to what's described in the link from @VanLogan. You would just apply the material to a flat plane instead of the lowpoly model. I tried it out, but the results are a lot mushier.

Still works great for tiling textures though. Check it out!

( P.S. I have no idea how to do this in Max/Maya, so don't ask.)

You'd better use node setup. It is more flexible.

pixelb

You'd better use node setup. It is more flexible.

How so?

Stromberg90

I've been working on a node setup in substance designer, to make a hand-painted map from a height and normal map.
Tried it with your maps then worked some on it, think it turned out ok, some parts, like the highlights are a bit underdeveloped right now.

Spoon

wow, thats a great substance!

Jakob Gavelli

!!!!RISE FROM THE DEAD!!!!
Holyshit it's already been a year what the fuck. It's time to go 3d with this shit! I wasn't really going to show this stuff to anyone, so I'm sorry for the messyness! Lets get this shit show on the road!

I just happened to be sculpting some random skull yesterday and figured I'd give this another bash. The understanding I got from this thread a year ago was that what really matters is the SCULPT. Like seen in the earlier examples it doesn't really matter what you do in PS, unless you re-paint the entire texture, if the sculpt isn't clean or looks good.

So here's the sculpt : 

And here's what I came up with, with minimum actual painting: 
Sitting on 220 tris and 256x256 resolution. ( Bad bake, hence the warping. I'm lazy like that)

Looking at it now I could've been more aggressive with the saturation. Without the " no handpainting " limitation the next step would be to add more color variation and adding subtle strokes and fixing up some of the darker areas that look very desaturated.

So there you go. The .zip contains the UVed low-poly model and AO, normals and bent-normals, same as before.
Disclaimer : I'm doing this for educational purposes and it'd be great if the files are used as such.

EDIT : Updated the kit with proper shading and edge padding!!!

Jakob Gavelli

@Erik Rading : Thanks man! Looking good! With a proper cage the bakes would look a lot finer, haha. I fixed the silly seams and added the much needed edge padding to the kit, so others won't have to go through the mess you did

Tectonic

what are you guys doing to get the lighting to come diagonally from the top left? are you combining the green channel of the normal map with another one? I'm able to get a similar result but it involves 6 photoshop layers with the red and green channels set to different blending modes.

pixelb

@Jakob Gavelli Nice, very nice. It's missing the final pass of highlights and cavities that would make it look really Faffish, but as it is it would sit just right as a base texture in a Darksiders-type game.

@Eric. Nice, though I would've gone with less-saturated shadow colors and a stronger specular reflection. I'm glad you mentioned the thing about normalmapped hard edges.

@Tectonic Speaking for myself it's because I'm using lighting from a matcap sphere, not normal map channels.

Jakob Gavelli

@Tectonic : Same principle as the earlier post. I have a Group set to Soft Light - Inside the group I have the green-channel and a 50% black layer with Lighten on. I then control the darkness of the top-light shadows by changing the opacity of the Lighten layer. And the overall strength of the top-light by changing the Groups opacity.
This is how I do it anyways, super simple and gives alot of control : 
On 3d object I bake a bent normal-map to do this, instead of taking the green channel from the tangent space normal.

@pixelb : Thanks! I'm trying to step away from cavity, it's just feels so over-used, so it's very subtle here but maybe I took it a bit too far! I do agree that it lacks alot of contrast looking back at it!  
More stupid funsies with the bakes, haha, there's a fine line between "Color Variation" and "Clown Face" : 

pixelb

Yes! You've got it! Now grab it and run!!

Tectonic

I did this on a lunch break. I think it might have turned out better if I had chosen a brick pattern with larger/less bricks.

pixelb

I did a quickie. Made two matcap bakes based on Darksiders color scheme- one blue, one yellow. Used one for the skull and the other for the teeth + eyes, then spent 15 min loosely painting over the results.

Jakob Gavelli

OOoooh nice! Damn these bad bakes are going to haunt me.
Very cool working with matcaps like that, looks very cool!
I need to investigate further how to do this in Maya!

wirrexx

One of the threads that should live forever. my try! working hard to achieve that stylized style. Wow, the sculpt does a lot of the work! but i\m understanding the workflow better and better. 
Gradient maps and not even 1% hand painted. 

riceart

Hi, here is my take on this.

@edit, Tried with floor texture. I used faf work as reference

wirrexx

riceart said:

Hi, here is my take on this.

@edit, Tried with floor texture. I used faf work as reference

Nice, how do you get your edge to pop out that Good?

Jakob Gavelli

Oh shit! Nice revive @wirrexx haha, thanks for taking your time to play around with the bakes!
Yeah, one of the biggest takeaways from this thread is that you can't run a mediocre sculpt through gradient maps and a nice stack in photoshop and expect good results. I was looking at tutorials from Faf and Orb etc a couple of years ago and thought "Man, that looks easy! Just a bunch of bakes in photoshop on the correct blend-mode!" NO! Wrong Past-Me! It actually takes a lot of sculpting effort and skill, aswell as a lot of touching up in photoshop, handpainting over details and creating lighting and gradients that you can't get from a sculpt only.

I'm going to shamelessly direct you do this thread if you're interested in more specific material-definitiony stuff : http://polycount.com/discussion/191774/hand-painted-sculpt-defining-materials#latest

Here's one of the loose breakdowns from one of the materials going sculpted > handpainted : 

@riceart : Really nice! I like that there's not too much Cavity coming through there, my only tiny nitpick would be to maybe lighten it up a bit so that the dark areas aren't aaas black, but that's just me haha, really good, thanks for taking your time and try stuff out!

riceart

@wirrexx ive overlayed cavity map twice, one bigger softer, and one smaller sharper.
@Jakob Gavelli yeah but its just a matter of contrast adjustment and some paintover, as for this new wood study, i will write in other thread