Matching diffuse/spec to normal maps 1:1 100%

odium

What I mean is, what are your methods of working with diffuse maps, when you come to match them up to the normals? Obviously normal maps are made from higher poly models, mapped onto the lower poly to fake the geometry, but what are your methods of getting, say, a trim on the normal map, to match up to the detail in the diffuse?

ATM, I'm simply using the local map as a seperate layer in photoshop, painting over the top, so I can see if it matches up. I've also been thinking about ambient occlusion to fake a little shadowing.

Anybody have any extra tips to get things looking spot on?

EarthQuake

In max you can simply throw colors on different pieces of your highres, and then bake a diffuse map as well. Then select color in photoshop to set up layer mask. In modo i'll just take an image, give it 4 colors, and just planar map certain sections to each 1/4th of the texture, and then render out a diffuse map in Xnormal.

Joshua Stubbles

Both of your ideas together makes a decent solution.
By applying the colors as EQ said, you have ready made masks, and can easily see what goes where.
Using an AO/crevice map is helpful as well. Not only does it actually add a little shadow to your texture, but it will certainly show you where the high poly elements are, in relation to your low poly UV's. Painting over the normal map as you're already doing, odium, is also good.

MoP

I always use an AO map on top of any local map generated from high-poly geometry.
In Max I always bake different "material" sections as different colours like EarthQuake said - as contrasting as possible so you can use them as selections in PS. Just assign different materials with different diffuse colours then add a DiffuseMap to the baking set, make sure it's not baking lighting and shadows on the same map, render out the lighting map separately.

Also it's handy to generate a Shadows/Displacement map from Crazybump by loading in the local map and setting the slider to 95 or so, just to pick out the high and low edges. Then overlay this on top of your specular to brighten the sharp edges.
I tend to only flick on and off the Local group (or layer, whatever) to check that everything lines up now and then, rather than keeping it over on a low opacity.
Also, I always always use layer masks for material types - easy to create clipping masks for sub-layers, and copy masks between diffuse/spec too.

Edit: Ryan agrees with me I tend not to slide it up to 99 because that causes banding artifacts. 95-98 is a good value for nice sharp highlights/shadows.

Ryan Clark

<shameless plug> Load your normals in CrazyBump, and render shadows with the sharpen slider around 95. Copy the output into Photoshop and paint over it! </shameless plug>

edit: I guess my shameless plug was unnecessary. MoP beat me to it

fritz

i just desaturate normal map....contrast them a bit.....set as a top layer and bring the opacity of that layer down to almost nothing.

EarthQuake

oh yeah, forgot the AO part, always have ao on top of my diffuse, that makes it easy to know where stuff lines up.

arshlevon

not exactly on topic, but slightly related.
if you take a copy of your color layer, and overlay that on top of your ao layer it will kill all the black in the ao layer, then collapse the 2 layers, and now you have what looks like a colorful ao layer, multiply this over the color layer instead of just an ao(might have to play with opacity a bit). looks much better, because your just adding color to your texture and not nasty black that usually makes your texture just look dirty or smokey. i also juice the saturation on this a bit...

MoP

Good point Arsh - a desaturated AO can give unwanted results - what I've done quite often is to use an inverted AO bake as a layer mask for a colour layer multiplied over the diffuse/spec - that way I can tweak the colours/saturation without losing the contrast and accuracy of the AO bake. That way you don't lose any info by collapsing down.

poopinmymouth

You can also color balance your AO layer. Hit Ctrl+b, and then click shadows, add colors till it looks how you want, click highlights, and go for an opposite. Arsh's way is nicer, but this is a good quick/dirty way too.

Ghostscape

I'll just take the AO layer, invert it, and use it as a mask for adding some color to my shadows. By adjusting the curves of the AO map you can use it as a nice mask for dirt and grit.

Ruz

I tend to have the ao map plus cavity map overlayed on my color map, whilst i paint/smudge around the colour layer underneath.
obviously then I get the colours right whilst visualizing with the detail on the layer above
I also colourise the ao/cavity map so its a more or less a fleshy colour.
as an added bonus I darken the blue channel of my tangent space normal map and overlay as a multiply layer for fine detail on ears/nostrils

JordanW

I use heightmaps quite a bit, they can be helpful for helping to change hues and shit in deep areas, or messing with it to help pop out some stuff that sticks up.

I've tried running a find edges filter on normal maps before then desaturating it, it makes a kind of edge mask that can "sometimes" be useful.

Often times i'll render out some passes on the lowpoly as well. Like map a gradient to another uv channel so that I have a B&W mask going from the bottom of the object to the top, or from the center out.

Mental ray also has some neat things you can plug into the diffuse chan of an object before you render, there's one called landscape that can let you mask out parts facing up.

Flipping the normals and rendering an AO map with the distance limited can give a sharp mask sometimes for edges of a model.