Baking out Diffuse maps (for quick selecting in Photoshop)

tristamus

Hey there guys!!

So, I have recently been studying hardcore about everything texture creation (especially for environment art)...

I was wondering, especially after watching the Eat3D tutorials "Old Pillar" and "Next-gen Texturing," how exactly he goes about getting that very useful flat color map that he brings into photoshop and uses to make selections with the wand?

I use Maya for my geometry creation needs in general, but I'm really not sure how to bake out just the color map via Maya for use in Photoshop, so that when I get to the texturing phase I can make selections on certain areas of the texture map way easier?

It is so strange, because I can create every other map wonderfully, yet something so simple as just a color map, I still don't know....I am willing to bet it's stupid simple to bake out one of these, but in my search online, I have not been able to find out how!

I know that you must assign different materials with different colors to whatever you want / need to on the geometry itself, it's just the "getting that to the texture map" part that I cannot fully understand.

For further explanation, I plan to use the base colors of this baked out color map in tandem with my AO etc maps...

I hope somebody can just point me in the right direction! Thanks guys

rasmus

If you've assigned various colored materials to your hipoly, it should just be a matter of doing a Diffuse-bake along with all the other map types in Transfer Maps.

tristamus

Alright, so via Transfer Maps, using the diffuse option. That's all there is to it?

renderhjs

since you are using maya and not max this might be useless to you, but in TexTools there are several texture maps to bake directly in the clipboard exactly for that kind of workflow. I often use a blockout map

To figure out which 3d piece sits where in the texture, and of course AO, lightraced or other dirt alike maps to get more depth into the texture.

I wish that those things would become standards in most 3d packages, baking special purpose maps on the fly directly into the clipboard.

tristamus

Yes! I agree, render, as I feel like that specific aspect is not as well thought out as it should be in some 3d packages.

Yes, a blockout map is exactly the concept I was getting at. I'll go and give TexTools a try. I have max 2011 as well as maya 2011 on my machine, so I'm sure I can give it a go.

SimonT

renderhjs is our hero

cw

SimonT wrote: »

renderhjs is our hero

hear hear!

onionhead_o

i would say to use convert to file texture in the hypershade editor. it should be faster than transfer maps. for some reason maya transfer maps is so slow. especially if u want 4k textures. even 2k is slow.

ZacD

In xsi there's a "matcolor" or material tag option that renders out solid colors based off the different materialed sections on the high poly.

Alphavader

Id like to reuse this thread.

Here a little question, ive done the uv so far.
Now in Cs i recently noticed that is probably good to have
a selection mask. Ive downloaded a nice plugin that put all my
selected objects into a different color (each into one surface shader
with color)
Id some issues to bake this mask in a texture. - in maya the texture is still black.
In xnormal the texture has only one color.
Can someone help me with this ? How would you bake a selection mask ?
(Maybe without adding tons of Lambert shaders and done the color manually)

- transfer map doesnt work cause of multiple objects and surface-shaders

Jeff Parrott

You'll need tons of lamberts and manually applying them (at least in Maya).

Alphavader

//EDIT:

Okay, my question was how to bake out a
selection map in maya for CSS.

Here is a great plugin

Under Rendering -- lightning and shading you can batchbake mental ray and bake the colors to a map.

EarthQuake

You don't need any special plugins, simply assign materials to your various highpoly meshes(and faces in some instances) with the following color values:
0,0,1
0,1,0
1,0,0
1,1,0
0,1,1
0,1,0
1,1,1

Bake a color/diffuse map from transfer maps

Then you can use a PS script to split them, Andy Davies (metalliandy) released a nice set of actions which you can find here.

Alphavader

Thanks, but this was my issue.. i have about 25 different
meshes and dont want to add values per hand - the plugin does it for me, by simple
clicking.

Ill try your method tomorrow morning..
thanks again

passerby

i really good way to do this, if your useing a maya to xnormal workflow, is to use vertex colour, since xn can directly bake that out, and it is easy to apply, in maya.

Joopson

Passerby, I've tried that a million times, but I've never gotten the vertex colours to be read by xNormal, no matter what format the mesh is. "Ignore High-poly vertex colours" is unchecked. Any ideas?

MM

am i missing something ?

xnormal had this feature for a long time now.
all you do is export as separate obj files and xnormal applies random colors to them.

Bek

Might as well also add the luxology Modo method for baking diffuse / colour / selection maps to this thread (since it was pointed out to me just recently):

Assign materials with solid colours (1, 0, 0 - 0, 1, 0 etc) to your objects (or specific polygons)
Go to the shader tree and click on Render > Properties tab, and set your frame width to your desired texture size
Go to the render menu and click Bake to Render Outputs. You might get better results by setting your output in the shader tree to Diffuse Coefficient rather than final colour.

passerby

Joopson wrote: »

Passerby, I've tried that a million times, but I've never gotten the vertex colours to be read by xNormal, no matter what format the mesh is. "Ignore High-poly vertex colours" is unchecked. Any ideas?

check the alpha, channel and make sure it is white, and try useing xnormals sbm format.

i do vertex colour bakes in xn from maya all the time, both for blockouts, and for baking curveature information i calculate in maya that is stored in vertex colour.

Ace-Angel

MM wrote: »

am i missing something ?

xnormal had this feature for a long time now.
all you do is export as separate obj files and xnormal applies random colors to them.

Most people don't know this because it's not documented in the latest help file.

Also, sometimes, your model will share the same mesh of different parts of the model (EI: Your helmet has a 3/4 different sub-objects which are defined as different material, people would like to apply different colors to represent said material, people can't do that, especially on a single mesh).

I honestly suggest if you want more refined control to simply used vertex colors to your advantage, especially if you're going after material definition (EI: Red = Wood, Blue = Metal, etc).

MM

Ace-Angel wrote: »

I honestly suggest if you want more refined control to simply used vertex colors to your advantage, especially if you're going after material definition (EI: Red = Wood, Blue = Metal, etc).

if i export all my highpoly separately according to material definition(which i always do anyways), wouldn't applying vertex color to highpoly become redundant ?

also, applying vertex color would mean importing a dense highpoly mesh back to maya/max is a hassle.

the base color u see here was baked in xnormal pretty fast. it is more than enough to create all the separate material base color masks.

cptSwing

Vertex colors are easy, yup. Either slap a VertexPaint (?) modifier on your highpoly, or use the less laggy option of setting up vertex colors in the Editable Poly rollout in your subD base object (both options are in Max). Export using xNormals SBM and check [x] Export Vertex Colors in the exporter popup.

With all of these options, there'll always be a little cleanup to do before you can really setup your masks (at least in my experience), but yeah in the long run it's a huge timesaver.

wirrexx

MM: care you make a tutorial, cause i've been searching for something similiar to this for a while!

MM

wirrexx wrote: »

MM: care you make a tutorial, cause i've been searching for something similiar to this for a while!

there is nothing to it really. all you have to do is export all your meshes or subtools from zbrush as separate obj files, source them as the highpoly in xnormal and then source your lowpoly which you want to bake onto and use the "bake base texture" feature in xnormal. make sure in the options of that feature you have "write object ID if no texture" checked.

wirrexx

cheers will try it out!

EarthQuake

MM wrote: »

there is nothing to it really. all you have to do is export all your meshes or subtools from zbrush as separate obj files, source them as the highpoly in xnormal and then source your lowpoly which you want to bake onto and use the "bake base texture" feature in xnormal. make sure in the options of that feature you have "write object ID if no texture" checked.

The biggest problem is see with this is sub-object selections. I virtually always have smaller polygonal selections that I want to mask with different materials, splitting your objects into multiple files wouldn't allow this.

If you're dealing mostly with really dense sculpts I could see how making these sort of selections and applying materials would be a pain though, and maybe polypaint or something would be a better option if you need more control.

Ace-Angel

Yeah, EQ said it, I meant sub-object of the current mesh are not bakeable as they would be via Max or Maya.

As to if the time taken it worth it or not...depends on the mesh honestly.

MM

EarthQuake wrote: »

The biggest problem is see with this is sub-object selections. I virtually always have smaller polygonal selections that I want to mask with different materials, splitting your objects into multiple files wouldn't allow this.

If you're dealing mostly with really dense sculpts I could see how making these sort of selections and applying materials would be a pain though, and maybe polypaint or something would be a better option if you need more control.

agreed. it all depends on how you plan ahead your highpoly mesh.

in some cases if you have small parts of your highpoly that are not different object initially you can either select those polygons, split them inside zbrush and export that or you can came quick vector masks inside photoshop based on the normal map. both of these work arounds take literally less than 5 minutes.

i dont remember spending more than 15 minutes extracting a complete color mask for all the elements of my mesh ever.

Jeff Parrott

Here's the same workflow for baking color maps in Maya:

[vv]55523169[/vv]

I can never apply the color as vertex color in Maya, export it (either as OBJ or FBX) to xNormal/Topogun, and get the diffuse map to bake correctly. Maya's OBJ exporter seems to not allow vertex color. The FBX exporter I'm not sure what's going on.

Froyok

I made a script for this, instead of trying to export the vertex colors, I try to directly export the vertex colors as a map from Maya. It's quite simple in fact, but there is one constraint : each uv shell must be separated on your mesh (aka : exploding your mesh in pieces), otherwise the vertex colors will bleed when you will export them.

I still have some bleeding sometimes, but nothing crazy.

Jeff Parrott

Ah good idea Froyok. Right now the aliasing between colors makes dDo think there are additional materials. Hopefully that will get fixed. There's no options with this in Maya. Does your method reduce that color blending at all?

pior

On a related note - anyone knows of a photoshop script or action taking a blockout color map as an input and spitting multiple layers from it, with each layer corresponding to each source color (with something like a +/- 10 color tolerance) with transparent pixels all around the islands ?

Now of course dDo can do that but I wonder if there is some kind of standalone script for that out there.

Thanks!

Ace-Angel

You mean this?
http://eat3d.com/forum/tips-tricks-and-free-videos/rgbcmyw-splitter-photoshop-actions

pior

Very cool! Will try it asap. Too bad that it requires "ugly" colors as an input (since there are cases where the blockout map is made from properly colored materials as opposed to random RGBCMYW colors) but still, it should surely make things faster in my case anyways. Coool...

Froyok

Jeff Parrott wrote: »

Ah good idea Froyok. Right now the aliasing between colors makes dDo think there are additional materials. Hopefully that will get fixed. There's no options with this in Maya. Does your method reduce that color blending at all?

There is not AA on my exported texture, so no bleeding on this side. The only bleeding I have is from the vertex colors. The script is not available in stand-alone, but you can find it in my froTools.

Here a little example :

bk3d

Ninja UV has a feature to bake out uv shells. It is shader based and there is an option to randomize the color or you can have it a specified color

Ninja Dojo Home Page
Ninja Dojo (Black Belt) on Creative Crash
Ninja Dojo (Grand Master) on Creative Crash
[ame="http://www.youtube.com/watch?v=FWrH_gAPCHY"]Ninja Dojo Demo Video on You Tube[/ame]

sculptn

Just on the ninja uv you would go into the bake options and edit the colour bake there is a randomize the color option there ?

Deadly Nightshade

MM wrote: »

agreed. it all depends on how you plan ahead your highpoly mesh.

in some cases if you have small parts of your highpoly that are not different object initially you can either select those polygons, split them inside zbrush and export that or you can came quick vector masks inside photoshop based on the normal map. both of these work arounds take literally less than 5 minutes.

i dont remember spending more than 15 minutes extracting a complete color mask for all the elements of my mesh ever.

That's the way I've always done it.
And with a bit of MEL you can fully automate it.

jibberishballr

What if you're working on a high poly model that has no low poly? Is there a simple way to bake out the colors? Can you just bake it to itself and use that map afterwards?

EDIT: Working in Maya

Chase

Well, I'm not familiar with Maya but I would think it'd be the same as doing it with the low poly where you'd apply material IDs to sections you want to be able to select separately

jibberishballr

Chase wrote: »

Well, I'm not familiar with Maya but I would think it'd be the same as doing it with the low poly where you'd apply material IDs to sections you want to be able to select separately

I'm a little confused on what you are saying....maybe my question was confusing...haha. If I apply a bunch of materials to a high poly object, let's say 6 different shaders of different colors, and want to bake that color map out (for use in ddo), but only have one high poly mesh (no low poly), how would I do that?

throttlekitty

Use Transfer Maps for that, and assign your high poly(s) as both source and target, then bake diffuse map.

If you're working in dDo, I question you only having a HP, since normal maps are a key factor in the magic, and you can't bake normals from a mesh to itself.

throttlekitty

Jeff Parrott wrote: »

Here's the same workflow for baking color maps in Maya:
-video snip-

I can never apply the color as vertex color in Maya, export it (either as OBJ or FBX) to xNormal/Topogun, and get the diffuse map to bake correctly. Maya's OBJ exporter seems to not allow vertex color. The FBX exporter I'm not sure what's going on.

obj doesn't support vertex colors, you'd need to use the xNormal .sbm 'porter to handle those.

jibberishballr

throttlekitty wrote: »

Use Transfer Maps for that, and assign your high poly(s) as both source and target, then bake diffuse map.

If you're working in dDo, I question you only having a HP, since normal maps are a key factor in the magic, and you can't bake normals from a mesh to itself.

That's what I was thinking...cool. I plan on using low and high poly for baking normals as that's my normal workflow, but have some high poly models that id like to tinker with in Ddo for diffuse, spec, and bump maps.

Deadly Nightshade

One thing I really DON'T like about Maya's tranfer maps though is bleeding. Even if you set envelope to 0 you get bleeding. This becomes a real pain in the ass when you have for example, one mesh that you want 2+ selections on. I don't want to have to select and separate those two (and later combine them and merge the verts again) as it's quite some work if you have many selections.

And the same problem arises with xNormal: You need to separate your selections, separating/extracting mesh pieces, exploding the mesh, exporting each individual part as OBJ. Why is this so damn hard anyway? It should all be automated!!!

Ace-Angel

^It's hard because no one used the technique furiously in the industry until much later on, so the tools in question aren't updated to show the new workflow.

Maybe in XN4, we will get this, but until then, exploding and moploding is the only solution.