Baking out vertex colors to a color ID map

DirtyBlueFL

Dear god, I am going insane. I have tried so many different ways to bake out a color ID map from this high poly. Maya's baking has never once produced a flawless bake of any sort of map, so I have taken everything into Xnormal, where baking is a pleasure.   I have painted my high poly with vertex colors, and exported them as FBX, yet when I bake the vertex colors in xnormal with low poly and cage, I get a white map. It spends time rending that area, but it's all white.

>I have "ignore per-vertex-color" UNCHECKED< 

I have no idea what's going on, as my FBX files retain their vertex color data when I bring them into a new Maya project and turn on the option to see said colors in the viewport. If you would like to take a look at the file, I have uploaded one of them here:
http://www.megafileupload.com/1qgsj/SuckerHiVerts.fbx

I have spent more than 15 hours alone on playing with baking settings across 3 renderers and waiting for failed bakes. I am dying. Please...

DirtyBlueFL

Solved it. 4 minutes after I made a post... All file types must be of .fbx format.
I'll leave this here for future people, as google has nothing on this.
As a novice, I have no idea why .obj low poly and cage would not serve as proper vertex data for a baker to pull vertex colors from an fbx, considering all its doing is translating color to a UV space.

pior

DirtyBlueFL said:

I have painted my high poly with vertex colors, and exported them as FBX...

For the record : you are overcomplicating the process as there is no need to rely on vertex colors to bake out an ID map. In order to bake such a map in Xnormal, export out parts of your highpoly as separate OBJ and FBX files and tick "Bake base texture" for output. Enable "Write Object ID if no texture" in the options. Xnormal will then apply random colors to each chunk thus resulting in exactly what you are looking for.

You might also want to transition over to using the Toolbag3 baker as it lets you interactively assign materials to the various chunks of your highpoly before baking, directly in a 3d environment. It is definitely the most advanced and user-friendly option available to this day.

DirtyBlueFL

So each part of that high poly would have to be split into a separate mesh and file for that to work?

For example, I have a high poly wheel, with tire. The whole thing is a cylinder with extrusions at various levels, to make the hubcap, rim, a ring for detail, and the rubber. Assigning faces their own material was how I was baking in Maya. To bake in xnormal, I swapped to paint vertex normals, all while still being one contiguous mesh. Because "write object ID" implies that its splatting a whole color for that specific file, I would need to break up each set of faces into its own file for this to work the same way, right? 

pior

Yeah - there is really no reason for two highpoly parts receiving two different materials (say, rubber and chrome) to be merged into one object in your authoring scene. So if you take the habit of always assigning materials to your highpoly components (which is something you should always do anyways for the sake of reviewing your work) then your objects will naturally be broken down per materials - thus making the export in multiple chunks for Xnormal trivial.

That said for that too Toolbag3 is a superior tool, because it can load an OBJ or FBX made of multiple sub models and lets you manipulate them in the 3d scene as needed, most remarkably allowing you to apply materials to them at will. Meaning that much less prior planning is needed, and no splitting into multiple OBJs or FBXs is necessary.

DirtyBlueFL

So just to make sure I understand:

In xnormal, split everything that would have a different ID into its own object. When exporting for xnormal, I'll have to make each individual piece its own file.

In TB3, I can just drag select the whole highpoly, with all submeshes, and export it all as one fbx or obj with groups preserved.

DirtyBlueFL

i did the thing.

Nam.Nguyen

@DirtyBlueFL   In TB3 you can bake ID map with materials,  it said "Material ID"  but you don't even need to apply matID to your object, just standard material, and voila . . . . no need to paint vertex colors