Baking selection masks in xnormal?

AlecMoody

Does anyone have a reliable workflow for baking selection masks (based on the highpoly) in xnormal? I tried assigning vertex colors to my highpoly but so far I haven't been able to get them to import correctly. Very highpoly meshes seem to have difficulty with the SBM exporter from max and enabling vertex colors either doesn't work for or breaks other formats.

I am just looking to assign a handful of solid colors to differentiate material types.'

Also, im working from max to xnormal.

Next

1. apply a vertex paint modifier paint the object.
2. export with sbm and check export vertexcolors
3. uncheck ignore vertexcolor thingy at the highpoly in xnormal.
3. bake the vertex colors!
should work atleast does for me!

edit: vertex color modifier should always be applied before any other modifiers like turbosmooth and so on...

passerby

exactly what next said, i do the same minus the vertex paint modifier bit since i use maya.

AlecMoody

Next- what kind of triangle counts are on your highpoly models? When I do this with a big mesh (decimated sculpts especially) the workflow breaks.

Next

mhh... out of my head i know i had once one piece with 350k.. but then again most time the stuff is always around 20k and then gets turbo....
but if max cant handle it which could be.....maybe use zbrush polypaint and fill object and then export as an obj and use this to bake.....

passerby

never tried it with sculpts, mostly just manually applying vertex colour to the cage on subd meshes.

are you using multiple subtools, there might be a way to colour each subtool, with material iD's or somethign like that.

cptSwing

ToTex has selection mask baking as well, if i remember correctly, using poly selections? And i've baked down vertex colors from a turbosmoothed mesh (VertexPaint modifier on top of the stack) with xnormal, must have been at least a million tris.

Vrav

Could always throw diffuse textures on your highpoly and bake base texture. Has worked for me before. Even if it's too much of a mess to unwrap, pretty sure a solid colour on a mesh element with nonsensical UVs (such as view projected or whatever) would still work.

Benefit of doing them all at once is you get anti-aliasing and one layer. If you can toggle elements of your highpoly (example, bolt heads as the only high mesh showing), turn off edge padding and render base texture at a res one step higher than what you need, scale down in PS, nice anti-aliased selection mask for selecting a colour range in or somesuch.

I also like to bake an aliased selection mask of the (untextured) lowpoly to itself for UV island selection, using bake base texture. Just another use for the option in xNormal. Hope this post is relevant to the original question.

Synthesizer

Vrav wrote: »

Could always throw diffuse textures on your highpoly and bake base texture. Has worked for me before. Even if it's too much of a mess to unwrap, pretty sure a solid colour on a mesh element with nonsensical UVs (such as view projected or whatever) would still work.

That's usually what I do. I have a texture with several solid squares of colour, and I just do a planar projection for each piece I want separated. Don't worry about overlapping UVs, just scale that projection down until it fits in a colour square.

FBX should be able to do vertex colour though, have you tried that?

Thane-

I've got an exploded high poly mesh in 3ds max with tons of different pieces, most having a separate material applied to them via the material editor. They are all individuals objects i export by selecting all and exporting to one file, which renders the other maps fine. When exported using fbx or sbm, xnormal only renders out the background color very quickly when doing a vertex color bake. Do i need to make a single multi-subobject material and apply materials ids for each one to get color the export or is there an easier way?

I know for Substance Painter to make a color map i had to apply material ids. But if its just a simple unique color for each object, maybe an easier way?

edit: Just fyi, i wanted to bake the id map in Xnormal because when i bake the colors in max the color for floating geometry is slightly off and i assume the projection skews differently between max and xnormal.

Bartalon

Colored materials applied to a model is not the same as vertex color data. Unless you baked the colors into the vertices as vertex color data.

Assuming you have vertex colors present in your model, make sure you have "Ignore per-vertex-color" unchecked in your High Poly tab in xNormal.

Keep in mind that this setting will interfere with some other bakes, such as an AO which requires that option to be checked.

If you don't actually have vertex color data, you should be able to bake out a color map from your materials right inside Max.

Alternatively you can flood the vertices inside ZBrush with polypaint data if you haven't changed the positioning between ZBrush and Max.

Thane-

I added this bit of info just as you were posting: "edit: Just fyi, i wanted to bake the id map in Xnormal because when i bake the colors in max the color for floating geometry is slightly off and i assume the projection skews differently between max and xnormal."

I did try in max but it didn't quite match up to the normal map i baked in xnormal so i wanted to try baking in xnormal.


EDIT: NEVERMIND, i fixed it (apparently...) using the "assign vertex color" function under utilities in max, I had tried the vertex color modifier but probably didn't use it correctly.

Neox

AlecMoody wrote: »

Next- what kind of triangle counts are on your highpoly models? When I do this with a big mesh (decimated sculpts especially) the workflow breaks.

just use zbrush

Bartalon

Neox wrote: »

just use zbrush

A response three years too late :poly142:

Neox

Bartalon wrote: »

A response three years too late :poly142:

haha welli didn't dig it up

still true!

so it might help someone in 3 years

:poly142:

Thane-

"haha welli didn't dig it up"

I just didn't want to create a whole new thread for something thought would be a simple answer.