vertex colors go rainbow on convert to poly

Justin Meisse

Here's the deal, I have to do alot of vertex color light bakes and through various plugs-ins and whatnot I sometimes end up with an editable mesh with vertex colors on it. When I convert the mesh to an edit poly I get random vertices turning rainbow colors and other visual bugs.

Anyone know why this happens and how to prevent it?

Noors

Yeah had this problem too some years ago. I don't think i fixed it ^^
Bad point/edge conversion i suspect.
What if you make all edges of your mesh visible before converting ?

Will Faucher

Usually what I do is I export the mesh as OBJ, but I uncheck vertex colors at the export window. Then, just reimport. Should work. Not sure what causes this though.

Justin Meisse

Prophecies: wouldn't that just throw out all my color info?

Will Faucher

Ohh my bad, I didn't realize you wanted to keep the color info. Disregard what I said.

*edit*: Granted, you COULD try exporting the vert colors with the OBJ and reimport. Who knows, maybe it'll fix it.

Justin Meisse

hehe, an easier way to get rid of vertex color info is to go to tools > channel info, right click on vertex color, usually called 0:vc, and select clear.

Noors

Should i understand my suggestion failed ? It worked on a little test i did though.

FicWill

Given that you don't know what those other applications are going to do to your vertex colors or edge orders, it'd probably be simpler in the long run to save a version of your mesh in max with the vert colors you want, then write a script to re-create those values on your updated mesh based on proximity. That way you could 'bake' the vert colors back without having to worry about anything except vertex positions.

Justin Meisse

Noors: I'll try to work something up at home - it's a pretty big problem, I asked the entire art team at work and the only answers I got were "reset xforms" "box trick it" and "I don't know, I just always paint the incorrect verts"

FicWill: interesting but that still seems to be an over-complicated alternative to painting out the rainbow verts manually. also, I don't know how to script in max :P

Noors

Mind to post one of your object, or some faces ?
By making edges visible, i meant triangulate the mesh, with Surface Properties>Visible in edit mesh modifer, at edges level.

Now if you don't want your edit poly to be triangulated, you can propably select just the "invisible egdes" (found a script by Bobo which make that on scriptspot), make them "visible", convert to poly, delete the egdes (which could be maxscripted) edit : well no, deleting the diagonal edges in edit poly fucks the vertex color too. Thank you Autodesk !

double edit : got it, ok, haha, wtf !
I don't know if this will work for you but :
the vertex color seams to work like UV. You can display them in the unwrap by switching to "vertex color channel" When you delete an edge (or convert from mesh to poly i presume), the vertices goes funky because everything is weirdly split.

SO, select your vertex colored mesh, apply an unwrap, switch channel to "vertex color channel", select all the "UVs" and do a "weld selected". Weld twice if needed. Convert to poly. win !

You can totally edit vertex color like uvs, doing symmetry and stuf,f that's pretty fun.

Justin Meisse

Noors: wow, I think it works! It's kind of hard to intentionally make a mesh that generates rainbow colors but I do get issues with weird seems popping up when I convert it to editable poly, but the weld trick seemed to fix the issue. Thanks

Noors

Yeah i think you need really smooth gradient so the "bug" could be seen, ie, if you bake a lighting. I tried to simply paint manually on a sphere and the bug couldn't be seen.
I tried on a teapot and some turbosmoothed extruded box, but i hope it works on complex objects too. My concern is, doest it export correctly when the vertices are welded ? You can still break them afterwards though.
The triangulate thing surely work too anyway.

edit : made some tests, and a mesh can handle different color on 2 triangles separated by an invisible edge. An edit poly can't, if you don't triangulate manually.
In this case, if you don't triangulate the poly, there will be differences.

When you unwrap a mesh, you see the invisible edges in the uvs. You can split a face along an invisible edge. When you convert to a poly, and apply an unwrap, you see the invisbile edges cuts don't exist. As the triangles are all split in mesh, max tries to reconstruct ngons when you convert to poly and it makes some twists. That explains the differences. Logical once you know vertex color works like UV's.


So the ultimate solution is to triangulate, else the weld trick is the best approach afaik but may not completely work. Remapping a vertex color as FicWill suggest would be tricky as a vertex can have a different vertex color value by adjacent face, like a vetex can have multiple uv's.

ok that's it, i couldn't let Autodesk win