Vertex paint -Mesh or Poly

Veronika

Hi Guys, 

I have a short question: I prepare imported .fbx object for painting in Substance Painter,
is there any difference, on what I put "Vertex paint modifier" - on "Editable Poly" or on "Editable Mesh"?

Noors

To be complete, there's a difference if you paint by face :
In edit mesh you have access to all triangles (faces) while in edit poly you only have access to "polygons".
If you apply a vertexpaint modifier to an editable mesh, and go to face mode, the mesh is diplayed triangulated, which isnt the case with edit poly.
I would avoid to paint only a triangle in edit mesh tho.

If you paint on vertex level, there's no difference.
Beware, I'm pretty sure fbx doesnt support vertex color by face anyway, unless you split the mesh.

Eric Chadwick

Ah, I do know FBX supports per-face vertex color. It duplicates the color vertices, but not the mesh vertices. Kind of like splitting UVs. 

There should be no export difference using Editable Mesh vs. Editable Poly as your base, except the triangulation. Which is important for normal mapping, if you've baked one from another mesh. http://wiki.polycount.com/wiki/Texture_Baking#Triangulation

IMHO it's better to keep it as Editable Poly in 3ds Max (more tools, easier modeling), and use a Turn to Mesh modifier to triangulate for export. Or use a Turn to Poly modifier with Limit Polygon Size set to 3, same result though.

cptSwing

Eric Chadwick said:

Ah, I do know FBX supports per-face vertex color. It duplicates the color vertices, but not the mesh vertices. Kind of like splitting UVs.

I too have heard of these ancient forlorn myths - when going from FBX to unity though, it never splits the vertex-colored faces, and I get a runny ol' mess.

Eric Chadwick

That's cuz Unity sucks when it comes to vcolor.

cptSwing

^ Haha.. very fitting, smash those multiple sharp-and-nicely-split hues!

I've had soul-crushing bouts of work where I had to split fairly complex meshes into hundreds of elements in order to transfer Vcolors correctly, even automated it's a pain. I guess one could prep an FBX export script with a routine that automatically split per-face vcolors before piping the mesh to the exporter, but I don't really have the leisure to root around maxscript for days

cptSwing

With some leisure time to burn, I did find this (by @PolyTools3D himself) ...

(
	   fn BreakVerticesByColorAndAlpha node =
	   (
		   mesh = copy node.mesh
		 
		 getmapface = meshop.getmapface
		 getmapvert = meshop.getmapvert
   
		   vertscolors = #()
		 vertsalphas = #()
		   colorsarray = #()
		   vertsbyface = #()
		   flagedverts = #{}
		   
		   for j = 1 to mesh.numfaces do
		   (
			   faceg = getface mesh j
			   facec = getvcface mesh j
			 facea = getmapface mesh -2 j
			   
			   for i = 1 to 3 do
			   (
				   vcolor = getvertcolor mesh facec[i]
				 valpha = getmapvert mesh -2 facea[i]
				   
				   if vertscolors[faceg[i]] == undefined then
				   (
					   vertscolors[faceg[i]] = vcolor
					 vertsalphas[faceg[i]] = valpha
				   )else(
					   if vertscolors[faceg[i]] != vcolor do flagedverts[faceg[i]] = true
					 if vertscolors[faceg[i]] == vcolor and vertsalphas[faceg[i]] != valpha do flagedverts[faceg[i]] = true
				   )
			   )
		   )
   
		   setvertselection mesh flagedverts
		   meshop.breakverts mesh flagedverts
   
		   flagedverts = getvertselection mesh
		   sharedfaces = meshop.getfacesusingvert mesh flagedverts
 
		   for j in sharedfaces do
		   (
			   faceg = getface mesh j
			   facec = getvcface mesh j
			   facea = getmapface mesh -2 j
			 
			   for i = 1 to 3 where flagedverts[faceg[i]] do
			   (
				   vcolor = getvertcolor mesh facec[i]
				 valpha = getmapvert mesh -2 facea[i]
				 
				 vrgba = vcolor + valpha
				 found = finditem colorsarray vrgba
				 
				   if found == 0 then
				   (
					 append colorsarray vrgba
					   append vertsbyface #([j,i])
				   )else(
					 append vertsbyface[found] [j,i]
				   )
			   )
		   )
 
		   for j in vertsbyface do
		   (
			   weldverts = #{}
			   for i in j do
			   (
				   face = getface mesh i[1]
				   weldverts[face[i[2]]] = true
			   )
			   meshop.weldvertsbythreshold mesh weldverts 0.0001
		   )
		   
		   node.mesh = mesh
		   
	   )
	   
	   gc()
	   st = timestamp(); sh = heapfree
	   BreakVerticesByColorAndAlpha $
	   format "time:% ram:% \n" (timestamp()-st) (sh-heapfree)
)

(via http://forums.cgsociety.org/showthread.php?t=1185723 )

The script breaks at various junctions, but at least I'm not the first person to contemplate this. To work for all cases, it'd probably need to also copy pre-existing smoothing groups even after vertices are split, but that's a different story I guess.

Eric Chadwick

Just tested this out again, and Unity is letting me use per-face vcolors. Not sure what's changed.

cptSwing

Eh, I tried it earlier with results as I'd have expected (using Max 14 and the 2014 FBX). But prompted by your reply I just tried it with Max'17 and their newer FBX exporter and yup - it transfers colors-by-face alright. Awesome!

Noors

Yeah Clanker is me on cgtalk ^^
Polytools is my hero.
Cool that fbx supports it now.
I do think poly and mesh trinagulate the same way, not 100% sure tho.

Veronika

Wow, thank you a lot for answers, I need to digest it. I did not suspect these two approaches has so much differences.

Veronika

Woho-ho , so many answers, I really need time to make a research to realize what they are about. 

Thank you for your help! 

For what purposes you could use Vpaint in Unity/Unreal? 

Eric Chadwick

So many purposes! Anything you want to mask on a per-vertex or per-model basis.

I'm using it in Unity to blend textures on terrains, to control wind bending on foliage, to bend water, etc.

monster

Eric Chadwick said:

Just tested this out again, and Unity is letting me use per-face vcolors. Not sure what's changed.

Dude! Glad to know this. I've got a chunk of code I can edit out of our vertex color baking tools.

Raffles

Eric Chadwick said:

So many purposes! Anything you want to mask on a per-vertex or per-model basis.

I'm using it in Unity to blend textures on terrains, to control wind bending on foliage, to bend water, etc.

That's actually a really good idea, I've never considered using vertex colours as a mask, as you would a pixel mask to blend textures etc.

As per your other examples, obviously some 0-1 vertex colours would be used to scale the amount of wind applied right? But what about the water, what exact is bending water?

Eric Chadwick

You can use vertex color as a cheap flow map. http://wiki.polycount.com/wiki/Flow_map

One way to think about vertex color is it's just data per vertex, and makes automatic gradients from vert to vert, so it can be used for lots of things. 

cptSwing

You can even free up memory by skipping on light bakes and baking shadows etc. to your vertices (sufficient mesh density given).. vertex colors and alpha rock.