Vertex colors FBX problem.

animulator

Hello I have a question. Whenever i want to export my model from 3dsmax 2012 as fbx file. My vertex colors having blurry result.


When i export this as open collada i dont have any problem.

I need  my model  in FBX file. Any idea to solve this problem?  

Obscura

Looks like the vertices gets welded together either upon importing into Unity, or when you export the fbx. So it can't have the sharp transitions anymore.

animulator

Yes very strange, when I export this with open Collada .dae. It's exporting it correctly. I guess fbx exporter have some problems. 

Obscura

I don't think so. Try exporting with smoothing groups enabled. Its a checkbox in the fbx export dialogue.

animulator

Actually its selected. I attached here my screenshot. if i miss something let me know please. 

Obscura

This setting should do it. Then something happens when you import it into Unity. Are there some import settings that you can set? If there are, try to find something related to smoothing groups.

animulator

i dont think Unity has a problem about this. Because when i import back this fbx model to 3dsmax. I have the same blurry result.  

When i export my model as collada and then if i use fbx converter software then it works. But i lose my rigging settings , if i do it so. 

Eric Chadwick

It's not smoothing groups. Unity only supports 1 vertex color per vertex. 

So, you cannot color per polygon (which causes hard edges in vertex color, because edge vertices are then storing multiple colors).

If you want hard edges in VC, either chamfer the edges with tight bevels (not recommended, long thin triangles are bad for performance), or split the edges making duplicate vertices (hard to control though, difficult to limit each vertex to just one color).

Best bet is to avoid hard transitions in vertex color, rethink your use.

Good luck!

Obscura

What I meant is to have separated smoothing groups for the different colors, so the vertices splits along the color changes, so he can have the hard transitions.

Eric Chadwick

No you need to manually split the vertices if you want to allow this.

Smoothing groups are simply a way to get multiple vertex normals into a single vertex. It doesn't actually duplicate vertices in the exported file. Only in the GPU for rendering.

animulator

Thanks for your reply.Maybe you guys right. just  i am just thinking  when i do same model in blender ,how blender fbx  export works. and how 3ds max fbx export doesnt work. 

Mark Dygert

animulator you might want to try a newer version of FBX. I ran into similar issues in Unreal when I was using FBX 2013, it became a non-issue after we started using FBX 2015 and 2018. It might have been something they changed in Unreal too, because we jumped from 4.12 to 4.20 about the same time.

But it wouldn't surprise me if Eric is right. It was an issue a long time ago in Unity when I was trying to blend materials based on per-face vertex colors and Unity kept making a blurry mess until I broke the geometry and doubled the verts, but that was a really long time ago (2010-2011). That is probably what blender is doing, breaking the geometry and doubling the verts along those seams.

It would be weird if they haven't gotten per-face vertex coloring to work in newer versions of Unity. I'm half tempted to download it and try.

I know for a fact that Unreal and FBX 2015+ support per-face vertex colors, which you probably can't switch to so you're probably stuck breaking off each color as a separate element, like Eric suggested OR using an exporter that does that for you. OBJ and Collada have been known to split geometry along smoothing and vert color seams so that might be a simple fix?

animulator

Thank you Mark , I will try the new FBX version. 

Noors

Yeah this is an issue with old exporter. I think you can update the exporter even on old max versions. But just in case, here's a maxscript to add support vertex to handle vertex color/alpha by face, thanks to polytools3d on cgtalk. Has to be an editable mesh i think.

</code><code>(
	   fn BreakVerticesByColorAndAlpha node =
	   (
		   mesh = copy node.mesh
		 
		 getmapface = meshop.getmapface
		 getmapvert = meshop.getmapvert
   
		   vertscolors = #()
		 vertsalphas = #()
		   colorsarray = #()
		   vertsbyface = #()
		   flagedverts = #{}
		   
		   for j = 1 to mesh.numfaces do
		   (
			   faceg = getface mesh j
			   facec = getvcface mesh j
			 facea = getmapface mesh -2 j
			   
			   for i = 1 to 3 do
			   (
				   vcolor = getvertcolor mesh facec[i]
				 valpha = getmapvert mesh -2 facea[i]
				   
				   if vertscolors[faceg[i]] == undefined then
				   (
					   vertscolors[faceg[i]] = vcolor
					 vertsalphas[faceg[i]] = valpha
				   )else(
					   if vertscolors[faceg[i]] != vcolor do flagedverts[faceg[i]] = true
					 if vertscolors[faceg[i]] == vcolor and vertsalphas[faceg[i]] != valpha do flagedverts[faceg[i]] = true
				   )
			   )
		   )
   
		   setvertselection mesh flagedverts
		   meshop.breakverts mesh flagedverts
   
		   flagedverts = getvertselection mesh
		   sharedfaces = meshop.getfacesusingvert mesh flagedverts
 
		   for j in sharedfaces do
		   (
			   faceg = getface mesh j
			   facec = getvcface mesh j
			   facea = getmapface mesh -2 j
			 
			   for i = 1 to 3 where flagedverts[faceg[i]] do
			   (
				   vcolor = getvertcolor mesh facec[i]
				 valpha = getmapvert mesh -2 facea[i]
				 
				 vrgba = vcolor + valpha
				 found = finditem colorsarray vrgba
				 
				   if found == 0 then
				   (
					 append colorsarray vrgba
					   append vertsbyface #([j,i])
				   )else(
					 append vertsbyface[found] [j,i]
				   )
			   )
		   )
 
		   for j in vertsbyface do
		   (
			   weldverts = #{}
			   for i in j do
			   (
				   face = getface mesh i[1]
				   weldverts[face[i[2]]] = true
			   )
			   meshop.weldvertsbythreshold mesh weldverts 0.0001
		   )
		   
		   node.mesh = mesh
		   
	   )
	   
	   gc()
	   st = timestamp(); sh = heapfree
	   BreakVerticesByColorAndAlpha $
	   format "time:% ram:% 
" (timestamp()-st) (sh-heapfree)
)