[3DS Max] ISSUE with Vertex color being converted to int8 when exporting

Rayoule

Hi everyone,

I'm currently encountering a big issue here, and everything makes me think that this is a big issue from 3ds Max.

Here is the problem:
I'm baking some pivot positions into vertex color using MAXScript. I use standard methods sush as "SetVertexColor" or "meshop.setMapVert" on channel 0, which is the vertex color. The precision here seems to be a nice float32 if I set it, and the same if I get it.
BUT when I export my fbx (2019), my vertex color seem to be shrinked into a int8 and then into a float8, meaning my I will only have values between 0 and 255 without floating points. When debugging vertex (color.z * 255.0), it gives me 127.0002 of 135.0001, that kind of stuff. If I make a comparison with the value I get using maxscript it gives me like 126.4856, stuff with precision.

So max is loosing quality when exporting vertex color ?

No, it's not that simple. Because if you leave MAXScript behind and use only the data channel modifier and you just bake vertex position into vertex color and then export.... All the precision is here !

So what the hell Max is doing with the vertex colors ?? Why is it not happening using the data channel modifier ?
Do people know that their vertex color is slaughtered at export ?
(don't tell me to switch to Blender, I already did it years ago but the environment artists are working on max....)

If you have any information please share it, this seems to be a huge problem and I can't believe no one has ever experienced such problems.

Thanks a lot

poopipe

I vaguely remember working around  this once - I'm not at a machine with max installed  atm but I did find this passage in the online maxscript docs 

https://help.autodesk.com/view/3DSMAX/2016/ENU/?guid=__files_GUID_D8DBE1AA_F3D7_4708_93FA_7C28845A1C03_htm

"Thus, the Color Value is not well suited for High Dynamic Range Image calculations with floating point color components.
For that type of operations, MAXScript provides a Point4 value that can be used to represent floating point colors without implicit conversion from/to the 0-255 range. This value was introduced in 3ds Max 6 to support the color parameters of mental ray shaders.
"

seems like you need to control the typing of the data 

monster

I can't reproduce this issue. In Max 2021 on three different spheres I set the vertex color to the geometry position using...

1. meshop.setMapVert
2. Unwrap Modifier MaxScript
3. Vertex Paint Modifier MaxScript

Then exported to FBX with default options, and reimported into a new scene. Using meshop.getMapVert the values were identical to before the export. I didn't convert to and from a color value, just set the vertex color exactly to the geo vertex position.

I'd need to see a sample of code that isn't working.

coven

Hey @Rayoule I reached out to one of the dev and am posting what he said.

maxscript is expecting the value to be set to be a color value.  Color values in maxscript are normally in the range of 0-255, but each component is a floating point number and can be outside this range.
When setting the color value, the mesh stores colors in the range of 0-1 (again float, can be outside this range). So when passing the color value to the mesh, maxscript is dividing the component values by 255. on write, and multiplying by 255. on read.
A little test case:
m = mesh()
defaultVCFaces m
getNumCPVVerts m --> 36
setvertcolor m 1 1.1 100.1 1000.1
c = getvertcolor m 1 --> (color 1.1 100.1 1000.1)
setvertcolor m 2 2.1 200.1 2000.1
c = getvertcolor m 2--> (color 2.1 200.1 2000.1)did 2019 export. First 2 values from LayerElementColor:[.00431372551247478,0.392549008131027,3.9219605922699]*255. --> [1.1,100.1,1000.1]
[0.00823529344052076,0.784705877304077,7.84352922439575]*255 --> [2.1,200.1,2000.1]So I am not seeing a float -> int -> float conversion occurring.

Hope this helps. If not, let me know and we can go from there.

Rayoule

Hi everyone, thanks a lot for your answers.

@poopipe This seems really interesting! But when trying to pass all values to point4 it appears to no change anything at then end... I really thought this would fix the issue. Thanks anyway !

@monster and @coven Well I tried what you tried and... It seems to work perfectly. Indeed, the precision is not lost at the export of 3dsMax, but its lost when read outside of 3dsMax !
That was highly unexpected. Considering that an import from a blender fbx is working properly.

Here is what's happening if I import the fbx in max and check the selected vertices vertex color:
(Look for the Z value)


These vertices have the same value because Vertex Color means pivot position here. But each leaves have a different one, like you can see  here :



BUT in UE4 the precision is lost :



Of course I checked with a step() function to be sure. The precision is truly lost.
I tried with different fbx versions, export in alambic... Nothing changes.

If I try to import the fbx in blender and print vertex colors (multiplied by 255) :




Well thanks to you I found out that re-import in 3dsMax seems to work so the information is here, but why is it misunderstood by UE4 or Blender ? This is a mystery. I'm currently exploring the possibility of signed/unsigned float, I will update this thread.

poopipe

That will be the ue4/blender importers.  

you could try storing the data in UV channels instead but they might knacker that as well 

Rayoule

@poopipe Yeah this is the last solution I can use, but I wanted to avoid to add 2 UV channels... But i'm afraid that's how its gonna end
Seems weird tho. Feels like something is broken

monster

You can try this for Vertex Colors. If not it should work for in a regular UV channel as suggested.



Also, come on man! I spent an hour and half yesterday trying to repro your problem in Max. You didn't even mention Unreal. This would have been my first suggestion if you had.

Rayoule

@monster Hey thanks for the info !
Sadly, it does not change anything.

Also, come on man! I spent an hour and half yesterday trying to repro your problem in Max. You didn't even mention Unreal. This would have been my first suggestion if you had.

Sorry to hear that! The use of UE4 should definitely have been mentioned in the first lines. But at the end its not an issue caused by unreal because it is also there when importing in Blender. Which means this is not a UE4 specific issue but a 3dsMax specific issue.

Also when having problems with an exporter, a good way to be sure that its working is to try it in a different environment than the one it has been created into (wow I hope this is correct English).

But anyway I'll try to be as precise as I can be in the future so I do not make people that helps me lose their time!
Thanks again for the time you spend trying to fix this issue.


I will now set my values in a new UV set and see how it goes. But this is not the end and I'm sure there is a way to fix this. Especially considering the fact that if you use the data channel modifier, the precision remains intact !

poopipe

Save an fbx in ascii format using the maxscript approach and another version using the datachannel modifier and compare the vertex color data (it'll be stored in a layer) using a text editor.  There's more than one way to store the data in an fbx so its entirely conceivable its being represented in different ways. 

if you get stuck ill be happy to take a look through them for you (as long as it's just a cube or similar)

Rayoule

@poopipe That is a great idea !
I did what you said and... the results were kind of interesting.

I appears that I was wrong in my way to debug vertex colors. The precision is lost at import in Blender and UE4. Also from Blender to 3dsMax.
But it appears 3dsMax to 3dsMax is the only way to keep the vertex color at full precision.

The export to UE4 does not work at full precision EVEN with Data Channel modifier. I was misunderstanding the values because I can only debug vertex color with a shader, and there was precision he because the color in interpolated between the vertices which gives float32 or float16 precision at pixels. But the raw values of the vertices are still loosing precision.

So the problem is now so different from the title of this thread that I don't know if I have to move this thread or rename it...

So I think showing you the ASCII files is useless since the problem is the same for the one bake using MAXScript and the one baked using Data Channel.

poopipe

In that case it's probably worth trying to put the values into a UV channel. 

If that doesn't work you should be able to store the data in a texture, either as a bake or just by storing values sequentially 

Rayoule

Yeah its what I did and its working great.

Its just that I'm really surprised that you can't bake vertex color with good precision for game development.... it is veeery surprising.

I could set the position range to the object bounds so the precision would be optimized but anyway, this is really bad issue. But not 3dsMax's fault !

Thanks for your answers @poopipe , they were very much appreciated