Normal Map displayed flawless in Marmoset, bad in 3ds Max & Vray

Klaus Hustle

I'm testing my normal workflow to build my wooden shack in 3DsMax and always struggled to get decent results with normals.

By browsing the polycount-forum i found lots of praise for the marmoset baker, so i tried it out, and it works like a charme!..

..Until i load my maps into 3DsMax sadly, here are some screenshots of the problem:



The highpoly details were made in zbrush. 

I'm not sure if it is done right, but this is how i load the normal-map in 3DsMax & Vray:

VrayMtl -> Normal-Bump with Flipped-Greens in the Bump-Slot -> VrayHDRI -> Bitmap

I always assumed that something with my baking-process is wrong, but now that i see that the results in marmoset are alright i'm curious how to get it displayed right in 3DsMax as well.

Eric Chadwick

3ds Max doesn't support the tangent basis that nearly everyone else uses... mikktspace. You will only get synched normal maps if you bake them inside Max using Render To Texture.

Klaus Hustle

God damn, i gave several 2 times before on this topic assuming i'm just too stupid to use normals in general. Thanks for the info, i recall having another set of issues by trying render to texture before, but i'll give it a shot. And i might try blender already...

Klaus Hustle

I downloaded blender, watched a tutorial on how to apply normals and it looks perfect...

I tried render to texture in max and i'm getting the exact same BAD results as in my original Post.

I don't feel like leaving max behind and learn blender just for normals, but i need these normals I can't get it to work with max, it's quite maddening.

You helped me alot by pointing out that this is a max thing, but i'm afraid i need an additional push in the right direction on how to solve it finally..

At least i know that my general workflow to create these maps is working.

Mark Dygert

I use 3dsmax for modeling, it's amazing for that, very flexible and powerful, BUT for materials and baking I work in anything but 3dsmax, mostly I use substance painter and Unreal.

Autodesk has abandoned/forsaken Render To Texture.

It barely renders normal maps, forget about curvature and AO. Even with normal maps I run into a lot of projection artifacts when baking. Not to mention the RTT UI throws up a bunch of maxscript garbage collection errors and crashes max if you don't relaunch and resize the window after changing selections in the viewport. Why does that crash max!?

I also run into baking errors like the mesh is penetrating the cage but it isn't, adjusting the cage doesn't help, editing the geometry (changing edge direction) seems to nudge the artifact around to another random part of the mesh that was previously fine, but it doesn't ever go away. I just chase it around until it ends up in a spot that I can live with. GARBAGE! Take the same models into Substance, or xNormal, they bake just fine.  

It's just a nightmare to bake anything in max.

Klaus Hustle

Damn, okay.. But do you get at least your normal maps you created with substance to display correctly in 3dsmax? I could bake it elsewhere, no problem. Like i mentioned, toolbag 3 is reaally good at it, but i can't use it for max because it won't be displayed correctly...

So basically this is doomed, i can't use other tools for baking because max doesn't support the tangent base, but also can't use render to texture..

Maybe some recommendations for a programm that's good for setting your models up, light and render it with all the maps? I'm definitely sticking to max when it comes to modelling for sure, but that's it..although i really like vray for max, i got used to it very well. 

Obscura

Klaus Hustle said:

Damn, okay.. But do you get at least your normal maps you created with substance to display correctly in 3dsmax?

No. Max uses its own tangent space, while Substance bakes for Mikk. There are 2 places to display max bakes correctly. In Max, and in Toolbag.

Personally I didn't get as much trouble with RTT in Max as Mark described, but yeah it sucks.

Klaus Hustle

Obscura said:

Klaus Hustle said:

Damn, okay.. But do you get at least your normal maps you created with substance to display correctly in 3dsmax?

There are 2 places to display max bakes correctly. In Max, and in Toolbag.

I havn't had the pleasure yet, none of my max-bakes were displayed correctly. So 'm curious how you do it, if you actually are working in max that is.

Obscura

Scanline renderer displays Max bakes correctly  But thats it.

Axi5

The others are right in that Max uses it's own tangent space but if it's V-Ray specifically you're having troubles with then I'm not sure it uses Max's tangent basis at all.

If you're using Substance (or even if you're not), take a look at this guide here:

https://support.allegorithmic.com/documentation/display/integrations/Vray+for+3ds+Max

It might just be you have your colour space settings incorrect and we should check that first.

Klaus Hustle

Hey thanks, i don't have Substance Painter, so i couldn't try to bake with it (My 30 Day Trial has expired).

But i applied everything from the documentation (except the last "color mapping only" thing in the last screenshot, i havn't got the option in my version, but i imported the maps with gamma override 1.0)

I'm having the same issue as before, although the overall quality of the normal-map has improved a bit, so that's something! But as you can see in the screenshot, it looks better without the hard-edge obviously, like it does in marmoset.

As you can see the lowpoly-model has changed, i'm trying different styles of lowpoly meshes out for best results, didn't help so far.

Obscura

It would be nice to figure out what tangent base is required for V-ray. Anyone knows?

Axi5

Obscura said:

It would be nice to figure out what tangent base is required for V-ray. Anyone know?

Nada, but this forum post by one of the developers seems to confirm that Chaos have checked Substance's (mikktspace) normals at least:

https://forums.chaosgroup.com/forum/v-ray-for-3ds-max-forums/v-ray-for-3ds-max-problems/69728-substance-painter-normals-in-vray

Another snippet of info there, which might be helpful and that we're overlooking, is that we could just bake an object space normal map which will guarantee it works. I doubt this model is going to deform, right?

Klaus Hustle

Quick Question; in general it's better to keep the smoothing-groups seperated for the low-poly, right? I'm experimenting with both right now, completely smooth and hard edges with smoothing-groups, doesn't help with my problem but i want to know anyway.

Obscura

That really depends on many factors. If the geometry is more dense, and edge shading or roundness is represented in the geometry, leaving them smoothed together is usually fine. On lower poly meshes however, its usually better to split the smoothing along the uv seams if the normal was baked from a highpoly mesh.

@Axi5 - So nothing means tangent space normal maps baked from a highpoly mesh should not be expected to work correctly in V-ray, is that it? Game engines doesn't alter the vertex normals when you tessellate / apply displacement, so you still need to use a normal map, but does V-ray do this when you apply displacement? If it does so, you could just go with a displacement, and finer details should be like in a detail normal map kind of tiling normal map? I'm not entirely familiar with the answers to these questions, as I'm mainly working with stuff for real time applications, but now I became curious.

Axi5

Obscura said:

That really depends on many factors. If the geometry is more dense, and edge shading or roundness is represented in the geometry, leaving them smoothed together is usually fine. On lower poly meshes however, its usually better to split the smoothing along the uv seams if the normal was baked from a highpoly mesh.

@Axi5 - So nothing means tangent space normal maps baked from a highpoly mesh should not be expected to work correctly in V-ray, is that it? Game engines doesn't alter the vertex normals when you tessellate / apply displacement, so you still need to use a normal map, but does V-ray do this when you apply displacement? If it does so, you could just go with a displacement, and finer details should be like in a detail normal map kind of tiling normal map? I'm not entirely familiar with the answers to these questions, as I'm mainly working with stuff for real time applications, but now I became curious.

I'm in the same boat as you. My only suspicion is that offline renderers probably don't have normal maps high up on their list of things to do since it's probably use for micro-detail and the rest is all geo, hence the lack of information. Maybe it could also be the quality of the lighting/lighting techniques don't play well with normal maps.

Obscura

Axi5 said:

 Maybe it could also be the quality of the lighting/lighting techniques don't play well with normal maps.

Sorry for the bunch of quote boxes, I'm not sure how to remove them. So this particular thing wouldn't make much sense, as a normal map is an absolutely correct representation of the directions of the surface of a mesh.

Instead of waiting for some V-ray pros, lets just summon @Eric Chadwick . From what I know he works on some arch viz stuff now, so maybe he can give some insight regarding these questions. If not, @musashidan, but I haven't seen him posting for a long time now, so maybe he isn't an active member anymore.

Obscura

@Klaus Hustle an object space normal map  (local to the mesh so it works with transformations) should indeed work, but the thing that this is the correct way sounds weird, as I think I haven't seen this method in any offline rendered art breakdowns. It is definitely a workaround though.

Klaus Hustle

First time hearing about, i look it up and try it, thanks!

Eric Chadwick

Yeah we only use normal maps in V-Ray for tiled textures. 

If we do a sculpt like that, we use the original sculpt mesh directly in Max. Though we'll optimize it a bit in Zbrush just for easier file handling.

Displacement does indeed subdivide, and it does it generally at the subpixel level. Not realtime friendly haha.

My current task is to convert these high-res models and materials into real-time ones. Fun times! No really, I'm enjoying the challenge.

Klaus Hustle

Would it may be easier, in terms of technical hassle at least, to create tileable maps in substance designer, for instance, instead of modelling individual highpoly and trying to generate maps from that?

Even getting displacement-maps to work properly is a pain and i'm growing a bit mad right now. I'm willing to buy and learn substance designer if it would make my life a bit easier.

As mentioned, i never worked with Substance Designer, but here's how i think it could work:

I could create 2 tileable materials in substance designer for the wooden plank, for example one material for the white paint and one for the underlying wood, then create individual handpainted masks to break the white paint on some areas..

Alex_J

explanation and tables of synced workflows

Useful charts that make it easy to see which programs work with each other. 

FourtyNights

If you really wish to render tangent space normal maps inside 3ds Max, you can always use the old "handplane3D tangent space calculator" to convert a baked object space normal map to 3ds Max's tangent basis.

http://www.handplane3d.com/download.html

Klaus Hustle

oh great, just downloaded the tool, looks simple, thanks alot, trying it out later.

i'm now having troubles with displacement maps in 3dsmax as well, i probably going to use marmoset for rendering for a while because it 100% works with all my maps i get out of zbrush with my workflow. but i'm still interested how it could work in max and vray for the long run.

Eric Chadwick

In Max, make sure you use the modifier (VRayDisplacementMod) not just the material, and make sure Keep Continuity is enabled.

Help file
https://docs.chaosgroup.com/display/VRAY3MAX/Displacement+Modifier+|+VRayDisplacementMod

musashidan

What a coincidence, @Obscura I'm just back up and running this weekend after a 6 month hiatus from 3D.

@Klaus Hustle  in your original post you have this: 'VrayMtl -> Normal-Bump with Flipped-Greens in the Bump-Slot -> VrayHDRI -> Bitmap' as your workflow.

I'm guessing it could be a double-gamma issue, as you're using an unnecessary VrayHDRI map. Just drop the baked normal map into the Normal slot of the Normal bump map channel.

Also, here's a shameless plug of my Substance Painter>Vray gumroad tutorial. Although it's Sub Painter most of the workflow is Vray and certainly applicable for any mikkt baker.
https://www.youtube.com/watch?v=XdCNht6vjoc

I'm not sure what tangent basis Vray uses but mikkt baked maps render perfectly. I would also follow advice above and completely forget about Max RTT. It is ancient, obsolete, and a dead toolkit.

Best of luck.

Klaus Hustle

Yes i'm using that modifier, but it gave me terrible results for whatever reason, i just experimented a bit with my zbrush-generated maps again and now i got good results,  with certain settings export-settings. So far so good! 

For normals in 3DsMax i went back to a simple cube with round edges, simple, baked normals and object space normals in marmoset and tried handplane 3d, with the following results:



Here's what i tried:

- adjusting the padding from moderate to extreme
- switched the tangent-space option for baking in marmoset between the option "marmoset" and "3dsmax"
- experimented with using the "Normal Bump" -modifier and the "VrayNormalMap" -modifier, with bitmaps and vrayHDRI maps
- always loaded my maps with gamme-override.

Not much of a difference between all these adjustments

Klaus Hustle

musashidan said:

[...]

Best of luck.

I've seen some people using the VrayHDRI map to get better results in quality over bitmap, but it hasn't helped me so far..

As you can see i just posted a new docu with my recent experimentation and i tried both with no success, but i'm checking your vid, thanks!

Eric Chadwick

Green channel is flipped in the bottom normal map.

Also, check the normal map strength parameters in the material.

Klaus Hustle

raising the strength-parameters also strengthens the harsh-edge but with bump-strength on 200 i can at least see the roundness underneath :>

Eric Chadwick

Did you try flipping the green channel?

Klaus Hustle

yes, most of the time i render flipped and unflipped anyway to get sure not to overlook it.

Klaus Hustle

@musashidan

i bought your tutorial, it's useful, just watched part 3 till you set up your normal-map in the bump-slot, but you've done nothing i hadn't done, and i can't get it to work on a simple cube even. Am i expecting too much maybe? Isn't it possible to display it in max the way it is displayed in substance painter or marmoset?

If someone likes to try this in max, just with a simple cube, i'm really curious how your normals turn out..

Klaus Hustle

small update, i'm still trying to get better results, now in comparison after finding the right light and angle, FOR BOTH, i think both are pretty equal...bad in my opinion, but yeah.

As you can see, problem is the darkening of the rounded edges in the back - and these pointy edges. 

Is it as good as it gets? Or do you sitll see room for improvement? 

Eric Chadwick

Is there space between the UV shells on your box? If you have hard edges (different smoothing groups) then you need a split in the UVs and enough space between them to prevent bleeding from other nearby UVs.

Klaus Hustle

I raised the spaces between the boxes. Both renders are 3dsmax&vray, just different backgrounds.

You see, it's not THAT nice, but it got better over time with your help.

Amiminoru

Add a directional light source to the scene

Obscura

Amiminoru said:

Add a directional light source to the scene

Best solution ever, lol.