A challenging problem with normals

Rhimor

Hey everyone.

I've been trying to solve this issue for about ~20 hours now and it's come to the point where I think it's quite nearly impossible to fix. The problem is my normal map seams.

Basically I have sculpted my creature in Zbrush, exported it for 3D-coat, retoped and unwrapped it there. After went to 3DS-Max to reposition and relax some of the UVs. Took the lowpoly back in Zbrush, divided and projected the detail of the original sculpt and then followed "http://www.vfxforge.com/blog/zbrush-to-marmoset-toolbag-baking-normals/#more-2430" this guide to export the normal map and model (repeated this step I think 4 or 5 times to make sure Im doing everything correctly). Additionally I have also baked my normals with Xnormal.

When I put the lowpoly and the normal map in Marmoset 2 I get seams everywhere that I can't fix.

I've tried to have separate UV islands with separate smoothing groups, but that doesn't work.
I've tried to export the lowpoly mesh as triangles and quads and it didn't make any difference.
I've tried rendering in 3Ds Max but the result is the same (seams everywhere).
I've tried Xnormal (with a cage), didn't work either.

Any tips/ideas I can try would be greatly appreciated, thanks for your time.

Here are the seams in Marmoset, normals from Zbrush:

Seams in 3ds Max:
 

Seams in Marmoset again, normal exported with Xnormals:


My unwrap:

Zbrush normals:

And finally the model itself:

huffer

In 3dsmax it looks like you have to disable gamma correction (or load the normal map with gamma 1.0), enable 1-Light not 2 in viewport settings and flip the green channel. Also make sure bump value is set to 100 in the material, not default 30. The seam might be cause by tangent space issue (baking in one program and viewing in another), its hard to say..but its not that noticeable.

Rhimor

@HUFFER

Thanks a lot for the reply! I'm going to try what you suggested in Max now and post results in a bit.

Also yeah, they're not very noticable and from a distance you can't tell at all, but I've just been through some of my old work which I've done, using the same pipeline as now pretty much, and is ALL has those damn seams. I know it's possible to have a clean seamsless model so I think that is my current goal :P

Rhimor

@HUFFER

I disabled the gamma correction, only 1 light in the viewport, baked the normal again with the green channel flipped in Zbrush and increased the bump value to 100. I tried ticking the flip green in Max as well, but it's all the same.

Seems to be better, but still visible... I'm starting to think that if it's improving, there's probably something going wrong with the baking and not the previewing. I wish I was still at uni, would pester my tutors to death :P

Here is the result:

Rhimor

So I see where the problem is I think. This is the normals baked in Zbrush and applied as a texture with a flat colour material. I'm really not sure what's going on. I've tried to rotate and reposition my UVs (was suggested somwhere) but Im getting the same result.

musashidan

You only need to split SG/UV shells where you want hard edges. For an organic character like this leave it all on 1 SG.

Rhimor

@MUSASHIDAN

Hmm, I don't think it's possible leave it all as one whole UV (if that is what you mean). But even if I do have it all on one, I'll still have a single split somewhere and the seam would be visible. I just don't understand how people can get seamless normals : ( I will continue trying tomorrow until it is solved! Thanks a lot for the reply!

huffer

You can keep a single smoothing group and the UV's, and bake the normal map in 3dsmax: decimate the high res from zbrush, import in max, triangulate the low poly, bake (without a cage), then preview the baked normal map in 3dsmax (it should be ok) or in Marmoset (setting tangent space to 3dsmax for the imported mesh). The seam you were getting in the Marmoset screenshot (file seems to be down, btw) I would say it's acceptable..it might be just a normal map limitation; on the other hand, in the screenshots from 3dsmax something is wrong (probably tangent  space difference).

musashidan

Rhimor said:

@MUSASHIDAN

Hmm, I don't think it's possible leave it all as one whole UV (if that is what you mean)

Leave it as a single smoothing group. The only reason to split uv shells to SG is when you want a hard/defined edge, which you don't as this is an organic character. You should be able to see the SG splits on your low poly in your 3d package viewport.

Rhimor

Hey again guys! Thanks for the advice again, I will try and bake in Max, I am not sure if my PC can handle it tho, I tried importing the high poly (about 5 million tris) once and it froze completely. Will give it another go tho definitely!

EDIT: I just properly read your comment today HUFFER, Im going to decimate it now and try, see how it goes!

Oh and now I understand what you mean about the smoothing groups. I've had it all as one most of the time, just when I couldn't figure out how to fix my seams I started looking for suggestions and one was to split ever UV island with a separate SG which was weird, but I've tried anyway without any results obviously.

Tomorrow, after trying to bake in Max, I'll try and bake some other simpler organic model in Zbrush and see if the same thing is happening... It's just driving me insane.

Rhimor

Right so baking in Max produced A LOT better results. I end up losing a bit of detail after decimation, but the seams are barely barely barely visible now. I'm guessing at this point (and after speaking with you guys) that the seams can't be 100% non-existant.

I am really glad and happy that I can now continue and start painting some colours in, but it still bugs me that I couldn't bake a good normal map in Zbrush or Xnormals < > Hopefully after some experience, I will figure out what I'm screwing up : P

Thanks a lot for the tips, really appreciate it!

Here are the results:

Rhimor

One more update. I've baked a normal in Zbrush again with a flipped G and put that with the Max normal in Photoshop. Now I'm using the edges (where the seams are) of the Max bake and the details of the Zbrush bake. It seems like the problem was how Zbrush bakes the edges so they don't "match" that well.

Scruples

Yeah Zbrush can't bake tangent space maps correctly, I thought you said you were using Xnormal? If you're really adamant about baking in Zbrush for some reason you can bake an object space map and convert it into tangent using handplane, you'll still get some warping but on organic surfaces it shouldn't matter.

Rhimor

@SCRUPLES

I used Xnormal yeah, but I ended up getting pretty similar result to Zbrush around the edges (not very familiar with Xnormal tho). Only tried using Zbrush to bake 'cause I've watched a few tutorials where people have gotten pretty clean seams and it's the software that I'm most knowledgeable of (plus it seems easier to me, no cages to set up, as far as I know, and also that's where I do my sculpting).

I'll try baking an object space map and convert it as well, see what kind of results that produces! Thanks for the suggestion!

musashidan

ZB can bake TS no problem but the quality is not on par with xnormal.

huffer

Glad you got it sorted. My standard workflow is to always bake in xNormal; once you create the low poly with UV's it's just a matter of loading the low poly, setting projection distance; loading the high poly and hitting bake. I don't even decimate the mesh from Zbrush anymore, I succesfully baked with 16mil+ triangle mesh. You do have to be careful where you preview the bake, xnormal uses mikk tangent space, so you can view it in Marmoset or UE4 for example.

kanga

I dont bake normals in max because it would mean importing a hi rez mesh.I also use xnormal.  Decimating a mesh from Z for use in Max for the bake is still decimation. Maybe you do this already but just in case.

Lowpoly in max plus uvs for the low poly.
Apply a projection cage to your lowpoly and use a decimated version of the hipoly (in the projection modifier) as a reference.
Export the lowpoly with the xnormal exporter with projection cage checked.
Close max, fire up xnormal and point it to your two meshes and the results are ususlly great.

Cheerio

musashidan

Also, to add to what the lads have advised, if your target is mikkt synced then just bake in xnormal: single smoothing group/no cage/tangents&binormals unchecked in .fbx export (baker and target will compute this) and you should get a perfectly seamless bake.

musashidan

Here's a quick example I just did using the method from my last post. Zremesher+UVMaster to obtain the low-poly(2 clicks!!   )

kanga

musashidan said:

Also, to add to what the lads have advised, if your target is mikkt synced then just bake in xnormal: single smoothing group/no cage/tangents&binormals unchecked in .fbx export (baker and target will compute this) and you should get a perfectly seamless bake.

What is mikkt synced? When I dont use a projection cage with xnormal I get pretty bad results, so this is interesting!

Rhimor

Cheers for the suggestions! I will try each one very soon as I've entered Allegorithmic's contest and will be baking more normals in the next couple of  weeks. Will post results with each method as I'm done.