Normal map problem

SamFisher45

Hey guys,

I just baked out a normal map from a high poly object onto a low poly model using XNormal. Suddenly, out of nowhere, XNormal renders this weird line in an area where there are no seams. I tried rebaking the maps from another PC, but the same problem exists. Please help. I uploaded the unwrap template to show where my seams are...



Here's the unwrap...



I am using Marmoset toolbag to view my maps here... The high poly model was sculpted in zbrush and the low poly retopo was done using the graphite tools in 3D Studio Max 2012.

Please help.

Thanks

Bartalon

Did you try baking with a cage? Also, is that weird line appearing exactly along edges or through the middle of faces?

SamFisher45

Bartalon wrote: »

Did you try baking with a cage? Also, is that weird line appearing exactly along edges or through the middle of faces?

The line is along an edge but that area does not have any seams, these sort of lines generally appear in the seam. The line runs throughout the model and does not follow any specific edge loops. I use the ray distance calculator in Xnormal. I let it calculate for couple of mins and then start baking the maps.

Bartalon

Do your smoothing groups match according to the UVs? You usually only want hard edges along UV seams, and smooth everywhere else.

I would also recommend making a cage, as it can have a substantial effect on the quality of your bakes. I believe the Push modifier will make a cage pretty easily in Max.

SamFisher45

Bartalon wrote: »

Do your smoothing groups match according to the UVs? You usually only want hard edges along UV seams, and smooth everywhere else.

I would also recommend making a cage, as it can have a substantial effect on the quality of your bakes. I believe the Push modifier will make a cage pretty easily in Max.

I think so... I do not have a harsh angle in the area and I used plenty of polygons in the low poly to avoid hard edges. I'll delete unnecessary loops later to optimize the model for a game engine. Baking on an extreme low poly map directly brings the smoothing groups issues.

And I dont know how to import and use a cage in XNormal. I've been using the ray distance calculator utility all the time. Please tell me how to do that.

Bartalon

Cages are super easy. With the Push modifier, you just dial up the value until your HP and LP are completely encapsulated. You want to keep as close to the original models as you can. If you have any tight spots where the LP or HP shows through, you can tweak the sub components around in those specific areas without having to push all the other parts further out to compensate.

Export this version of your model. In xNormal, right click on your LP mesh, and choose "Browse External Cage File." You should get a prompt about making sure your vertex count is exact on both the cage and LP. From here, just click generate and your results should improve.

EarthQuake

I don't think this has anything to do with the cage etc, it looks like your mesh normals aren't being loaded into Toolbag correctly. The lighting in general looks off here.

What app are you exporting your model from?

What does the normal map itself look like?

SARGIY576

Just try invert Y channel

SamFisher45

EarthQuake wrote: »

I don't think this has anything to do with the cage etc, it looks like your mesh normals aren't being loaded into Toolbag correctly. The lighting in general looks off here.

What app are you exporting your model from?

What does the normal map itself look like?

I am using 3D Studio Max for the model. The maps are baked from XNormal. The normal map doesnt have this line... weird. Great!! so now I discover the problem is not with the normal map. God help me!!

SamFisher45

SARGIY576 wrote: »

Just try invert Y channel

No effect, just the lighting info gets opposite (the positive areas look negative), the line still is in place.

EarthQuake

SamFisher45 wrote: »

I am using 3D Studio Max for the model. The maps are baked from XNormal. The normal map doesnt have this line... weird. Great!! so now I discover the problem is not with the normal map. God help me!!

Ok, so when you go to export your OBJ file from max, make sure "export normals" is turned on.

STEVONATRON

Does it render with this crease in Max, or only Toolbag? I often find Toolbag messes up my normals with weird patchiness and other strange effects, to the extent I've stopped using it.

SamFisher45

STEVONATRON wrote: »

Does it render with this crease in Max, or only Toolbag? I often find Toolbag messes up my normals with weird patchiness and other strange effects, to the extent I've stopped using it.

I just tried the normal map using Xoliul Shader in 3D Studio Max 2009 and the weird line is gone. I dont understand the problem. They say Marmoset is the best tool available for previewing game assets. And I personally like Toolbag for its unmatched post effects and user friendly interface...

Anyway, here's a screen cap.

STEVONATRON

Yep, just what I thought. Out of curiosity, did you use a different program to Max for unwrapping your model?

SamFisher45

STEVONATRON wrote: »

Yep, just what I thought. Out of curiosity, did you use a different program to Max for unwrapping your model?

No... just default 3DS MAx unwrap tools. That's all. But I dont understand why toolbag is creating such a huge problem...

STEVONATRON

I don't know if it's Toolbag itself or the way you exported the texture, for e.g. when saving the file after modifying it in Photoshop, it might have problems if you export a targa file with compression or a particular resolution it doesn't like. There's all kinds of variables that can result in bizarre Toolbag renders.

This was my experience anyway. Hopefully someone with more experience using that program can narrow down your problem, or you could just try experimenting until you find the solution?

EarthQuake

SamFisher45 wrote: »

I just tried the normal map using Xoliul Shader in 3D Studio Max 2009 and the weird line is gone. I dont understand the problem. They say Marmoset is the best tool available for previewing game assets. And I personally like Toolbag for its unmatched post effects and user friendly interface...

Have you tried re-exporting the OBJ with "export normals" enabled? In the GW:OBJ export options you should see:
Export Normals (or vertex normals, or mesh normals, don't have max installed on my laptop)
Export Texture Coordinates
Export Smoothing groups
etc

Marmoset Toolbag does not read smoothing groups, so you need to make sure export normals is checked.

If you're still having issues, send your normal map and OBJ to: earthqke@marmoset.co

STEVONATRON wrote: »

I don't know if it's Toolbag itself or the way you exported the texture, for e.g. when saving the file after modifying it in Photoshop, it might have problems if you export a targa file with compression or a particular resolution it doesn't like. There's all kinds of variables that can result in bizarre Toolbag renders.

If you're having issues specific to Marmoset Toolbag, please post in this thread: http://www.polycount.com/forum/showthread.php?t=74848 Someone may be able to find a solution for you, and at worst we can log the issue as a bug in internal bug system.