Normal Map: Meshes projecting into eachother

daniellooartist · Jun 2015

Hello everyone. I'm having an issue with my normal maps. This isn't the first time I have had it either. As illustrated below my meshes are projecting onto one another.

I have no idea why. I followed EarthQuake's "You're making me Hard" thread. All 90 degree angles have cuts and I am also using the script from this thread to harden the edges.

http://www.polycount.com/forum/showthread.php?t=52722

My cage is perfect and covers my entire high poly. I have a screenshot below. So any ideas as to what is happening? I generally solve this problem by launching each of my meshes to opposite ends of the solar system in a cartesian space represented by "long doubles" but many credible individuals said that this is not necessary. If that is true, then what am I doing wrong?

I have also provided my source file. If you want to try to find out what is what, then look for the following outliner groups.

High poly: high02>export
low poly: lowTRIANGULATED
cage: cage

Source: https://www.dropbox.com/s/xvgna0tlsx2rlcd/table15.ma?dl=0

EDIT: RESOLVED
solutions...
>never mirror half, just offset mirrored UV's
>always mirror pieces, never rotate them. It's ok if the UVs are flipped, just make sure it's off the 0-1 space
>Never bake instances. If you have insistence disable them by going to Modify>Convert>Instance to Object
>if all else fails, play with the use cage function.
>ensure the cage covers BOTH the low poly and the high poly
>if all else fails, try your bake once by disabling use cage in xNormal.
>For some reason my bake fails when i select the outliner group and export. It works when i select the group, isolate selected, them marquee select all the object i want to export. I do not understand why but this is what I needed to do.

m4dcow · Jun 2015

I forgot to mention that the projection errors still in my image can be solved by exploding the mesh for the normal map anyway.

daniellooartist · Jun 2015

Holy crap thank you so much for looking in to that! That worked like a charm! A charm crafted from Olympia no less. For years people keeped telling me my cage was wrong or I was experiencing an error that happened every once in a blue moon (except it happens to me literally ever time.). I was like "I swear to God it's 1 thing, just 1 piece of the puzzle i'm missing! I bet it's something stupid like a setting or some function i have to preform." I was like the guy from the movie Pi, neurotically trying to throw every solution and the kitchen sink at the problem to solve it and literally losing sleep over it. So, I can't thank you enough for the help m4dcow. You made my day!

daniellooartist · Jun 2015

Actually. Is it ok if I do this with multiple objects? It's all 1 combined mesh right now.

m4dcow · Jun 2015

It should be okay with multiple objects, although sometimes there can be issues with keeping the cages in order.

daniellooartist · Jun 2015

I spoke too soon. I didn't actually fix the problem. For some reason the projection error is happening again. This is really frustrating. My low poly and cage are 100% perfect. The bake will work fine if I use Maya transfer maps but not if I use xNormal. And yes I have the Use Cage disabled.

Source File (just the leg): https://www.dropbox.com/s/ctlhb1xnhsgi77q/quarantine%5Bisolated%5D.ma?dl=0

kanga · Jun 2015

I also bake any parts that overlap or are not connected separately. Gives me clean results.

m4dcow · Jun 2015

daniellooartist wrote: »

I spoke too soon. I didn't actually fix the problem. For some reason the projection error is happening again. This is really frustrating. My low poly and cage are 100% perfect. The bake will work fine if I use Maya transfer maps but not if I use xNormal. And yes I have the Use Cage disabled.

Source File (just the leg): https://www.dropbox.com/s/ctlhb1xnhsgi77q/quarantine%5Bisolated%5D.ma?dl=0

You do actually want use cage enabled, the only reason you wouldn't have it enabled is if your mesh was totally smooth, or like just a plane. I just thought your issue was that you didn't actually assign the cage mesh to the low poly properly.

What version of xNormal are you using, because it maybe different from mine and have a bug.

Tzur_H · Jun 2015

use custom cages, explode parts so you don't get intersections in the baking process. these 2 steps will give you clean results.

daniellooartist · Jun 2015

I dont quite understand. What do you mean by overlap?

daniellooartist · Jun 2015

m4dcow's response did not show up on my phone for some reason. I enabled use cage like you said and it... worked!?

So what does "use cage" actually do? I've been googling the crap out of this and it's not making any sense. You say to use it when you mesh it "smooth" but at what point does smooth become "not smooth?"

daniellooartist · Jun 2015

I'm talking about the xNormal "use cage" checkbox in particular. I know how cages work but I don't understand the circumstances in which you check "use cage" in xNormal. I did this bake twice and both times i used a cage. I unchecked the button and it worked. I seperated the objects and for some odd reason it didn't work. The solution was to recheck the check box that caused the problem in the first place. It makes zero sense.

m4dcow · Jun 2015

So by checking use cage in xNormal, it does an averaged projection. If a cage file is not present xnormal projects from an infinite distance thereby projecting distant pieces of the mesh on the wrong spots.

Even if you have a cage assigned and don't check use cage, it ignores the cage and projects along the vertex normals, which if you a mesh with hard edges it creates projection errors. Its kind of weird and I would just work through the quirks. I started baking with substance designer now, version 5 handles exploding easily (or rather not having to). Plus I do most texturing in substance now too so baking with it is logical.

daniellooartist · Jun 2015

So I am beginning my final bake. I baked each piece out separately to see if they work independently. Each piece works on it's own just fine as illustrated below. HOWEVER, once i export the whole thing (high, low, and cage) then bake it,the error starts happening again. I have done with with a perfect cage and I tried it with the use cage on, then i tried it with the use cage off. It looked screwed up each time. What the actual Hell is going on!? As usual, the source files are below along with an illustration

Outliner guide
High: high02
Low: lowTRIANGULATED02
cage: cage02

Source: https://www.dropbox.com/s/nodo03i8s7l89vo/table18.ma?dl=0

Stromberg90 · Jun 2015

Could you screenshot your xNormal settings from High definition meshes, low definition meshes and baking options?

daniellooartist · Jun 2015

What do they need to be at? Its all default with a 2048 bake and a cage assigned.

Stromberg90 · Jun 2015

Well there can be something that is easy to miss, that others could spot.

m4dcow · Jun 2015

Haha, well I found the issue with the WTF parts. When you export from maya... it only exports one instance of any instanced objects, so if you open the 3d viewer in xNormal, and only show the high poly you can see what is missing. So select all of your high poly objects, and go to Modify-Convert-Instance to Object before exporting.

After doing that I also found that one of the table legs (2 since they are mirrored) were projecting weirdly and it was because the cage didn't encapsulate the low poly so there was artifacting. This also happens on the section where the table top connects to the legs.

Also a lot of the ornamental stuff still doesn't line up with high poly and low poly versions, and with most of it I would suggest not having separate geometry pieces, maybe cutting in some simpler geometry and letting the normal map work for you.

daniellooartist · Jun 2015

It seems like we are getting closer. I disabled the instancing and it already is looking much better. However there is a new problem. For some reason only 2 legs are getting messed up. I extended the cage even more on the areas you mentioned. I have a new source. The reason I'm trying to model this in separate pieces is so that if I ever need to add anything I will only have to rebuild the cage for that one part as opposed to 1 tweak = remake the entire cage.

Outliner guide
High: high02
Low: lowTRIANGULATED02
cage: cage02

Source: https://www.dropbox.com/s/5r2tn5qsfl6rxnu/table19.ma?dl=0

daniellooartist · Jun 2015

I am also having issues with this area. This is when i bake it out in isolation so i know it's not effected by any of the other objects. It only happens on 2 corners. Cage looks just fine in that area.

m4dcow · Jun 2015

Yeh, I mentioned the bit about the legs in the previous post, I fixed them by enlarging the cage (change translate axis to move along normal in vertex mode). I'm not sure why it won't work for you.

I noticed that other part too, but I didn't really look too closely into it.

daniellooartist · Jun 2015

Were you able to sucessfully bake all 4 arms of the table?

m4dcow · Jun 2015

daniellooartist wrote: »

Were you able to sucessfully bake all 4 arms of the table?

Yup

Here's a low poly, a cage and a 2k normal map
https://dl.dropboxusercontent.com/u/10105386/pc/table150606.zip

I deleted the ornamental geo to make it simpler for me (many didn't line up with the high poly, and might also require exploding the mesh). I changed some vertex normal here and there like the table top where regular smoothing will produce artifacts. There's still an issue with the table top though, but that's in the high poly.

Also when doing UVs, there are always seams where there is a hard edge, but it shouldn't be like this the other way around. So on the center spindle of the table I smoothed the edges where there were seams, no need for a hard edge there.

I also noticed that you didn't actually mirror the table top around, you rotated it, which can and did give me some issues with seams. On a contiguous piece like that it is better to actually mirror, whereas things like the legs you can rotate them after duplicating.

daniellooartist · Jun 2015

This is making no sense. How is it that yours works and mine does not? I baked the center spindle and the arms and i got the problem in image 1. Bake out both of the 2 arms by themselves and this happends (image 2).

m4dcow · Jun 2015

The 2 arms by themselves might be uvs not offset, ie: overlapping in UV space. The 1st might be not having the cage loaded.

Try a bake in xNormal with the files in the zip and see how it turns out.

daniellooartist · Jun 2015

wait a second? Does the cage have to surpass the low poly as well as the high poly?

m4dcow · Jun 2015

daniellooartist wrote: »

wait a second? Does the cage have to surpass the low poly as well as the high poly?

Yes it needs to encompass the low poly.

daniellooartist · Jun 2015

Dude, it worked! Thank you very much I'm going to edit the initial post with everything you taught me in case someone google's this in the future. I don't want others to lose sleep over this.

Normal Map: Meshes projecting into eachother

Replies