This is not the case, but does it matter if there are n-gons in the high poly? (but not in the low poly)
N-gons are fine in your high poly so long as it smooths alright.
What is your normals like on your low poly? Are they all smoothed + UVs sewn together?
Also, I don't know what this piece is exactly, but it looks like some sort of bolt. If it is, I wouldn't even have the indent modeled in. Could just be done with your normal map.
So in xNormal you want to check "use cage" on your low poly, even though you don't have a cage file. It is the only way xNormal does an averaged projection, and this file is simple enough that you won't run into issues without using an actual cage.
You had some results with mesh skewing the way you have your low poly setup, so I did a version where I just poked the face. Now my version looks like it has no depth but I exaggerated the angle of the hole a bit and got this:
You could also have a larger bevel on the edge of that area so it shows up better on the normal map.
I exaggerated the skewing in the high poly. I got the same result as you in the 1st image. I think this happens because of the averaged projection, while the vertex normal are flat, the averaged projection is skewed, if you move the verts along the normal this is basically what happens.
So, I don't want to get too complicated and/or introduce bad habits but as mentioned before, your the flat looking normal in this case is cause by a skewed projection.
In the following image, I used the right model to bake, and the left and right models share the same normal map.
You have to be careful doing this as any added geometry to prevent skews must not change the vertex normal. A good rule of thumb would be to limit this technique to planar geometry with hard edges.
I made a low poly with extra geometry to fix the projections, and a cage because there were some intersections. it also has a normal map, and a screws.obj file with all the different screw head model geometry, but identical vertex normal. If you apply the supplied normal map to all of them... they all look the same.
So use these files and try baking in xNormal yourself, the zip has your original high poly too, not a modified one.
In xNormal load the HeadHP.obj into the "High Definition Meshes" section.
Then load the head_low.obj file in the "Low Definition Meshes" section. Click the "Use Cage" checkbox, then right click on the same row select "browse external cage file" and load the head_cage.obj file.
Bake a normal map and it should come out exactly like what I have.
I stress again, adding geometry and then removing it is a thing that should only be done in certain situations, ie: when you don't alter vertex normals.
I don't understand why it did not come out the same as yours:
I followed your steps:
added the HP mesh from your dropbox file
added the LP mesh from your dropbox file
checked use cage
right clicked and added the cage mesh from your dropbox file
had those Xnormal settings
Saved as a PNG (results below)
Tried again but saved as a TGA (again, results below)
The three meshes on the left are from your dropbox file, the one on the right is my low poly model.
I even imported your files to Maya, created a new material lambert, and applied the PNG/TGA and this was the result:
Replies
I'm not sure what to do..
What is your normals like on your low poly? Are they all smoothed + UVs sewn together?
Also, I don't know what this piece is exactly, but it looks like some sort of bolt. If it is, I wouldn't even have the indent modeled in. Could just be done with your normal map.
Also, how close are you going to get to this particular screw? It seems really excessive to model in the indent, unless its a huge screw or something.
I created a new HP and got rid of the modeled X shape on the face of the LP.
Result:
- Normal map from XNormal looks good.
- The Normal map when applied in Maya looks bad.
Here is the HP:
Here is the LP:
I don't know what I am doing wrong/How to fix this..
If you can, try it in toolbag and set the settings for the mesh to -xnormal.
You are actually smoothing the HP before you send it to Xnormal, right? You don't just have smoothpreview on by pressing the 3 key?
Are you using a cage?
Maybe upload your LP and HP here and me or someone else more experienced could troubleshoot it better.
https://www.dropbox.com/s/xjysb8l8t6kllrs/ScrewHPLP.zip?dl=0
You had some results with mesh skewing the way you have your low poly setup, so I did a version where I just poked the face. Now my version looks like it has no depth but I exaggerated the angle of the hole a bit and got this:
You could also have a larger bevel on the edge of that area so it shows up better on the normal map.
This was my process:
HP Beveled and skewed (so there isn't a 90 degree drop):
Then I clicked smooth:
And this is the result: (Used Xnormal with LP cage checked)
In the following image, I used the right model to bake, and the left and right models share the same normal map.
This thread explains what is going on:
http://www.polycount.com/forum/showthread.php?t=147227
You have to be careful doing this as any added geometry to prevent skews must not change the vertex normal. A good rule of thumb would be to limit this technique to planar geometry with hard edges.
Could you show a breakdown of what you did to achieve this?
https://dl.dropboxusercontent.com/u/10105386/pc/Screws.zip
I made a low poly with extra geometry to fix the projections, and a cage because there were some intersections. it also has a normal map, and a screws.obj file with all the different screw head model geometry, but identical vertex normal. If you apply the supplied normal map to all of them... they all look the same.
So use these files and try baking in xNormal yourself, the zip has your original high poly too, not a modified one.
In xNormal load the HeadHP.obj into the "High Definition Meshes" section.
Then load the head_low.obj file in the "Low Definition Meshes" section. Click the "Use Cage" checkbox, then right click on the same row select "browse external cage file" and load the head_cage.obj file.
Bake a normal map and it should come out exactly like what I have.
I stress again, adding geometry and then removing it is a thing that should only be done in certain situations, ie: when you don't alter vertex normals.
I followed your steps:
added the HP mesh from your dropbox file
added the LP mesh from your dropbox file
checked use cage
right clicked and added the cage mesh from your dropbox file
had those Xnormal settings
Saved as a PNG (results below)
Tried again but saved as a TGA (again, results below)
The three meshes on the left are from your dropbox file, the one on the right is my low poly model.
I even imported your files to Maya, created a new material lambert, and applied the PNG/TGA and this was the result:
On the Bump2d you need to change it to tangent space.
Well you still have to use the bump mapping to apply it to regular materials, but just select tangent space normal.