I am not that familiar with either of those tools. It is going to depend on how they work and what process they use to blend micro texture into the normal map. I would just test it out early in your process so you can see if it is causing issues before you spend a ton of time on it.
yeah, I tested it and Substance Designer works alright.
going from 3d max2012 object space - maya 2013 tangent(using .h3d) I some time get random edges baking into the tangent map. I have looked at a tangent map in max (max is not baking the artifact) and the object space map in maya (before using handplane) and it has no edge artifact but when processed through handplane it gets edges visible in random areas. Any thoughts on what is causing this?
Yeah, Unity's forward rendering has some minor issues. I use the deferred rendering pipeline but for some people that isn't going to be a viable option.
There's a couple of things you can do to work around it, but they're more expensive to compute and the good version isn't easily compatible with Surface Shaders.
To improve things, you can normalize the light and view vectors per pixel in the surface shader.
To fix things, you need to convert the tangent space map to object space inside of the pixel shader, then do lighting in object space. This is not really wish doing in a Surface Shader as it results in a lot of code duplication and hacky workarounds.
I have never used this software but today I wanted to try it. I usually bake my maps in xNormal so I know how to set up proper my meshes, but handplane don't work for me, it shows really strange normals. I even watched this video but nothing helps:
[ame="https://www.youtube.com/watch?v=l59_R7RENxE"]handplane baking tutorial - YouTube[/ame]
And here is result after render my baked textures from xNormal in handplane. It looks terrible:
If someone use 3ds Max 2012 and everything works fine, then it would be great to know what I doing wrong. I know that models are for game engines but it's good to have good maps also in 3d software.
You need to set the normals to qualified mode via the 3dsmax.ini file.
Enabling qualified normal mapping is done by adding the following lines to the end of 3dsmax.ini (C:\Users\UserName\AppData\Local\Autodesk\3dsmax\2011 - 64bit\enu\3dsmax.ini).
Just been having a quick look at your website, I'm wanting to implement this into my pipeline for a small indie studio I work at. I know this used to be a paid app but I have no idea about now, your website doesn't seem to mention it. Can you clarify at all?
So, I'm condensing all the info on the issues I have in this thread. I did more tests and documented the artifacts/errors that were occuring, hopefully somebody can help me get to the bottom of this.
OBJ export settings
FBX export settings
Newest tests done in Maya 2012, files can be found HERE. The only bakes working correctly are the ones done using smoothed normals and baking in Mikk T-Space natively in xNormal.
Running into an issue, I reformatted my PC and now xNormal (using the Handplane plugin for normal maps) is returning some strange artifacts on my normal map, any ideas?
AO and BentNormal maps are baking correctly and the issues in the normal map along the top of the head are present in engine too.
EDIT: Sigh, never mind, I forgot to harden the normals along the UV seams, that fixed it ^^
Pior: Yep, that's what's been driving me crazy. Before the reformat, for 99% of meshes I would just do a nice smoothed normal for the whole mesh and bake without issue. It's only since the reformat that I've been having these issues.
belkun/Andummy: Just tried that, causes the same artifacts, though to a lesser degree. Tried baking the object space with both +y and -y which yields the same results.
Yep, I'm aware of that fact/post but the issue persists in-engine, see image below. Additionally, I've never had those issues before the reformat with the handplane plugin. If the angle of the edge could be a factor, I've also baked way harsher angles without it trying to compensate with these artifacts
I posted this over in the Dota2 subforums some time ago, reposting here for visibility. The OS map, generated in xNormal looks fine, but when I convert it in handplane or bake it directly in xNormal using the handplane plugin, artifacts happen, especially along UV seams.
Check the other thread, it doesn't. Background is, I've been baking this way for about a year now, never had normals come out looking like this using this exact workflow. Then I reformatted my PC, something somewhere broke along the way and I cant for the life of me figure out what.
Alright, more info. As usual, here's a link to the files I'm working with.
Conditions of these tests:
1) Object Space (OS) maps are baked in +X +Y +Z
2) Tangent Space (TS) maps are baked in +X -Y +Z
3) Software versions are xNormal v3.18.6.28468 and handplane v1.4.1
And the results:
OS baked in xNormal with mesh normals hardened at UV seams -> TS baked in handplane using the Source preset
-> visible artifacts when using normal map as diffuse, but no artifacts in engine
-> no visible seams, but seam on mirror axis
OS baked in xNormal with mesh normals softened -> TS baked in handplane using the Source preset
-> visible artifacts when using normal map as diffuse, and same artifacts in engine
-> no visible seams, but seam on mirror axis
TS baked in xNormal using the handplane plugin with mesh normals hardened along UV seams
-> visible artifacts when using normal map as diffuse, but no artifacts in engine
-> seams on UV edges
TS baked in xNormal using the handplane plugin with mesh normals smoothed
-> visible artifacts when using normal map as diffuse, and same artifacts in engine
-> no visible seams, but seam on mirror axis
I did another test where I split the vertecies along the edges that seemed to be the primary source of problems, in this case the edge of a blade. Both TS maps were baked in xNormal with the handplane plugin.
This should be done before baking object space normals since the object space normals need to correspond to the correct side of the model. I was able to grab the correct set of shells and recalculate the tangent space map and it looks like this:
"Running into an issue, I reformatted my PC and now xNormal (using the Handplane plugin for normal maps) is returning some strange artifacts on my normal map, any ideas?
AO and BentNormal maps are baking correctly and the issues in the normal map along the top of the head are present in engine too.
EDIT: Sigh, never mind, I forgot to harden the normals along the UV seams, that fixed it ^^"
Adding a hard edge will fix those patches but we are trying to come up with a solution so that you don't need to worry about those edges.
That looks like an issue with overlapping UVs to me. Try to keep in mind how all the software works.
So first off your baking software casts a ray from your lowpoly model to the highpoly and grabs the orientation of that normal which is then stored in object space. That object space normal is specific to a single angle in the world so the mirrored normal isn't the same color. If you have overlapping UVs that sit in the 0-1 square the baking software can be confused about which of the possible pixel colors to fill into your object space map. Even if your bake software happens to pick a continuous set of shells during the bake there is no guarantee that handplane will grab the exact same set when it does it's calculation.
Handplane works by going to each pixel in the object space map, checking where it sits on the lowpoly model, and then doing the tangent space calculation. Just like in the baking software, when you have overlapping UVs that are inside the 0-1 coordinate space, handplane can't know which copy of the shell is the correct one to do the tangent space calcution with.
Does handplane automatically flip the Y channel when outputting for Source? :poly121:
If you are talking about the xnormal plugin then you should leave your xnormal swizzle settings at default and the channels will be oriented correctly. If you are using object space bakes inside handplane then everything should automatically orient itself.
As a follow up-
Here is some testing with the head model Spudnik posted that was giving him issues. I spent hours on this model a few months ago and never thought to check the UVs.
First I took the model and smoothed the whole thing in order to stress handplane. I re-exported that lowpoly, did a bake with his highpoly and cage using the source engine xnormal plugin. As you can see it has the issue with the off color patches:
Next I fixed the lowpoly UVs so they weren't overlapping in 0-1 space:
The main patches are all gone!
There were still some issues around the edges of the shells. I overlayed the UVs to see if they were just in the padding but they are issues within the normal map.
Big image: http://i.imgur.com/wjpLYus.jpg
The first step in debugging this is to see if the problem is caused by how the normals are being calculated or if the cause is a projection error from the bake itself. The easiest test for that is to bake to object space since that requires no tangent space math and the lowpoly normals are totally ignored.
The patches are still present, meaning the issue is with the projection not the normals calculation.
Next up I loaded up xnormal's viewer and checked out the lowpoly, highpoly, and cage together. There were a few places where the cage was too small:
Most of the other problem spots look like they are caused by the highpoly forms that are close to being silhouette scale but aren't reprsented in the lowpoly model.
For example, the main exoskeleton/plate surfaces that make up the outside of the model are seperate objects and there are some small gaps:
If you email me your files I can take a look. I can probably figure it out from just your low poly model and object space map. Also, what tangent space? alecmoody@gmail.com
Unreal uses smoothing groups to calculate normals (unreal calculates normals and tangents when you aren't using the explicit normals/import tangents workflow). I took a look at your smoothing groups and they correspond to the areas where you are having errors. This is what smoothing group 1 looks like:
I am assuming modo uses a hard edge system like maya. Does anyone know of a reliable way to get smoothing group data out of modo?
Can you post an image of the modo FBX export options?
Edit:
Doing some research I think you are going to have a very hard time getting your modo smoothing information into unreal. Your best bet is probably to smooth everything, no hard edges at all.
MODO does export smoothing groups to FBX, but I'd recommend not using them as they're kind of broken.
Vertex normals export properly, but there is a bug in 801 SP3 for vertex normals if you're exporting multiple items from within a group.
UDK had some issues importing stuff but I haven't used it in years, but UE4 seems to work just fine. Although you won't get MikkTSpace out of Handplane currently.
I'm using SP3 but I also have SP2 installed. Will give it a try
@Alec thanks for looking into this, what's strange then is that UDK doesn't give me "no smoothing group" error when I import the file and I've done extensive testing with a cube before starting this project. I've tested different cubes with smoothing groups/without them - baked with a cage and with averaged projection and everything went alright - the cube with vertex normals baked with a cage, UV splits, smoothing groups, converted in Handplane was the best one obviously.
I'm actually ok with how the final mesh with diffuse and a couple of overlaid normal maps looks like but I want it to be as perfect as possible so I'll do more testing today and post results here.
I think the file does have smoothing groups. They are just really oddly set up.
You could fully split your verts where you want smoothing splits and then smooth everything. It's pretty hacky but it seems like that might be your most reliable way to control smoothing.
MODO does export smoothing groups to FBX, but I'd recommend not using them as they're kind of broken.
Vertex normals export properly, but there is a bug in 801 SP3 for vertex normals if you're exporting multiple items from within a group.
UDK had some issues importing stuff but I haven't used it in years, but UE4 seems to work just fine. Although you won't get MikkTSpace out of Handplane currently.
*sigh*
So what to use then, if FBX is STILL (!) broken? I mean, we got your wonderful Vertex Normal Kit, could harden/soften edges and such... (which is STILL not "on board in MODO")
-but what is it all worth, if you can not export it proper?
Can you tell, which workflow/format would be working for export?
(Maybe DAE or OBJ, I guess?)
It´s a pitty having to switch to another app, just for setting up hard-edges and proper fbx exporting... kind of weird
You can export FBX from MODO and into UE4 just fine. I do it all the time. You don't get smoothing groups (as Farfarer said you should ignore them - they suck in MODO) but you can certainly set hard edges and such using his Vertex Toolkit. It works great. You don't need another app.
*sigh*
So what to use then, if FBX is STILL (!) broken? I mean, we got your wonderful Vertex Normal Kit, could harden/soften edges and such... (which is STILL not "on board in MODO")
-but what is it all worth, if you can not export it proper?
Can you tell, which workflow/format would be working for export?
(Maybe DAE or OBJ, I guess?)
It´s a pitty having to switch to another app, just for setting up hard-edges and proper fbx exporting... kind of weird
Cheers
As Warren said... as far as I know exporting stuff using my toolkit works with everything I've tried it with (Dota2, UE4, Unity3D, 3DS Max, xNormal, Handplane).
Smoothing groups do actually export correctly to FBX, it's just that MODO doesn't always blend between them correctly. So what you get from your mesh in-engine isn't guaranteed to be what it looks like in MODO.
That said, if you want to use smoothing groups, my plugin will blend between them correctly.
There is a bug in 801 SP3 with exporting objects with custom normals if there are multiple objects in a group.
It's actually a bug that's arisen as a result of a bug I filed to try and get meshes exporting to UE4 correctly. But that turned out to be a bug with UE4, not with MODO. So, uh, sorry about that :P I'm told it'll be fixed in SP4 (or you can stick to SP2 for exporting to FBX).
UE4 4.7 preview will sync properly to MikkTSpace (xNormal's default) for static meshes (not skeletal ones yet) and so no longer has that bug.
I was using Vertex Toolkit 2.4 and Modo SP2/SP3 for this. I've tested multiple versions of the file and Modo SP2/SP3 and still get the same error. Actually if I don't triangulate the mesh there are less errors. I haven't tried EVERYTHING (like default Modo smoothing groups setup) and that's the reason why I haven't posted screens. I just can't figure out how you do smoothing groups in Modo without Farfarer's plugin.
I'll do everything from ground up in SP2 once again just to be sure but I think it's not the reason in this case.
and yes, everything looks peachy in UE4 4.4 :
This is the same mesh done with Vertex Normal toolkit 2.4 and baked in Substance Designer in Mikkt space.
I don't know what 2.61 has fixed but it works as intended now.
Anyway even if you have no access to Vnorm toolkit you can set smoothing groups manually by going to Geometry - Polygon - Assign smoothing group and it'll work too.
Now everything is ok! Also it probably explains why I was having problems with smoothing in mid-poly meshes after sending them from Modo to Maya.
My toolkit doesn't really do anything with smoothing groups - in fact the only thing it's capable of doing to affect them is if you tell it to remove all smoothing groups as you convert a mesh to hard edges.
Although it does support them and will respect then if they are on a mesh.
v1.5 changes
Added support for blizzard engine/ Starcraft 2
Added a button to open output directory
Recompiled Xnormal/source plugin for version 3.18.10
Does someone have a link to where Epic said they will be changing how normal maps work in UE4.7 I couldn't find anything online, we got our UE4 maps working perfectly using MightyBake so I hope this update doesn't break our existing maps or invalidate our current workflow as we've had error free bakes for a long time now in UE4.
Just to clarify, besides being able to sync the normals via the upcoming UE4 Handplane support. Is there any other way to sync the normals? I.e. we still can import tangents and binormals in 4.7? Not sure what the mikk tspace support changes in regards to that.
Nevermind I found it, this seems a bit sketch actually if it's going to auto convert everything we've already made, I'll have to test an existing asset and see if the normals are now out of sync.
Replies
yeah, I tested it and Substance Designer works alright.
I checked this thread with regards my issue which I think is related,
http://www.polycount.com/forum/showthread.php?p=2194972#post2194972
All down to Unity shaders. I am asking my programmer colleague to document what he did to the shader but all good now.
cheers,
To improve things, you can normalize the light and view vectors per pixel in the surface shader.
To fix things, you need to convert the tangent space map to object space inside of the pixel shader, then do lighting in object space. This is not really wish doing in a Surface Shader as it results in a lot of code duplication and hacky workarounds.
[ame="
And here is result after render my baked textures from xNormal in handplane. It looks terrible:
https://copy.com/DAVA9BO4ZnKi0DUF/mmmm.JPG
I tried everything - flip green, red channel, nothing works.
@edit SOLUTION FOR MY PROBLEM
I made a mistake in handplane, proper normal map is showed bellow
Ok, so I have noticed that normals rendered in Handplane doesn't work proper in 3ds Max 2012 shader. Take a look on this:
You can read about this also here:
http://www.laurenscorijn.com/future-xoliulshader-support.html
If someone use 3ds Max 2012 and everything works fine, then it would be great to know what I doing wrong. I know that models are for game engines but it's good to have good maps also in 3d software.
Enabling qualified normal mapping is done by adding the following lines to the end of 3dsmax.ini (C:\Users\UserName\AppData\Local\Autodesk\3dsmax\2011 - 64bit\enu\3dsmax.ini).
[ViewportNormalMapping]
ViewportNormalMappingType=Qualified[/quote]
A few questions:
Do I need to triangulate my low poly before importing in to handplane and/or convert it to editable mesh?
Are these settings correct? It was -Y by default, I tried +Y but it didn't change anything.
Thanks in advance!
Wasn't too sure as in the program is says not for commercial use still.
OBJ export settings
FBX export settings
Newest tests done in Maya 2012, files can be found HERE. The only bakes working correctly are the ones done using smoothed normals and baking in Mikk T-Space natively in xNormal.
Older stuff:
http://www.polycount.com/forum/showpost.php?p=2129312&postcount=17
http://www.polycount.com/forum/showpost.php?p=2129588&postcount=21
http://www.polycount.com/forum/showpost.php?p=2129629&postcount=23
http://www.polycount.com/forum/showpost.php?p=2202968&postcount=26
http://www.polycount.com/forum/showpost.php?p=2203066&postcount=223
http://www.polycount.com/forum/showpost.php?p=2203127&postcount=225
http://www.polycount.com/forum/showpost.php?p=2204008&postcount=228
There are a couple of things going on in your posts. First, the cube and sphere mirroring test:
You need to offset mirrored UVs by one full square like this:
http://i.imgur.com/7nf80JQ.png
This should be done before baking object space normals since the object space normals need to correspond to the correct side of the model. I was able to grab the correct set of shells and recalculate the tangent space map and it looks like this:
http://i.imgur.com/y3XT3GA.png
As far as this section goes:
"Running into an issue, I reformatted my PC and now xNormal (using the Handplane plugin for normal maps) is returning some strange artifacts on my normal map, any ideas?
AO and BentNormal maps are baking correctly and the issues in the normal map along the top of the head are present in engine too.
EDIT: Sigh, never mind, I forgot to harden the normals along the UV seams, that fixed it ^^"
Adding a hard edge will fix those patches but we are trying to come up with a solution so that you don't need to worry about those edges.
That looks like an issue with overlapping UVs to me. Try to keep in mind how all the software works.
So first off your baking software casts a ray from your lowpoly model to the highpoly and grabs the orientation of that normal which is then stored in object space. That object space normal is specific to a single angle in the world so the mirrored normal isn't the same color. If you have overlapping UVs that sit in the 0-1 square the baking software can be confused about which of the possible pixel colors to fill into your object space map. Even if your bake software happens to pick a continuous set of shells during the bake there is no guarantee that handplane will grab the exact same set when it does it's calculation.
Handplane works by going to each pixel in the object space map, checking where it sits on the lowpoly model, and then doing the tangent space calculation. Just like in the baking software, when you have overlapping UVs that are inside the 0-1 coordinate space, handplane can't know which copy of the shell is the correct one to do the tangent space calcution with.
If you are talking about the xnormal plugin then you should leave your xnormal swizzle settings at default and the channels will be oriented correctly. If you are using object space bakes inside handplane then everything should automatically orient itself.
Here is some testing with the head model Spudnik posted that was giving him issues. I spent hours on this model a few months ago and never thought to check the UVs.
First I took the model and smoothed the whole thing in order to stress handplane. I re-exported that lowpoly, did a bake with his highpoly and cage using the source engine xnormal plugin. As you can see it has the issue with the off color patches:
Next I fixed the lowpoly UVs so they weren't overlapping in 0-1 space:
The main patches are all gone!
There were still some issues around the edges of the shells. I overlayed the UVs to see if they were just in the padding but they are issues within the normal map.
Big image:
http://i.imgur.com/wjpLYus.jpg
The first step in debugging this is to see if the problem is caused by how the normals are being calculated or if the cause is a projection error from the bake itself. The easiest test for that is to bake to object space since that requires no tangent space math and the lowpoly normals are totally ignored.
Result:
http://i.imgur.com/GwfDKap.jpg
The patches are still present, meaning the issue is with the projection not the normals calculation.
Next up I loaded up xnormal's viewer and checked out the lowpoly, highpoly, and cage together. There were a few places where the cage was too small:
Most of the other problem spots look like they are caused by the highpoly forms that are close to being silhouette scale but aren't reprsented in the lowpoly model.
For example, the main exoskeleton/plate surfaces that make up the outside of the model are seperate objects and there are some small gaps:
And those same gaps with the lowpoly overtop:
thanks, I offset those by 1U, now it bakes well.
alecmoody@gmail.com
Thanks for support, I'm sending you an email!
I am assuming modo uses a hard edge system like maya. Does anyone know of a reliable way to get smoothing group data out of modo?
Can you post an image of the modo FBX export options?
Edit:
Doing some research I think you are going to have a very hard time getting your modo smoothing information into unreal. Your best bet is probably to smooth everything, no hard edges at all.
Vertex normals export properly, but there is a bug in 801 SP3 for vertex normals if you're exporting multiple items from within a group.
UDK had some issues importing stuff but I haven't used it in years, but UE4 seems to work just fine. Although you won't get MikkTSpace out of Handplane currently.
@Alec thanks for looking into this, what's strange then is that UDK doesn't give me "no smoothing group" error when I import the file and I've done extensive testing with a cube before starting this project. I've tested different cubes with smoothing groups/without them - baked with a cage and with averaged projection and everything went alright - the cube with vertex normals baked with a cage, UV splits, smoothing groups, converted in Handplane was the best one obviously.
I'm actually ok with how the final mesh with diffuse and a couple of overlaid normal maps looks like but I want it to be as perfect as possible so I'll do more testing today and post results here.
You could fully split your verts where you want smoothing splits and then smooth everything. It's pretty hacky but it seems like that might be your most reliable way to control smoothing.
*sigh*
So what to use then, if FBX is STILL (!) broken? I mean, we got your wonderful Vertex Normal Kit, could harden/soften edges and such... (which is STILL not "on board in MODO")
-but what is it all worth, if you can not export it proper?
Can you tell, which workflow/format would be working for export?
(Maybe DAE or OBJ, I guess?)
It´s a pitty having to switch to another app, just for setting up hard-edges and proper fbx exporting... kind of weird
Cheers
Smoothing groups do actually export correctly to FBX, it's just that MODO doesn't always blend between them correctly. So what you get from your mesh in-engine isn't guaranteed to be what it looks like in MODO.
That said, if you want to use smoothing groups, my plugin will blend between them correctly.
There is a bug in 801 SP3 with exporting objects with custom normals if there are multiple objects in a group.
It's actually a bug that's arisen as a result of a bug I filed to try and get meshes exporting to UE4 correctly. But that turned out to be a bug with UE4, not with MODO. So, uh, sorry about that :P I'm told it'll be fixed in SP4 (or you can stick to SP2 for exporting to FBX).
UE4 4.7 preview will sync properly to MikkTSpace (xNormal's default) for static meshes (not skeletal ones yet) and so no longer has that bug.
Can you post a link to your plugin for dzibarik/other modo users?
I'll do everything from ground up in SP2 once again just to be sure but I think it's not the reason in this case.
and yes, everything looks peachy in UE4 4.4 :
This is the same mesh done with Vertex Normal toolkit 2.4 and baked in Substance Designer in Mikkt space.
Here is how smoothing groups look in Max now:
I don't know what 2.61 has fixed but it works as intended now.
Anyway even if you have no access to Vnorm toolkit you can set smoothing groups manually by going to Geometry - Polygon - Assign smoothing group and it'll work too.
Now everything is ok! Also it probably explains why I was having problems with smoothing in mid-poly meshes after sending them from Modo to Maya.
Although it does support them and will respect then if they are on a mesh.
v1.5 changes
Added support for blizzard engine/ Starcraft 2
Added a button to open output directory
Recompiled Xnormal/source plugin for version 3.18.10
Direct link to build 1.5:
http://www.handplane3d.com/handplane_v1_5.rar
8)
https://docs.unrealengine.com/latest/INT/Support/Builds/ReleaseNotes/2015/4_7/index.html
https://www.unrealengine.com/blog/unreal-engine-47-released