Correct way of baking normals for UE4
Feeling lost.
I've been having trouble trying to bake correct tangent space maps for UE4.
My first approach was the traditional one. I use Maya 2014.
I exploded my mesh, triangulated, managed smoothing groups according to UV splits, and baked a Tangent Space normal.
The results looked clean in every application I had experience with, including 3DO, Marmoset and obviously Maya itself.
(picture related)
However, upon trying to view it in Unreal Engine 4, the results were less than satisfying.
I am aware tangents are calculated differently in each application, and that for better results you are supposed to synch them.
Unfortunately I am not familiar with this kind of workflow. So I did some research.
I found this:
https://docs.unrealengine.com/latest/INT/Engine/Content/Types/Textures/NormalMaps/Creation/index.html
Apparently, there is this new workflow involving "tangents and binormals" which supposedly would remove the need for supporting geometry and produce clean results on the spot.
It would be great, IF I could figure out what I'm doing wrong...
This is what I did so far:
1) I exported an FBX with smoothing groups, without tangents & binormals (UE4 doc says T&B are important, but in the settings screen they are NOT checked)
2) I also exported an FBX with smoothing groups AND T&B checked, just to be extra sure
3) I imported the FBXs in xNormals 3.19.1, AND the hipoly
4) Baked separately the one with and the one without T&B. Got 2 maps.
5) Imported the lowpoly FBXs in UE4.
I DIDN'T assign the map in maya and re-export them. I imported the lowpolys I used for baking, and assigned the normalmaps in the UE4 material editor.
Yes, I changed the compression setting and texture group. The software recognized the files as Tangent Space Normals.
Here is where the firts problem appeared, however.
The documentation shows a checkbox, in the FBX importa settings, which says "import normals". This checkbox doesn't exist in my FBX import settings windows.
(see picture)
6) Anyway, both models had horrible normals. They didn't look nearly as clean as I hoped. More, they looked bad in other applications as well, because I also tried to make a single smoothing group regardless of UV splits, as suggested in teh UE4 docs.
I also tried to keep my original smoothing groups, they still look bad in UE4.
PART 2: object to tangent
So I read xNormals can convert an object space map into a tangent space one, and this workflow is used to bake object space maps in your application of choice and then convert them in a tangent space map
This is what Handplane does, if what I read is correct. xNormals has this function as well, so I tried this too.
I baked my object space map in Maya. It looked super-clean as expected from an object space map.
Then I imported the lowpoly and the object space map in xNormals and converted it to a tangent space.
It suddenly looked terrible again, both in Maya and in UE4
conclusion
I feel like I've ran out of options and ideas.
I don't know what to do anymore.
Please help me.
Here are the lowpoly and the hipoly, if you want to examine them or attempt the bake yourself.
http://www.filedropper.com/barrellbake
If you can make this look good in UE4 and explain how you did it, I couldn't thank you enough. If I solve this, I promise I will help anyone I see struggling with similar issues, no matter how slow or lazy they look.
(baking considerations:
-since it's a cilinder, when baking in the traditional way I used to make a separate bake for the front, so that it was planar, and then blended things in Photoshop. However, after discovering NDO2 this is no longer needed... but the bullets were already modeled by the time I catched up with NDO2
-As you can see from the LOWPoly UVS, the object takes a very small part of the map. This happens because the barrel is just a small part of the whole weapon. I only shared the barrel as it's the part that looks the worst)
I've been having trouble trying to bake correct tangent space maps for UE4.
My first approach was the traditional one. I use Maya 2014.
I exploded my mesh, triangulated, managed smoothing groups according to UV splits, and baked a Tangent Space normal.
The results looked clean in every application I had experience with, including 3DO, Marmoset and obviously Maya itself.
(picture related)
However, upon trying to view it in Unreal Engine 4, the results were less than satisfying.
I am aware tangents are calculated differently in each application, and that for better results you are supposed to synch them.
Unfortunately I am not familiar with this kind of workflow. So I did some research.
I found this:
https://docs.unrealengine.com/latest/INT/Engine/Content/Types/Textures/NormalMaps/Creation/index.html
Apparently, there is this new workflow involving "tangents and binormals" which supposedly would remove the need for supporting geometry and produce clean results on the spot.
It would be great, IF I could figure out what I'm doing wrong...
This is what I did so far:
1) I exported an FBX with smoothing groups, without tangents & binormals (UE4 doc says T&B are important, but in the settings screen they are NOT checked)
2) I also exported an FBX with smoothing groups AND T&B checked, just to be extra sure
3) I imported the FBXs in xNormals 3.19.1, AND the hipoly
4) Baked separately the one with and the one without T&B. Got 2 maps.
5) Imported the lowpoly FBXs in UE4.
I DIDN'T assign the map in maya and re-export them. I imported the lowpolys I used for baking, and assigned the normalmaps in the UE4 material editor.
Yes, I changed the compression setting and texture group. The software recognized the files as Tangent Space Normals.
Here is where the firts problem appeared, however.
The documentation shows a checkbox, in the FBX importa settings, which says "import normals". This checkbox doesn't exist in my FBX import settings windows.
(see picture)
6) Anyway, both models had horrible normals. They didn't look nearly as clean as I hoped. More, they looked bad in other applications as well, because I also tried to make a single smoothing group regardless of UV splits, as suggested in teh UE4 docs.
I also tried to keep my original smoothing groups, they still look bad in UE4.
PART 2: object to tangent
So I read xNormals can convert an object space map into a tangent space one, and this workflow is used to bake object space maps in your application of choice and then convert them in a tangent space map
This is what Handplane does, if what I read is correct. xNormals has this function as well, so I tried this too.
I baked my object space map in Maya. It looked super-clean as expected from an object space map.
Then I imported the lowpoly and the object space map in xNormals and converted it to a tangent space.
It suddenly looked terrible again, both in Maya and in UE4
conclusion
I feel like I've ran out of options and ideas.
I don't know what to do anymore.
Please help me.
Here are the lowpoly and the hipoly, if you want to examine them or attempt the bake yourself.
http://www.filedropper.com/barrellbake
If you can make this look good in UE4 and explain how you did it, I couldn't thank you enough. If I solve this, I promise I will help anyone I see struggling with similar issues, no matter how slow or lazy they look.
(baking considerations:
-since it's a cilinder, when baking in the traditional way I used to make a separate bake for the front, so that it was planar, and then blended things in Photoshop. However, after discovering NDO2 this is no longer needed... but the bullets were already modeled by the time I catched up with NDO2
-As you can see from the LOWPoly UVS, the object takes a very small part of the map. This happens because the barrel is just a small part of the whole weapon. I only shared the barrel as it's the part that looks the worst)
Replies
Ok, this is a little embarassing. Thank you, this improved the normal map drastically.
Really the devil is in the detail.
Thank you very much.
*next step*
This worked, to a certain extent.
This is the current result:
These are the same (small) artifacts you can see in Maya as well, on a dark material with high reflections.
It would be fine I guess, as with other maps it should become nearly unnoticeable. Or I could add supporting geoemtry.
But since it's not perfect yet, and since handplane and UE4 say nowadays you can get clean results without supporting geometry... I have to try. I'm watching this series right now:
https://www.youtube.com/playlist?list=PLJHtayLGTyCGKTQR1FZ6aKS34Vg4dsPRw
I'll be making some more attempts in the next hours/days until I'm satisfied or completely devoid of energy (aka dead)
Thank you again for the help so far
-object space map baked in xnormals from FBX with TB checked, converted in Handplane to TS
-object space map baked in xnormals from FBX without TB checked, converted in Handplane to TS
-object space map baked in Maya, converted in Handplane to TS using both the FBXs as base meshes
-object space maps baked in xnormals from both FBXs, converted in xnormal itself
-object space map baked in Maya, converted with xNormals to TS
+
tried both straight green or inverted green (thanks, UE4 node system)
and finally
-just inverted the green channel on the whole old tangent space map baked in Maya
results:
-all maps that involved Handplane looked very bad. Maybe it's the lack of a specific support for UE4?
-all maps baked in xnormals looked decent, but had visible shadows with high reflections
-maps baked in Maya looked great, just the same as in Maya and Marmoset. Bothe the Object Space converted in xnormals and the Tangent Space baked directly in Maya produced good looking results.
pic: the original TS map baked in Maya, as it looks in UE4 after inverting the G channel
Conclusion:
-Maybe it's me, maybe I don't understand what this "new" baking workflow is all about. I don't know. (and if so, please tell me) What I know is, in the end, the original map was the one that looked the best, once the green channel was inverted.
* Should I stick with the traditional way of doing normals? *
I am comfortable with baking in Maya and it seems like my techniques are still valid.
However, I can't stop thinking about the fact that handplane, synched normals and the "new workflow" explained by Epic Games, all promise to produce even better results.
(if only I could figure out what I'm doing wrong, that's it)
http://www.polycount.com/forum/showpost.php?p=2257792&postcount=1845
Also you may need to change this setting in xNormal http://www.polycount.com/forum/showthread.php?p=2254880#post2254880
I'm gonna give it a shot.
I'll be trying to bake this:
EXPORT SETTINGS:
ACTIVATING PLUGIN FOR MIKK TANGENTS, and RAY DISTANCE CALCULATOR
As I posted this image, the RDC was still running. Has been almost 3 minutes so far. I'm going to stop it now and press "copy results"
SETTINGS USED AND RESULTING MAP:
UE4 IMPORT SETTINGS (I'm using the original FBX I exported for xNormals baking)
NORMAL MAP IN UE4:
INVERTING GREEN CHANNEL IN UE4:
RESULTS
artifacts everywhere.
- Setting the normal in xNormal to x+ y- z+ is the same as reversing the green channel in UE4? Did I skip an important step?
- Did I miss any UE4 checkbox or settings related to tangent space or normals in general?
- Should have I opened the FBX in Maya, assigned the normal and then exported with Embed Media, instead of importing the FBX and the normal separately in UE4? Does this even make any differences?
- Should have I imported a custom-made cage instead of using the automated process?
- Is it just the shape that's too weird to compute, or the UVs that are too bad? I thought synced workflow was supposed to produce clean results regardless of how messed up the geo and how steep the angles were, without needing supporting geometry or carefully planned UV splits... I guess this has limits as well?
- The documentation even suggests to use a single smoothing group
(should have I used a single smoothing group instead of making UV Splits hard?)
changes made:
-Base FBX is a single smoothing group, as suggested in the UE4 documentation
-Normal was created with x+ y- z+ in xnormals, all other settings unchanged
-Opened in Maya, exported with Embed Media, imported in UE4, green channel not flipped
changes not made:
-No manual cage
-UVs are the same as before
RESULTS:
This is getting worse
(EDIT: NOTE THAT INVERTING THE GREEN CHANNEL DOES NOTHING TO IMPROVE THE LOOK IN THIS CASE. I TRIED)
_________________________________________________________________________
THIRD ATTEMPT:
New UVs
Hand-adjusted cage in Maya. Baking an Object Space
What I did here:
-Still left the mesh with some difficult angles to better test the limits of the synced workflow.
-Baked an object space map
-Converted said map in xNormals
Result:
Same artifacts as the first attempt.
CONSIDERATIONS:
- THE OLD RULES STILL APPLY:
Unless you can get a clean old tangent space bake in your application of choice, (i.e. Maya) you still have to mess with UV splits on steep angles, adding support geometry where needed, fighting incorrect shading and carefully plan each triangle orientation.
Synced workflow doesn't make any of this easier. (but it should! the doumentation said it would!)
- WHAT SYNCED WORKFLOW DOES:
It solves ONLY the problems that can happen if your baking software isn't aligned with the destination engine.
This has nothing to do with how you plan your normal map, your lowpoly, your hipoly etc.
- what I think of this paragraph:
It just doesn't work. (???? but it should!)
Did they have several UV islands, one per plane, to achieve clean results with that? Or something? What am I missing?
How. The. Heck.
Why can't I make it work.
I'm working with modo and doing this workflow when baking using the rounded edge shader and it's working perfectly.
Could you upload the model so we can have a look?
Also, just so you know, you can flip the green channel via the textures editor:
Yes, the last attempt was all about baking an object space and converting it into xnormals.
Here are the models.
http://www.filedropper.com/archive_8
Cubostrano_LowP = the first cube, edges hardened where UVSplits exist
Cubostrano_LowP_OneSG = a single smoothing group
Cubostrano_LowP_ObjToTang = the new UVs. Edges hardened where UVSplits exist. I used this for the Object to Tangent workflow
Cubostrano_Hip = the Hi Poly
Thank you for the information about flipping the green channel
I use Modo as my 3d application.
Here's what I modified in your low poly model:
1. A lot of stretching occurred in one of the side pieces, I had to cut down the middle part of it and relax the UVs.
2. Set all edges to soft, then set only UV borders to hard (there are scripts for this, 1 click process - I would have give you more details on where to find it but I don't use Maya, sorry)
3. I duplicated the low poly, pushed all verts along their normals to create the cage, couple verts had to be modified manually since some intersecting occurred.
- Imported into Xnormal, set files, cage and bake into tangent space normal with Y-
- Test in unreal
Here are some images:
Modified UV: * Highlighted edges are set to hard.
UE4 Import settings:
Test screenshots: * I made a shiny metallic material, makes it easier to spot the errors and artifacts.
For production quality I would have bake in Xnormal as a 16-bit TIF, then convert in Photoshop to 8-bit, save and use that in unreal.
Here are the low poly and the cage FBX files:
Low Poly
Cage
I hope this helps you!
First, I see your cage has the same smoothing groups as the original model. After reading some information online, I was worried this could be a problem. Looks like it isn't.
I'll try a bake right now in xnormals. If it comes out clean (and I really hope it does) I will try to export a cage from Maya and repeat the process.
If it doesn't come out clean, then my problem lies within xNormals, and can be hopefully solved once identified.
If the bake using your cage is clean, but the bake using my cage is not, then the problem is about cages.
Experiments time, updates soon
EDIT1:
Baking worked with your cage! Wow! This is, like, super clean. It's amazing.
I can't thank you enough!
(tinypic captcha: Helping Hand :P )
NOW
The next step is about making a cage myself, and testing it. If it works, I will be independent from now on.
EDIT2:
Just to be extra sure, I unlocked the normals on the base lowpoly shape, and re set every edge by hand.
I created a cage in Maya. I unparented it, and exported it as an FBX.
I imported my own cage in xNormals. Baked.
The results were perfect.
My reaction:
[ame]
There's really nothing I can say at this point. (I just had to use a custom made cage in xNormals and invert the green channel.)
It wasn't a big deal but - man, it kept me blocked for oh so long.
I am infinitely grateful to everyone who contributed to this thread.
I'm looking forward to sharing this knowledge with other people struggling with the same issues.
Thank you guys. You are the best.
Where have you read that?
As far as I know, the SGs of the cage doesn't affect anything, the cage is there only to affect the distance that the rays are being cast.
I might be wrong though, I'm fairly new to this whole process myself.
* I ran a test with 3 different cages, 1 with the same SGs as the low poly, 1 with all edges set to hard, 1 with all edges set to soft - all resulting bakes are the same.
Glad you got it sorted out :]
Yep. Thank you again.
My map is clean now. I rebaked everything and it's looking just like I wanted.
Of course now it looks bad in 3DO but it's not like I really care or anything, as long as I can comfortably add detail and it looks fine in UE4.