xNormal Workflow question
I wanted to make normal maps to bake my High Poly to Low Poly. I have a model ready to go. Do I need to unwrap the model? If I do then do I need to collapse it back into an Editable Poly? I have pretty much tried both methods but the normal isn't coming out the way I expect, or at all really. So in general I am just lost. I have been watching tutorials and it looks like xNormal can make great normal maps but not when I do it.
Here is what's going on.
This is my High Poly:
Low Poly:
and the Normal Map xNormal made:
The models have smoothing groups cleared and set to one (1). They are aligned over each other. There are no n-gons. I exported them as .sbm with cage turned off on the low poly. The only thing is the model itself is not set a 0,0,0 in the coordinate space because there are other models in the overall scene. In xNormal the baking options are set: Size 1024x1024, Edge Padding 2, Bucket Size 512, Default Render, 4x Antialising, Normal Map checked with Y- set on Swizzle coordinates. Ray distance was ran for 5 seconds. What am I missing or what am I doing wrong?
Also any suggestions on how I could improve any of these models as this is my first time doing a high poly to low poly model.
Here is what's going on.
This is my High Poly:
Low Poly:
and the Normal Map xNormal made:
The models have smoothing groups cleared and set to one (1). They are aligned over each other. There are no n-gons. I exported them as .sbm with cage turned off on the low poly. The only thing is the model itself is not set a 0,0,0 in the coordinate space because there are other models in the overall scene. In xNormal the baking options are set: Size 1024x1024, Edge Padding 2, Bucket Size 512, Default Render, 4x Antialising, Normal Map checked with Y- set on Swizzle coordinates. Ray distance was ran for 5 seconds. What am I missing or what am I doing wrong?
Also any suggestions on how I could improve any of these models as this is my first time doing a high poly to low poly model.
Replies
For me it seems there is some mismatch with the UVs regarding your normal map result.
Froyok. That's the thing, I never unwrapped them. I was trying to find an answer online but every video I seemed to watch just show people exporting the Hi and Low poly model so thats what I did.
So I should unwrap them? If so do I collapse the model after I unwrap it or do I just keep the unwrap UVW in the modifier stack? Also when I unwrap them do the need to look exact? Like if I exported my Hi and Low poly unwrap and brought them into photoshop on their own layer should they line up exactly on top of each other?
It's important to load a cage before baking, especially for models with enclosed areas like your grating holes. The ray distance calculator can only do so well and doesn't work so well with some surface topology, but making a cage lets you determine exactly how far out the rays are cast across your entire model.
What I think you are saying is unwrap the sides (Left, Right, Front, Back ,Top, Bottom) and spread them out on the UVW Map but make sure they aren't touch? Does that mean the Top and Bottom can not overlap even though they are the same?
I also turned off cage because one tutorial I was doing said that xNormal would cage for me.
Quack!, Thanks as well. I assume the workflow you are explaining is starting from Low Poly and going to High? Also when you say to add another UVW Modifier doesn't that negate what I unwrapped before. So I would have to completely unwrap it again to add the new polys.
2) If you have any islands that are stacked on top of others (from mirrored or duplicate objects), move them 1.0 units in any direction in UV space. This is to prevent xNormal from baking multiple sets of information in the same texture space.
3) Create a cage with the Push modifier. Keep the cage as close to the surface of the low poly as you can but make sure it encapsulates the low poly. Triangulate and export both the low poly and the cage.
4) Load your HP and LP as usual into xNormal. On your LP, right click and choose Browse External Cage File and assign your cage. You'll receive a prompt to make sure the cage is exactly the same as the LP. Just click OK and then Generate. Your results should improve.
Bartalon, thanks for being descriptive. I understand that more. So basically when I unwrap nothing should be touching at all. Ill work on that as well.
It looks like floor grating, if this is something you intend to repeat many times down a corridor or something you'll be better off using a flat plane and alpha mapped transparency. Otherwise you're going to end up with a crazy high polycount if you need to instance this piece a lot.
I need to figure out how to do alpha mapping. My main concern is that if I alpha map it, will the holes look solid around the interior or will they be transparent as well? Or basically how would I go about making it work?
As you can tell I am very new to this. I know how to model for the most part but this is my first time trying to actually make a whole scene for a portfolio piece. I am hoping i'll learn a lot by doing this so that I can continue making pieces quicker and flesh out my, so far non-existent besides school work, portfolio.
So thank you all for your help so far.
I made a box, unwrapped it, collapsed it, then made a copy. I edited the copy to test out angles and general curiosity of something slightly more detailed. I then unwrapped the High Poly. The UVW's were all messed up except for the basics from the Low Poly. After a lot of breaking and stitching I came up with the second UVW Unwrap below.
Low on the Left, High on the right:
Low UVW:
High UVW:
I then made a cage by copying the Low Poly box and adding a push modifier on it. The white is the Low and the green is the Cage. I then converted the Low Poly and the Cage into an Editable Mesh so that it would be triangulated. I did not convert the High Poly to Editable Mesh, so it is still an Editable Poly.
I exported each object; High, Low, and Cage as a .sbm. I then loaded up xNormal. Put the High in the High, Low in the Low and added the Cage to the Low. I did not run Ray Trace as I assumed having a Cage would negate running it. Here is the normal map that was created.
So everything seemed to work except that I have weird lines through two sides of the Normal. I also applied the normal map onto the Low Poly in 3DS Max and no features show up.
I made a standard material, added a Normal Map into the Bump Maps, then in the Normal Map I added the Normal. I set the view port to realistic and then placed an Omni Light to move around and the surface is still flat.
I would NOT use the convert to editable mesh technique to triangulate your model. Bad things can happen there. Instead use the "Turn to poly" modifier and "Limit Polygon Size" to 3. Non destructive.
Here is the new Cage and Low:
And the new Normal:
I still don't get why I have weird lines going through the Normal. As far as I can see there aren't any lines doing that in the model. Over all though I am glad I am getting closer to sorting this out. The real challenge will be doing it to my grate model.
I also just found out that the Normal does show up when I render, it just doesn't show in the view port.
At first I gave every side of everything their own smoothing group, but I did make sure they matched up. Example: Top was smoothing group 5 while Bottom was smoothing group 6 on all of them, Low, High and Cage. Obviously this made everything too smooth.
I eventually realized that I did not want my High Poly smooth so I cleared those groups. I then cleared the Low after and told 3DS Max to Auto Group it, at this point I believe I did the same for the cage, I did A LOT of tests and they all blend together at this points.
Either way I came out with the lower right Normal.
I still see some problems, especially when looking over at the other Normals and what I feel should look right. Those are outlined here:
Anyways I feel like I am getting a better grasp on making normal maps. I still want to solve these problems, or get an answer on whether I am doing this right, before I continue onto my other pieces.
1. If you are using an unsynced workflow then harsh angles > 70° to 80°(varies depending on topology) need to be their own smoothing group. If synced, you may still need to, but to a much less degree.
2. Where ever you have a UV island, you need to have a smoothing group break.
There are some exceptions to this, but these are the basics.
The reason that some of your normals are skewed, is because they are being pulled toward the low poly verts/normals during the bake. If you want to remove the skewing, cut in a vertex or add a loop to that uv island. If the detail is small enough, it probably isn't necessary, but on some large details it is important to spend a few triangles to fix normal map skewing.
The portions of your normals that have uneven edges are due to your LP surface topology not matching your HP model closely enough, and possibly also the orientation of your vertex normals. Read this post regarding those types of problems, though the previous post I linked is probably more pertinent for foundational knowledge on normal mapping.
Where ever you have a smoothing split, you need a UV split as well.
If you unwrap a cylinder or whatever you don't need a smoothing split everywhere you split the UVs. If you do that the normalmap will have to compensate for that by creating gradients which are widely hated as they lead to additional problems.
They are sort of one in the same, it just depends on how you unwrap. If you go through and unwrap, the islands have their own smoothing groups. If you tweak your smoothing groups at harsh edges, you need to go back to your UVs and make them reflect the changes.
Or on organic models.
If these UV-splits were set to hard (like they are in the image) the normalmap would need to compensate for this to restore them to how they look on the highpoly more than if you had not placed the smoothing splits there. Splitting either UVs or smoothing groups is free if you've already split one or the other in that place. The normalmap will however compensate with gradients which can look like crap if the tangents aren't synced, there isn't enough bitdepth and it won't compress or mip as well as without them. These examples would probably work fine but you should keep this stuff in mind anyways as it will only benefit the model as there are no downsides to it.
Yes, I agree. It's more of a rule of thumb than a concrete fact. I suppose I was trying not to be confusing to someone trying to learn the basics by introducing the "exceptions to the rules."
More often than not, the UV seams will require a smoothing split. Surfaces like cylinders, spheres, and other elements that loop back around and meet with themselves should have smoothed edges. Either way, one should be testing his bakes to make sure they come out properly.
Why?
Again, why?
If you're using baked normal maps, the normal map will account for both of these situations. I use simple scripts to set hard edges from uv seems for basically all of my models, and I never go back and soften the seam edge on cylinders, because it generally doesn't matter.
The only situation where I've ever had an issue with it was specific to LODs.
I suppose both methods work but one will result in a normalmap with a few more (albeit small) gradients and therefore a greater (still small) chance to look worse. In-game the models would probably look the same so it wouldn't matter, but technically I'd say my method would result in a normalmap that would be a tiny bit better (while, for me at least, not yielding any additional work).
I find if you set your smoothing first, you may end up with far too many UV islands, and if you don't think about your smoothing while doing UVs, you may end up with too few UV islands (and smoothing errors as a result).
Lets take my test model for instance:
Now I understand why Hard Edges are a problem when normal mapping. But what if you want those hard edges? I understand that you put everything into their own UV Island and give it a smoothing group but in my test case my Low Poly only has 6 polys so I'm a little confused about how I would go about this.
Is my problem that my Low Poly doesn't supply enough polys for the High Poly to be projected onto it, as explained in Earthquakes "Understanding Averaged Normals..." post? Lets looks at my Low Poly with a Normal Map and my High Poly side by side.
As you can see some sides worked while others didn't. The Top of my model, the middle most picture, looks pretty much exact. Same with the left side as nothing was done to it. The right side which was a positive Bevel looks right as well.
Now the Front, the part just under the Top, doesn't even show detail. This was a negative bevel. The Bottom, lowest picture in the group, has some parts correct and some not. It seems the upper right and lower left quadrant of the Bottom section show up the best as well as the quadrant boarder. But the opposite quadrants are messed up. Same with the Back, the upper most picture. The third section from the left is almost perfect, but it looks like the Normal map just gave up on the other 5 sections.
Now everyone said that I don't need to unwrap my High Poly. So what would I need to do to my Low Poly to make hard edge features stand out, as well as getting my other features to look correct throughout, like fixing my Back poly.
A HP with rounded corners will bake into the normal map for the LP, thus causing the LP to appear to have rounded corners, despite having a smoothing split.
The left cube is my (messy) HP. I threw in some supporting edge loops and applied a turbosmooth to round out the corners. The closer the supporting edge loops are to the corners, the sharper the corners will be.
The center cube is my LP with the resulting normal map applied, and the right cube is my LP with no textures.
I decided to avoid creating an indent on my LP for the front face because it doesn't affect my silhouette. Notice, though, that I added in some geometry at the top for where the high poly pushes out, because that part of the model contributes to the silhouette of the shape. If I were to use just a cube for my LP, there would be distortion due to the surface inconsistencies. The reason some of your normal details appear shifted on the sides is due to a sort of parallax distortion. Imagine observing the needle of a gauge from one side rather than straight on. The needle will appear to read the wrong value from that perspective. When baking normals, a similar effect happens if you don't have enough supporting topology on your LP to match the HP; your normals are basically baking from a poor perspective, the perspective being the vertex normals. There are no vertices between the corners, so any non-flat details will become skewed. This picture from Earthquake's thread is a good example, you can see how the details are stretched depending on how the vertex normals face:
So the short answer is yes, you need more detail in your LP model. All those parts on your HP that are extruded and affecting the silhouette of the object need to also be there in the LP. Of course, it all depends on the size of the object, the importance an element plays on the silhouette, etc.
If you add a vertex at the center of each of the extruded/chamfered square bits protruding from the sides, your bake should come out a bit better, but you may want to consider just matching the HP topology more closely in general.
Starting from the Top Left image. This is the most reduced and triangulated I could get my Low Poly before is turned into a 6 poly box. The Top Right is a set up from that, though through my trials I must have over written that model.
The Lower Left takes out most polygons from every side while leaving enough to get most of the features to show. So for the set of 6 small squares they take up 18 polys each but is reduced to 10, which isn't much. The Lower Right picture is a direct copy of the High Poly with every side flattened.
Doing this allowed to see that more intricate sides, like the set of 6 squares, should have more polys. While the Top face, because it is chamfered, only needs one poly, which is what I gathered while I was reading.
I still have a question about the Cage though. For one it seems like the Cage only works properly if I use a Push modifier. If I try to move the faces myself it doesn't work right. Why is this or am I not doing something right?
Secondly, just to make sure, when I export the Cage with xNormal I am supposed to be exporting it as a Low Poly with export cage unchecked?
Cages control how the high poly details are transferred to the low. So when you use the push modifier, it sends the verts out in the direction of their normals. This gives a nice average between all of the verts.
If you move the faces/verts/edges by hand, you are now not averaging, you are artifically affecting the bake. You can change the cage by hand, but you have to do some practicing first to understand how changing the cage will influence the normal map.
Yes, export your cage as a seperate file. And import it as a seperate file within xnormal. The checkbox is if you have a workflow that can read the low poly and the cage as seperate elements within the file format.
c:/del /brain
I would suggest applying what you have learned so far to an actual model. It doesn't have to be a huge project; just something small, like a simple prop.
As for your second question, I usually just export the LP and cage separately as FBX files.
I also forgot to ask. Do I triangulate the High Poly and Low Poly or just the Low Poly?