Edges - when to normal map them?
So it just occurred to me that I have no idea when I should bake a corrective normal map to my edges and when I should just leave them be. Having recently had a problem with backing normal maps, I was instructed to just 'paint over' the sides of my models because, apparently, I didn't need the detail on the edges (90 degree). Just today, I tried to bake more detail into my 45-degree edges on another model and it seems that even this creates problems (an issue which I just posted to the xNormal master thread btw.).
Thus, my question to you is - when should I bake detail (aka high-rez to low-rez) to the edges in my model and when should I leave them be?
Thus, my question to you is - when should I bake detail (aka high-rez to low-rez) to the edges in my model and when should I leave them be?
Replies
Edges are the most important parts of your model. They're the bits that catch the light and define the readability of the forms.
EDIT:
Also, nice examples pretty please?
Your lowpoly with normal map should match your highpoly as closely as possible, that is the goal. There isn't a set of rules specifically relating to "edges", this seems sort of silly. I suggest reading up a bit more on normal maps, what they do and how they work.
Start here: http://www.polycount.com/forum/showthread.php?t=81154
Also, post some images of the specific problems you're having.
http://www.polycount.com/forum/showpost.php?p=1472238&postcount=2243
Secondly, look here (UDK forum) on the very last post to see engine rendering issues that I'm facing with the normals on edges:
http://forums.epicgames.com/threads/821132-BSP-lightmass-error-and-static-mesh-shadow-error/page2
A guy helping me solve the problem on the UDK problem told me to paint over the sides' normal map with the neutral 127,127,255 color (or whatever its values were), essentially getting rid of the normal mapping on the sides. I dare say such a solution seems weird.
These two, however, also lead me to an interesting conclusion - that I've no idea when to normal map edges and when not to. Looking at games, sometimes they're normal mapped, sometimes they're not and sometimes it's hard to tell.
As another example, in this article:
http://www.chrisalbeluhn.com/UT3_Asset_Creation.html
Chris Albeluhn seems to deliberately NOT normal map the edges as there's 'no need'. When is there need?
I believe you're referring to either beveling/chamfering edges or using smoothing groups to denote hard edges.
A. Your smoothing error problems are caused by a lack of synchronization between the baker and the renderer. Baking in XN or Blender and then importing into UDK is going to give you problems. Generally each app does it a little differently, which gives you inaccurate shading when combining two counter techniques. This is the root cause of smoothing errors.
This is a complex matter, but essentially you want your baker and renderer to read the normal map data in the exact same way. Unfortunately this is hard to do with UDK, there have been some efforts to use max and export the custom tangent data via FBX, I believe this resulted in significantly improved results, but not perfect.
A few examples of sync'd normals:
A. Baking normals in Maya and displaying in Maya viewport
B. Baking normals in Max and displaying in Max viewport with 3point Shader Quality mode
C. Various proprietary 3d engines that have a properly sync'd workflow(UDK isn't one of them).
Alright, so now that we understand the problem you're having is smoothing errors caused by lack of sychronization, we can start to look for solutions. When we KNOW we're using a broken pipeline, using hard edges or smoothing groups instead of beveling edges is often the best solution.
There are however some things we need to consider when baking with hard edges/smoothing groups:
A. For every hard edge/SG you have, your uvs need to be split(with space inbetween) or else you will get seam artifacts.
B. Your projection mesh must be averaged(explained in detail in the stickied thread that I linked with my last post). If not, you will get seam artifacts, from the "split" projection mesh creates gaps.
PS: Painting a flat-blue in your normal map is a really bad idea. A normal map is a collection of data, the purpose is to counter-act the smoothing of the lowpoly on a per-pixel basis. Randomly painting on it isn't going to help anything.
http://wiki.polycount.com/NormalMap?action=show&redirect=Normal+Map some more general info here as well.
+1
If you can, try to get pretty much all edges beveled through normal maps or geometry. They will help catch specularity and as sprunghunt mentions, help the readability. The exceptions would be edges which would really portray sharp edges such as the edge of a knife or something similar. I bet that 99.9% of all edges in real life in your every day has some form of man made bevel for you not to hurt yourself.
Everything has a bevel, just like fresnel,
Never forget, the example of a bell.
One very important thing to note, if you're working with a broken tangent space, adding in lots of small bevels, as apposed to hard edges will likely result in a worse end result. While bevels can help lessen smoothing errors, they can still be quite apparent, as in the OP's example with a broken tangent space.
Not to mention increasing triangle count even if your vertex count remains about the same, tris are not free. As well as introducing many long, thin triangles that can be a bottleneck for rendering, and cause issues when your triangles are smaller than the pixels on your uvs(even with a properly sync'ed tangent space) ie: resolution based smoothing errors.
So no, I certainly wouldn't say "pile on the bevels" for lowpoly models. But there isn't a "always do this, or always do that" rule here, its more important to know and understand the pros and cons of bevels vs hard edges/smoothing groups instead.
However, with highpoly modeling, edge widths and bevels are extremely important, having nice punchy bevels so your shapes not only read well, but catch interesting highlights. This doesn't mean you need to match every bevel in your high in your low as well though.
Perhaps it's because it's late and I'm tired... but you seem to have varied opinions on whether or not to always bevel the highpoly etc. Huh?
Also, I got some useful info from the thread but I'm still not sure how to go about fixing the problem. So, instead, I propose this - take a look at the two threads I mentioned above (the post in the xNormal master thread and the rendering problem in UDK) and tell me: how would you fix these two particular issues? Would you bevel the edges on the highpoly bookshelf? Would you use a different workflow for generating the normal map for the keypad thingy?
As a reminder, I am currently using a setup that consists of Blender, xNormal and UDK.
I don't really get A, as I've often seen UVs in games that are not split at hard edges (unless not every 90-degree edge is considered a hard edge? It's late and I'm tired so sorry if that's the case...).
Regarding B, I assume you're talking about a cage? Having worked with Blender for the last couple of months (and earlier with Softimage), I don't really use cages, although I do know the theory behind them (or at least I believe I do). I don't remember what the non-cage method is called atm but I use that instead. I tried using a cage in xNormal - after all it's really not hard to get a good cage for a model like the bookshelf pictured in one of the threads above) - and I got the same results. If your 'averaged' means the same as my 'averaged' (basically: reset so that it conforms to the mesh) then it was averaged.
EDIT:
Also, a bonus question in hopes of also answering the thread question: How do most people do it in games? Do they bevel the highpoly edges, or leave them be like in the article on UDK mesh pipeline that I linked to a post or two ago?
Also, Blender and xNormal use the same tangents (latest versions) for the Normal Map, so baking in one of the other shouldn't matter. So it's better to bake in Blender from what I can see since you have proper control on the cages.
The only time you shouldn't bother with cages, is if your piece is a non-hero piece (EI: something like say a piece of pipe in an angle that the player won't ogle at or hold in their hands.
Also, the reason many people have conflicting opinions is because UDK is such a mess currently, that it can get in the way of things, plus combined with each persons on way of working, you have the equivalent of a nuclear recipe for disaster. UDK currently, simply put, is not synched or stream-lined in a way that makes many people happy.
Ofcourse, everyone so far has been agreeing on one thing, bevel will serve your models better in both the long and short run.
Of course. I agree upon there not being any certain "always do this" in this matter. My point was rather that bevels in what ever manner achieved will give make your model seem more believable and generally more appealing, as opposed to unnatural polygon hard edges all over.
And as you point out, there's always technical aspects to consider. If this isn't possible due to whatever limitations you have to follow, then ofc don't do it. Not sure where you got the "pile on the bevels" for any lowpoly's, that's not what I was trying to say.
I read your post a little too hastily before realizing that most of what you said was in relation to highpoly, but didn't bother to edit it as what I wrote is hopefully helpful information anyway. Sorry if it comes off as snarky.
I think you're confused here, you wont solve your smoothing error issues by doing anything to the highpoly.
I took a look at the threads you linked and have tried my best to explain the cause of your problems, so that you are able to work out your own solution.
You may see this often, its not the correct way to do it though. Again, it causes visible seam artifacts. This is a fairly common mistake, so its not surprise you've see it done this way.
This is explained extensively, like for multiple pages(check the last few pages) what exactly a cage, or projection mesh is, when it is averaged, etc in the "waviness" thread. I know its a long thread, but I would suggest reading the entire thing.
A. Yes I mean a cage. If you have hard edges or smoothing groups(max term) you get gaps in your projection. You should always use a cage/averaged projection mesh if you have any sort of hard edges.
B. If your mesh is on one smoothing group, or has no hard edges, using a cage or not shouldn't affect your end result. In general, it is good practice to use a cage/averaged projection mesh.
C. Basically, averaged means that the projection mesh itself(cage, or whatever you want to call it) has averaged normals, soft normals, or no hard edges/smoothing group splits. Again this is essential for getting gap free bakes.
Averaged refers specifically to the lowpoly mesh's vertex normals being averaged between it and its neighboring vertex. "Averaged normals", may be a Maya specific term but its about as literal as you can get.
Too broad of a question, its also not clear exactly what you're referring to in that article, as its a massive article covering many topics.
Blender doesn't have cages, actually, so Xnormal still wins out.
edit: Also, since EQ glanced over it: 90 degrees angles are not hard by definition. A hard edge is where the normals are split to achieve a break in shading. Blender currently also doesn't do this, you need to physically split the edge to do so.
The need for splitting the edges only arises when the normals are split, as otherwise it will introduce a black line along the seam.
One good idea to minimise shading artifacts is hardening the edges that make up uv splits (before baking, otherwise you'll need to rebake), although as mentioned before: You'd have to physically split the edges in Blender (you can use the modifier for this). Where you can set a model to soft or hardshaded in Blender, other software gives you more finegrained control over this in the form of smoothing groups (max) or hard edges (everything else)
if only this worked for skeletal meshes =[
a)
I've seen it not done in many games (including UT3 I think)
b)
As evidenced by this picture hotlinked from the article:
PICTURE
Either I'm blind or this guy didn't bevel his edges. And he's, supposedly, a pro at this.
Now, getting back to the issue of rendering issues, I still can't seem to tune in to how to solve those. It seems to me that everything you mentioned I'm doing correctly in regards to the technique used. Could you tell me what exactly I'm doing wrong, for example in this mesh:
That gives me the artifact? The lowpoly is smooth-shaded. If I wanted to do hard-shading, according to your instructions, I would have to map out each and every polygon separately on the UV map, which is not a really nice option (that would be almost impossible to texture sensibly). But I digress, as I really want to (as you say I should) get these bevels in the bake. And it's not working. Why? What am I doing wrong here? What am I missing?
A.You can deal with it, the error in this case is fairly minor
B. You can split up your mesh with hard edges/smoothing groups(again make sure to follow all of the rules for using hard edges). You don't need to have a hard edge on every face, you could split it into 5 groups(front, bottom, top, left, right).
C. You can work with a system that has a sycn'd tangent space
D. You can add even more geometry to the lowpoly to lessen smoothing errors.
Also, you do not need to literally match every bevel that is in your high in your low as well. If it was a simple box shape, that would mean less uv islands. Again you'd still end up with about 5 uv islands though.
Instead of posting and asking more questions, try experimenting with these methods and seeing how your results turn out.
Here are some example, this one is a bit more creative, you can do hard edges, uv seams but keep it all on one uv island.
and basic example of hard edges on a 5 sided layout
You can do whatever you want. If you just want an excuse to have aliased, jaggy hard edges all over you model feel free to do it. The linked result looks bad to me, its covered up with a noisey texture so its less noticable.
You need to decide yourself how important the asset is, how it will be viewed, and whether or not you think its important to have nice bevel edges. If you're just looking for a rule book from the "pros" I dont know what to tell you, you've gotta use a bit of critical thinking and have a subjective opinion on this stuff of your own.
When I do work, I create a highpoly model. When I bake it down to the lowpoly, the goal is to get a seamless bake that looks as much like the highpoly model as I can. The advice I've given you is with that in mind. Does every asset need this sort of treatment? Thats entirely up to you.
Keep in mind that many "Pros" do pretty dumb things, being a "Pro" doesn't mean you've figured it all out and are using optimal workflows. There are many industry professionals that barely understand how to bake a proper normal map.
Hell sometimes being a pro means you're knowingly doing something in a poor way to save time or resources.
Understanding all of your options, experimenting, and thinking for yourself is really the best thing you can do.
Ummm... Which one? Doing it with hard edges? I'm slightly confused here.
It's what I've been doing for a lot of time. Then I ran out of ideas, after hundreds of combinations, and so asked here. Then I experimented some more. I'm not simply trying to get someone to solve the error for me. Well, I am, but I'm still working on solving it myself using your advice.
This is basically my UV layout:
Although I didn't try that hard edges layout. I shall try it out! Once I actually figure out how to get this result in Blender...
My point is, though, you can't really say xNormal has a broken tangent space setup, can you?? If you can then life has officially lost all meaning as not only I did not understand what you mean but also there seems to be no application with a correct tangent space setup.
But, since everything is within xNormal, I'd expect xNormal to treat its own bake well and not produce errors like that. So what the hell's happening here?
Regarding the hard edge or highpoly bevel question, I shall now ignore the advice in that article regarding it and try my best to bevel all the things. I do, however, feel I need to research this topic some more. An epic adventure awaits me in this case, I guess!
I just tried the ol' basic-example-of-hard-edges-on-a-5-sided-layout, it seemed to display fine enough in xnormal
As previously explained, there isn't currently a baker that syncs with UDK. Thus, any workflow involving UDK is a broken tangent space workflow. This is hopefully an issue that Epic will fix at some point.
The best I've seen is that max bake +qualified FBX export thing with Max.
When you get this into UDK, the differences between UDK's tangent space and Xnormal's will be amplified further, and likely result in worse smoothing errors.
So, when you know you've got a broken system, you need to do all you can do minimize what causes smoothing errors, which is extreme changes in vertex normal direction on your lowpoly. using hard edges, or extra geometry or whatever you can, it can be quite messy at times but its all you can do.
Doing it with hard edges, and following the rules outlined for using hard edges but avoiding the classic errors that generally happen because you're using hard edges. Blender may not even be possible of this, I've read some people complain about how blender handles and exports hard edges/vertex mesh normals.
I can absolutely say Xnormal has a broken tangent space, as it quite obviously does. Its a bit funny, you would think the baker and renderer would be sync'd up, but its not quite, even when baked and displayed in Xnormal's 3d viewer.
Again, I gave some examples of some workflows that are synced up. Fortunately this is an issue that is getting more traction in the games industry, I've done work for various studios that have properly synced up tangents in their internal pipelines. Splash Damage's engine for Brink for instance was perfectly synced with Maya, I can't really talk about any of this more than that for legal reasons, however the Splash Damage/Maya thing has been mentioned here on polycount so it's safe for me to say that much.
If you're looking to get the best results when baking normals, you may find yourself a bit too restricted using Xnormal and Blender. Just something to consider going forward.
Sorry for not replying earlier. Thanks to your advice I was able to greatly reduce the artifact (or whatever it is) so that it's barely noticable. While it doesn't look as great as I'd want it to under more extreme angles, I think it should be fine when a texture is put on it, right?
Everything has a fresnel? Example of a bell?