Normal Map edge padding 'streaks'
Hey guys, this problem has always been there for me and I've just ignored it, painting it out where it's noticeable and moving on with my life. However, I'm currently on a bit of a normal map pilgrimage of truth and would really like to get to the bottom of why this happens and fix it at the source.
I vaguely remember reading somewhere that it might be to do with some kind of anti-aliasing thing within Max (which I'm using) but I don't think it was elaborated on past that. I'm using the scan-line renderer and have included the settings I'm using in the render setup. I understand very little about rendering so hopefully I'm missing an obvious trick. Thanks for any help you guys can give on the subject!
I vaguely remember reading somewhere that it might be to do with some kind of anti-aliasing thing within Max (which I'm using) but I don't think it was elaborated on past that. I'm using the scan-line renderer and have included the settings I'm using in the render setup. I understand very little about rendering so hopefully I'm missing an obvious trick. Thanks for any help you guys can give on the subject!
Replies
The workflow I've adopted is to drop AA entirely and render at say 4x or 8x size, then scale down at the end. Dropping AA makes it a lot easier to process the image; things like magic wand select work better when the threshold doesn't leave halos where the AA blurred the edges a bit. It takes the same amount of time to render 4x size on both axes as it does 16x AA, and produces results that are comparable, but defers the AA until the end of the pipeline.
I also now avoid using built in baker padding, because if you want to do any post editing, the padding is no longer "correct".
At the end of the pipeline, with the island space still transparent without AA, THEN padding is added using a filter before resampling down to the final size.
I've taken the opportunity to mimic your bake workflow James and I'm dead pleased with the results. I baked at double resolution with absolutely no fancy stuff, no supersampling etc, and then used the Photoshop Action found at the bottom of the wiki article (http://wiki.polycount.net/EdgePadding) to add padding before downsizing the whole texture to 2048:
OLD Normal Map -
NEW Normal Map -
Doesn't make a vast amount of difference on the model at the resolutions I'm looking at but it makes for a cleaner, overall map which compresses nicer and will shine through on lower mips. Thanks for all the help guys!
I usually do like 64 pixels of padding, because you can literally never have too much, and as long as your bake is done correctly, padding should have absolutely no effect on the end result.
I guess adding padding after the fact is more accurate to the actual content if you've done a lot of editing, but really it seems to be a lot of fussing over something very trivial. Unless your content is massively different, it shouldn't make any perceivable difference in reality. Remember, padding is there to avoid artifacts and visible seams showing up when using very low mip-levels, so at that point your texture has already been sampled down to a much smaller size, any minor inaccuracies will be moot.
Though it may make more of a difference for diffuse and specular maps where the color information is more varied(normal maps are by in large various shades of blue with predictable values). Most baking apps will export an alpha channel with your bake so you can redo the padding if you need to, which is something I would do on a case by case basis only for the 1% of the time where it makes a difference.
Sp0nge: very specifically to your last post: You should have more pixel padding than you do there, I like to crank it up to the point that there is virtually no "dead space" between the islands. The amount of padding you have in your "new" image there is only enough for 1 or 2 mips down.
I've had the same issue with padding as well...thanks fellas
Regarding padding, I totally get your view on 'never enough padding' and have to admit I would definitely prefer tons more, the action I got however only adds 8 pixels. Probably great for a 512 map but with this size map it is pretty thin on the ground...
In answer to your question, Notes, padding will just bleed until it meets more padding, it wont ever go over UV islands.
EDIT:
Also, it's worth pointing out I was getting the occasional area of weird coloured pixels in random spots of the bake - easy to paint out as there wasn't many but still, maybe some light can be shed on this? I imagine my usual AA process nukes these out of existence but interested to know what causes them in the first place...
The reason I do the downsample method is that it makes it so the UV islands don't have antialiased edges and padding filters know exactly what to fill and what not to; had some weird behaviour with padding filters otherwise.
Also, any changes you make after baking like painting over bake errors (again, tight rapid deadlines, don't have the luxury of a week to make 15 textured world assets, I have a day or two) get a bonus downsample which usually covers them up.
I use infinite padding, via my GIMP plugin, also designed to reduce streaking to a degree:
A. How "clean" the padding is has no relevance in real use, as the only time you need padding is when the lower mips kick in, and the image is already sized down/much blurrier than your original res content. So fussing about this is some serious OCD stuff.
B. DXT image compression(what every game uses) uses a fixed compression ratio, in most cases 4:1, so your content has nothing to do with how small the compression will be. A 3mb file will always compress to 768kb with standard DXT compression. With alpha it will compress to 2:1 or 4mb = 2mb.
C. The random odd pixels is a max specific thing, AA will help as will super-sampling, or rendering at a higher res as sizing down(which is exactly what super-sampling is). There have been a few threads on this on polycount, maybe you can try and find them, I don't remember if there was a specific solution to the issue.
D. Yeah, padding will bleed until it hits its nearest neighbor in the middle, it won't overlap. The only time it would be an issue is if you're baking multiple different objects to different maps and then combining them all later, then you would have to mask it, but really you should try to bake everything at once, it saves a lot of time.
Just to be clear, you mean overlapping when you combine different assets that were baked at different times but all on the same UV layout, correct? Because this is the only situation where you would ever have padding overlaps.
If you're doing multiple bakes for different objects this is a good point. However, this is generally a slow way to work. If you can, bake all of your objects at once, it will save a lot of time waiting for bakes, setting up bake scenes/cages etc. I've done a lot of environment props and sets that all share the same uv space and baking it all at once makes a huge difference time wise.
Unless you don't have the objects all ready to bake at once or something, then this wouldn't be an option.
Yeah, this makes sense. Getting proper padding from a mask can be tricky sometimes and you might have to fudge with the selection, so no AA would help here.
Yeah if you end up drastically changing your maps after the bake this is a good point, however generally in my experience doing a lot of hand editing takes MORE time than just fixing the bake with your geometry in the first place. Though it really depends, if your triangle counts are really low there is only so much you can do with the mesh itself.
I mean, thats cool and all, but you're looking fullsize at padding, show me an asset in game with a mip level of 3 or 4 where this makes any difference whatsoever, otherwise its just obsessive. Its sort of like looking at a normal map in photoshop to see how well its going to work on the model, it just doesn't work like that.
I don't want it to seem like I'm just hating on you here or something though. I'm posting for clarity here. If an inexperienced user skims this thread they can easily come away with two conclusions:
A. Too much padding is bad
B. "Streaky padding" is bad I and I must develope special workflows to avoid it.
Both of which are wrong in all but the most extreme cases, and these things tend to sort of snowball once people start repeating them.
Yea point taken, it's entirely possible that I'm getting carried away with this one.
Good shout again. I'm also getting my terms confused here - I was initially thinking that the actual base image would be lower in memory if the image was less noisy, but after a quick test to make sure I wasn't crazy, I discovered I was in fact crazy... They are exactly the same. I'm not sure where I got that idea from, I think maybe working with .gifs on the web is warping my mind.
Hehe yeah, with gifs and jpegs the image content has a huge impact on the compression ratio, but not DXT.