Advice for solving various normal map/edge bake artifacts?

Interface

So I have this flaregun model that I've been working on for some time. It's nothing special but I've been using it to experience the different steps of creating an asset. And well I've hit a bit of a bump in the baking phase of the model. Obviously I'm using Marmoset toolbag to bake down my maps, and I'm getting quite a few different artifacts in my bake that I would love to solve. I've read quite a bit about the baking process and different things that help result in better bakes, but nothing that I've read so far has REALLY helped me. Like it has helped me improve other aspects of the bake such as cleaner baked down details. But nothing has really helped me get better edge bakes and it's something that I've struggled with for all my time using Marmo. Anyways enough of that, here I'll put some screencaps of both my Hi and Low Poly models of the flareguns as well as screencaps of the artifacts I'm talking about. I'll talk about the specific artifacts where I place their respective screenshot.


Hi Poly


Low Poly

So now that you know what my model looks like, let's talk about the bake artifacts.

Above I have a few screenshots of some of the edges where I have these artifacts. I've also gotten these kinds of artifacts on other smaller projects so this issue has plagued me for quite some time now. Here you just see that some of the edges are super hard and don't look soft whatsoever. And then for the edges that kind of look soft, you get this hardening/creasing on the vertices that are between the two edges. One such area that this happens is the trigger guard/main body area. Here you can really see this effect happening. I've done all that I can UV's wise and geo wise in Maya to solve this issue but nothing has REALLY worked. Some methods have worked a tad but nothing has helped me solve the issue in its entirety. I've heard that hardening your UV borders and hard edges in Maya tends to help the bake out a bunch. But I haven't really seen this at all in this project, but softening the edges doesn't help any more than hardening them does. If anything it makes it worse so I'm kinda just lost as to what I should do there. In Maya, my geo has been shrinkwrapped onto the HP many times and I have checked over and over again to make sure that the verts are on the hard edges like they should be. So I don't think it's an issue of the geo just not encompassing the HP well enough, because I can assure you that it most certainly is. Okay now, let's talk about this other type of artifact I've been getting which are UV seams. 


Okay, so here I'm not all that clueless as to what's causing these artifacts. I'm fully aware that these are just my UV seams, however, I'd love to know any methods that there are for reducing UV Seams appearing in the bake. I know that you can prevent them by planting them carefully or only using them on hard edges. But I do need these seams there in order to separate my islands. So other than just careful and mindful placement, are there any other methods for reducing UV seams in a bake? Also, alongside the UV seams, I'm also getting this weird black artifact in the middle of the inside of the trigger guard. Although it's more so to the back of the trigger guard right below the UV seam there. I have no idea what's causing it and there's nothing apparent in my UV's or geo in Maya that would cause such an artifact. For that, I would just love to know what the heck COULD be causing it if anyone's experienced a similar-looking artifact. And well that's pretty much it for the artifacts I've been getting. I have something else I want to ask about but that pertains more towards Maya so I'll post in the correct category for that. But yeah just overall I'm more so looking for guidance on this issue rather than a full-on solution. Since I know that each case is unique for every model. So if you guys could help me out in any way then I'd really really appreciate that! I've just been struggling with this for so long and I'd love to get past this bump in the road. Thank you in advance to anyone who helps me out! I really do appreciate it

Interface

Update: Since my post last night I've been thinking about different techniques that I've heard about that some people use to get better bakes. One such technique was loosening up my edge loops on the hi-poly so that my bevels would be made much more pronounced than before. I gave this a shot and it appears to have worked in most places. While I still am getting some of the same artifacts from before, it's not as bad as it was before. Most edges are baking a lot cleaner than before and other areas just look better than before. So I'm going to keep messing with this technique in hopes that it'll continue to improve my bakes. 

Vertrucio

Your high poly does not have enough of a bevel on those edges for the baker to bake down to lower geometry.

Interface

Thanks for the reply man! Yeah, I've actually kinda figured that out already as seen in my previous update. Where I showed that giving my HP more of a bevel greatly helped the bake out. I've continued to do this throughout my HP which has resulted in a much better bake than I had before. I only wish that I knew of this sooner. Well, I technically kind of did, but I didn't think it was the solution I was looking for. Whelp, the more ya know! I'm just glad that my bakes are looking a lot better now. Now I'm finally able to move forward with texturing this bad boi in Substance! But here are some screencaps of my improved bakes. Definitely much better than before. 

Vertrucio

I'd still say for game art, especially for anything you plan to bake down normal maps, you'll want to make your bevels over double the current size/strength you have now. For example, the side panels there don't actually have any normal differences for the map to work with. It'll look like a flat thing with a bit of shadow around it. You need a change in height that's visible for the light to play off it.

Interface

Huh, you know you're right. Those parts do need more normal differences. It most certainly would look a lot better with more normal detail on that part anyhow. So I'll see what I can do on that end. At the moment I believe those parts are just basically rounded polys I extruded inwards. So I'm not too sure what I could do there tho to show more height difference or any sort of normal difference. Do you think perhaps a much more exaggerated bevel on the border edges of those panels would suffice for this detail?? Or what would you recommend I do here? 

Interface

So since my last reply, I've loosened up my edge loops around the panels so that the bevel could be exaggerated more on the border. Overall I feel like it does improve the normal detail on this part. Although perhaps it could use a bit more of that normal detail you were talking about. But it definitely is better than what I had before I think. I could be wrong here tho so I'm open to hearing what you think @Vertrucio

Vertrucio

Move that model away from the camera enough to view it at say, arms distance. Can you really see any of those bevels? They'll just be hard edges at that distance.

But yeah, we're not in marmoset territory anymore.

Interface

@Vertrucio I'll go ahead and get right on that. I'll post again once I've loosened up the loops a lot more and have baked them down in Marmo. 

And I'm fine with being a tad out of Marmo territory if everyone else is    Like, in the end, it circles back to Marmo so eh. 

Interface

Annnnnd here's another update! Went ahead and loosened up those loops more like I said I would. And I even loosened up the loops overall in other parts besides the body. So the edges look a lot smoother in other areas as well. Also, I moved the face of the bigger panel more inwards. Which is giving it more of this AO effect as you see in the bottom more rounded panel. Which is helping a great deal with depicting some sort of depth there imo. But now it's judgement time!!! @Vertrucio

Vertrucio

You have started to reach an acceptable level of bevels... if you were examining the weapon up close all the time.

But you can double the bevel widths and it'd look better. Some of the areas you could easily go 3x or 4x.

http://wiki.polycount.com/wiki/Normal_Map_Modeling

Right now, it's close to a Realistic level of bevel. However, this is game art, so you want something that's functional. Let me refer you to the polycount wiki on edge thickness from racer445.




Now for that inset, take a look at:



As with your bevels/edge widths, it may not be realistic to slope the walls of insets/protrusions, but if you don't, then there's no info for the normal map to pickup, and it'll just look like some flat shading out of nowhere.

Interface

Oh wow okay that makes a lot of sense tbh. I'm definitely starting to see your point now. However, some of my edges on the HP are already pushed to their max. By this I mean that I've loosened up the edge loops as much as I can without any of them intersecting one another. If I were to push some of them any further I'd get intersections that would cause issues. But I'll definitely go back and see how much more of a bevel I can get on these edges. I'll do my best to squeeze out as much of that bevel as I can! As for the insets, I was actually thinking of doing this method for them. As a matter of fact, I was going to ask you about this yesterday but I just didn't know how I should word it. But it's nice to see that I was getting on the right track for those insets. Anyways, I'll also go and edit those insets whenever I go in to mess with the bevels. Well thanks so much man for helping me out so much with this! Honestly, I'm super grateful since I've struggled with this kind of stuff for a while now. And the outside help really goes a long way since I tend to overthink things a lot. So yeah man just thank you so much! I'll post another update once I do the things you recommended to me. 

Interface

@Vertrucio Annnd I'm back!!! Sorry for taking a day or two to post another update. Got a bit busy but I have been messing around with my HP to get those better and more exaggerated bevels that you talked about. Although I admittedly have pushed them as far as they could go just yet. I've more or less just been messing around and testing different bevels lengths and stuff in Maya. Also, I've been messing around with the panels to see how far I should scale them in to get that needed depth. And I've kinda been experimenting with that as to see what looks best. But I'm kinda feeling conflicted here. Because on the one hand, I feel like it's fine for the most part even from afar. But on the other, it's kinda hard to not look at it and see just some shading out of nowhere you know? Like I've tried different things to combat this, such as messing with the bevel lengths of the inetted faces to try and get a stronger and more prominent bevel there. But doing that hasn't really helped the baked detail look better from afar sadly. Although admittedly, the AO does help out a bit here since it reinforces that insetted edge by shading the border a darker shade than the actual insetted face itself if that makes sense. But anyways, here's where I am with the bake so far. I'm not entirely done here because I do want to make this bake as good as it possibly could be. But I just wanted to get your input first on what I got so far. Oooh and I forgot one thing. For the little circular details that I have on the body, I've decided to inset those faces as well so that I could get this extra detail there since it just felt like empty space. And yeah that's about it. Looking forward to seeing your feedback man! Although I feel like I already know what you're gonna say... "MOOOOORE" xD (Also last little note, I went ahead and left a couple of screen caps of my HP down below. More specifically screencaps of the areas we've been discussing. Just thought it might help you help me if you had a visual ref of what the actual HP detail looked like)




(Distant shots ftw!!)

Vertrucio

Check out Simon Fuchs' handgun tutorial.
https://www.artstation.com/artwork/lVy0bV



I bought this tutorial a while back, and haven't gone through all of it, but from these images you can see how chunky he makes his highpoly bevels. Admittedly, he does separate in zbrush using strategic polish uses, but you can get similar in sub-D.

But you can see my results from his military radio tutorial here. Simon is basically where I got my understanding of just how unrealistically wide your edge bevel widths can, and should, be to look good in-game.

There's points where you will have to make bevels that do wreck the highpoly topology in places, but remember that it's a highpoly, it doesn't need to have great topology, it's going to be used to bake down onto a low poly and that's it.

Interface

Ah that Simon Fuchs tutorial. Yeah, I've seen that tutorial being recommended quite a bit actually. It seems like a pretty great tutorial from what I've read about it. I would pick it up right now but I'm tight on cash at the moment. But the moment that I can, I'll definitely pick this one up. Seems like a good workflow that's definitely worth picking up. But I also definitely see what you mean here about the bevels and how chunky he makes them. It's a really good visual ref for what you were talking about here. Anyways, after I initially read your post and read a bit more about the tutorial, I went ahead and messed around more with my bevels and I'm starting to push them even further than I did before. To the point where some of my loops almost intersect but the result so far seems to be worth it! The bake is looking a lot better than initially thanks to you. So yeah man thanks for all your help and advice! I really appreciate it dude. It's helped me get a lot further than before and I now feel comfortable taking this bake into Substance for some good ol' texturing 

Vertrucio

Also, as someone who has gone through multiple of Fuchs' tutorials, I'd actually recommend starting with this tutorial, which is a lot faster, albeit it skips over a lot of hand detailing. https://www.exp-points.com/exp-tutorials-portable-music-player-maya-painter-zbrush

Also, I should never write posts at 4am again because I can't even decipher what I was saying.

Interface

Lol man it's all good. I was able to understand what you were getting at xD but I'll for sure check out that tutorial more. Heck I may even go ahead and buy it. I already gave the description a look over and it seems like a pretty good tutorial. And it's not even that expensive at all either. So that's even more tempting! May just go ahead and buy it and see what this workflow is all about. Better yet it's based in Maya rather than Max like Simon's was. So that's even more reason for me to check it out. Then hopefully after this, I can move onto that Simon Fuchs tutorial. Thanks for the recommendations man. Can't wait to get into this workflow!