[MAYA] Hard edges on UV seams when baking normal map.

apebit

So I have this problem as you can see in the picture below. I have made the model in Maya and made the entire object have soft edges. For some reason when I bake the normal map from the high poly model it still shows hard edges where the UV seams are. What gives? Why is this happening? I used to be able to do this without problems. As you can see also, there are soft edges on the model and they show just fine without seams. PLEASE HELP I'M GETTING SUPER FRUSTRATED! TT_TT

EDIT: I've now tried baking it with Maya, Xnormal and Substance and a friend of mine baked it in Blender. All of them have the same result. Wtf?!
EDIT2: If you have time and want to take a look at the files and see if you can figure out what the porblem is, you can dowloand the files here https://www.dropbox.com/sh/dvhiuuty46jvubi/AAAElpaFPQ0srwvnR6eiKbIga?dl=0

Here's the topology, as you can see it's all soft edges.

Here's the high poly model.

Here are the UV's as you can see there's plenty of space between most of them.

icegodofhungary

Are you baking the normals within maya? If so, that's probably it. Maya's baker is bad. You should get xNormal, which is free. You'll need to make a custom cage mesh for it and that should solve your issues.

Also make sure your vertex normals are actually soft. Maya sometimes won't smooth the vertex normals for some reason and you have to do something like Mesh Display > Set to face, and then try to soften them again. You can view the vertex normals by going to Display > Polygons > Vertex Normals. I created a shelf button to quickly check my VN before exporting for baking.

Check the Polycount wiki on normal maps, lots of info there about cages and troubleshooting issues.

DavidCruz

Try doing some tests, add lines i show above re-unwrap and try it again, i.e. do test bakes till it works out. 

I link here a post by a user that knows how to make hard surface work with loop support

FrankPolygon said:

@Klo Works

thread is here:https://polycount.com/discussion/56014/how-the-f-do-i-model-this-reply-for-help-with-specific-shapes-post-attempt-before-asking/p171

Just cntrl+f seach for frank and you get the rest of the above post ^

If you want to do hard-surface pieces you have to learn how to do the information linked above.

I would also go so far as to say jump on another baker entirely or put the object into marmoset and use these options

and switch between them and see if the seams disappear under a different tangent setup.

Mikk usually works for me to make them go away. (i am not a maya user but wanted to try to help you anyway.)

Think i remember someone using maya also complaining about this and just going the above routes ^ to find a solution. (i could be wrong, only one way to know, try the above.)

Neox

icegodofhungary said:

Are you baking the normals within maya? If so, that's probably it. Maya's baker is bad. You should get xNormal, which is free. You'll need to make a custom cage mesh for it and that should solve your issues.

Also make sure your vertex normals are actually soft. Maya sometimes won't smooth the vertex normals for some reason and you have to do something like Mesh Display > Set to face, and then try to soften them again. You can view the vertex normals by going to Display > Polygons > Vertex Normals. I created a shelf button to quickly check my VN before exporting for baking.

Check the Polycount wiki on normal maps, lots of info there about cages and troubleshooting issues.

What o_O

it may be slow but its a great normalmap baker. recommending xnormal which isnt baking in maya tangentspace will only look worse inside Maya.


@op: looks to me like color correction is turned on, set the viewport to raw, make sure the map is not color corrected and it _should_ work. sorry, speaking from memory, don't have maya here to check.  but looks to me like this is the issue. Mayas normalmaps should look perfect in maya.

now the question is, will your target renderer stay maya or will you switch to something like unreal, unity etc?

if thats the case, a mikkt baker such as xnormal, marmoset amongst others would indeed be the correct choice.
Mayas normals will look broken in anything but maya, unless that engine is synced to Mayas tangentspace.

icegodofhungary

Neox said:

What o_O

it may be slow but its a great normalmap baker. recommending xnormal which isnt baking in maya tangentspace will only look worse inside Maya.

Sorry if I was mistaken. I've read somewhere in one of the normal threads that it's better to use xnormal or something that allows for custom cages for better control of the normal maps. It was in the context of exporting and rendering in a different engine though.

Neox

maya has custom cages as well

they are called envelopes. make sure to turn off "reset envelopes on bake" why that option is default, is absolutely beyond me

apebit

@icegodofhungary Yeah I baked it in maya but I have the same problem in Substance painter. I've tried softening the edges again after doing the set to face and checked the vertex normals as well. I don't really want to download a dedicated normal map baker cause I want this to work on substance or maya to keep my workflow as simple as possible. Thanks for the help though! EDIT: I tried baking with xnormal and it still shows that hard edge.

@DavidCruz Thanks for the help even if you don't use maya. I appreciate you trying. I need to look into that thread you mentioned

@Neox My target renderer is Maya but I mean to bake on Substance painter. However, I have the same issue on substance painter as well. I tried turing the vieport to RAW and then doing the bake but it didn't help. EDIT: I tried baking with xnormal and it still shows that hard edge.  

Neox

Upload the .ma and normalmap  

apebit

Neox said:

Upload the .ma and normalmap  

https://www.dropbox.com/sh/dvhiuuty46jvubi/AAAElpaFPQ0srwvnR6eiKbIga?dl=0 The .ma is a bigger model so I don't want to upload that but instead this one object from the model. The entire model has the problem but I'm trying to pin point it on a smaller scale first. There's the high poly version as well if you'd like to try and bake it yourself

apebit

@Neox Do you have time to take a look at the files? I'd appreciate it a lot! 

apebit

Bump! Still need help with this

icegodofhungary

Took a look at the low and high poly. Several points:

The hard edges are UV seams. I don't want to say something wrong but I feel like you're not going  to avoid seams all the time. There's always going to be seams somewhere. There could be some software feature or technique to get rid of them all that I'm unaware of. That being said you can reduce the seams you have and place them in an area you won't really see. How is this asset being used? If you're only going to see three sides of it at once, then you can place the seam on the fourth side. And trust me, that if you're playing a game you're not going to notice the right kind of seams. They'll be hidden or very hard to see unless the camera is right on them.

I would place the seam on a flat area of the plank as well. This lets your edges stay intact and will bake the nice edge detail without error. For example:

Third is that your lowpoly has unnecessary bevels. If this asset is supposed to be for a game that is. You really don't need all the little bevels on most edges. Think about how close the character will get to this, think about how close the camera will get to it. Also how much of the screen space it takes up. You can get away with 90 degree edges on your lowpoly as long as your high poly has beveled edges. You won't really notice the illusion, again, unless you're very close or at a weird angle. The bevels are in your way imo.

Finally, your high poly isn't right. It looks like you sculpted it then decimated it before exporting. There's not enough high poly detail to bake properly to the low poly. And decimation made a mess of the topology which adds to the errors. You should keep it at the max subdivision level you sculpted on and then use that to bake to the low. This is why you're getting some shading issues outside of the hard edges.

Ghogiel

I had a look at it marmoset, there is something wrong.

The only thing I did really that changed anything fundemental that looked definitvely problematic, was I triangulated the low poly. Since you were baking in maya and maya is the target renderer, not sure if thats the fix you are actually looking for as triangulation shouldn't change unless you do something to the model or move it around to different apps.

UV looked like it should bake ok tbh, I did press unfold on it again just because. I made sure there were no hard edges just to rule that out, in practice with chamfered edges like that you don't really need them. And then I reset transforms, exported and baked n marmoset.

The advice above isn't bad either. You aren't going to get rid of seams entirely,  you can minimise them with UV changes, since this is mid poly looking with double chamfered edges, you don't need or really want to break UVs so much as it'll just introduce visual errors. I mean it's a plank that is beat up, so I expect it'll be pretty rough surface and that'll hide the seams well enough so it'll not a big deal imo.

original

slightly cleaner

You can still see the seams with a glossy material. nature of the beast

apebit

icegodofhungary said:

Took a look at the low and high poly. Several points:

The hard edges are UV seams. I don't want to say something wrong but I feel like you're not going  to avoid seams all the time. There's always going to be seams somewhere. There could be some software feature or technique to get rid of them all that I'm unaware of. That being said you can reduce the seams you have and place them in an area you won't really see. How is this asset being used? If you're only going to see three sides of it at once, then you can place the seam on the fourth side. And trust me, that if you're playing a game you're not going to notice the right kind of seams. They'll be hidden or very hard to see unless the camera is right on them.

I would place the seam on a flat area of the plank as well. This lets your edges stay intact and will bake the nice edge detail without error. For example:

Third is that your lowpoly has unnecessary bevels. If this asset is supposed to be for a game that is. You really don't need all the little bevels on most edges. Think about how close the character will get to this, think about how close the camera will get to it. Also how much of the screen space it takes up. You can get away with 90 degree edges on your lowpoly as long as your high poly has beveled edges. You won't really notice the illusion, again, unless you're very close or at a weird angle. The bevels are in your way imo.

Finally, your high poly isn't right. It looks like you sculpted it then decimated it before exporting. There's not enough high poly detail to bake properly to the low poly. And decimation made a mess of the topology which adds to the errors. You should keep it at the max subdivision level you sculpted on and then use that to bake to the low. This is why you're getting some shading issues outside of the hard edges.

Welp first off, I wanna thank you for taking some time to take a look at the models and giving your opinion, it's much appreciated. However, I'm afraid you wasted some time and maybe didn't really read the post. I already knew that the hard edges were the uv seams, the problem is that I don't know how to get rid of them. I'm aware I could've hidden the seams "behind" the object or underneath it so that it wouldn't show that much, but they shouldn't be showing like this in the first place, that's the problem. As I mentioned I've done this before without this problem, that's why I came to search for help. It SHOULD be able to be done without hiding the seams, since I've done it before. I will keep in mind the bevels tip though. When I was modeling this I didn't really think enough about the render. The highpoly should be fine as well, just because I decimated it before exporting, it shouldn't have an effect on the bake. Again, I've done it with this workflow before with no issues. If I would keep it on max subdiv, it would take ages to export from zbrush and even longer still to import to maya. Anyways, thanks again for trying to help

apebit

Ghogiel said:

I had a look at it marmoset, there is something wrong.

The only thing I did really that changed anything fundemental that looked definitvely problematic, was I triangulated the low poly. Since you were baking in maya and maya is the target renderer, not sure if thats the fix you are actually looking for as triangulation shouldn't change unless you do something to the model or move it around to different apps.

UV looked like it should bake ok tbh, I did press unfold on it again just because. I made sure there were no hard edges just to rule that out, in practice with chamfered edges like that you don't really need them. And then I reset transforms, exported and baked n marmoset.

The advice above isn't bad either. You aren't going to get rid of seams entirely,  you can minimise them with UV changes, since this is mid poly looking with double chamfered edges, you don't need or really want to break UVs so much as it'll just introduce visual errors. I mean it's a plank that is beat up, so I expect it'll be pretty rough surface and that'll hide the seams well enough so it'll not a big deal imo.

original

slightly cleaner

You can still see the seams with a glossy material. nature of the beast

Thanks for trying! I tried triangulating the model as well and baking in maya and substance painter but I still had worse results than your marmoset bakes. Now I just wish I had marmoset tbh...

Neox

Finally have the time to take a look at your data.

at first glance, a few pointers:

A. I wouldnt optimize the highpoly, but if you really want to do it, don't do it this harsh. you just introduce a new errorsource to your baking process. Highpoly should be your baking highpoly, an extra step is always a risk.

B. You have quite a lot of long thing triangles, which might not give you enough texturespace dependent on the resolution you are baking, so without testbaking i would assume, that some of those might be problematic. On the sides you can certainly kill some of those long thing triangles by distributing the geo differently.

C. I think you might have overdone it with the chamfers, depends on how close you will get, i dunno. didnt check the objects size.
But this is also contributing to having very long very thin triangles.

Anyways

will bake now

Neox

Okay first testbake,

my result is the same as yours:



But as I assumed, this has to do with color correction being turned on


Turn this off



looks a lot better already huh?



So yeah what you can see here are issues based on too little texture resultion, seems like my Viewport 2.0 is set to clamp the texturesize at 2048px

Switching to 4096 and ramping up the edge padding (Fill Texture Seams in the transfer maps dialogue) killed quite a few of the other issues


So what is left looks to me like a combination of still too little texture space to put the info in and possibly the too strongly optimized mesh causing issues upon bake

Looking at the the UVs at a 4k texture reslution already shows you quite good where the issues likely are coming from



I just gave the outher edges some more UV space and rebaked



Quite a lot better. tje rest i would put on straightening the UVs, giving them more space in general and dropping the mesh decimation stuff you applied to your highpoly.

apebit

Neox said:

Okay first testbake,

my result is the same as yours:



But as I assumed, this has to do with color correction being turned on


Turn this off



looks a lot better already huh?



So yeah what you can see here are issues based on too little texture resultion, seems like my Viewport 2.0 is set to clamp the texturesize at 2048px

Switching to 4096 and ramping up the edge padding (Fill Texture Seams in the transfer maps dialogue) killed quite a few of the other issues


So what is left looks to me like a combination of still too little texture space to put the info in and possibly the too strongly optimized mesh causing issues upon bake

Looking at the the UVs at a 4k texture reslution already shows you quite good where the issues likely are coming from



I just gave the outher edges some more UV space and rebaked



Quite a lot better. tje rest i would put on straightening the UVs, giving them more space in general and dropping the mesh decimation stuff you applied to your highpoly.

Thanks for taking the time and figuring things out! Much appreciated The problem is, I still can't get the bake to work in substance xD 

Neox

apebit said:

Thanks for taking the time and figuring things out! Much appreciated The problem is, I still can't get the bake to work in substance xD 

If Maya is your target renderer, you will not get happy with substance painter. as substance painter is not synced with mayas tangentbase

Marmoset Toolbag is, if you wanna give it a try. I wrote a guide for our artists on how to do it.
https://marmoset.co/posts/baking-for-maya-with-marmoset-toolbag-3/

I believe there are a few other bakers which are. Forgot the names, Knald not sure if it is, never baked high to low there.... the other one.... really no idea what the name was.

apebit

Neox said:

apebit said:

Thanks for taking the time and figuring things out! Much appreciated The problem is, I still can't get the bake to work in substance xD 

If Maya is your target renderer, you will not get happy with substance painter. as substance painter is not synced with mayas tangentbase

Marmoset Toolbag is, if you wanna give it a try. I wrote a guide for our artists on how to do it.
https://marmoset.co/posts/baking-for-maya-with-marmoset-toolbag-3/

I believe there are a few other bakers which are. Forgot the names, Knald not sure if it is, never baked high to low there.... the other one.... really no idea what the name was.

Oh neat, thanks for letting me know! Marmoset is definitely on my "to buy" -list!

Neox

one question tho, why is your target renderer maya?

apebit

Neox said:

one question tho, why is your target renderer maya?

It was supposed to be at first but after a while I gave up with it and rendered with substance, with the issue still there. I decided that since the render is going to be small enough, people won't notice it. At first I thought that it would be nice to render it in maya with neat lighting, but I'm not good enough with maya yet to do that. SO, to save time I ended up using trusty old substance painter

Neox

well i think what i wrote still applies for substance painter, what are your issues in painter?

apebit

Neox said:

well i think what i wrote still applies for substance painter, what are your issues in painter?

The same issue persists in painter. I get the hard edge and no idea how to get rid of it in painter. I can't export the high poly without decimation because zbrush crashes whenever I try. So I can't try baking the nondecimated version and see if it still leaves the hard edges. As for UV space, I've tried giving the outer edges more uv space and bake then but it doesn't seem to give a better result.

Tried giving the edges even more UV space and it seems to eliminate the problem, not entirely but mostly. So it was the UV's all along I guess(?) ¯\_(ツ)_/¯

Neox

you can also try to decimate it to a higher polycount. but all the artifacts you have in your high, you will have in your low

how are you exporting? obj? fbx? i usually use obj and no problem with even the densest stuff - my bakefiles usually are like 6GB thanks to the highpoly meshes

apebit

Neox said:

you can also try to decimate it to a higher polycount. but all the artifacts you have in your high, you will have in your low

how are you exporting? obj? fbx? i usually use obj and no problem with even the densest stuff - my bakefiles usually are like 6GB thanks to the highpoly meshes

I usually do fbx. My file size for the undecimated file is around 1gb but it still crashes zbrush. The entire model has around 37mil polys. Is that good or bad?

Gladioluslab

It seems that your normal display is incorrect. In these parameters switch to "raw"