Milky normal maps - xnormal

Sir Apple

Hi polycount.

I see a lot of expertise on this forum, so I've been reading around a lot the past few days, trying to understand the source of my latest woes (I have found some really good information). Despite my best efforts things seem to be going a bit south with my latest bake on a scooter prop.

Its a more complex bake then my past, and while a bit simpler things bake fine, I am running into nothing but trouble on this.

Certain things I've fixed with ease, flipped normals, xform, cage issues etc. But I am unsure of how to go about getting this thing to bake properly from there.

I'm using max 2012, and baking in Xnormal.

Here is the normal map, the super funky edges is a newer issue that happened after I started playing with dozens of baking methods. The pictures explain better then I can



And on the model in marmoset.









Any advice and information on how to fix some of this would be awesome, in the mean time I'll keep trying. :thumbup:


Thank you for your time,
Apple

16bit

Post a picture of the model with a flat gray material in 3ds max.
And some wireframes.

It looks to me that you might have smoothing groups applied to the seat incorrectly.

Farfarer

Looks like you might have duplicated geometry there, too.

MephistonX

a lot of the time smooth or flat parts of your low poly model simply don't need a normal map - so you can use the background colour of your normal map (the colour that represents flat on a normal map) to colour out those areas of the map - as long as your low poly has proper smoothing groups then it should eliminate a lot of your issues

Millenia

Xnormal bakes tend to look a bit like that, try baking in Max

Ace-Angel

Millenia wrote: »

Xnormal bakes tend to look a bit like that, try baking in Max

No they don't, if a surface is flat, it will be neutral blue, just like Max and countless other apps will bake out, one app shouldn't be giving out a better bake in the case of Marmoset.

Even the 8M guys recommend XN for baking for Marmo in most cases by the way.

Many people had the issues he's having and it has been discussed with a plethora solutions in the Marmoset thread, they usually boil down to a lack of proper cage, Symmetry UV's that got touched up later on, etc.

EarthQuake

MephistonX wrote: »

a lot of the time smooth or flat parts of your low poly model simply don't need a normal map - so you can use the background colour of your normal map (the colour that represents flat on a normal map) to colour out those areas of the map - as long as your low poly has proper smoothing groups then it should eliminate a lot of your issues

Do not do this, it will only result in skirting the problem, not understanding what causes it or how to avoid it. If it even has any effect at all that is.


This is probably an issue with overlapping uvs. I think I've seen similar in the marmoset thread.

Try moving your overlapped(mirrored) uvs 1 unit exactly and then reimporting, and rebaking as well if that doesn't solve it. If that doesn't help it may be a problem with your export/low poly mesh normals.

The "funky edges" are edge padding, and a good thing. It helps with mip mapping, and obviously doesn't show on the model.

edit: Looking at your bake it looks like you're not really mirroring anything? Probably an export error if thats the case. OR some sort of corrupt geometry or something like having some faces that do not have uv co-ords.

joebount

Did you use .obj for exportingb your mesh to Marmoset ? If yes, try to use fbx instead.
How does it look like in Max?

Sir Apple

Wow lots of replies, where to start

I've never been able to get normalmaps to work in 3dsmax unless its with viewport shader, thats why I'm using marmoset.

I will say, aside from the other issues the normal map hardly shows up without an AO in the diffuse slot.



To answers the questions;

Duplicated Geometry - I'm assuming this means a clone of the mesh hidden under one another? I don't believe there is any, unless of course its some annoyingly hidden thing.

Export - yes I use an .obj of the mesh in marmoset. Its weird as its never happened with previous work, I could try an fbx, not sure if that will fix it though.

Earthquake;

Forgive me I'm not as savvy as you, there aren't really any overlapping UVs on the UV layout. I have a few things of exact geometry stacked on top of each other, but none of those areas are really bugging out. For instance I am unsure of why the seat would be acting up so severely, or the other areas. Unless I misunderstand.

When I set it up for bake, I modeled one half of the model, mirrored many of the parts, then attached everything as one object, UV'd it and went through bake process. Because of mirroring it I had to flip some normals.


Here is model with cage;




Hopefully I answered some questions, its difficult as I don't know if the issue is something small that I understand, or something more complex that will take me a bit to figure out - I've found with baking its normally either :poly121:


Anyway thanks for your time,
Apple

16bit

Cage is kind of tight. Expand it out a lot more. It's probably not catching all the geometry and that's your problem.

EarthQuake

Can you upload an obj+ normal map? I'm curious now and want to have a look.

SnowInChina

16bit wrote: »

Cage is kind of tight. Expand it out a lot more. It's probably not catching all the geometry and that's your problem.

second that

looks like some parts of your model are outside of the cage

Sir Apple

Sure thing guys, I will upload tomorrow I'll upload the exploded file, as well as the file with the hi and low in it so you can see the untouched version.

Thanks for your time,
apple

EarthQuake

SnowInChina wrote: »

second that

looks like some parts of your model are outside of the cage

We've had a few comments like this, but if you look at the baked normals he posted there really doesn't seem to be any noticeable ray trace errors, so I'm not sure what relevance it has. (Unless I'm wrong and the errors simply aren't visible from his posted texture).

Sir Apple

Does this work for you? http://www.gamefront.com/files/22383173/scooter+prop.rar

I can't seem to upload any files through the attachment feature on this site. It attempts to upload but then the connection times out. I've tried just the files, then after that didn't work I tried a .zip or a .rar but neither upload.

Perhaps the files are just to large.

Thanks for your time,
apple

GeeDave

Sir Apple wrote: »

I've never been able to get normalmaps to work in 3dsmax unless its with viewport shader, thats why I'm using marmoset.

I will say, aside from the other issues the normal map hardly shows up without an AO in the diffuse slot.

To get normal maps working in 3ds max without a realtime shader, you apply the map to a "Normal Bump" map, which goes in the "Bump" slot in the material editor, this will not show up in the viewport, you need to add lights and render an image to see the affect.

Your second point there leads me to conclude that the problem is with your AO, not your normal map.

Edit
I downloaded your file, why is the AO not included? I really don't think the problem is with the normals at all. They're a little weak (for tests sake I had to overlay two duplicates) but aside from a few gltiches here and there, which are visible in the map anyway, I'm not finding any problems.





There is what appears to be an "odd" line on the seat as an example of possible "glitchyness", but that's visible in the texture itself... so, nothing really out of the ordinary.

I'm now 99.8% convinced this is an AO issue. That's a 0.4% increase.

Sir Apple

Hmm does look pretty good to me minus the visible bugs that are fixable anyway. Thank you for giving it a go.

But I don't see how it could be an AO issue, as posted above here is an image of just the normal map that I baked, it comes out like this;




I suppose it could be a previously mentioned export issue or such on my end with marmoset or something? Although if you even look at the floor boards or such on my image, the normal is very warped and tiled, rather then a smooth bake in those areas.

thanks for your time,
apple

GeeDave

Aw shucks, I'm a bit stumped then. Can't test it in marmoset either.

Have you tried the export that you uploaded for everyone else? I mean, if that works fine for me, and not for you... then the problem has to be marmoset related, n

Sir Apple

I'm really not sure at this point I feel I've tried most everything, at least everything I know could be the issue along with the suggestions.

Also sorry for late reply, busy week.

Apple

dirigible

Do me a favor, take all the overlapping UVs and move them into their own UV space.
Even if they aren't the areas that are having problems in marmoset.
Like in this image (though I didn't finish).

Sir Apple

When you say overlapping UV's, do you mean UV's that are stacked on top of each other? Just double checking before I do so

dirigible

Yep, that's what I mean.

Sir Apple

I tried doing such, the thing is there are only very few parts that are stacked. It didn't seem to solve anything.

I'm wondering if its some random export issue? I'm baffled as to why it worked decently well for GeeDave.

I spent today trying to figure it out, so I exported and did a test bake on a past model and displayed it in marmoset, and got this error. I had some other errors, but I fixed them as they were a result of smoothing errors.




This issue seems to not go away, which leads me to believe the source of all my issues is possibly a bad export? Unless there is a way to fix the above issue, because its a similar issue the scooter has in many areas.

Really wanting to solve this and move on lol

Thanks for your time,
apple

EarthQuake

Sorry I haven't had a time to look at your files yet.

But one quick thing, you're using max right?

When exporting obj, turn off(or on) "use compression" or whatever that setting is. I vaguely seem to recall one or the other producing more reliable results. So what it is set to in the obj export options, toggle it.

Also, try FBX if you haven't already.

dirigible

Post your files online, I'll try to trouble-shoot manually.

Fomori

joebount wrote: »

Did you use .obj for exportingb your mesh to Marmoset ? If yes, try to use fbx instead.

EarthQuake wrote: »

Also, try FBX if you haven't already.

I had a very similar problem in Marmoset and an FBX export solved it for me.

cptSwing

Sir Apple wrote: »

I tried doing such, the thing is there are only very few parts that are stacked. It didn't seem to solve anything.

I'm wondering if its some random export issue? I'm baffled as to why it worked decently well for GeeDave.

I spent today trying to figure it out, so I exported and did a test bake on a past model and displayed it in marmoset, and got this error. I had some other errors, but I fixed them as they were a result of smoothing errors.




This issue seems to not go away, which leads me to believe the source of all my issues is possibly a bad export? Unless there is a way to fix the above issue, because its a similar issue the scooter has in many areas.

Really wanting to solve this and move on lol

Thanks for your time,
apple

fullbright white polies in marmoset are usually a sign of bad uv's. check for single polygons overlapping on themselves (wonky quads or some such).

Sir Apple

Thanks earthquake, I will check into that right away No worries, I know life can be busy.

@Dirigible, I posted them earlier in the thread; http://www.gamefront.com/files/22383173/scooter+prop.rar

@cptSwing - thanks, I will look into that and see what happens.


Thanks for your time,
apple

dirigible

Woops, I missed that link somehow. Sorry.

When I opened up your model, the vertex normals were so incredibly jacked up, I don't even know.


I unlocked them, which reset them back to default (all normals hardened). Softened the normals, threw it into marmoset, and everything looked fine.


So yeah. I think the normals on your model are jacked up.

Paulod

A quick google didn't reveal any decent results. how do you Unlock normals?

Thanks.

poopinmymouth

In Max, add the 'edit normals" modifier, ctrl+A for select all, then hit the reset button, plus collapse. In maya I think it's under one of the geometry edit menus there is a lot of normal options and one is unlock all, but sometimes I think you have to set to face first, unsure on this.

dirigible

Yeah, I can't say for Max, But in Maya it's in the polygon menu, Normals > Unlock Normals.

Xoliul

Or just use a (my) shader in max, then you won't get these issues.
i got this all the time when i tried marmoset, never see it with my shader.

Sir Apple

@Xoliul I love your shader, but I can't get your shader to work anymore. Ever since I got max 2012. Perhaps its my ATI video card. From what I've read ATI can cause issues with it?

@dirigible thanks very much for that I will give it a go, not totally sure I understand what you did so I'll give it a look and come back with my woes

Also when you say;

I unlocked them, which reset them back to default (all normals hardened). Softened the normals, threw it into marmoset, and everything looked fine.

Does that mean you re-fixed the smoothing groups afterwards, or left it the way it is? Because in the image link below you show one model with hardened smoothing, and one thats soft, but I don't know which one is the before and which is the after.

http://i.imgur.com/0On3H.png

Thanks for your time and help Very much appreciated.
Apple

dirigible

The one on the right had the original normals that were all messed up. The one on the left had the soft normals. You can see that the lighting on the right model looks totally different from the left model. That was my first clue that the normals were weird.

I could tell that you had some smoothing groups set up, but unlocking gets rid of those. I didn't fix them, since I wasn't sure what exactly they were supposed to be, and figuring out would have taken a long time. You're going to have to go through and set them up again (or just rebake your normal map).

Sir Apple

I tried to reset the vertex normals, but still the same exact results in marmoset. What does your normal map look like when you test baked?

Here is mine. Still a few weird bugs here and there but yea. I can't figure out why its showing up so weird on the model.



Here is a picture to perfectly explain what is confusing me;



I don't know if its a marmoset, or model side issue?




I will say, nothing visually happened at all when I selected all the normals and hit reset, I thought maybe the settings were weird but I tried a lot of combinations and nothing worked. Also told it to 'select by' and reset "vertex" at one point, thinking that was the issue, but still no

Here is what I see before I 'reset' the vertex normals, it looks the same afterwards though.



I don't know, perhaps the model is a bit to poorly done to accompany a fix to my woes? I'm 100% positive I'm doing something terribly wrong on his though, if its been proven to bake for other people here.

Thanks for your time and reading my lengthy post,
apple

dirigible

I didn't rebake your normal map when I tested it, I just used the one you sent. Since all the normals were softened it didn't look 100% correct, but I could at least tell that there wasn't any random seams.

I'm a maya guy, so I really can't help you with normals in max. I can't tell if they look correct in your max screenshot, but they definitely look wrong in your marmoset screenshot. It might be something to do with your exporting, I don't know.

Okay I just baked out a couple different maps and noticed that some of your high poly geometry has inversed normals. I don't think that's what's giving you issues, but it's something to check out.

Anyway, with the maps I baked out and the model I have with the correct normals, it looks fine. There's nothing inherently wrong with your model, it's just something weird about your vertex normals.

Side note, you should probably be mirroring more parts of your model, and packing your UVs a bit more efficiently. I baked out a 1024, and the resolution on model was terrible.

Sir Apple

Okay, well thank you for your help Lots of good information in this thread regardless.

In the mean time I will keep trying, and press on! If its okay I may use this thread for any future woes I could encounter, as I don't want to redundantly make threads about my long winded baking troubles This is the only object though thats given me this many issues, so I'm hoping it stays that way. :P

Thanks!
apple

keelhauler

My spoon to the soop.

Did some test bakes with your model also and fixing the vertex normals indeed gives clean normal map bakes.