How can i solve this normal problem ?

jimpaw · May 2011

Hi,

I am working on some propps with normal maps, for practise. I am really new to this normalmap stuff. I have made a highres barrel in Mudbox,then a lowpoly in Maya. I am using Xnormal to extract the normalmap from the highres-lowres. The problem is that the barrel is suppose to have hard edges on the lowpoly model (or should it ?) but when i extract the normals Xnormal creates some strange edge in the normalmap. When i soften the edge the normalmap looks good on the edges but instead it gets a fade over the entire normalmap, i guess that this is from the soft edge which makes the light fade over the edge right ? ,

so how can i solve this ?

darthwilson · May 2011

Hey

Look at were I have marked your image.. use photoshops healing or stamp tool to fix these wee bad patches.. Should do the job

almighty_gir · May 2011

keep 1 smoothing group where possible.

this is working as intended.

healing brush will never get you the result you're after, in this case.

McGreed · May 2011

gir, are you user about that 1 smoothing group, because from my experience (and some other places I read) you should make your smoothing groups fit your islands as well, this way you will avoid seams. I would keep the sides in one group and the top and bottom in another.

almighty_gir · May 2011

just personal experience i guess... but i can't think of a time when i've needed to use multiple smoothing groups for baking normalmaps.

Mark Dygert · May 2011

McGreed, its more like making sure your UV seams fall where you have hard edges, instead of turning your UV seams into hard edges. It's ideal if you can make them match but the unwrap shouldn't drive that choice the smoothing on the model should.

The gradient/fade you're seeing is counteracting the wonky smoothing/normals on the low poly so it matches the high poly smoothing/normals. You're better off going with the smoothed version because you don't really have any hard edges on the high poly barrel.

EarthQuake · May 2011

Really, both of these methods should produce the exact same end result(hard edges or no), unless the bake is set up poorly. Are you using the cage? If you're doing the "use offset" thing, you'll get seams from hard edges, as it breaks the projection there.

Sorry i see you're using XN here not max, you need to either export a cage from max/maya/etc, or go into the 3d viewer and set up a cage, when using hard edges.

The gradiations in the 1 SG example are exactly what you want to account for the lowpoly mesh normals.

Darthwilson: what you're suggesting isn't useful, what you see there is pixel padding to aid in mip-mapping, this in not an error.

almighty_gir · May 2011

oooo actually i have a question regarding this (sort of).

something i'm having to do at the moment involves segmentation of a mesh (for dismemberment). am i correct in assuming that the best way to bake the normals for this, would be AFTER segmenting the low poly mesh, and then using the projection modifier in max, make sure that the vertex cage at the borders are alligned identically?

jimpaw · May 2011

EarthQuake wrote: »

Really, both of these methods should produce the exact same end result(hard edges or no), unless the bake is set up poorly. Are you using the cage? If you're doing the "use offset" thing, you'll get seams from hard edges, as it breaks the projection there.

Sorry i see you're using XN here not max, you need to either export a cage from max/maya/etc, or go into the 3d viewer and set up a cage, when using hard edges.

The gradiations in the 1 SG example are exactly what you want to account for the lowpoly mesh normals.

Darthwilson: what you're suggesting isn't useful, what you see there is pixel padding to aid in mip-mapping, this in not an error.

okay , i will look into that, never used a cage before. i solved the problem in Photoshop by pasting the middle of the first normal to the second. ( have a look at the attachment). But if there is a easier way to solve this problem i will do that.

Thanks everyone !

Mark Dygert · May 2011

EQ brings up a good point about using an unbroken cage over whatever kind of geometry you have. I didn't know you could do that in xNormal so that's good news, its crappy that it derives its cage from the smoothing instead of the actual geometry. I can't think of too many times you would want your cage to be broken where you have smoothing breaks... That's just shitty...

Max sets up a welded cage regardless of the geometry smoothing, getting around the issue of a broken cage. The only way to break the cage in max is to break the geometry into separate autonomous pieces.

Left: Example of a broken cage, because it isn't contiguous, the bake won't be seamless, (this can only end in disaster).
Right: Separate smoothing on the low poly, but contiguous welded geometry and cage, (this is ideal).

WeldedCage..jpg

Personally I'd bake in Max but you're using Maya so xNormal is probably preferable... you just need to make sure your cage is contiguous. It's kind of crappy that the smoothing is effecting the cage generation.

With a contiguous cage, the fade/gradient will be part of your normal map, that is perfectly... normal. You get that anytime a normal map bends around a corner, unless your projection is perfectly planar (which is happening when you're baking with hard edges and a broken cage).

Its that fade that makes it look ok on the final model. That fade is spackling over the horrible smoothing on the low poly model and making it look like the high poly.

If you're used to creating planar normal maps like for floor or wall tiles then you'll probably never see the fade/gradient but you want that whenever it wraps around a corner. What you have in your latest attachment is probably going to have seams.

Also if the normals shouldn't be changed after you bake (soft or hard edges effect the normals ) If you change the smoothing you should rebake.

Left: Split, separate smoothing groups, (notice the double normals, bad for a cage, good for the final low poly)
Right: All in one smoothing group (good for a cage, doesn't always look good for low poly)

Long post short, whatever the smoothing on the low poly is, make sure your cage is welded so there aren't any gaps in the projection. Those gaps lead to seams.

jimpaw · May 2011

Mark Dygert wrote: »

EQ brings up a good point about using an unbroken cage over whatever kind of geometry you have. I didn't know you could do that in xNormal so that's good news, its crappy that it derives its cage from the smoothing instead of the actual geometry. I can't think of too many times you would want your cage to be broken where you have smoothing breaks... That's just shitty...

Max sets up a welded cage regardless of the geometry smoothing, getting around the issue of a broken cage. The only way to break the cage in max is to break the geometry into separate autonomous pieces.

Left: Example of a broken cage, because it isn't contiguous, the bake won't be seamless, (this can only end in disaster).
Right: Separate smoothing on the low poly, but contiguous welded geometry and cage, (this is ideal).

Personally I'd bake in Max but you're using Maya so xNormal is probably preferable... you just need to make sure your cage is contiguous. It's kind of crappy that the smoothing is effecting the cage generation.

With a contiguous cage, the fade/gradient will be part of your normal map, that is perfectly... normal. You get that anytime a normal map bends around a corner, unless your projection is perfectly planar (which is happening when you're baking with hard edges and a broken cage).

Its that fade that makes it look ok on the final model. That fade is spackling over the horrible smoothing on the low poly model and making it look like the high poly.

If you're used to creating planar normal maps like for floor or wall tiles then you'll probably never see the fade/gradient but you want that whenever it wraps around a corner. What you have in your latest attachment is probably going to have seams.

Also if the normals shouldn't be changed after you bake (soft or hard edges effect the normals ) If you change the smoothing you should rebake.

Left: Split, separate smoothing groups, (notice the double normals, bad for a cage, good for the final low poly)
Right: All in one smoothing group (good for a cage, doesn't always look good for low poly)

Long post short, whatever the smoothing on the low poly is, make sure your cage is welded so there aren't any gaps in the projection. Those gaps lead to seams.

thanks alot man for this ! priceless !!

EarthQuake · May 2011

Oh also, if using maya, just bake in maya. =P And do AO in XN.

How can i solve this normal problem ?

Replies