Newbie Normal Mapper - Kinks in the workflow, black blotches

DudeBro

Hey guys! New Polygon poster and member. Been doing the modeling and texturing thing for a few years now but it's only recently that I've been trying to step up my game with, well, game knowledge, and normal mapping seemed like a logical thing to be working on.

I have the basic workflow down, and have even seen success with it before:

Low Poly Maya Model, sent to Mudbox to sculpt detail, sent back to Maya as a high-poly model where I attempt to do a normal map transfer bake. I was struggling with the results so I attempted to break the model down into pieces and attempt to transfer the map piece by piece.

The results seem...sloppy to me. A lot of the detail seems lost and worse are the black blotches (also attached):

http://imgur.com/WWIcEK3

A bit of research tells me this is from "overlapping normals" in the high poly, but that doesn't seem to like it would be a problem...the fingers all look separate enough, for example. Could this be a setting I'm missing somewhere? I'm searching inside envelope, by the way.

Thanks for the help guys, this is an intimidating new frontier!

Bartalon

Maybe it's just me, but it's hard to say exactly what's going on from just that image.

Depending on how you sent your stuff back from Mudbox, the mesh normals could become locked which may result in dark spots like that... Normals > Unlock should reset them. It may mess up any hard/soft edges you had previously but it should still look better (black spots removed).

Does your high res version have all smoothed normals?

Do your hard/soft edges match your UV shells?

Are you using an envelope when you bake with Transfer Maps?

cryrid

The left image almost looks like it needs to have its green channel (on the normal map) flipped.

DudeBro

Hey guys, thanks a ton for your feedback. Sorry if I'm not providing a clear enough perspective on what's going on. I tried again and took more workflow screenshots to show what I'm doing, hopefully that helps (they're attached).

I also tried the unlocking normals you mentioned, Bartalon. Crypid, I'm not sure what you mean (yet) about the green channel. It's y axis should be inverted?

Bartalon

I see you're using Viewport 2.0. I've had some instances where my normals absolutely would not display properly with that rendering mode. Switching to High Quality usually does the trick for me.

As for those completely black spots near the elbow, that is probably due to the envelope not being far enough out. You can see some clipping in the screen shot with your envelope near the armpit, so perhaps it is happening on the elbow as well. The envelope can be manually tweaked in specific areas if your HP is poking through.

DudeBro

Thanks again Bartalon. I gave High Quality a shot this time.

I'm seeing better results now, with the occasional black blotch but I think it's about pulling some envelopes around at this point. There's a touch of aliasing problems, especially around the folds toward the cuff. I had heard this might be because the normals of the high poly are overlapping a bit much but I'm not sure how to work around that one...

Looking much better though, thank you

Bartalon

Excellent! I would also recommend giving the following threads a read, as they helped me get normal mapping down pat when I was struggling with it.

http://www.polycount.com/forum/showthread.php?t=107196
http://www.polycount.com/forum/showthread.php?t=81154

DudeBro

Thanks! I will definitely check it out.

When I first started out I suppose I should have considered that "overlapping wrinkles" would not be a particularly friendly first effort, huh?

Any thoughts as to those aliased patches? Is that normal? I know it will hardly be noticeable when I apply a diffuse map but it's interesting to be thorough nonetheless.

Thanks for making this less bang-my-head-on-the-keyboard, Bartalon.

Bartalon

I'm glad to help!

Those aliased patches are usually caused by intersecting HP topology, cages, diagonal UV shells combined with low res texture maps, or a combination of all of the above.

DudeBro

I follow ya on all but "diagonal UV shells." Does that refer to UV shells in the map that are located physically diagonal from each other? Does this negatively affect things? The workflow I followed for this was to unwrap the whole body and then split it into pieces to make the sculpting less intensive and not drag down my computer.