Strange Normal Map... Please help!

lymiller

Hi everyone. I've been having an issue with Mudbox as long as I've used it. Hopefully someone can help me...

This is a work-in-progress I'm doing, but I figured I'd try an export. I'm working in Mudbox 2010 and applying the Normal Map in Maya 2008. I applied the normal map to the low-poly model and got this.

The high-poly model in Mudbox currently looks like this.


And, since this is probably important, this is how Mudbox exported the normal map. It was exported at 2048x2048.


What's going on? Please help!!!

Archanex

I think it would help people troubleshoot your problem if you went into more detail about the steps you took to get what you got. did you unwrap the model in maya before bringing it into mudbox?

CheeseOnToast

First off, is that a render or a viewport grab in that first shot? Maya's software renderer treats normal maps as bump maps by default, with results similar to what I'm seeing there. If you want to view the normal map, set the viewport to "high quality" to see it in realtime.

rollin

the errors are within the normalmap.. so pls show us the lowpoly and the bake setup if possible

Mark Dygert

Aside from the technical Maya rendering issues.
- The uglies around the hanging bits on the chest, is due to the unwrap, not enough pixels to capture much of anything.
- It also looks like your bake failed to include padding?
- You also have mirroring? Did you move the mirrored (inverted) piece our of the 0-1 space?

Personally I think your map is big enough and you have enough space to uniquely unwrap the whole thing.

lymiller

Thanks, guys. Let me try to respond to all of you...

I had exported the model from Maya (where it was modeled) to unwrap in Max (I'm better in Max and figured I'd save time.) I made sure to export quads and not tris. The images shown I tried creating the normal map straight from Mudbox.

The first image (with the black background) is a render from Maya. I had set the viewport to high quality like you mentioned.

A collegue actually recommended I try baking the normal map from Maya, so I followed this video tutorial: http://www.moddb.com/tutorials/bake-normal-maps-in-maya (he's using ZBrush, but I figured the logic was the same.) Basically he imports the original low-poly and the high-poly and uses the "Lighting & Shading>Transfer Maps..." function in the menu. Only problem is mine doesn't look smooth like the Mudbox model AT ALL. In fact, Maya seems to be paying attention to the edge loops in the low-poly too much. Here's what it looks like:



I actually rendered the high poly from Maya and it looked fine. It must be my export settings...

Vig, what do you mean by "uniquely unwrap the whole thing"? You mean have fewer seams between the different elements? (more things joined in the unwrap?) And you say "not enough pixels" -- I'm not sure about that, either. I've never gotten a normal map to export correctly so I believe I'm missing some basic theory behind the rendering process...

Also, if it's important, I'd unwrapped the model in Max as a half and had a Symmetry modifier on top of that so I wouldn't have to unwrap both sides. I assume that if this is a problem I can merely close symmetry, re-unwrap it, and import the new UVs into Mudbox without losing any work.. I hope.

I guess I'll leave it at that for now. Thanks a lot, guys. I really appreciate the help...

lymiller

On a side note, in case I was unclear, the initial post was when I tried exporting the normal map from Mudbox. The image from the post I just made is the other technique of baking the normal map from Maya.

I hope that wasn't confusing.

Mark Dygert

Yea it will cause a problem if you just run symmetry on an object and leave the mirrored pieces inside the renderable area. You want to move the inverted pieces 1 tile to left, right, up or down. That way they sit outside the renderable area and aren't effecting the bake BUT are still sitting on a tile and will display correctly.

Taken from http://joeyspijkers.com/t_normalmapping.html
Notice the mirrored bits are hanging 1 tile to the right.

If you don't offset the mirrored pieces, you end up rendering each side twice to the same area and it can really make a mess.

By uniquely I mean each UV island gets its own space inside the renderable 0-1 space. So no mirrored pieces, everything is facing the right way and everything has its own unique space.

There is a lot of dead space in the unwrap layout. Things could be scaled up quite a bit and use more of the pixels to give better normal map definition.

lymiller

Vig wrote: »

Yea it will cause a problem if you just run symmetry on an object and leave the mirrored pieces inside the renderable area. You want to move the inverted pieces 1 tile to left, right, up or down. That way they sit outside the renderable area and aren't effecting the bake BUT are still sitting on a tile and will display correctly.

Okay, so let me see if I understand... basically, if I collapse the geometry in Max and move everything from the symmetry-ed side OUT of the renderable space the Unwrap will still effect those pieces even though they're not rendered? Or does it do what you're saying if I don't collapse symmetry?

Does any program other than Max see the "symmetry" algorithm from Max, or do those programs just import the symmetry-ed model as a solid piece? I'm trying to understand how an unwrap of only the renderable area will cause both sides of the model to be effected...

Yeah, I haven't yet mastered minimizing what you call "dead space". I'll try to resolve that and reimport the UVs into Mudbox. Thanks.

Mark Dygert

They see it as a solid piece of final geometry. Some might not read it at all and you'll have to collapse it before it will read everything.

To take care of the mirrored bits you use the type in transform tools in the UV editor window inside of max, which are just like the type in transform tools just below the time line in Max.

Select the symmetrical half click 1 (this changes it from scale to move)
In the U box (2) type 1 and hit enter.
Then uncheck 1 so you don't accidentally move a piece around.

lymiller

Thanks a lot. I'll have to try this tomorrow since I'm home sick today and Max is in another location. I'll let you know how it goes once I try it.

kodde

Just making sure here, but when you baked the normal map in Mudbox and imported it in Maya did you change the Bump2D node's setting "Use as:" from "Bump" to "Tangent space"?

When baking the Normal Map in Maya with Transfer Maps it will by default create a new lambert material for you and hook up the normal map to the shader correctly (bump2D with tangent space). I can see how you can easily miss this part when connecting it manually.

You might know this already, just making sure.

lymiller

kodde wrote: »

Just making sure here, but when you baked the normal map in Mudbox and imported it in Maya did you change the Bump2D node's setting "Use as:" from "Bump" to "Tangent space"?

Yes, I did make sure to change that. That video tutorial I linked to mentioned that.

mLichy

Hmm... I've always moved the symmetried pieces out of the 0,1 space when rendering, but then guys at work said it didn't matter, so I haven't most of the time, and it renders fine?

EarthQuake

mLichy wrote: »

Hmm... I've always moved the symmetried pieces out of the 0,1 space when rendering, but then guys at work said it didn't matter, so I haven't most of the time, and it renders fine?

Different apps seem to handle it differently. For example i think for both max and XN the normals will render fine, but AO will bug out. In maya normals will rendered wonked if you dont offset.

lymiller

EarthQuake wrote: »

Different apps seem to handle it differently. For example i think for both max and XN the normals will render fine, but AO will bug out. In maya normals will rendered wonked if you dont offset.

In Maya, is that why my model is rendering with SUPER deep geometry? Because it's basically rendering the normal map twice because of the overlapping?

lymiller

Hey again. I went ahead and dragged one half of the UVs out of the renderable area. Now I'm actually not sure: how do I import the new UVs to the Mudbox model without losing all of my work? It won't let me import a .uvw file. I can't find a tutorial explaining how to export UVs from Maya, or how to import new UVs into Mudbox.

lymiller

Sorry, nevermind. I just figured it out. Maya just works differently than Max...

Let me see if this fixes my rendering problem.

lymiller

Oookay, I definitely have a new problem.

So, as I mentioned, I dragged the mirrored side of the model's UVs out of the renderable area. I imported these new UVs into Mudbox and it seemed to work.

So here's what's happening: Before this, when I would step up a level, the UVs would disappear (which, I don't know if that's suposed to happen.)

Now I'm getting this. The top is the UVs at the lowest subdivision, the bottom is going only one step up:



What on earth is happening?? I can only assume this is not going to render a normal map correctly. What can I do to fix this?

Mark Dygert

The UV's on the high don't matter, its only the UV layout on the low poly that matters.
There are some very specific cases you would want to unwrap the high but this is not one of them.

What it's doing is creating geometry between the seams, it attaches each new face to its neighboring seams.

lymiller

Vig wrote: »

What it's doing is creating geometry between the seams, it attaches each new face to its neighboring seams.

Oh, okay. So that isn't a problem? Sorry, I must seem like a total newbie. (I guess I kind of am.)

Okay, I'll go ahead and try a render today and see if it works. ::crosses fingers::

Thanks for everything. I really appreciate the help.

lymiller

Okay... it still didn't work. Here's what I got when I rendered from Mudbox:



Is there really no specific reason why I'd be getting crispy edges like this? I have no idea what I'm doing wrong!

Ott

Those tiny little bits have almost no UVW space, and as such when you are trying to bake you are going to get all sorts of messed up information from the render. I see no reason to waste your UV space, and as such you should scale all that stuff up.

How you UV is among the primary determining factor for how your normal map is going to look.

As far as the edges, those are most likely edge misses where the high and low poly geometry aren't perfectly aligned when the cage does a check. If you have a little padding turned on, it might not really matter too much on the final model, but you would have to apply it to your model and see if it is noticeable or not. (It depends on how your model and UV splits are set up)

lymiller

Ott wrote: »

Those tiny little bits have almost no UVW space, and as such when you are trying to bake you are going to get all sorts of messed up information from the render. I see no reason to waste your UV space, and as such you should scale all that stuff up.

How you UV is among the primary determining factor for how your normal map is going to look.

As far as the edges, those are most likely edge misses where the high and low poly geometry aren't perfectly aligned when the cage does a check. If you have a little padding turned on, it might not really matter too much on the final model, but you would have to apply it to your model and see if it is noticeable or not. (It depends on how your model and UV splits are set up)

Everything you've mentioned is what I am having the most issues with. I guess because I was never taught the mudbox pipeline I'm struggling against that now. Can you recommend a tutorial or book that lays out IN DETAIL what needs to be done to get optimal results?

Also, I'm planning on expanding those squished areas, but do you think rendering a larger map would fix the problem? (for instance, rendering 4096x4096 instead of 1024x1024 or 2048x2048)

Mark Dygert

lymiller wrote: »

Everything you've mentioned is what I am having the most issues with. I guess because I was never taught the mudbox pipeline I'm struggling against that now. Can you recommend a tutorial or book that lays out IN DETAIL what needs to be done to get optimal results?

Also, I'm planning on expanding those squished areas, but do you think rendering a larger map would fix the problem? (for instance, rendering 4096x4096 instead of 1024x1024 or 2048x2048)

4096x4096 was the entire texture buget for this years Unearthly Challenge, so I think that's a bit of overkill. Even at 4096 the UVs would need to make better use of the space. If you can post the low poly model I can show you how I would unwrap it.

lymiller

Vig wrote: »

4096x4096 was the entire texture buget for this years Unearthly Challenge, so I think that's a bit of overkill. Even at 4096 the UVs would need to make better use of the space. If you can post the low poly model I can show you how I would unwrap it.

Here's a link to the zip file containing a Maya file and the OBJ -- whichever you need.

http://www.impdragon.com/lymillerModel.zip

Unless you mean for me to post images -- that I can do too. Let me know.

.Murder

just render it with maya hardware not maya software

Mark Dygert

Quickie lunch time unwrap.

The model: http://www.vigville.com/forum_images/lymiller_model01.zip
There are a few areas that could be finessed a bit more but I ran out of time.
Hopefully that gives you a few more pixels here and there to work with.

lymiller

Wow, thank you! Just seeing the unwrap like that gives me a beter idea of what to try.

I actually began welding symmetrical pieces earlier today (because I think symmetry might be causing a problem) and otherwise spreading tightly-clumped UVs apart. I'll keep that up and let you know if it works.

Thanks again!

lymiller

Okay, so I fixed up my UVs quite a bit. Here's what they look like now:



I just tried a new render from Mudbox and got the following. It looks better, but it's still doing the weird crispy thing...



Before this I did have a lot of overlapping verts (don't know how I missed it) so I fixed a lot of that up; also spread out a lot of clumped UVs to provide more breathing room.

I'm... not quite sure why this is still happening. I read the troubleshooting Maya help doc today and it stated that this was the problem (same as you guys said).

Let me know what you think.

lymiller

Are tightly clumped UVs the ONLY reason why it would be rendering this way?

I just need to know if I need to focus all my energy on unwrapping it till it works, or if that would be barking up the wrong tree.

I have nothing else to try. I don't know how my UVs could be "bad" if the checker pattern is displaying as undistorted as possible.

If anyone has any opinions, PLEASE let me hear them.