major baking problems please help

polygonfreak

I have spent many hours trying to figure out what I am doing wrong with this bake. I created the high poly model in mudbox, then I used xnormal to bake the model. I keep getting these lines on some of the corners of the house. I have tweaked the uv's, and tried all types of different smoothng groups. I am hoping that its a simple beginners mistake. I think I have all the pictures to give you an idea of my problem. I am thinking its a smoothing group problem, but I am not sure what needs to be done. any help would help me out so much. Thanks.

SaferDan

Im pretty sure you just need to harden up all the edges that have a seam, which looks like all of them?

Aerial_Knight

I think you just need to stench the edges that might help

Sean VanGorder

You'll want to have different smoothing groups wherever you have UV seams.

Also, make sure your green channel isn't inverted.

polygonfreak

I am confused about smoothing groups and what to use... here is a pic of my seams and the smoothing groups.

polygonfreak

Just thought of something I will try... instead of the seams on the corners, what if I broke them in the middle of the model?

Artifice

That's going to make texturing a pain in the ass. Just break them where it seems natural...along the corners. That means you need to have different smoothing groups wherever there's a seam.

For example, make your wall with the door Smoothing Group (SG1). Make the wall with the window SG2. Make the roof SG3, because it's adjacent to both SG1 and SG2, so it needs to be different. Otherwise Max will think that there should be a smooth transition between your wall and your roof, where there's really a hard corner. The same goes for the edges of the doors and windows. They're at 90 deg to your walls, so they need different smoothing groups/UV islands.

REMEMBER: Every poly that has a different smoothing group needs a different UV island. Every UV island needs a different smoothing group. Break your UVs up by smoothing group, and set your smoothing groups by UV island.

polygonfreak

Artifice, thanks for taking the time to help me out, I really appreciate this. So I placed everything into different smoothing groups, fixed the uv's to be separated per group, and then rebaked the whole thing. I saw a difference but the front face still has the dark edge. If you or anybody else knows what I am doing wrong... you would help me out so much.

Artifice

It looks like the majority of the smoothing issues are better (along the walls and such). A couple of things to check now. Is your low poly intersecting your high poly at all, ie can you see the HP edges sticking through your LP, or visa versa? If it is, that can cause artifacts.

Second, are you using a cage in xNormal or ray distance? You might try a test render in Max's RTT using scanline. It's quick and doesn't require exporting, and you can see what your cage is doing. Alternately, check the cage in xNormal's 3D preview.

Lastly, how are you viewing this? Is it in Max? If so, what shader are you using?

EarthQuake

The first shots look simply like an incorrectly applied normal map, so basic questions:

1. Where did you bake your normal map?
2. How are you trying to render your normal map? - What program, what material type, settings etc

[Edit] I see you baked in XN, displaying in max, this is likely the source of your problems -
Have you tried inverting the green channel of your normals?


Now on a more practical level, this sort of asset would rarely be done with a unique normal map, instead you should be using a tiling texture.

Also, some pretty strange advice in this thread, c'mon guys.

polygonfreak

EarthQuake, I inverted the green channel and it completely fixed the problem. Thank you so much... I was getting really confused.

But now you have my attention, so in the industry, this asset would not be modeled the way I did it in mudbox? I am currently not even a few months into making next gen assets and I dont want to start bad habits that will suffer when I have a job in the industry.

Artifice

EarthQuake wrote:

Also, some pretty strange advice in this thread, c'mon guys.

In all fairness, he did have some pretty nasty shading errors in that pic. I figured he already tried the swizzle since SeanEG had mentioned it before. I'm not really familiar with the issue as I don't use xNormal.

Anyway, glad you got it figured out, it's looking much nicer.

EarthQuake

Yeah, in this case especially you'd just use a tiling texture, as the detail is all uniform and easily represented that way. There isn't really any unique detail here, so it doesnt make much sense to use a unique layout. Your texture resolution is going to be much lower that way. You can get the soft bevels by just using a little bit of geometry in your lowpoly(which you have already on some parts.

In general, you can optimize your mesh quite a bit as well, you have lots of redundant edge loops that aren't defining silhouette or smoothing.

Another thing you'll want to do is look into a realtime shader, max is known to have some issues displaying normals much differently in realtime than it does in the scanline renderer, so its important to know how your assets will look ingame.

Look into modular design, you'll want to learn how to do this for environment art: http://wiki.polycount.com/CategoryEnvironmentModularity?highlight=%28\bCategoryEnvironment\b%29

polygonfreak

EQ, modular is on my list of things to dig deeper into. What is a good render software. I will use udk for bigger scenes but for smaller assets, what is the best route?

EarthQuake

In max, 3ps shader(check the link in my sig), Xoulio's shader, or you can use Marmoset. All good for viewing realtime.

polygonfreak

Thanks, i will give those a try.

LateWhiteRabbit

I use Max to model, and Earthquake is right, you need to use a real time shader to view your work.

Even for things like flat diffuse textures, Max's viewport and render will show your colors as darker and grayer than they actually are. I personally use Xoulio's shader while I work, then render final images in Marmoset.

polygonfreak

Just installed 3 point shader and boy what a difference. Thanks again.