Maya normal maps and transparency sorting?

KICKER · Apr 2008

Hey all,

Having an issue viewing my normal mapped model in mayas viewport (high quality rendering) with an alpha channel. Rendering in hardware gets the same results. I tried changing the hardware attribute in my material to "Combined Textures" but it didnt help. I also have transparency sorting on. I should also mention that the normal map contains the alpha channel if that makes a difference.

Any ideas, tips, tricks?

using Maya 8.5

Thanks!

Kicker

ivars · Apr 2008

Have you tried using the alpha-channel in the colormap instead? (If that's an option)

TWilson · Apr 2008

Hey me too. I have my alpha in my diffuse tga. From what I can tell it's fine when you render at least. Just not in the viewport hardware.

Helpful response I know ...just letting you know you're not alone

MoP · Apr 2008

Maya has always had horrible transparency sorting in viewports. One of the ways you can maybe make it better is trying to detach some of the transparent parts to separate objects, that usually helps Maya sort them better.
The only way to get good results is using the software renderer. Or use a different model viewer (try XNormal?).

pior · Apr 2008

Hey Kicker,
Ever tried this?
http://boards.polycount.net/showthread.php?t=49920

I think it is the best solution available for realtime display of normalmaps in Maya so far, and it supports masking. It won't give you gradual transparency, but will cut the masks in a very clean manner.

Also, regarding sorting. I found out when working in Maya7 that you could force the sorting by gradually attaching the alphaed bits from the inside out. Select the inner shell, select the one overlapping it, hit unite, select the next one, unite, aso.

Hope this helps!

Brice Vandemoortele · Apr 2008

hehe Merci Pior pour ton soutient :P. I wanted myself to point at Vailias Shader, which supports progressive alpha while mine is only alpha test. But as I read more carefully his tutorial it seems you'll end up with the same sorting issue as you do with regular maya rendering...

KICKER · Apr 2008

I tried using my normal map as my color map with the embedded alpha just as a test and it didn't seem to make a difference, but I'll test it again with a file used primarily as the transparency map.

The objects are separate from the main mesh, but I'll experiment with combining them.

Pior - Does this shader work with 8.5?

This would be perfect for me because I have a lame ass Ati card with SM2.0 support only and I can't use HQ rendering! My model also has no gradual transparency which sounds like it would work well.

I'll post some screens later today if I can get it to work.

Thanks for the quick replies guys!

KICKER · Apr 2008

Here are some results. Haven't tried the shader yet though.

If I could get hardware rendering to work I would be happy!

Yozora · Apr 2008

I also use Maya and have this problem, the way I "fixed" it is by applying a seperate material on the transparent polygons. The diffuse texture is still the same for both materials, but the hair/fur has a transparency map as well whereas the body material doesnt.

These are 3 screenshots of the viewport;
#1 shows the model using 2 materials
#2 shows the model using the diffuse+transparency material
#3 shows the model using diffuse only material

I have no idea why it does this but hey, it works so whatever

Vailias · Apr 2008

Maya really does have horrid alpha sorting problems. It seems to always sort by object, and when you combine them it sorts in the order the combination was done, last on top.

Brice: thanks for the link there.

The shader does support graduated alpha. There are still some sorting problems, mainly that the alpha areas are transparent straight through the model, but its loads better than maya's basic default viewport sorting. Reminds me.. I need to re-visit that shader now that I have a better machine with an nvidia gfx card to test on.

ivars · Apr 2008

High Quality Rendering has Per Face Sorting, but it has to be turned on in the optionbox.

Vailias · Apr 2008

well sure, assuming your video card can render properly in high quality mode (unlike the one on my laptop). But that still doesn't solve all the issues. Just loaded up a tree I built the other day.. and strangely the poly's with alpha are semi translucent all over, not just in the alpha area, and the transparency is additive. So my tree is glowing. Plus the transparency sorting is inconsistent, sometimes it sorts right, sometimes not. .. this was actually a major factor in me learning enough cg to write that shader anyway.

oh, also yozora. It works that way because transparency is only calculated for the materials that have it. So since you've applied the transparent material to some portions of the mesh, but another material for the other portions, you've basically created two separate meshes as far as the rendering engine is concerned, though only one mesh from your perspective.

funshark · Apr 2008

Well, have you tried the "polygon transparency sorting" option since you're using maya 8.5 ?

I don't know if it's working with high quality rendering but you can give it a try.

Prior to maya 8, yeah, maya did only object sorting, while 3Dsmax does poly sort for ages...

KICKER · Apr 2008

The main body has its own material as does the shoulder guard, the hood, and his waist cloth which are the only 3 objects with transparency.

As you can see I'm still having sorting issues, any more suggestions?

This is hardware rendering at default production settings.

I'll try some other options and some general tweaking and get back asap.

Ged · Apr 2008

Im not sure but I think one guy on cgsociety seems to be using mental ray in maya(no gi or ao just normal lights) instead just so that he can avoid these kind of issues http://forums.cgsociety.org/showthread.php?f=234&t=598732

As you can see from his results it doesnt quite look the same as a viewport render :P

rooster · Apr 2008

for the winning pose mental ray is prohibited.. which is damn frustrating as I'd like to just use it without any fancy effects, just to get normal maps to work and gloss maps, since they dont work in high quality (afaik...)

KICKER · Apr 2008

Its been a real pain in the ass! Mental ray is alright but like rooster said, we can't use it for the winning pose.

I'm thinking about rendering out the character exactly how I want and then rendering out the objects by themselves and then adding them in post! and erasing overlapping areas.

Is that prohibited?

I'll have to look at the rules again

Ged · Apr 2008

rooster wrote: »

for the winning pose mental ray is prohibited.. which is damn frustrating as I'd like to just use it without any fancy effects, just to get normal maps to work and gloss maps, since they dont work in high quality (afaik...)

Well this guy at cgsociety seems to have got permission to do just that

he says

"Octostatue: Thanks as always! I'm using MentalRay. Everything is in the textures. There's no FG or GI and all lights are straight up Maya lights. No CIEds, blackbodys, MR area lights, etc. etc. I had raised this question before as I was running into the issue of my normal maps rendering wrong in naitive Maya rendering engines. Loukor ultimately responded and said that "they don't say you can't use mental ray, you just can't use mental ray for advanced lighting techniques", i.e. FG, GI, or MRlights. Which totally made sense to me. There is also the huge issue of transparency maps in the native Maya rendering, just as you've stated. So since you raised the question, I thought I would take it to the powers that be. Heavyness checked into it with the DW people. They said you can use MentalRay for rendering, just no fancy rendering techniques. They allow it as Maya people have issues with the transparency maps like you said. Hopefully this will help you and speed some things up. Good luck buddy!"

on this page.
http://forums.cgsociety.org/showthread.php?f=234&t=598732&page=7&pp=15

so if you really want maybe just ask for permission or something?

rooster · Apr 2008

awesome

Maya normal maps and transparency sorting?

Replies