Having Some trouble getting an Enchantress Headpiece into the game

Esuka

Hello Everyone,

I've been working on a peach blossom brooch for Enchantress as part of Dota's upcoming Year of the Horse event. I've gotten the model and the texture where I want it, but I wanted to upload it into Dota2 to check the shaders. That's where I started to run into problems.

At one point, I had the model load properly and the color texture appeared as it should. However, the shaders didn't appear to be loading, and the texture needed a bit of cleaning up. On subsequent uploads, the texture was fuzzy and uneven, as if the UV's were bugged in the export. When I reimported the file to Maya, however, everything checked out. The shaders still refuse to load properly, and I decided to call it a night when subsequent exports (in .fbx) no longer attached the hairpiece to her head. It now sits in the ground between her hooves.

I haven't attached any files yet, as I do not know what would be useful. If you would be so kind as to let me know, I will get them along to you asap. Thank you for your help, and Happy New Year.

Reza

The reason why it is not placed properly is one either of these(or both):

1. The location and rotation of your object is not on 0,0,0 (x,y,z)
2. You didn't bone the object or you boned it improperly.

Esuka

It is quite possible that I have a setting flipped somewhere that is causing the export with the skeleton to bug, but I've yet to spot it. The joints are left untouched from how they came in the zip file, so I don't think it's their position. The model is properly connected to the skeleton when I test in Maya, though that evidently isn't helping in the end result.

Reza

Maybe upload your smd model (the one you submit to dota) here, so we can take a look.

belkun

Make sure to Freeze Transformation>Center Pivot>Delete all History every time after moving the mesh into place and before binding it to the bone using Smooth Bind.

Also, are the masks finished? They are the ones responsible for the shading to work in-game. Also make sure that you're saving them as 32bit TGA and that you have some white pixels at the Alpha channel of both masks, even if you're not using them, because if not, the Alpha channel won't get loaded.

Esuka

@Reza, thanks for your help. I posted the fbx to dropbox, let me know if the link doesn't work. I have a high poly model as well, but I get an error when it tries to make a .mdl out of it, so I've not been working with that as much.

@belkun, I figured out to save in 32bit, but I forgot about the extra pixels. I'll double check that, thanks.

Reza

I couldn't point out the problem, I will let you know if I find out

Reyne

I have had a look at the file. Your UV's aren't quite right. My guess is that you had them right and then modified the mesh in a way that interrupted the seams.

To elaborate: some parts are not in the 0-1 space. This includes things that are not symmetrically mapped.

The second problem is that the mesh does not seem to be skinned to any bone, that is why it appears at 0,0,0. When you export the skinning modifier still needs to be at the top of the stack (in max anyway).

I export to .smd so I cant say for sure, but you also need to have the appropriate bone visible when you export and choose an option like 'ignore non-renderable meshes'.

Hope that helps.

Esuka

Thanks for your feedback, guys. I will try to apply some of these tomorrow, and hopefully I will have some good news.

@Reyne, I think I get what you're referring to the with the UV's being outside of the 0-1 space. It is how it is because I was trying to avoid having the UV's overlap for sculpting in mudbox. If two UV shells overlap, it causes all kinds of mess in the normal map. I was producing the low poly and high poly models at the same time, so the high poly model is more properly arranged in the 0-1 space. The low poly model got arranged more in favor of the -1-0 space as a result.

As for the bones, that's very peculiar. I did smooth bind the mesh to the head joint, and everything seemed to be fine. It's quite possible I missed a checkbox in the export settings, so I will double check that tomorrow.

Esuka

Hey everyone,

So I made sure to freeze transformations, delete history, etc. before binding the model, and I checked on deformed skins in the fbx export options, and it worked. Her hair is on her head again, so there's that at least.

However, the textures remain bizarre. I followed belkun's advisory and added spots to each mask channel, but it didn't seem to fix the problem. It seems to me like the UV's are getting screwed up, but a double check of the fbx file in Maya confirms that they are fine. I would double check the masks, but that's what I'm trying to do by importing into Dota. Any ideas?

Thanks for your help, guys.

belkun

Looks like you have some problem with normals. Try smoothing the whole mesh or just averaging its normals. Also, how is the normal map looking?

Esuka

I tried smoothing the normals, then resetting them to face and smoothing them, but neither produced any results. The normal map isn't going to win any awards, but it works well enough for this one.

belkun

The normal map appears to be right. I'm pretty sure it got something to do with your vertex normals. Try selecting all vertices (before binding the mesh to the bone) and averaging/smoothing their normals. It's that or something weird is going on with your UVs.

Esuka

I checked over all the face and vertex normals, and everything is in order. The UV's are all fine too, I even loaded the texture onto the fbx to double check. Honestly, I'm at a loss as to what could be causing it. Maya is reporting everything fine, Dota says otherwise.

Esuka

Strangely enough, things have become much better. I was able to upload the low and hi poly models, and the textures look much more like they should. However, two problems remain. There is a certain glossiness to the entire model that might be a result of the rimlight shader, but adjusting it seems to have little to no effect. Perhaps from the same source, there are peculiar dark spots that distort the texture. I don't see anything in the shaders that might account for it, could it be the normals? Thanks for all your help, guys.

Xajai

Ive had a lot of issues like this, while working on masks, i find helpful to load into photoshop the standard textures for the hero, this way you can compare the masks, and figure out and channel by channel what might be causing the weird effects.

Esuka

Happy New Year everyone. Unfortunately, my goal of having this all fixed and ready by year's end did not come through. The textures are still odd, and now I'm back to being unable to import the high poly model. Since I don't really know what caused it to work before, I don't rightly know how to fix it this time.

Xajal, that makes a lot of sense, and that's how I originally did my shader masks. However, I found that, for whatever reason, it was actually better for me to deviate to get closer to the desired effect. Of course, now that I can't upload the new high poly, I'm not sure how things are looking.