So im working on consolidating a bunch of different objects onto a single texture sheet. I selected all the objects added a unwrap uvw modifier and rearranged them so they all fit into a single sheet using uv channel 2. Now when I do render to texture the diffuse transfers over fine, but the normals go to hell. First I tried just selecting normal map in the RTT options, that seems to weaken my normals and screws with the channels a little bit causing some seams. So then I tried putting the normal map file into the materials diffuse slot and then using the RTT to render a diffuse map of the normals. That worked but then the lighting information is wrong if an object was rotated in uv ch 2 while I was aranging things. To fix this I tried to swap around the channels inside photoshop, this worked for the most part but again caused problems with the seams. So I am completely out of ideas, if anyone has some tips I would love to hear them.
Replies
The only "proper" way to do this would be to bake down from your highpoly source again to the new layout.
Certainly, baking it across as a diffuse map is not going to work.
I'm still working on the more in depth version which covers how to handle normal maps, but MoP already hit on the short answer.
If you bake out a new normal map, it will work, although the OP is talking about losing some quality (which is to be expected whenever you re-bake, really). It will not work if you bake the normal out as a diffuse map, since if an island is rotated the channels need to be flipped.
this was all done is 3ds max 08 or 09, by the way--not sure which version we were on at the time.
No dude this makes perfect sense. You were baking the model + normal map applied to the model, which would be in the correct worldspace.
Now, rotating your uvs and baking from one to another wont work, because you're just baking colors, not real normal data, as you are when projecting from one mesh to another.
Generally it would be much better to just rebake from your highpoly source, because you may end up baking down smoothing errors, if you're duplicating, applying your normal map and projecting it onto another mesh.
Yep loss of a bit of quality is to be expected ofc, I went and checked and this did work - it was from one mesh to a cloned and remapped mesh similar to silversteez and recomputed the normals more or less correctly from the differently aligned uv chunks.
It was an emergency bugfix. Worst part is the loss of layered-ness of the psds for subsequent adjustments but as I say needs must!
I was glad to find that I wasn't imagining things though, I can tell you - my little heart almost stopped for a second when I read the OP. ;-)
here i finally cleaned up and 'documented' this little trick:
http://boards.polycount.net/showthread.php?t=61438
Anywho...
Render settings effect the quality DRASTICALLY. Most notably the default Anti Alias filter and Global Super Sampling settings. Consider them EVIL when transferring uv layouts. Just remember that AA + GSS spells ASS. Yea it drops so much quality it loses the G.
This is on a sharp image with high contrast, it gets much uglier when the colors are closer in hue/sat. In this example turning both AA & GSS off rendered it perfectly, mostly thanks to the high contrast, don't expect that kind of voodoo every time. I'm surprised the AA only render came out as well as it did.
Turn both AA & GSS off to get the closets thing to per pixel as you can. UV island scale/rotation changes will also factor into quality. Don't scale but if you do, go down not up. If you rotate try to stick to 90 degrees, Max has handy buttons for this, look to the lower right of the unwrap editor window.
Also rendering to TGA (uncompressed) will also help you preserve pre-pixel quality. Let the engine make a blurry mess of your textures when it compresses. Don't help it make a mess by feeding it already blurry images. Or worse blurred 3x, like use a png as your texture (1st loss) then RTT with AA-GSS on (2nd loss), then save again to png, oh the horror...