Creases n' edges n' happy times in mudbox.

fredriklars

So, Here is a thread, that maybe should be a post, but i didn't find the thread for it, so i went ahead and posted.

I have a rather specific inquiry regarding mudbox's support for hard edges and creases.
The feature is nice, because sometimes, creases make for really neat stuff in subdivisions. It is said to support hard edges AND crease sets(possibly only the latter from maya), but i have found that it rather supports hard edges OR creases, because in order to get the creases from Maya, to FBX to Mudbox, you have to disable the checkbox for smoothing groups, which in turn kills your hard edgees and the ability to use the mesh as recipient for neat and dandy normal maps.
However, if I leave that option on, the mesh is an excellent mesh to bake to, but I will face the smooth or no smooth dilemma when subdividing, and i have been back and forth with this and neither option is very apppealing. Large shapes become shit when not smoothing, hard edges become impossible when i smooth.



So, I want to:

Make a lowpoly model in maya.
Crease it for nice subD's.
Harden edges to make it a good baking target.
Paint and sculpt and then pull it all in one single magnificent bake.

In order to:

Never switch models, so i can bake via uv- matching or subdivision method(which IMO is superior, correct me if I'm wrong(and say why)).
Stay in maya and mudbox as long as possible, rather than back-and-forthing with xnormal, maya, mudbox and photoshop and keep track of all files during trial and error to get the perfect bake.

Because,

If this was to work, the workflow would be maya>mudbox>WIN(PS for final touches of course, but you know...)
The workflow would fairly non-destructive, which is happy times.
There would be no cage-tweaking and no retopo, which is happy times.
I am insanely curious if this could be a functioning workflow.

I do realize that this method would require some compromise regarding modelling and/or optimization but if this was to work, i believe one could regurgitate quality content in the hastiest of manners, and stick to the fun parts of content creation for as long as possible.

This, to me, should not be that hard, since mudbox lets you choose specifically to check "preserve hard edges" and/or "preserve creases", which leads me to believe that someone had this in mind when designing the features.

So, i await your replies, regarding my problems or anything i have said here. Thank you for your time and awesomeness.


Also... it hit me, might there be a format besides fbx that carries creasing AND smoothing? I tried obj, fbx and the "send to mudbox" feature.
No dice.

EarthQuake

Well the first problem with this is: Good topology for lowpoly models, and good topology for sculpting are two different things. For your lowpoly, you want to optimize it to use geometry for silhouette changes, and joints for animation. For sculpting, you want even sized, square quads to avoid stretching when you sculpt. So rarely will these two meshes be the same.

Second problem is: Doing your lowpoly model, uvs, smoothing, etc first is very restrictive. Generally by the time you finish sculpting your high you will have made some drastic changes. If you move your model around any significant amount during the sculpting phase, you could introduce a lot of UV stretching.

IMO you're really over complicating things here, I would simply invest some time and research into how get nice bakes reliably, out of maya, xnormal, whatever.

RE: UV matching, I posted my thoughts on that recently in this thread: http://www.polycount.com/forum/showthread.php?t=116502

As far as hard edges go, if you're baking normal maps, you'll need uv seams everywhere you use hard edges. So even if you can't import your hard edges, it will be very easy to recreate them in Maya before you bake. Here is a script to turn your uv seams into hard edges: https://dl.dropbox.com/u/499159/UVShellHardEdge.mel

fredriklars

That was an interesting read and we seem to draw similair conclusions in some cases, although i mostly disagree with you. I dont mean to be an ass about it though i just think you might be missing my point a bit. I'll try to further elaborate.

Well the first problem with this is: Good topology for lowpoly models, and good topology for sculpting are two different things. For your lowpoly, you want to optimize it to use geometry for silhouette changes, and joints for animation. For sculpting, you want even sized, square quads to avoid stretching when you sculpt. So rarely will these two meshes be the same.

Yes, this is what i meant with saying compromises will have to be made in regards to modelling and optimizing, one would have to find a topology that is good as a in-game model and at least half-decent for sculpting.

As far as hard edges go, if you're baking normal maps, you'll need uv seams everywhere you use hard edges. So even if you can't import your hard edges, it will be very easy to recreate them in Maya before you bake. Here is a script to turn your uv seams into hard edges:

Yes, this is the sole reason i need my hard edges separate from my creased edges.

I do not think that i am over complicating things here, i have invested a lot of time in learning to baking and as all here knows it is usually a case by case thing how you solve that, bake parts separately, paint on a normalmap and so on. That is boring and destructive.
While my methodis only fit for specific objects, I really think that there is something there.

Let me try and explain it a bit more.


Imagine I was an idiot.
If i was, I would probablymodel the left shape and hope to bake the right shape on it. Note, the right shape is simply subdivision with creasing.
In this example I have put hard edges and uv-seams along all edges i want creased, to bypass my earlier mentioned problem.

Now, looking at the idiot shapes together, you see where this idiot bake is going wrong. You got extreme mismatch in geometry, the low and high poly intersects themselves and each other in the middle of a round hole shape in the middle of a big cube shape. On the rright there is some more common jaggedness one might encounter.

And yet... this normal is baked. in one bake. NO cage, NO retopo.
All i did was put it in mudbox and drag the smooth-brush on it.
Now, granted that this is an idiot shape, it is ugly. but it gives as nice a beveled edges as a normalmap can. Even in the grotesque mismatch it holds, it provides a fine bevel across the entire thing. Never mind the color issues, i hit something in PS and its 3 am so i wont redo it.



And think on this, the whole example i set up here took 30 min to do, and 20 of those was for uv-mapping the idiot shape and painting angry red arrows the second image. To me that renders this method insanely useful in regards to simple props where one can make the model low-poly and half-sculpt friendly.
IMO this could be used if it was possiblle to crease edgfes without making them hard in mudbox.

For reference, here is the normalmap:

EarthQuake

Well, sure its sort of cool that you can bake a normal map with completely mismatched meshes, but in reality, if your lowpoly is nowhere near matching your high, its just not going to look good, or how it should look. Making a lowpoly that matches your highpoly very well will solve a LOT of baking problems, and result in the best looking end result as well.

If inter-sectioning issues are a big problem for you, do a key framed exploded bake(only takes a couple minutes to set up) then you don't need to fuss about with doing separate bakes, hand painting stuff in photoshop, etc.

If you're getting waviness or other ray projection errors, you can re-think how you design your meshes, as well as how you construct your lowpoly meshes(both of which generally will result in better looking assets as well), more here: http://www.polycount.com/forum/showthread.php?t=81154

You keep mentioning "cages", but if you've created your model in a logical manner, setting up a cage should be a quick setting of the push % or whatever(depending on app), if you're spending a lot of time tweaking cages to get good results, you've got issues somewhere else.

Again it seems like you're trying to come up with solutions to the problem of baking, but really it seems like you just don't have a solid baking workflow down, that is what I would focus on.

fredriklars

Considering you are are the only one replying to this(thanks for that btw, your posts have taught me much in the past), and you seem pretty not stoked about this, i'm thinking i should re-evaluate the usability of this.

I seem to have forgotten to mention the actual baking i do, i bake it in object space and converting with handplane. I have found that matching uv's and baking in object space gives a very good normal map, incredibly clean results, as far as i can see it makes the perfect normal-map, but then again, im no pro.
I'm feeling i might be on to something with this, am I?
I dunno, im skipping this for now and going back to cageing til i can get some time to study this more.

EarthQuake

Yeah, and sorry if I'm being a buzzkill here, but there are plenty of valid reasons why the industry standard methods for baking are exactly that: industry standard.

However, I'm actually fairly curious to see if you can get this to work. For certain types of assets it may be incredibly fast. My general hunch towards this workflow is that it will be stupid awesome, or just stupid stupid. So I wouldn't tell you to give up on it, I was just playing devils advocate here trying to make sure the issues you want to fix are really issues in the first place, if you know what I mean.

And yes, baking OS and converting to handplane, especially for engines that do not have any other proper synced workflow, is a very good thing to do. Alex and Luke(the programmer behind it) have made an excellent tool.

Theres also the maya rounder corner shader trick which you may be interested in doing, which some people have had success baking, more on that here: http://wiki.polycount.com/NormalMap/#mental_ray_Round_Corners_Bump

fatihG_

This method definitely works for simple meshes or meshes with a defined topology that needs to be sculpted, like various heads from the same base mesh with already laid out UV's and what not. Though if the starting topology wont be the same as the final topology, this method can be used.

I have used this method in the past, but like said by EQ if your base is merely a base mesh than it wont be good enough for the final bake. If the base mesh however is planned out properly, and it is good enough to be used as a game ready asset or it is the "final" mesh, than using this method should be fine imo.