[ZBrush] ~ Normal map gets cut out around creases

Steamy_Steve

Hello everybody!
Newcomer here.

I've just recently started to follow a game design course and we've started to tinker with ZBrush for hiPoly-to-normal-map baking. We're not supposed to try sculpting, yet, as we're still at an early stage with the lessons....but I just couldnt wait!!

Basically, this is the workflow:
.: LowPoly model w/ UVs in Maya
:. Import lowPoly into ZBrush, subdivide and sculpt
.: Bake normal map from hiPoly and export as PNG
:. Splat normal map onto lowPoly in Maya

Well, let's just skip all the problems I've encountered with understanding ZBrush's quirky workflow *sigh*; now I'm having problems with baking the normal maps.
My teacher told me to use the "Crease" function to maintain certain edges during subdivision, so that they wont get eaten away by the smoothing process. And it works, it really does, but now it seems this thing is also compromising my normal map output.

What I get (or rather dont get!) is that all the faces around the creased edges are missing, in the normal map. The quads around the creased edges are, like, completely ignored!
I tried to "UnCrease All", but I get the same result: [ https://ibb.co/mFHTgS ]
See the missing teeth and that flat stripe around the temples? It's soooo frustrating! >.<

I've also tried to use XNormal, but the output is....<unusable>, to use a euphemism. I get odd squares in place of the object's details, or I get the right-ish figure but with improbable colors (and still missing strips of details!).
And the funny thing is that I cant tell which settings result in one outcome or the other!

Personally, unless XNormal produces way better normal maps than ZBrush, I'd rather use the latter to bake them. It would mean one step less in my still-evolving-workflow.
Ideally, I would like to have a steady, standard workflow that works independently of what I'm modeling/sculpting. A man is allowed to dream, after all.

Anyway: crease = no normals. Why?!
I've tried googling for hours, but it seems I'm the first human being on this planet that is facing this issue. My luck!
Thus here I am.

Any help, hint and suggestion is very welcome. Thank you!

PS: this is my third sculpture (and third skull too!) EVER. Please be kind with your comments. ^^'

Eric Chadwick

Use the "question first" posting strategy (start with your Q, then add clarifying text... don't bury your call to action behind a bunch of introductory stuff!). 

And embed your images.
http://polycount.com/discussion/63361/information-about-polycount-new-member-introductions/p1#tricks

Make it absurdly easy for people to give you feedback, and you are more likely to be rewarded with answers.

cryrid

Personally, unless XNormal produces way better normal maps than ZBrush...

It really does. Xnormal gives you far more control over the bakes. It lets you pick the tangent basis used for the normal map, it lets you choose how the vertx normals of the lowpoly are handled (including custom normals ), it lets you choose padding values higher than 16, it lets you use multiple meshes for each resolution (one lowpoly object could use 20+ highpoly  sources for its info), it lets you use different topologies for the two resolutions (which also means you don't have to worry about the subdivision smoothing the UVs or even the base mesh itself ), you can use offset UVs to share the same texture space (whereas even touching the 0:1 borders will cause issues in zbrush), etc. 

Baking in zbrush might seem simpler at a glance, but its the kind of simplicity that comes with many limits - limits that will only complicate your workflow as opposed to the alternatives. Meanwhile dedicated baking solutions like xnormal are the "steady, standard workflow that works independently of what you're modeling/sculpting".  

As for your problems in both xnormal and zbrush, it would be more helpful to show full images. Show the full unsmoothed UVs. Show your bake settings that you use and the "oddish squares" and "improbable colors" that they create. This is the kind of thing where a picture is worth a thousand words, and 1000 > 4. Those issues you're not sharing could very well be easily recognizable and telltale symptoms of what the real problem is (especially if they occur in the same areas regardless of which program is used), which would lead to discovering the right solution for you.

Personally I'd start by checking for duplicate faces/overlaps. You can do this externally in Maya, or by creating a "New From UV Check" texture map inside Zbrush to see if it is detecting any such overlaps.

Steamy_Steve

Alright.
First of all, thanks to both of you for answering my question so promptly.

I have some updates.
Yesterday night, minutes before falling exhausted onto my bed, I discovered what went south with my model: the "Crease" function added new edges to my model.
What's odd is that the model didnt lose the UV mapping, which already happened several times with much less intrusive operations.

I noticed it only because I toggled the "Line Fill PolyF" on and these two areas with a different colouring popped up.


Seemingly, that's the reason why the normal map shows holes: the topology around those areas has been changed.
I would've expected ZBrush to destroy the UV mapping, after this nasty trick of his, but apparently he's perfectly fine with it. Except, maps no longer bake properly.
Nonsense.

So, now I have to learn how to use the original low-poly skull to replace the one in ZBrush as the base mesh for the hi-poly sculpted version. And without losing the UV mapping, or it'd be utterly useless.
I think I've read somewhere that it's possible, but I couldnt find how. A brief list of instructions would help a lot.
OR, alternatively, how to revert this madness without losing the UV mapping.

For the ZBrush/XNormal quality difference, this is how they look (ZBrush on the left, XNormal on the right):


For XNormal, the "Baking options" are:
.: Edge Padding: 2 (an exaggerated padding was the cause of the initial weird results)
:. Bucket size: 32
.: Closest hit: on
:. Discard back-faces: on
.: Default bucket renderer (OpenRL doesnt seem to make any difference and Optix/CUDA makes XNormal crash)
The LoPo and HiPo obj files have been both exported out of ZBrush, so to assure a 1:1 correspondence. Ironic indeed.
I've also calculated the ray distance with the specific tool, which probably is the reason behind those artifacts in the upper/outer corners of the eye sockets (?).
All the other parameters should be the default ones, for what I can remember.
By the way, is there any way to use the hi-poly model from ZBrush without having to export it as an obj? The resulting file is huge!

For those who might be wondering, yes, I did try asking my teacher (since he's the one who taught me this Maya->ZBrush workflow), but it seems he doesnt like me to explore so much ahead of my fellow students; he dispatched me with a wink and a phrase that sounded something like "dont bother me any further, just wait for the next lessons to see how to make it or find a solution yourself".
Devastating frustration, my newly found companion.

I really hope you can help me solve this problem with the missing areas, I'm not sure I could survive sculpting a fourth skull. It's for the teeth...all those teeth.
And, if it's just not possible, I'd appreciate if you could teach me a different technique to prevent the smoothing process from eating away specific edges.

Thank you.


PS: if you need me to post any other screenshot/parameter, just tell me.

PPS: this is what happens if I try to use Maya's LP and ZBrush's HP in XNormal with the "match UVs" option on:

With "match UVs" off I get some small squares with the same improbable colors. Not worth showing.

cryrid

the "Crease" function added new edges to my model.

Creasing really shouldn't add any new points to the model since it is designed to act more like a dynamic tag that you can remove or change around as needed. However there are other options in the creasing menu that could lead to the actual altercation of the model if they were used instead (like the bevel option with a low width). 

I would've expected ZBrush to destroy the UV mapping, after this nasty trick of his, but apparently he's perfectly fine with it. Except, maps no longer bake properly.

Plenty of programs won't delete the UVs when you change the topology. They don't always lead to clean UVs though, as you've noticed.

So, now I have to learn how to use the original low-poly skull to replace the one in ZBrush as the base mesh for the hi-poly sculpted version. And without losing the UV mapping, or it'd be utterly useless. I think I've read somewhere that it's possible, but I couldnt find how. A brief list of instructions would help a lot.

This is where the workflow of xnormal and other baking apps really comes in handy, since they don't care if the topology matches. You can just load back the original file as the low res and bake away. 

For Zbrush though, if you want to use the topology and UVs of the original/uncreased model then you'll need to project geoemtry of the sculpt back onto it. You'll have two options for this. 

1. First, since the difference doesn't seem major, you could use Tool: Geometry: Freeze Subdivision Levels on the sculpt. This will temporarily remove all subdivision levels from the sculpt and drop it down to its base level. When it is in this state, just use Tool: Import to reload the lowpoly OBJ that you want to use, and Unfreeze. This will automatically try to subdivide the new mesh and project the details back. 
2. If the results aren't pretty, you can do this manually. Just add the new lowpoly model as a subtool, and use Tool: Subtool: Project All. Subdivide it once and Project again. Keep doing this until the lowpoly mesh now has subdivision levels and all the sculpted detail. 

Both methods are covered here: http://docs.pixologic.com/user-guide/3d-modeling/topology/zremesher/transferring-detail/

Xnormal Results:

First up, crank the padding as high as it goes. It may look weird to human eyes, but this dilation only happens in areas where there are no UVs (so it wont be noticable on the mesh) and having this padding can be beneficial when mipmapping kicks in (in other words, not having that padding can be noticable on the mesh instead). 

You can change the background of the normal map to something like 127, 127, 255 so that empty area isn't white. There's no harm in playing with the ray distances if you think that is causing the area around the temple. 

By the way, is there any way to use the hi-poly model from ZBrush without having to export it as an obj? The resulting file is huge!

You can export as FBX, but chances are the file size will still be huge when you're dealing with a model that has millions of points/polygons along with additional data for each vertex. 

You can keep the file size lower by deleting any extra data what you wont need like polygroups, and polypaint (assuming you're not using it). Don't use the Match UVs option in xnormal, don't even worry about UVs on the highpoly sculpt, so delete those too for an even lighter file. 

If you sculpt is finished, you can always make a copy of it and run the Decimation Master plugin to crunch the sculpt down to something that uses a mere fraction of the original points while still retaining all of the surface detail. 

Steamy_Steve

Ok, time to update once more.

I've tried your techniques (@cryrid), the "Project All" one first because it sounded more solid, from a topological point of view.
.: I have imported the low-poly skull as a new subtool
:. Manually aligned it to the sculpted one (apparently, it doesnt work unless the two models are overlapping)
.: Selected the low-poly one and left the other visible
:. Lowered "PA Blur" to 0
.: Given the cage a positive value (one that made it look suitable, or just big enough...?)
:. Set the distance to a higher value
.: Hit "Project All"
:. Subdivided
.: Hit "Project All"
...and so on with subdivision and projection up to the desired level

Alas, the result wasnt good, as many artifacts (vertices gone crazy!) showed up.
I've also tried to subdivide the low-poly model straight to the desired level and then use "Project All", but I only managed to get a bit less artifacts (since they didnt pile up through each subdiv).
This is the result:


The one on the left is the target/resulting model, while the one on the right is the source (please, ignore the silly polypaint, I was bored).
Notice the nose cavity and the eye sockets?
Fact is I have no idea what might've caused them, so I dont know what to change in the procedure.

The "freeze/unfreeze" method gave artifacts too, but of a different nature:


Again, I have no idea what I could/should've done different in order to get the correct result.
What I did is:
.: Cloned the model
:. Froze the clone
.: Replaced to resulting low-poly with the the one exported from Maya
:. Manually aligned the imported model with the original one and scaled it down a bit
.: Unfro....unfree....well, hit the freeze button again
...and thas is the result.

If you could give me a more detailed explanation on both the procedures and on how to pick values for each parameter...I moved around a bit too casually, to be honest.

Thank you.


PS: the default padding value in XNormal was 16, which caused the padding to overlap the UVs to the point you couldnt tell what you were looking at. Not to mention all those yellow and orange tones.

cryrid

The models have to share the same space for projecting (and baking). If they don't then projection is going to be a mess. And when they share the same space you shouldn't have to touch the blur setting, distance value, or cage settings. Those weren't part of the steps/wiki page, and increasing those particular settings can be a good way to give you errors in concave zones like the eyes & nose cavities (you are  essentially telling the projection to look further than it needs to, which can cause it to project onto opposite surfaces like you're seeing).  

The freeze method also requires that the models to be in the same space. It is intended for changes in topology, not in surface positions. 

You should not need to manually align and scale anything, as that would suggest there are more differences between these models than just a few additional unwanted vertices around the creases. If they no longer match in size and volume then you might just have better luck fixing/retopologizing the high poly sculpt. 

Lastly, padding does not overlap UVs. Ever. Even if you crank it to 128. If it does overlap, your UVs are probably more screwed than you realize. Do you mind uploading a .ztl file somewhere that includes both the sculpted skull and the lowpoly UV'd version you want to use? 

Steamy_Steve

@cryrid
Here's a link to the ZPL file: [https://www.dropbox.com/s/8pk6jaixufq6r44/Skull%20HP.ZTL?dl=0 ]

I hope it's what you're looking for, I'm not sure whether I did it the right way.
I've left the two subtools visible and deleted all the unnecessary ones, and then saved the "Document". The ZPL file is a 46MB beef thus I think we're safe assuming at least the hi-poly subtool is in there.

cryrid

Yeah, there's definitely a size difference between the two models, and that won't make automatic projection methods fun (especially with the thinner areas around the eyes/nose).  

There are two fixes that are simple enough, depending on your end goal:

1) If you absolutely need the original topology / UVs, then you'll need to make that model match the high poly's size and shape as close as possible. Automatic projection is going to suck since the two models no longer match up. On the plus side there's not a lot of vertices, so you could load the model into Maya (along with the base level of the sculpt), and just start snapping the vertices to match the sculpt's. You'll need to take some minor liberties around the temples and teeth where the topology no longer matches up, but otherwise you can get the overall skull to match the same size and shape this way (like this). Then you can use the project all method (I had to bump the Projection Distance from 0.02 to 0.07). 

2) If you're not bound to the original UVs and don't think the extra 20 vertices are a big deal, I think it would be quicker just to work with that model instead. You can drop to the lowest subdivision level and export that you re-UV in Maya, or you could do it in Zbrush. For the latter just drop to the lowest level and create new polygroups ("Autogroups with UVs"). This will turn each UV shell into it's own polygroup. You can quickly add the temple polygroups back to the skull's polygroup (I would also do a slight change on how the teeth are grouped in order to split the very backs off into its own group), and then use UV Master to create new UVs based on the polygroups. Then you can bake away either in zbrush or xnormal (I didn't detect any issues with padding). 

Steamy_Steve

@cryrid
Thanks a lot for your very quick answer.

Nevermind.
I guess I'll just have to redo the whole thing starting from the obj exported out of Maya.
This is sooooo frustrating! If only I could know how the heck I got those extra edges!! To prevent any future disaster...

Ideally, I should not need to tamper with the low poly and its UVs at all, once I'm done exporting it from Maya.
My problem with this thing is that it's meant to be a prop for a videogame, so I'd use a fast and steady workflow that, hypothetically, should be something like: Maya LoPo -> ZBrush HiPo -> ZBrush (or XNormal) normal maps -> Maya texturing (and animation?) -> Unreal Engine.
I cant really afford to go through this misery every time ZBrush decides it's time to screw up my topology.
If only this game design course wasnt so slow..... -.-

By the way, is there any way to, like, lock the number of vertices/edges of the LoPo in ZBrush?
Like, you know, you teel ZBrush that's the base mesh and nothing can be added or subtracted....?
That would help a lot! =O

cryrid

This is sooooo frustrating! If only I could know how the heck I got those extra edges!! To prevent any future disaster...

You very likely clicked the wrong button, either completely by accident or through your general experimentation without knowing what everything does. For example, look what coincidentally  happens when I click on Bevel:



I wouldn't say its a disaster. Both options I presented only require 30seconds - 5 minutes to fix and left me with models and maps I could load into an engine to preview.

By the way, is there any way to, like, lock the number of vertices/edges of the LoPo in ZBrush?

You could either subdivide at least once before you start doing anything, or create a new layer if you really want to lock those subdivisions down. Both of these options rely heavily on the existing geometry, so as long as either of these things exist on the tool then zbrush will prevent you from taking any actions that will modify the topology. 

Steamy_Steve

@cryrid
First of all, I'm impressed by the animation. I really am! =0
Thank you for all the work you're putting into helping me.

My mouse is being a bit quirky, these days, so who knows, I might've actually misclicked without noticing.
But I almost exclusively use the graphic tablet, with ZBrush, so.....well, it can still be.

Anyway, thank you for those solutions for locking the base mesh. Especially the "subdivide once first" one, it's that kind of simple solution I like so much.

What about the other one, though? How does the "layers lock" work? Step by step....?

Steamy_Steve

If anybody's interested, I did manage to fix it all....

model