Airborn Studios Tools
godlike master sticky
Hey Guys, we just started creating some simple tools to enhance our workflows, one of these tools is now ready to lift off. Thanks a lot to benjamin for creating the stuff we need!
Backstube NOUN die Backstube | die Backstuben
Bakery NOUN a bakery | bakeries
https://bitbucket.org/benjamin_sauder/airbornstudios-backstube
we have a few artists using it already, it seems to be stable. Right now it supports raydistance settings and cages via Xnormals .SBM data. Soon it will support custom exported cage meshes.
Several Artists use it differently, i always create a cage and export this, as it gives the most predictable results, other have created a couple of test presets, baking out a normalmap at different raydistances, picking the best one and bake that.
Others also don't want to fiddle too much with cages and export a short range cage and a long range cage and bake those and mix and match in in Photoshop. it might be somewhat bound to our needs, but how you use it, is totally up to you. If you feel like baking 25 AO maps with different settings, it will do that for you.
But beware, it is made for mass automation, it bakes everything you give it with a fixed set of settings, so if your settings are set to 2048x2048 at 4x AA, it will bake everything it finds at that ratio.
There is more to come, the biggest chunk of our tools is in the works and might take a while, we will have to talk to our current client if it is okay to release it.
But to get a small glimpse, it goes one step further by assembling all baked maps in a given Preset and create the normalmaps and about 90% of the diffuse work for you if you already textured via polypaint in zBrush.
The plan is to expand that to create gloss and spec maps for you as well.
This is more specific to our workflow and our current clients needs, so yeah it might not be a solution everyone can use.
to get an idea:
nothing here was texturized in Photoshop, this is a near final diffusemap the toolkit generates
okay here is a simplified version of what our own tool does, we are preparing the switch to the new ddo and need to change some parts of the workflow
Packstube
Backstube NOUN die Backstube | die Backstuben
Bakery NOUN a bakery | bakeries
https://bitbucket.org/benjamin_sauder/airbornstudios-backstube
Core-Concept:
The whole application is based on the idea to use xNormal settings files as a blueprint for baking an arbitrary number of highpoly and lowpoly meshes. Or even better an arbitrary number of setting files with any meshes you want. This really speeds up the process, if you find yourself switching models and output in xNormal all the time, but never really touch the settings themselves.
Requirements:
python 3.3+
xNormal 3.0+
Constraints:
saves only .tga images
looks only for .obj or .sbm modeldata
Quick step by step guide:
Start the application, and supply it with a valid xNormal install directory
Setup xNormal with one highpoly and one lowpoly mesh, define the maps and all other stuff
Save these settings somewhere, you can setup multiple settings files too if you need some different things.
Close xNormal
First, export all your highpoly models with a prefix "hi_" or "high_"
Second, export all your lowpoly models with a prefix "lo_" or "low_"
To avoid intersections you can use a simple folder structure, these get baked one after another.
Example structure:
* lo_gun.sbm
* hi_gun.obj
* hi_trigger.obj
* hi_scope.obj
* Magazine - lo_mag.sbm
* Magazine - hi_mag.obj
* Magazine - hi_ammo.obj
Doubleclick on Backstube.pyw ( if nothing happes try, assocation it with the pythonw.exe in the python install directory) If it still doesnt run, type cmd in the explorer and start the Launcher.pyw from the commandshell, and look what it says - and contact me
Copy&Paste path to modelsfolder into the "Model Directory" entryfield
Copy&Paste path to settings files into the "Settings Directory" entryfield
(optional) setup a destination folder for your baked maps if you do not specify this, everything gets saved into the model directory
Hit "Bake Now" and wait...
Have a lookt at the TestData folder - and try to bake that one first.
The tool generates "xN_'your_settings_filename'.xml" files in each of your modelfolders, these get rewritten every time you click on bake now!
we have a few artists using it already, it seems to be stable. Right now it supports raydistance settings and cages via Xnormals .SBM data. Soon it will support custom exported cage meshes.
Several Artists use it differently, i always create a cage and export this, as it gives the most predictable results, other have created a couple of test presets, baking out a normalmap at different raydistances, picking the best one and bake that.
Others also don't want to fiddle too much with cages and export a short range cage and a long range cage and bake those and mix and match in in Photoshop. it might be somewhat bound to our needs, but how you use it, is totally up to you. If you feel like baking 25 AO maps with different settings, it will do that for you.
But beware, it is made for mass automation, it bakes everything you give it with a fixed set of settings, so if your settings are set to 2048x2048 at 4x AA, it will bake everything it finds at that ratio.
There is more to come, the biggest chunk of our tools is in the works and might take a while, we will have to talk to our current client if it is okay to release it.
But to get a small glimpse, it goes one step further by assembling all baked maps in a given Preset and create the normalmaps and about 90% of the diffuse work for you if you already textured via polypaint in zBrush.
The plan is to expand that to create gloss and spec maps for you as well.
This is more specific to our workflow and our current clients needs, so yeah it might not be a solution everyone can use.
to get an idea:
nothing here was texturized in Photoshop, this is a near final diffusemap the toolkit generates
okay here is a simplified version of what our own tool does, we are preparing the switch to the new ddo and need to change some parts of the workflow
Packstube
from http://www.polycount.com/forum/showthread.php?t=133325
okay how about now? but it is unpolished and not tested much, i would like to add a bit of graphics to the input window down the road.
put it into presets/scripts/airbornstudios there might be more coming
how it works is simple:
Lets assume you baked everything with xnormal (or with backstube to not have to touch xnormal), your files will always have the same suffixes like _occlusion _vcols _normals and whatnot.
It will collect your files and pack those files based on suffixes into layergroups.
You also have the option to turn on padding, however, you will need xnormals dilation filter to be able to use that. We will add a routine that asks if that filter is installed, but right now you have to take care yourself.
Note, always bake without padding and let dilation do the job, because then you will never ever have to touch up overlaps based on padding you baked. padding is a postprocess in Xnormal as well, so stay non destructive and keep it a posteffect in your PSDs as well.<<<<< CLICK !
Replies
I was kinda rusty with python, so forgive any unconventional solutions.
So, I just pushed another small update:
- its now possible to have cage files next to the lowpolies - these get automatically asigned as the "external cage model" for each lowpoly. xnormal is VERY picky about vertexorder - so do your homework there..
- you can now either use a single settings.xml file or a directory (containing a one or many settings.xml) as input for the settingsdirectory
its all pretty much untested.. so we'll see if it all works
double derp, tried associating the file with python and I don't know what I'm doing on the command prompt window sooooo I'm just gonna go back to playing with my legos in the corner now -.-
don't use the command line tool, you don't have to, you can. But i'm an artist i hate commandline, thats what we have the Gui for
sounds like you didnt install python correctly, or it messed up setting the path variables, anyway try this: http://swaroopch.com/notes/python_en-installation/
Simple enough fix in order to keep from breaking other things was just to change the extension of the file and have window run that specific file with python 3.3
Got it all set up and running some tests now
Knowledge is power
but...
It keeps naming all of the maps "settings_" for some reason, Is that something I've done some where in xnormal or a bug?
*edit* nvm, It names the output files what ever the xnormal settings are.
When it comes to baking vertex colors you still have to save out separate settings for that pass which is no big deal.
It would be awesome when it sees "Bake Highpoly's vertex colors", it stops after all of the other maps are finished, disables the "ignore vertex color" on the high definition tab; Then continues rendering that pass.
Otherwise you have to re-open Xnormal and save a second settings parameter just for all of the diffuse bakes.
main reason for that is the multi seetings bake stuff - so i have to differentiate the bakes from each setting, otherwise they would overwrite each other.
where the main reason for this multi setting stuff, is the vertex colors problem - as you described yourself.
yes one could think of a different convention - to get stuff rolling we didnt want to do any fancy stuff. its not meant as a gui replacement of xnormal. which is something to think about, as I now know how it all works.. but well - too much work right now.
But one thing i did for my own tool, is for vertexcolor or OSnormals, it'll generate extra xml's for you with the proper settings so they wont overwrite existing bakes and the like - itll just be another XML to bake
we have hundreds of characters to do right now, all with the same specs, so the idea is to bake a lot at once
also xnormal just doesn't support that setting, so the intial idea was to get rid of that on and off ticking of the vertexcolorsetting all the time. Basically just give it all the settings you want and it will run through them. Yes you will have to set them up, but in most cases only once for a project
chris: you can do this
I am still a bit confused about what it does - the core concept seems a little hard to grasp.
So I guess I'll simply ask some questions about a specific scenario.
Let's say I already have personal settings ready that I always use, for all the Xnormal options : ray disance settings for the low poly meshes (the thing that says 0.5 by default), precise AO settings, convexity settings, the need to bake both Tangent space and Object space normals (which setting is exclusive in the xNormal UI), and the need to bake Zbrush polypainting data (amongst other xNormal things that I might forget). I would also need to bake selection masks based on random colors applied to the different components of the highpoly meshes.
Let's also imagine that I have a whole bunch of meshes ready - all low poly meshes and their corresponding highpolys, some with projection cages and some without. nameofthemesh_low.obj, nameofthemesh_high.obj, nameofthemesh_cage.obj.
My quesitons would be :
Can your tool bake all these maps, for all these objects, at the click of a button ?
Is there some kind of integration with the xNormal viewer, allowing me to see all the baked lowpoly meshes with at least their normals and local colors applied ? And in the case of no polypainting being baked, maybe showing the AO as diffuse, or the random colors from the highpoly region masks ?
Lastly, what can I do if I realize that one object of the whole batch got screwed up because of an error on my end (bad UVs, incorrect placement relative to the high) ?
Thanks !
i just rendered 8 maps from ~ 30 separate meshes without exploding and the folder grouping method is perfect for that.
highbred3d: what i don't want to end with, is a cluttered xnormal like interface with tons of options. But we could look into just reading that format out of your setting files, lets see maybe beni can find a good way.
The problem is, where do those options start and where do they end? We can't predict what people use it for, all we want is mass automation for always the same settings within one production. We don't want to rule out xnormal as your setup tool, as we also use it to bake
pior:
1. short and simple, Yes.
Longer: Just set up a few settings files. I'd say as vcol rule out AO, put AO, cavity, TS normals and base texture (for objectID masks) into one setting file. And OS normals with Vcols in another. Put both (or make 20 settings if you feel like baking 20 different AO solutions) into one folder, thats your settings folder.
The tool will bake as many settings at once as you give it, it doesn't care what you use it for.
2. No, simply because i dislike the xnormal viewer, so i have no clue how to really use it, if you can load those maps in there you can surely preview your stuff this way.
The second part of our tools, which is very far from beeing in a releasable state covers this for us, by assembling all maps automatically in Photoshop. So we can quickly iterate and look at the whole texture in 3dsmax on the full model in just a few moments. But this is highly specific to our current pipeline so i'm not even sure it would make sense to release it to the public.
3. we have artists using test bake settings, like 4 different lower res normalmap bakes with different ray distances,they bake out quick, so bugs are visible fast. Just load up a testbakes folder.
Mashru: thanks a lot, i'm glad if we can help!
I like the idea of a viewer, but like neox said, we have an assembly script ready which makes it possible to view the finished result very quickly. But i might investigate it a bit more - as I think it could be interessting to create..
"Lastly, what can I do if I realize that one object of the whole batch got screwed up because of an error on my end (bad UVs, incorrect placement relative to the high) ?"
The tool saves all generated settings files next to each part, so you could reexport your meshes and just rebake that settingsfile in xnormal directly
highbred3d: you can easily switch that yourself, look into src/xnormal.py line 14 -> bakeImageFormat = ".tga" in theory, you should be able to insert any filefomart xnormal supports
i GUESS this would be possible to fetch from the settings.xml, but you are the coder ^^
I would like to thank you alot for this as it saved me alot of tedious work. Not only could I bake all the maps with one click but also since I supplied it with models having same UV sheet, so it produced combined vertex color, cavity and normal maps in one go which otherwise i would have been combining in photoshop.
thanks alot again
It might be safer to have it as a zip download, with the tool and a simple .txt in it for instructions - and maybe also a simple step by step guide on how to use it, with a simple example. This could be a big vertical image showing all the steps of the process.
I'm impressed you can write such cool tools with that language. I need to pick it up ASAP. Got a few "tech art" ideas of my own.
Thanks for the inspiration! The tool looks great!
@Deadly Nightshade I think youre vastly undervalueing python, its the perfect choice for stuff like this..
haha yeah it's not easy to find, but all this is in the zip
thank you very much
I decided to give a try on this tool. It looks amazing.
I think im missing something here, i click bake, it loads xnormal and start to load the models, but it dont create the maps and after all the load it appears that the bake was aborted
Any clues on what im doing wrong?
i started testing with fewer models till it breaks. In the end i was trying with couple models, some with cage files and some without. That is when the problem occured
since i checked the "use cage file" on xnormal it was probably trying to find the cage and returning with an error.
i made a cage for each file and its working perfectlly now. and i need to say, IT IS AMAZING! =]
i wasnt expecting it to return just one map with all the models baked, that is really great!
next item i submit to the workshop i will send an email to Valve asking to add you guys to the 5% share
thanks!
Downloaded Python 3.3.2 and installed to "C: \Python33\"
Extracted "benjamin_sauder-airbornstudios-backstube-92f3afa3af22.zip" to "C: \Program Files\benjamin_sauder-airbornstudios-backstube-92f3afa3af22\".
Set my Environment Variables as follows
Before I added the PYTHONPATH variable, Python would just crash at startup with the error
What happens now is that when I doubleclick 'python' in the "\Python33\" directory. I get this:
When I doubleclick 'pythonw' in the "\Python33\" directory, nothing happens.
When I doubleclick "Backstube.pyw" nothing happens
Is there a way to have Backstube take the settings file from the model directory instead of a separate directory? That way I could just dump a setting file into the diffuse/specular/normal directory root and have only vcols rendered for the first two and normals for the last.
Oh and btw, it seems it doesn't work if the files are named simply cage/high/low.obj, but they do need more than that. Is that on purpose?
Either way it should prove a great time and sanity saver. Honestly I always wonder how people organize all their objects and whatnot as to not go crazy while ticking checkboxes and renaming exported and imported meshes and such. With this tool at least I can automate a big part of the work once I have set up the proper structure. Thanks a lot again!
gLitterbug: im not sure what youre descriping in the first paragraph..
having the settingsfile in the same folder, yeah that should be an easy change, ill have a look at it. only convention would be that you dont prefix them like the generated settings.
I guess we missed the trivial case of having just one low and one highpoly - so for now name it "low_foo.obj". But I'll look into this, as Im not sure why its not even working..
I thought by assembling, Neox meant that the baked maps will be merged together, like you can do with the Multi Map Exporter script in Zbrush.
For now only objects in the same folder go onto the same map, since they are visible together at rendertime, right?
However if some intersect, you have to put those into subfolders to handle visibility, but that means each folder gets a separate texture which you then have to merge together by hand. In my case for example, that means I have several subfolders and just one high and low in each and I end up with 8 maps per type which I have to add together manually to get the finished texture sheet.
This I thought means that the tool will be able to bake out polypaint data from more than one high poly model, while using the same lowpoly. In my case I have one high poly for diffuse and one for specular. So if I want to get the vcols baked out for both, I have to duplicate my folder structure. One with the diffuse polypaint high and one with the specular. Since only the polypaint is different on those, I get the same normalmaps rendered twice unless I change settings files in between and render vcols for both and then normals for only one of the folders (either diffuse/specular).
Sorry for all the text, but I hope that explains it better now. I really hope I'm not being annoying, as that is the last thing I want to be if someone releases some sweet tools for free. Maybe I have the weirdest way of working or am doing it all wrong. If that's the case I'd appreciate any guidance. I'd be really interested how you manage models and the different (intersecting) parts when it comes to baking.
Oh and I only mentioned the trivial name thing in case someone was wondering why bakes get aborted and doesn't think of that. Sometimes when things go wrong, the most obvious stuff costs the biggest amount of nerves.
Thanks a lot again for releasing this awesome tool.
For example It collects all maps with a specific (ie. _normals) name and puts them into the according folder in photoshop, creates a duplicate and applies edge padding. this way you don't have to erase overlaps due to wrong padding. We bake out all maps without padding and let photoshop do the rest. Unless your uvs overlap it will generate perfect padding for you within seconds.
This merging of individual textred doesn't have to be done by hand, it's monkey work and should not be done by hand.
the idea is, to later one use masks to generate gloss and specvalues from a predefined file, so you run a sript and all values get applied to the masks. It will need a bit of handwork but it will also make sure that values throughout a production are always the same.
Guess this is what happens when you are working on your own stuff mostly isolated like I am; you don't really know how those basic but time consuming monkey things are done quicker unless you specifically try to find info on how to automate it or how it's supposed to be done.
Thanks again for the tool and for clearing up my confusion.
* now possible to use stuff like "cage.obj" "lo.obj" "high.sbm"
* generated settings xml are not valid as input settings anymore (ie all "xN_".xml files are ignored)
" 'list' object has no attribute 'clear' "
I thought it might just be an error running through some of the meshes I was baking (like cages not being correct) but I had the same error using the default test project.
Hope this is useful, and maybe someone can tell me what I'm doing wrong
In production most issues we had where just due to python not beeing installed properly.
Or virus scanners deciding Python is a virus, deleting the executable without any notice.
Makes it a lot easier to package up and distribute.
I've written some tools at work using it, seems pretty neat (fwiw, I've tried py2exe and cxfreeze... pyinstaller was the only one I was happy with and worked well).
you can now define the tangentspace you want to bake in, in case you want to use Farfarers great Unity Tangentbase Plugin for instance
and you can now define per lowpoly if you want to use an alternate texturesize you can add that as a suffix behind your lowpoly name.
so if your lowpoly was named
lo_xxx.sbm
you can now use
lo_xxx@512x1024.sbm
and it will bake at this resolution
so far i always just baked at 2k and scaled it after baking but now it's much easier to set up if you need something that doesn't work within the norm.
also is there a way to add custom file format for the lowpoly ?
for example, can we bake using cgf/chr (crytek) formats ?
there are are xnormal plugings that allows to load such files in xnormal but this backstube doesnt not seem to load it.
also, is there a place we can donate to for the development of this tool ?
1. What do you mean by merging? as in open in ps and compose them? we have a tool for that, it combines the non padded layers and adds padding as a seperate layer underneath. i am really not sure if we should release this, it is very very limited to our workflow, uses a template psd to put all baked layers into, and was never meant for customization
2. whatever xnormal can read should be readable, i'll answer in more detail next
3. about the cgf, i will have to test, basically it should be able to read and write whatever xnormal can read or write, it might be that it needs to be added to one of the python setting files, after all we just use xnormal without having to touch it, besides for the setup per project.
in the xnormal.py there are supported formats and bake formats, but i didn't test it yet.
the problem with xnormal is, that it doesn't write all settings into the xml, some are in the registry. It really is a horrible habit if you ask me, to have settings here and there.
4. nope but i can ask beni, i only add ideas and from time to time he works on it, but i think this is a good idea.
when i tried cgf, it gave me error that no mesh was found. i did save out a new settings file with cgf loaded inside xnormal but it didnt work.