Please Help with Photoshop: Editing Bakes

BradMyers82

Hey there, I was just wondering if there is a good way to solve this issue...

I baked a Normal Map and AO Map as usual.
For the most part the bake is great, but there are a few wavy lines, and stretched areas I want to clean up with the smudge and liquify tools.

Now I can do one (the normals), then the other (the AO); (put one layer on 40 percent opacity and line things up) but it's not such an exact way to do things.
Also, it's hard to get things to match up and I think it would be much better if I could automate the process.

For example, if I could record the actions of liquify tool on the normals, then just re-apply it to the AO, that would be perfect.

I know I could also adjust the cage, and re-bake the specific area, but this becomes really time consuming and tedious, depending on what you are baking.

I'm just curious how other people do it, and if there is a better way, like mentioned above.

Thanks in advance!

Slum

Liquify already supports this (at least in CS4 which is what I am using). In the liquify panel, hit "save mesh". Then, on your next layer do filter->liquify->load mesh

Mark Dygert

Depending on the maps, and the unwrap I normally only get waves on the ends of cylinders. To get rid of those particular waves I copy a slice of pixels and drag it out over the waves.

Or do what Slum suggested, ha that's great never thought about using the save/load on separate layers. I always thought that was going to be specific to that liquefy.

BradMyers82

Whoa, cool man, I am using cs3 extended and it DOES have this feature.
I thought it had something to do with using the new 3d features in the extended version.
But... I tried it out, and it works great!
Thanks a ton Slum!

While I am on the Photoshop subject, anyone know if there is a way to get more handles on the warp tool. Like in 3ds max, there is the FFD Box, where you can specify the dimensions. It would be really useful to have a few extra handles at times.

MoP

Just out of interest, do you find that using Liquify, or similar methods, is faster than editing the cage?

Especially since if you're doing something like baking out vertexcolour/material colour as a guide for texturing, plus AO, plus normals, that's 3 maps you now have to edit and keep consistent if you do it the Photoshop way, while if you just edit the cage you can fix everything perfectly in one go without having to worry about stuff not lining up... just a thought.

BradMyers82

Opps, looks like you posted while I was Vig. Thanks for the tips!
Yeah, in this case I have a bunch of buildings I am baking, with like a thousand little windows and stuff. Only a fewof them are warped which is why I asked the question earlier.

BradMyers82

heh, MOP, you anwsered while I was posting this last response. I can't keep up with you guys.
The bake has a lot of detailed high poly stuff, and just to render it takes a while (over 15 minutes). where the edits on the maps as slum suggested would take a couple of minutes. The edits I want to make are very minor.

Slum

As for the warp resolution, there's no way I know of. Unless you want to load the image in After Effects :P

You can just select a smaller portion at a time, which kind of helps. Still not a great solution though.

BradMyers82

Yeah, I was thinking I could apply the texture to a plane in 3ds max and apply a ffd box; then render to texture it at a higher resolution than the texture itself.
But this is sort of a lot of work, and you might loose some texture quality.

[Edit] Its good to know I am not missing out on anything there though, thank again Slum.

MoP

Didn't you do any test bakes at lower resolution / quality for a fast check of how the bake is turning out?

Usually I only do the final, highest-quality bakes with everything turned up to the max once I'm sure that there aren't any ray misses or dodgy warped areas.

A 1/4 resolution bake with no antialiasing (and no advanced lighting - all you need to check this is the normalmap, turn off any lights which might slow down the render!) is all you really need to see what the issues are, and even with lots of highpoly objects a bake like that shouldn't take more than half a minute or so.

BradMyers82

MoP: Yeah, I should have done it that way. But what I did is do a test bake, then just look at it in Photoshop, thought everything was good, then proceeded to final bake. I didn't bother opening up another 3ds max scene and test it on the actual model first. I'll keep this workflow in mind next time for sure.
The other thing is, there are times when I adjust the cage, and it messes another area up, or doesn't quite make things perfect, and then there is a lot of back and fourth. Which could waste a lot of time. I think it's just always good to have different options available, and I'm sure the trick Slum pointed me to will be extremely useful. I really appreciate the tips though

MoP

I'm curious as to why you would bake something and have to "open up another 3ds max scene" to test your normals on the model. Surely you're baking to a lowpoly model, which you can apply the texture to and check it immediately?

And yeah, the Liquify trick is pretty handy.

BradMyers82

MOP: Well, the scene chugs a little with the high poly in it, and I want to make a proper lighting setup that is optimized for viewing once I'm done baking. So I just got in the habit of doing it this way.
But honestly, there is no real reason I can't check the normals right in the same scene to see that everything is correct. At least for a quick and dirty look. I'll keep that in mind, because I do tend to get in certain habits that are not necessarily the quickest way to go about. Thanks again MoP!

MM

MoP wrote: »

Just out of interest, do you find that using Liquify, or similar methods, is faster than editing the cage?

Especially since if you're doing something like baking out vertexcolour/material colour as a guide for texturing, plus AO, plus normals, that's 3 maps you now have to edit and keep consistent if you do it the Photoshop way, while if you just edit the cage you can fix everything perfectly in one go without having to worry about stuff not lining up... just a thought.

there is a "save mesh" and "load mesh" option in lequify so you dont need to individually liquify everything separately. you can just liquify the normal map, save that mesh and load that mesh for AO and it would line up exactly as long as your selecting withing the same boundary.

for quick fixes i rely on lequify most of the time.

MoP

MM: Heh, that's what Slum said in the 2nd post on this thread, but thanks for the info again!

Mark Dygert

MM, slum already posted that info, but its good to have it well documented I guess.

(...see what I did there...)

MM

lol didnt read carefully. seeing MoP's post i thought no one mentioned it.

BradMyers82

Perna: Because by posting this thread I learned something new, that I think is extremely useful and will definitely use
I agree, MoP is correct and a great help. He answers like 90% of my technical questions on Polycount, lol.

[Edit] Oh and look, MoP is going to make 10,000 posts soon! Is that a polycount record or something?

MoP

Yeah I'm a right spammer

(9 to go)

Muzzoid

:poly142: you need a 10'000 post party with strippers!