Environment Art Beginner Questions

Nintendo46 · May 2014

Hey there guys.

My name is Joel, im a Hobby 3D-Artist from Switzerland. I practiced 3D Art as a Hobby, creating some architectural Visualisations, but ive always been interested in videogames so i decided to create some environments which i later could put into the new unreal engine. However im not getting anywhere...

Whenever i try to bake normal maps i end up with very weird results. like in the attachments.

Its not like i never read any tutorials on baking, its just doesnt work, no matter what obect i try. I started with a simple Cylinder with some creases, then moved to a sphere but nothing worked.

I hope i have to ask here only once so i can start building what i had in mind.

I would be really thankful for any advices on how i can make this happen, so i can actually start building and show off my stuff...

Thanks in Advance and have an nice day

Nintendo46 · May 2014

Oh, and here is the scene in fbx, if you dare to take a look

http://bayfiles.net/file/1f50x/HVE681/weirdScene.fbx

EpicBeardMan · May 2014

If you are baking normal maps and AO maps from high-poly models, you should make a cage derived from low-poly model and your maps should be alright then.

Nintendo46 · May 2014

Hey there EpiBeardMan

Thanks for your answer. Bit with nonflat geometry i just use my low poly as a cage object? I made the low poly slightly bigger than the highpoly. Is is right to do so?

Thanks an advance

EpicBeardMan · May 2014

Duplicate your low-poly object but first you should triangulate it if you are going to import it in some game engine but if not then there is no need for it, but you need to duplicate your low-poly object and that duplicated object scale just a bit so it covers your whole low-poly object, and dont scale your low poly for it wont be compatible with the highpoly

reverendK · May 2014

Ideally your layers should look like this:

| | |
Lowpoly - highpoly - cage

You're casting rays from the lowpoly surface to the highpoly surface. when the ray gets a hit it will render that hit. The cage is just there to help you control the length and direction of those rays.
generally you'll duplicate your lowpoly and inflate it so that it acts as a baloon around the highpoly geo. (in most programs it has to be a 1 to 1 comparison of the lowpoly though. if you modify the geo of one or the other the Vert order changes and it doesn't know where to cast the rays)

also make sure your UV shells are broken where you have a sharp plane change in the lowpoly geo - generally trying to render softer normals on a 90 degree edge is going to cause problems.

And I don't use maya much - haven't baked normals in it in a long time, but keep in mind that any hard edges on the lowpoly geo will show up in your highpoly bake.

Nintendo46 · May 2014

alright, thanks EpicBeardMan and reverendK for your answers.

i will take in account what you said and implement it. I will rework my uv layout which currently looks like this:

aAoCcaAFH

So if i understand you right the lowpoly has to be smaller than the highpoly. And the cage should be bigger than the highpoly right? I bake with xNormal, which says the cage file has to be identical with the lowpoly. Does it mean also in size?

and when i bake out the normal in maya itself, the normal map looks much less messier. like this :

aaOcKaafh

But anyway, i will rework the thing and then post my update here. Thanks both of you
for your help.

Nintendo46 · May 2014

and am i right that the image upload isnt working?

i tried both bayimg and dropbox and none of them seems to be working..

ClusterOne · May 2014

its not loading for me, so I guess its not working

edit: did you try imgur?

Nintendo46 · May 2014

well here are the links to the uvs and the maya normals:

mayaNormals :https://www.dropbox.com/s/ssw8ygatit41quh/mayaNormals.jpg

uv layout :https://www.dropbox.com/s/8z8njl9nqytjt5d/uvs.jpg

CDeniz · May 2014

Not working for me. I took a look at your scene, you need to set the normal map as tangent space normals in the bump attributes.

Nintendo46 · May 2014

Hey clusterOne

yes i tried imgur as well, but it doesnt work.

Seems we have to wait a litte

but thanks for the tip

Nintendo46 · May 2014

Hey CDeniz

i was doing that but its still not working as it should...

what do you think of my low poly??? is it modeled right? im not very expirienced so i also thougt that the model itself causes the problems, since i used a lots of bevels.

Do you think a model like that could "go through"??

thanks for your comment

CDeniz · May 2014

Did a quick bake in xnormal:

Looks as it should.

http://bayfiles.net/file/1f5tY/NcX7SL/box_maps.rar

EpicBeardMan · May 2014

Nintendo46: To your question before. Lowpoly should be left the same it was before making the highpoly, but some forms of the mesh you could change to look more like the highpoly. Do not ovescale it. And for the cage, the cage should be slightly bit bigger than the LOWpoly, it should have the same geometry as the lowpoly so you should finish the modeling on the lowpoly, duplicate it and scale it a little bit. ( In Blender there is a tool called fatten and I use that to scale it just a little bit while it saves the original form).

Nintendo46 · May 2014

@CDeniz

...how were you able to get other results than i do

lol. Did you use a cage mesh? did you change any settings inside xnormal? what file format did you use for exporting to xnormal? or better ask, what exactly did you do??

and are those litte black "noises" on the edge of the box normal? because you seem to have them too.

@EpicBeardMan

Thank you for your advice. I finally understand what you mean. i will bear it in mind for future modeling and baking excercises. Thank you very much.

CDeniz · May 2014

I exported the low and high poly as obj's. In xnormal open the 3dviewer and click "show cage", then scale it up (the cage) so that the high poly is not touching the cage.

Those black edges are caused because you don't have enough space between the the uv's. Im sure there are tutorials in the poly count wiki, but briefly, if you have hard edges you need to split those uv edges and give them enough space. Then when you bake your normal you need to set the edge padding high enough (I think I use 16).

EpicBeardMan · May 2014

You welcome, always happy to help.

Nintendo46 · May 2014

@ CDeniz thanks for the explanation. I will change the uv-layout accordingly, as soon as i read through the wiki. i will then post another box, hopefully with a clean bake.

Anyways, thanks for your help, i will post as soon as i got it done

cheers Joel

reverendK · May 2014

Regarding the cage being identical to the lowpoly:

It's not the size or even the shape of the two that needs to be identical.
Every vert is listed in a specific order for any given program to read - adding or removing a vert, connecting two together or removing an existing edge will re-order those verts. If the vert order of the Lowpoly and cage meshes aren't identical you can't line up the raycasts (it's drawing from vert #n on the low to vert #n on the cage).
So if you do anything but MOVE verts in space on your lowpoly you're probably gonna change the order in which the verts are listed - and you'll have to make a new cage.

Spoon · May 2014

reverendK wrote: »

connecting two together

Are you certain this is the case? Cant see why the vertex numbering would change because of this.

Nintendo46 · May 2014

Hello there guys, im back but yes..

the bakes still return ugly as you can see here
http://bayimg.com/oAOfgAAFH

i remodeled my lowpoly completly and created a new uv layout which currently looks like this
https://www.dropbox.com/s/8z8njl9nqytjt5d/uvs.jpg

i created it accordingly to the wiki-thread "hard edges and normal maps".

I also wasnt able to scale the cage in xnormal. everytime i scale it up it gets distorted in different directions, so it doesnt look like the original anymore.

https://www.dropbox.com/s/nn1z64v2c61t0ev/xNormal_01.jpg

Now im not really shure, is it the model, the uv-layout or my lack of handling inside xNormal?

because no matter what i try, the normals have errors and look crappy.

i reuploaded my scene if anyone could take a look at.
https://www.dropbox.com/s/q84jjlp19qvot86/box_scene.fbx

im at the end of my latin, so should i try a more simpler object, maybe a paint bucket, consisting only of a cylinder with some creases??

thanks in advance, Joel

CDeniz · May 2014

Now your low poly and high poly dont match, the low poly is quite a bit larger than the high poly, the rays will never touch the highpoly surface.

I took your old scene to give you a demonstration of scaling the cage beyond the high poly in xnormal:

reverendK · May 2014

Spoon wrote: »

Are you certain this is the case? Cant see why the vertex numbering would change because of this.

I couldn't tell you why it does, and maybe it doesn't always, but it's definitely messed up my cages before.

JEmerson · May 2014

I'm going to piggy back off of this and ask a follow-up question. Is it ever appropriate to allow the low-poly object to have a larger volume then the high poly when baking normals map? I am thinking for when you want the appearance of recessed edges/areas or are working with more organic like objects (for instance a helmet) . I.E. #2 Low Poly line here instead of Low Poly #1 http://i.imgur.com/RJSNnWz.jpg

Also to further clarify, is this a correct visual representation of the optimal normal map baking set-up? http://i.imgur.com/ba3kowh.jpg

Shogun3d · May 2014

Took your fbx and quickly adjusted your high res mesh and your low res uv's, adjusting smoothing groups afterwards. What I did notice is that your high res mesh doesn't have smooth edges to really exaggerate the effects of baking down to a normal map. Thus resulting in little to no effect on the normal map itself, remember based on resolution the effect is still pixel based so this is why often high res meshes are exaggerated. You also had a large amount of empty space.

I just did this quick and baked once, often you'll want to bake, adjust, bake again to get a proper bake from the normal map. You want to avoid painting over your normal map as much as possible. A proper normal map is baked from the result of a properly uv'd mesh with smoothing groups and a cage that properly supports ray cast from high res mesh.

Hope this helps.

https://dl.dropboxusercontent.com/u/505968/box_scene.zip

Nintendo46 · May 2014

@ Shogun3d

Thank you so much, i will check the scene out as soon as i have time and try to learn as much as possible from your work.

Basically what i did to create the highpoly was to create the box, then add divisions and then smooth it like 2 times. I guess thats a real wrong way of creating a highpoly, am i right?

Also in maya there are no smoothing groups there are just hard and soft edges. Could you tell me where you put the hard and soft ones?

regarding the cage, you set that up in xNormal? just extruding the lowpoly a bit like it was suggested by other users? Isnt it safer to create the cage in the 3d application for maximum control?

huhh, lots of questions, but thanks for your help. I try to replicate your work and post an update here soon. Thank you so much

With best regards, Joel

Shogun3d · Jun 2014

Nintendo46 wrote: »

@ Shogun3d
Also in maya there are no smoothing groups there are just hard and soft edges. Could you tell me where you put the hard and soft ones?

Its basically the same thing. Just put hard edges where your uv seams are.

Nintendo46 · Jun 2014

hey guys im back

@Shogun3d

i was able to recreate your normal map from my box scene and it looked like yours:

https://www.dropbox.com/s/vulwjfvxcliqbcq/box_normals_new.jpg

thats a huge step for me. Didnt thought i could go that far

but i still have a lot of questions to you Shogun3d and that would be:

what where you thinking when changing my uv layout from the box:

https://www.dropbox.com/s/ykl91mcf0dc6unb/UV_Change.jpg

i would like to know your process of how to set up uvs for my low polys and how i could transfer that knowledge to future objects. Could you explain that to me?

i took a look at your highpoly and if differed indeed very much regarding the one i created...
i dont have a lot of highpoly modeling skills right now, but im watching tutorials and try to learn from them.

alright and then i created a new prop, a pallet.

heres a screenof the low poly:

https://www.dropbox.com/s/smalb6q2zy40ssw/palette_LOW.jpg

and heres the highpoly:

https://www.dropbox.com/s/s15u6pg8r0j4x5t/palette_High.jpg

i assume the highpoly is still modeled badly, am i right? i tried my best...

at least the normals look not so messy like the first time but i still have some errors in it, probably due my bad uv layout and bad high poly? nor shure if the low poly is modeled right either...

https://www.dropbox.com/s/o21f08hp2rt52tu/palette_normals.jpg

@Shogun3d could you take a look at my scene again and tell my if my uv layout is correct and what you think about the lowpoly and highpoly?

for example the uv layout. is it correct this time?
if not why is that? i tried to make the cuts at the most extreme angles for better lighting but im not completly shure. i think you know how to approach an object and i dare to know how:) could you tell me your approach? that would really mean a lot to me.

and thanks to everyone else who has been posting here for your help.

Thanks in advance

best regards, Joel

Nintendo46 · Jun 2014

and i still cant post images, i dont know why. Is there a scale limit? or do the images have to be at an exact scale? should i try an other provider? sorry for the question

Shogun3d · Jun 2014

Set your UV's up the way you would see fit to paint them and allow proper texellation. Think about the angle of the object, and what the players would see the most. For example, I scaled down the UV for bottom of the box, to allow more density resolution on the rest of the box since most cases you will see only the top of the box and sides.

This of course is different if the object is dynamic and can be seen on all sides, but I would still take similair approach.

I also can't see your images. The links aren't working.

Try uploading to IMGUR and using the IMG tags.

ScottHoneycutt · Jun 2014

You need to get this image uploading issue resolved immediately. We can't help you unless we can quickly and easily see your images. Below is an image I uploaded using the method it explains. It's free, quick and shows the image in the post without the need for a link to click on.

Nintendo46 · Jun 2014

hey there guys.

thanks for the responses. i try photobucket now.

ok here are the boxnormals.

then here is the uv comparison

heres the low poly pallet with the normal map applied

heres the highpoly from which i baked the normal map

and here is the generated normal map

and heres the scene as a fbx file(dropbox link should work now)

https://www.dropbox.com/s/2x2npz348w4o1so/palette_scene.fbx

Nintendo46 · Jun 2014

ok image uploading work now, thanks ScottMichaelH

Shogun3d

what do you mean with "texellation". is it something i should search the wiki about?
dont the uvs have to be split at "hard egdes"? this is suggested in a thread from the wiki
http://www.polycount.com/forum/showthread.php?t=107196 But im confused whats meant with "hard egdes". Are these 90 degrees angles? or just all extreme angles? because by a larger object this would produce many shells, right?

because as you said shogun3d its obvious that the uvs have to be laid out so that i can paint/texture them properly. I didnt quite understood many things in the thread i mentioned above, but it gave me a feeling that the uvs have to be laid out in certain way in videogame production in general. Man im trying so hard to understand and make sense of this all

and in my pallet scene, would it be wise to bake out the top and bottom part seperatly? maybe the rays cant quite go inside the pallet, which is why their normals dont look as good as the outer sides?

thanks to all who help me

with best regards, Joel

ScottHoneycutt · Jun 2014

Real quick I just searched and found these basic examples through Google. There are a lot of basic UV layout tutorials of boxes out there making it shaped liked this:

http://www.ozone3d.net/~geeks3d/public/satyr/200909/gettingstartedwithblender3/images/uvmatching.jpg

However, with game engines, normal maps and hard edges placed on low poly meshes, you will want to split the UVs up based on hard edges like so:

http://i229.photobucket.com/albums/ee301/knak47/Polycount/BoxUVS.jpg

I'm not saying "use this layout" just that the "UV islands" or "shells" should have some space between them at hard edges. Yes you can definitely have quite a lot of shells on a major piece. Below is a layout I did recently for a gun (I'm in no way saying I'm the expert but I can explain what I did with my own work). I placed a lot of hard edges on the model so many shells were necessary. It was something I had a hard time wrapping my head around because it seems like you are making it impossible to paint like this with so many seems. However it was necessary just to get the checker pattern in the layout stage to not distort. So yes, many shells is common:

Its a lot to take in at once, I've been there. As for hard edges, are you asking more about where to place them, or what they are and what they do?

Nintendo46 · Jun 2014

Hi ScottMichaelH

Wow thats really a lot of shells

Guess it was quite energy consuming creating the layout.
But thats a real nice example, thanks man.

And for the hard egdes, well i think i could ask both, what they are and where to place them. what i (think i) know about them now is that hard edges are there for the normal map calculation, helping the rays to render out smooth gradients. And i should place them at my uv seams, which i currently do automatic through a script in maya.

So basically my knowledge is very litte, if not incorrect. i would really appreciate if you could give me your version of hard edges and where to place them.

Thanks in Advance,

with best Regards, Joel

Nintendo46 · Jun 2014

@ shogun3d and also ScottMichaelH (if you like to help me out)

i created a new higpoly piece for my palette prop, changed the uv layout and got some nice results with the bake as seen here:

but theres some sqewing or distortions at the top, it looks like this:

is it a geometry issue (to less geometry maybe?), or should i change the angle of the edges. Also are these edges there to be considered "hard edges"???

thanks for your time, Joel

ScottHoneycutt · Jun 2014

Look at this if you haven't. As mentioned previously, 3ds Max uses "smoothing groups" and Maya uses "soft and hard edges" to do the same thing:

http://wiki.polycount.com/SmoothingGroups

Test the hard and soft edges. Place them on models and see what you see. If you want the edge to look hard and sharp, then place a hard edge there. If you want the edge to look soft, as if it were a part of a curve or flat surface, then place a soft edge there. Click the edge and hit the button (The MEL script is a good way to do it, just make sure you understand what it is that the script is doing ... apply the hard and soft edges manually on at least something if you haven't much yet). It will take time to have the Wiki and Tech Thread stickies information sink in. I'm still processing it all myself. It won't happen over night so keep going.

Nintendo46 · Jun 2014

@ScottMichaelH

I will read through your suggested thread. I know it will take me some time to get into this and im in no hurry, i was just thinking that you maybe have figured out some way to explain it to me, as the tech threads really are "techy". But thanks for your help anyway.

With best regards, Joel

Nintendo46 · Jun 2014

So hey guys

Thanks for all your comments, i tried to make the best of it and created this two props, a pallet and a thing which i dont know the name in english

what do you think about them?

im open to any kind of critiques, suggestions whatever, am thankful for all constructive responses, so thanks in advance

with best regards, joel

Environment Art Beginner Questions

Replies