Lighting, Seams, and Smoothing Problems

gutty333

Hello everyone, having an issue where my model is receiving bad lighting and has a seam on all the mirrored geometry. I did a test in Marmoset and everything appears fine, but is only in UDK were the issue appears. Also the smoothing on the model looks really nasty.

Also added my FBX export settings, I am using maya 2014 with fbx version 2013

Hope to get some feedback, I am fairly new with the engine, so any tips would be greatly appreciated.

UDK screen grabs





Marmoset screen grabs

gutty333

Ah, sorry for double post, did a quick test in Marmoset, and I think it might be how I am exporting my model, but still clueless on what the issue could be

Next

hiho .
what u see in udk comes from the FACT! that the way they calculate the tangent space for their meshes is a different one from the one which you used to bake the normal map itself.
therefore the numbers of the normal map and the mesh do not match thus those ugly vertex shading atifacts. which are just the difference of the current meshs tangent space compared to the normalmap being used. read more here: http://image.diku.dk/projects/media/morten.mikkelsen.08.pdf

there is only one option really... assign smoothing aka hard edges to flat areas of the mesh. unwrap it so that at least the hardedges mirror the uv shell seams. - for this u also will need to use a proper cage to bake the texture otherwise u get wired seams where there is a hard edge!

more about this u can read here in the wiki http://wiki.polycount.com/NormalMap?highlight=%28%5CbCategoryTexturing%5Cb%29 or at http://tech-artists.org/wiki/Normal_mapping

Money

Give hadplane a try, it should provide better shading for unreal.

http://www.polycount.com/forum/showthread.php?t=116899

Unreal:
Similar in quality to the new xnormal -> UDK workflow released in the July UDK build. However, this DOES work with skeletal meshes since it does not rely in importing tangents/normals from the object. It is very important that you do not enable any options in UDK to import this information (NO EXPLICIT NORMALS BUTTON) as this will break the result. It is our understanding that there are some internal precision compromises being made inside unreal that ultimately limit the final quality. This is a huge improvement over the old explicit normals pipeline and using normal maps from max, xnormal, and maya. Also, be sure to uncheck “remove degenerates” on your import as this can also create errors.

Also Epic made improvements in normals pipeline through xnormal pipeline, though works only on static meshes.

http://udn.epicgames.com/Three/XNormalWorkflow.html

Xendance

Don't use baked lighting with assets that'll be dynamic in game. Make it a skeletal mesh (or a mover, can't remember what is needed) so it'll use dynamic direct and indirect lighting.

gutty333

Thanks alot for the info, managed to fix the lighting issue, but the smoothing is still quite noticeable. Just to be sure is this a problem of my tangent normals not reading well in UDK. I am using Xnormal to bake my normal maps, and I believe I am doing all the export and import steps correctly, but the problem is still there.

Kon Artist

what does your second set of UVs look like?

gutty333

The second UV set has unique UVs for everything. Have been reading many articles, tuts, etc so I am not sure if UDK hates me, or there is something I am still not seeing. Note, I am using the latest version of UDK, February 2013, just wondering if this has anything to do?

Second UV set


Also went and created a test package and had same errors, had a clean normal bake but UDK still does not read it well.

Kon Artist

plug a spec map in and crank up your light map res on your mesh to 128 or 256. Does that look any better?

gutty333

Not going so well, tried it and same results, also tried hand-plane and no luck. Really running out of options and getting to the point were I am blaming the engine:poly127:

pics from UDK and marmoset results

Money

I think your green channel might be flipped for udk on that cube. That black enge on top looks like it should be facing down.

Though with non synced workflow engine like UDK I guess it's better to go with more geo rather than trying to get prefect shading with less geometry. Try posting in handplane support thread, alex helps with advices how to bake out those properly.
There are several threads here regarding mirroring in UDK, however I'm not sure if anyone got proper shading when using it.

gutty333

I tried seeing if that was the issue and the results were the same. Did some more testing and I think the issue might be with the hard edges of my fbx. The normals read well on the surface but when it comes to the edges I get weird artifacts. Got the same result in marmoset also. Any ideas, what is causing this issue?

@ Money, thanks also ran into many threads that explain how mirrored normals are hard to deal with. Also noticed myself that the default UT3 mirrored assets have a seam going through the middle(some of the NEC_Windows for example).











Here is the UV layout, the second UV set has unique space for all.

Money

Here are 2 most recent on it, I haven't tested if it works though.

http://www.polycount.com/forum/showthread.php?t=116922

http://www.polycount.com/forum/showthread.php?t=116474

gutty333

Okay tried just about all of the options mentioned in that thread. Tried exporting with different FBX versions and still no luck. Used maya 2011 instead of 2014 and also no luck. Played with my UDK import settings and no luck either. For marmoset the result were better when I turned off export tangent and binormals.






Also want to apologizes for spamming soo many pics and constantly bumping this thread, I am reaching the peaking point of working with UDK, really blank on what else I can do, aside from uninstall UDK and getting a new one. Or switching to new engine, which I hope it does not have to go there:poly127:.

Edit: Okay made mistake early, my fbx export works great with marmoset even when exporting tangents/binormals, the issue is apparently that marmoset does not like fbx with 2 uv sets, from what I can tell. In the end the normals look perfect in marmoset and in maya, the only issue is still UDK.

wenglish

I don't use unreal engine myself but I remember seeing something on eat3d about swizzle settings in xnormal. the video says that most programs use the default of X+ Y+ Z+ , but unreal uses X+ Y- Z+

the guy talks about it at 11:15 in the video, you can see it on eat3d since its one of their free tutorials

http://eat3d.com/free/xnormal_overview

hope that helps!

Nebularium

You tried inverting green channel of your normal map like Money suggested? Not exactly expert on these things, but but from your pic with cubes in UDK that would seem the problem. Oh and IF normal look right in Maya, they will look weird in Max (afaik), because max and Maya use different green channel information from normal map. UDk goes with Max way of rendering green channel on your normal map.

To invert it just go to Photoshop, open your normal map, go to to its channels, select green one, click ctrl+I.

gutty333

Yes I have tried that method, I inverted in Photoshop, even created a material in UDK and manually inverted with the one minus expressions, and have also baked my normal map with y- in the end I am getting same results.

Okay guys, this might be my last post (I promise)depending on if I find a solution in the next 16 hrs or if I receive any feedback.

So last confession, I have read countless articles, threads, tuts, downloaded different versions of maya, UDK, tried out handplane, exported/imported hundreds of time with different options, and still no luck in achieving perfect normals inside of UDK.

Going to showcase my complete workflow and hopefully someone can let me know if I am just stupid and am making a mistake or whether this is a problem with UDK, which I doubt seeing that other great artist are producing great content.

Step 1: Create base mesh inside of Maya



Step 2: Create high poly from the base mesh and make sure to smooth at the end


Step 3: Begin working on Low poly mesh and optimized when necessary. BTW I did not bother to optimized here since this is just a test model I created.

Step 4: Do the smoothing groups for low poly


Step 5: UV unwrapped the low poly model and make sure the smoothing groups match.


Step 6: Start getting the low and high poly model ready for export. BTW I export both as an obj file since I find it easier when working with Xnormal. I also make sure that no geometry is overlapping so no errors come from the normal.

Obj export settings


Step 7:Create a cage for the low poly model for use in Xnormal


Step 8:Import file into Xnormal and begin baking the normal map. Using recommended setting such as "use exported normals" and the cage from maya. Also ignore the tangent space box not being check, that was when I was getting an object space normal


Step 9: Make sure the normals are fine. I check both in Photoshop and also take a quick look inside marmoset

Step 10: Assemble the final low poly model. Here I just mirrored/duplicate anything parts that are sharing UV space. In this case it was just the three small tubes


Step 11: Combine the mesh and create a new UV set. Give everything a unique Uv space for proper lightmapping inside UDK. After I always delete the history, freeze the transformation, reset transformation, and I make sure the pivot is in the correct location.


Step 12: Begin to export as a .fbx. Here are my settings


Step 13: Begin to import my content into Unreal. Here are the settings I use for the mesh and the textures. BTW I have tried a lot of options, by turning off "Import Tangents" but enabling "Explicit Normals" also have turn both off, etc.



Step 14: Get everything set up, I create a new material with the normal and I also bump up the light map resolution to 128.

Step 15: I pray that hopefully it works, but it doesn't Anyways these are the results I get, and I find it hard to believe that this is considered as a great normal map inside of UDK. I am getting seams in some edge areas and nasty dark gradients.




Okay everyone that is the complete workflow. Would love to know if I am doing something wrong and thank you to everyone that has provided some tips.

Money

I haven't been working that much in UDK for the last year, so maybe better take it out to tech forums and support threads for xnormal and handplane, generally more users read those, but I here are some tests I did.

Last version I've been actively using was March 2012, IIRC that explicit normals workflow with xNormal was added in July 2012 so back then I didn't want to experiment and rebake all the meshes when I already had all my assets set in March build. Handplane also came out later so I used old method of just adding more geo and avoiding sharp anges.

Also 2 tips. I do all the normals in Y+ then during import I check "Invert normal maps" checkbox and let it auto do that for me since I'm using other apps which are mostly Y+ so it's a hassle to manually invert channel or check if it's flipped or no.
For better looking normal maps in UDK, I often check compression settings to NormalMapUncompressed, this will drop the resolution twice and remove the blue channel which will be recalculated back again in the shader, but it does give cleaner results for the normal map with less compression artifacts.

Now for the pics, here are low poly meshes I baked inside of Maya, again using that older workflow that gave me best results before. On one I ran script for hard edges along UV borders, another one is all soft edges. However since there are no 90 degree turns here, that all soft edges actually works better for this mesh.

http://abload.de/img/lowpolyaxus5.jpg
http://abload.de/img/lowpolysho6u55.jpg

Normal Maps

http://abload.de/img/normslhut5.jpg
http://abload.de/img/normsh69uj7.jpg

Export dialogue, maya 2012 with updated fbx to 2013.3

http://abload.de/img/exportdcucu.jpg

Import into UDK

http://abload.de/img/import23utd.jpg

UDK March 2012, pretty good looking

http://abload.de/img/march201280u0s.jpg

UDK Feb 2013, installed it, used same settings and files and woah. Tried checking tangents on/off, same with explicit normals but always got terrible looking result

http://abload.de/img/feb2012gju86.jpg

So in short, I have no idea why it looks so bad in feb 2013, in that older verions I had mostly good looking shading on my assets. Since I have been using mostly Unity now I'm not quite up to date with it so someone might be able to help better, but from other users handplane or that improved workflow with xNormal should give better results.

Kon Artist

I pretty much use the exact same pipeline that Money uses.

I still think this could be a light map issue... For tiles you have to match up the boarder edge with the grid in the UV editor for your light map. I believe you will get better results with a hard edge (like those on a cube) if you follow the same guild lines.

Here's a video that explains this...

[ame="http://www.youtube.com/watch?v=3eDjhOtiyuo"]UDK: Lightmap Basics - Introduction to Lightmapping with UDK Part 1/4 - YouTube[/ame]

wenglish

I use unity, not unreal, but i heard on eat3d that unreal uses different swizzle coordinates (i believe it is X+ Y- Z+ )

the eat3d video is one of the free ones, the guy talks about swizzle around 11:

http://eat3d.com/free/xnormal_overview

hope that helps!

Obscura

Hi. I also can't get perfect result in UDK, just saying. Unfortunately this looks ordinary to me. Multiple sgs = hard edges. One sg = bad shading without support edges, and hard edges along the uv borders sometimes.

Anyway, my results:
http://www.polycount.com/forum/showpost.php?p=1837800&postcount=129
These are made with dynamic lighting with/without Handplane.
If anyone know the solution, I also would be interested

joeriv

From my experience:

UDK is worst case scenario when it comes to normal maps, like the max viewport without qualified/quality normals.

xnormal is/was supposed to be somewhat synced, but it's just a lot of headache.
So either use handplane, or just (what I personally prefer) to do it the "classical/non-synced" way and split your smoothing groups/UV shells on harsh angles (90degrees).

So in your case it would involve for example the cube, splitting off every face of the cube into a seperate smoothing groups/uv shells.

I personally find that it just works, and it's overall just less of a hassle when working with UDK.

(and don't forget, UDK is Y-, so change that in the baking process or flip your green channel in PS)

Maybe this is wrong, but I do recall reading somewhere that you have to do your lightmaps in the same way (smooth/continues surfaces = 1 UV shell, harsh angles = break the uv's).
Besides that, if you are going for quality (portfolio stuff), might want to bump up the lightmap resolution a bit more.


On the hard edge thing, afaik this is just how it works:

Max viewport, it is supersubtle (probably hard to see in the image, but it's there), you still see the hard edge.


Inside UDK (latest build that is out atm):

Nothing special, just a normal static mesh with a lightmap and a normal normal map, and just a vector3 as diffuse with spec/gloss set(so no dynamic lights/interpactor/uncompressed normal maps or things like that).

It is more noticable, especially if you start tweaking towards lower gloss settings. (but then again, so does the "pixelation" on the corners).

But this being an actual game engine,
Keeping in mind that there will be other textures on it, the asset won't be like 1000x1000 pixels large on your screen, there will be postprocessing, effects, action etc I don't think anyone will notice.

So in short, imo it's not that there is a problem with UDK, it's just that because of texture compression (or something related) this (the hard edge thing) is just slightly more noticable.

Logithx

I always try to remove all gradient on flat surfaces (by using proper smoothing groups to create hard edges) because it looks terrible inside UDK. Almost every UV island on your weapon's normal map has gradient, this will cause visual problems inside UDK.

gartht3d

'So in your case it would involve for example the cube, splitting off every face of the cube into a seperate smoothing groups/uv shells.

I personally find that it just works, and it's overall just less of a hassle when working with UDK'

I agree with this

AlecMoody

gutty333 wrote: »

Not going so well, tried it and same results, also tried hand-plane and no luck. Really running out of options and getting to the point were I am blaming the engine:poly127:

pics from UDK and marmoset results

FYI something isnt set up correctly here for handplanes output. Check out the handplane unreal video:


[ame="http://www.youtube.com/watch?v=syDH7aLfqv0"]handplane UDK comparison - YouTube[/ame]

I would guess you are importing either normals or tagents, both of which will break the smoothing result.