Unity pro- lightmaps + specular?

AlecMoody

I'm new to unity work and having a problem with a scene I am setting up. I can't seem to get a good result for lightmaps + specular. The camera view is similar to an RTS view and I need the specular/normal maps to work at all distances in combination with my GI/lightmapping solution. I currently have the scene set up running deferred rendering with directional light maps. The shadow areas are getting all kinds of weird specular artifacts and it looks totally broken. Is there an easy way to set this up? Additionally, I would be happy to limit the specular effects to 1 primary (sun) + 1 nearby light source.

Farfarer

Generally, lightmaps end up 'nixing all the dynamic lighting.

Options are to turn on dual forward lightmapping (blends between lightmap for far and dynamic lighting for near shadows). For an RTS, that's probably not too helpful.

Or to use RNM lightmaps, which should let you get specular highlights from the lights you baked the lightmaps with - not sure if it lets you stick dynamic lights on top. Although you could hack that in with a shader I think.

That said, it is possible to hack together a shader that'll do a lightmap pass then add passes for dynamic lights on top. Might have one somewhere but won't be able to post it up until tomorrow.

It's a bit harder to do with the deferred pipeline, though. It's more locked down in how it handles things. You won't get coloured spec, which might be the cause of the white oddities?

Also not 100% sure how those other options tie into the deferred pipeline.

AlecMoody

switching over to forward rendering + directional lightmaps cleaned up most of those errors. The lightmap bake is going to have something like 100 lights in it. I only actually need a few to provide specular information- Am I right in thinking that the lightmapping dropdown settings do this? (auto providing both lightmap info and dynmaic lighting, and the other two are self explanatory)

AlecMoody

Nevermind- the problem is still happening. I took a closer look at the maps today and the errors only appear in the specular lighting and are baked into the directional/scale lightmaps. So far, to fix this I have tried both importing and calculating tangents and normals, using my imported lightmap UVs, letting unity generate lightmap UVs, toggling the split tangents button, and I have mesh compression and optimize mesh turned off.

and image summary of what is happening:

Farfarer

Are you using a custom shader or one of Unity's built-in ones?

AlecMoody

I am using the standard spec shader. Since the artifacts are literally painted into the scale maps I would expect that this is some sort of bug or fbx compatibility issue with that system.

AlecMoody

Doing more thorough testing leads me to think this is vertex order related. I tested a cube, it looked fine. Then I added a little steeple, and again it looked good. When I started extruding more complicated surfaces off of the mesh the shading errors came back. The larger mesh on the left was made by extruding pieces off of the smaller one:

AlecMoody

simple test with two cube shapes. On the left, made by extruding a face of the cube. On the right, I pulled those verts out and then added an edge loop. Similar topology, different vertex order:

Farfarer

That's really strange. Perhaps it's attempting to retain lightmap UVs as much as possible, even though vert IDs have changed?

Third shader on this page'll let you visualise UV2 coords;
http://docs.unity3d.com/Documentation/Components/SL-VertexProgramInputs.html

AlecMoody

Yeah, I'm not sure. The UVs for both the channels were made after all the geometry work was done (simple auto unwrap) and they live on the same sheet. I submitted this as a bug today. How quick are they to respond to these kinds of things? Doing work for a client means I can't wait around for months for this to be resolved but I also can't dump a huge amount of time into something non productive.

I will test out that shader tomorrow morning.

I did some looking around today and couldn't find any current examples of people using directional lightmaps. I am wondering how easy this is to reproduce.

Farfarer

I'm not sure, I've only submitted a couple of bugs (related to bad tangent generation) and I never heard back from them. That was at the end of the 3.5 cycle and it's still broken in 4.

AlecMoody

I tested that debug shader. Also, desaturating the scale maps fixes the errors (not surprising).

AlecMoody

I have resolved this 90%. The errors are actually being caused by smoothing splits that do not match with the channel 2 UVs. Much like a tangent space normal map, you need to have a UV split in your second channel UVs if you have a smoothing split on the model. What is weird about this is that the directional lightmaps don't mind most of the time, just on some polygons it will come up with those wierd errors.

Stromberg90

Could this be the same that happens with specular in udk when using lightmass?

AlecMoody

Not sure, havent used UDK lightmass enough to see that problem.

Here is an image of the problem and the fix:

Farfarer

In the Inspector for your mesh, if you set Normals and Tangents to Calculate, ramp Smoothing Angle up to 180 and then set Normals and Tangents back to their previous settings, does it fix it?

AlecMoody

It does but I need to bake the scale maps with those settings for it to do anything and it also bakes a lot ugly smoothed normals into the scale maps.

AlecMoody

Oh- after setting everything back the error remains gone. Like that had some kind of continued affect on the model even after I changed the settings back? Going to test this more.

Farfarer

Yeah, that's the unfixed bug I entered regarding the tangents.

If you set Normals to Import and Tangents to Calculate, it will create vertex splits in the tangents in the same place it would create smoothing splits if it were auto-calculating the normals using the Smoothing Angle value. Even though the normals are set to import and it shouldn't be calculating it's own normals. Which means you end up with your tangent normal maps showing some weird errors and you can't edit the Smoothing Angle setting unless Normals are set to Calculate, so the cause isn't obvious.

AlecMoody

What is required to get the error to show back up again? Im just trying to verify this is really what is happening (and not some mesh change I didn't notice) but I can't reproduce the error on that mesh.

Farfarer

Err, not sure. To get to the default values it should be...

Normals - Calculate
Smoothing Angle - 60 (Can't edit this if normals are set to import)
Normals - Import
Tangents - Calculate

And you'll need to re-bake the lightmap.

AlecMoody

yep that works. I can toggle this error off and on now simply by changing the import settings to 180 and then switching back to import normals.

AlecMoody

Do you have a link for where you reported this bug?

Farfarer

Not sure if you'll be able to see it or if it's linked to my account;

https://fogbugz.unity3d.com/default.asp?472207_4q31e3dl8a60gqoc

cupsster

This is really strange, in 3 years working with Unity newer before saw this problem.

Farfarer

Yeah, it's a really noticeable problem when working with normal maps baked to sync with Unity's tangent basis. And, apparently, when using directional light maps :P

cupsster

Maybe I had just simple luck :P