Shadow Artifacts when Rendering with Ray Tracing

Sidney Eliot

When rendering in Marmoset, I get these arifacts when Ray tracing is on. It's not due to the AO or Normal maps, as the problem still happens with them off (and the maps themselves look clean). The shading of the low is also good.

My guess is, that the topology of the low is to sharp (I attached an image of that), making the Ray Traced shadows mess up in those areas. 

Do I really have to increase the topology in this area solely for the ray tracing's sake, or can this maybe be fixed by changing render settings?

Ray Tracing ON

Ray Tracing OFF

(All of these images are the same topology, there is no subDiv at play.)
(My Marmoset render settings are default.)

Sidney Eliot

I was able to somewhat fix it by adding in this extra face strip (a new unwrap of the UV's was thankfully also not needed). I assume that's the only way of fixing this issue, so occasionally turing on Ray Tracing during retopology is defenitly a good idea.

ZacD

Looks like typical shadow bias artifacts, I'm not sure if Marmoset has any controls for it.

Sidney Eliot

ZacD said:

Looks like typical shadow bias artifacts, I'm not sure if Marmoset has any controls for it.

I would love it, if you could explain this more, or link relevant resources.

What I could find to this: 

- https://roar11.com/2015/05/dealing-with-shadow-map-artifacts/

- https://docs.unrealengine.com/4.27/en-US/Resources/ContentExamples/Lighting/6_1/

Most of the other shadow related problems people are having, don't seem to be related to my specific problem.

Joopson

Shadow offset, in the Render settings, under Lighting>Common can help remove these issues most of the time!

sacboi

"My guess is, that the topology of the low is to sharp (I attached an image of that), making the Ray Traced shadows mess up in those areas. "

Also possibly a similar issue -  https://polycount.com/discussion/226419/toolbag-4-ray-traced-shadow-artefacts

(...scroll to third response in that thread for a potential solution, alongside Joopson's advice above : )  )

Sidney Eliot

Thanks for the help.

Dialling up shadow offset starts creating all kinds of other shadow issues. But as earth quake said, this is simply not avoidable with ray tracing, as the rays use the low to cast shadow. I wonder why I haven't ever seen shadow issues like these in games with lower poly count.

Sidney Eliot

Maybe preventing specific objects from casting shadows onto themselves could be a fix in a way that doesn't create any other noticeable issues. And the AO map would then compensate for the areas that will be getting less shadows as a result. Disabling Mesh > Cast Shadows does this, but that option also prevents the mesh from casting shadows onto other objects. I guess in Unreal one can more finely tweak this though.

sacboi

Merely an additional workflow snippet thought worthwhile sharing.

I'd previously bookmarked an interesting example when searching for a possible workaround, that I could cope with which involved baking non contiguous grouped HS meshes, via Painter mitigating self occlusion cast by floating geometry in object space, as opposed to tangent.

Sidney Eliot

I feel like my recent topology (hands) is suffering from this new found knowlege of shadow bias being a thing. Seemingly no retopology approaches I use that aren't getting to high poly, prevent the shadows from messing up when ray traced. I've also found that triangulating quads properly beforehand can help in some situations.

Taking shadow bias into consideration when retopologizing really drives one mad, I feel that just not letting some objects cast shadow onto themselves (maybe even preventing every object from self shadowing and only enabling the ones that really need it) is the way to go. The only thing one should care a bout at the end of the day when it comes to game ready retopology, is keeping the poly count as low as possible without overley affecting the magical trio of silhouette, smoothing/ shading and animation friendliness.

I would really be interested to know how AAA games deal with this.