Toolbag 4 - Ray Traced Shadow Artefacts
polycounter lvl 12
Hi,
I've recently moved up to Toolbag 4 and love the ray tracing capabilities, but keep getting these horrible shadow artefacts only when ray tracing is enabled. It's as if the lighting is looking at a hard-edged version of the model to light rather than taking into account the normals properly with the textures.
With ray tracing:
With raster:
• I've checked the tangent space for the model. it's set to 'Marmoset'. The model was made in Maya, and changing it to that makes no difference. No other option there fixes it.
• Disabling 'Cast Shadows' on the mesh fixes it, but then means it wont cast shadows on itself.
• Latest video drivers.
Any help would be much appreciated!
Thanks,
Ricky
I've recently moved up to Toolbag 4 and love the ray tracing capabilities, but keep getting these horrible shadow artefacts only when ray tracing is enabled. It's as if the lighting is looking at a hard-edged version of the model to light rather than taking into account the normals properly with the textures.
With ray tracing:
With raster:
• I've checked the tangent space for the model. it's set to 'Marmoset'
• Disabling 'Cast Shadows' on the mesh fixes it, but then means it wont cast shadows on itself.
• Latest video drivers.
Any help would be much appreciated!
Thanks,
Ricky
Replies
Bump. I'm also finding this kind of issue with GPU VRay for Maya and in Substance Stager (all different scenes and models). I read somewhere someone say its an issue with GPU renderers, so not sure how to fix this other than changing the lighting setup. If anyone else is experiencing this or similar would be handy to know how others deal with it.
This is generally expected. What you're seeing here is shadows cast from the low poly geometry. If the low poly geometry had more detail or was more smooth, you would see more smooth, detailed shadows as well. A few things you can try:
Please try some of the suggestions in the response before yours.
This solution is not practical, as it's a good practice to force triangulate specific quads in modeling software, that aren't very flat and might flip incorrectly + even good retopologized models occasionally have tris, which won't yield the best results when subdivided.
This is definitely not a good fix. There are things which should dictate the structure of the low poly mesh like what one can see in the silhouette, or making it animation friendly, but ray tracing shouldn't be a reason to massively increase the poly count. Also, shadow terminators will never ever go away just because the topology density is higher, it will only become less visible, so with close up shots one will still see them. This can be especially bad in the face area (even with quite a high poly face) as one often has very close shots there, and just throwing more polygons at this problem will not fix it. (Allthough higher cast shadow detail could be a reason to increase silhouette affecting topology)
This will fix the issue but depending on the model, will make the shadow lighter/ less and thus worse (one can compensate with AO maps, although they will mostly only carry smaller shadow details)
Disabling self shadow casting for specific objects
This is probably one of the best fixes and is a better version of disabling shadow casting from the object entirely. Sadly Marmoset doesn't have this feature yet
Shadow Offset
Changing the shadow offset will make it better, but the result still has quite visible shadow terminators.
Disabling normal map or adding a blank one
Does nothing in regard to this issue
Enabling and editing render Shadow Cascade
Does nothing in regard to this issue
Checking maps like normal and AO for grid like patterns
This can help but in my case is not an issue (proper checking is done in image editing software like PS by adding adjustment layers like exposure to reveal hidden detail)
GPU/ Other Hardware
For any one thinking it might be their hardware, I'm very sure that this is not the case
This works but is obviously not desired as one doesn't want to change lighting just for shadow terminators. However, just reducing the intensity by a bit might fully fix the issue, and your lighting might have been to intense/ physically unrealistic anyway. A better option to having super bright lights might be to add more lights or add a HDRI/ let the HDRI take over more of the lighting
This could be an issue for some, but I still have these issues with the default HDRI
Also, if you have issues with this then your AO maps that you baked with Marmoset might have experienced the same issues. As having a low which is very different from the high, surface wise, will introduce topology imitating grid patterns into the AO map. I however made sure that my topology density on the low mes was good enough to achieve clean AO map bakes, but am still having ray traced shadow terminator issues.
- https://blender.community/c/rightclickselect/XBcbbc/?category=rendering&sorting=hot
And even the super high poly flat forehead is having the issue, so as I said topology won't fix this issue (this image has ray tracing OFF, turning it on makes the grid pattern disappear)
From left to right:
- The Diameter setting in the Area/Shape panel for the lights has been increased, this converts the light into an area light which softens the shadows and makes them look nicer in general.
- The Shadow Offset setting has been adjusted in the render settings. I'm using Toolbag 5 here so this only needs a very small nudge (0.003), but you may need a larger value in Toolbag 4.
Top to bottom:
- Base lighting
- Shadow casting disabled. Since there are not any other objects in the scene to cast shadows on the hippo, this simulates turning off self-shadowing. With no self-shadowing, the body doesn't shadow the legs, the upper jaw doesn't shadow the inside of the mouth, the back doesn't shadow itself from the rim light, and so on.
- Shadows enabled again, but light width and shadow offset have been adjusted similarly to the last post with the spheres.
You also brought up a good point with poly density variation (face size variation) and I never feel like I quite know what to do. On the one hand one wants to have fewer faces on flatter areas that don't need them, and will probably see not too much deformation action, but on the other hand it's probably better to keep poly density very uniform across the entire model. (The reason I didn't give the breasts more polygons, was because silhouette & deformation wise they had enough polygons, but as we can see not enough for shadow terminators.)