UE4.27 - VR - Forward Renderer - Custom Edge Detection PostProcess

TechArtDL

Hey hey everyone! 

I’m running into a pretty specific issue, and I’m wondering if anyone here has faced something similar or might have an idea on how to fix it.

Setup

- Engine version: custom Unreal Engine 4.27 branch based on the Oculus branch

- Project: VR (PC, not mobile)

- Renderer: Forward Renderer

- Antialiasing: MSAA (mandatory for VR)

Goal

→  I’m trying to implement a global outline (edge detection) post process that’s as performant and visually controllable as possible.

 I don’t want to rely directly on depth differences, or at least I want a way to hide unwanted visual artifacts (like grass outlines).

Solution 1: Depth-based Post Process

The idea was to do an outline pass using the Scene Depth (since it’s the only available texture in Forward), and then mask unwanted elements such as grass or UI using a Custom Stencil.

Problems:

- It technically works, but with MSAA enabled, the Custom Stencil texture doesn’t produce a clean mask, I get visible edge artifacts.

- Even after applying a multi-sample blur pass, the visual result is still poor (see screenshots).

- On top of that, it seems that Custom Stencil with MSAA does not properly support stereo rendering in VR (see video).

Solution 2: World Normal-based Post Process

I extended the engine to write a World Normal texture in the Forward Renderer and made it available to materials.

 This fixes the issue of using depth differences, I no longer get outlines on every single grass blade (since they share the same normal), and it removes the need for Custom Stencils.

Problem:

 This setup works well and is quite performant, but again, with MSAA, the visual result at long distances becomes unstable, with small flickering artifacts.

 I’m applying the same MSAA resolve logic to my World Normal texture as UE does for Scene Color, but I suspect this isn’t ideal, since I’m dealing with direction vectors, not colors, the resolve step tends to blur or distort the normals instead of smoothing them properly.

Question

Has anyone faced a similar challenge — creating a stylized outline in VR + MSAA + Forward Renderer?

 Any idea how to improve the aliasing or visual stability while keeping MSAA active?

 Already Tried

- Rebuilding the world normal from depth in the regular pipeline for edge detection → doesn’t look good visually, since the generated normals are too hard.

- Hull inverted outline (WIP for mobile version, but too much drawn call)

Huge thanks in advance if anyone has an idea or any experience with this kind of setup! 

poopipe

that's a big ask - I have no idea how you'd stabilise stuff at distance - it's inherently not right?. 


Have you tried something along the lines of the difference of gaussians ?  not sure it'd be controllable enough, stable over time or performant but it looks pretty good and will probably work pretty well on depth. 

acerola did a video on it a while back  - here's a wikipedia  https://en.wikipedia.org/wiki/Difference_of_Gaussians


and another thought...
if the issue with inverted hull is draw calls, why not build it into the models and mask the inverted hull off  so you can manipulate it (i like shoving things out of 0:1 UV space for that) ?

TechArtDL

Thanks for the reply!
Unfortunately I can't use depth because it creates an outline around each blade of grass, which I don't want, and the stencil is buggy in stereo rendering on 4.27 so I can't mask it properly. 

That's why I set up the system to render the world normally, which allows me to avoid outlines on the grass or the UI. But I have this problem with AA, which makes the result unstable, with too many artifacts. Only temporal AA does the job, but it's too expensive. 

I wonder if SSAA would be suitable? 

iam717

I want a way to hide unwanted visual artifacts (like grass outlines).

Never played with any VR stuff, & just posting cause i want to add in my thoughts on possible solutions.(trying to help)

i forget what its called atm, but the only way to describe it is, you know when things are only rendered by the view distance of the player, like how things behind the player won't be rendered, that thing.

Can you control that & in the forward towards the outline renderings, & specify it towards whatever is applying the "lines of grass", & use the "do not render this", at this distance?  

& just use that or you still want some outlines of grass in front of the user/player?

Is it costly to use that?

Also if you look nearly directly at the img provided, that bunch of greenery near the waterfall, is the look i would assume the rest of the blades of grass would look like if you "controlled", or pulled in the render, things at this distance, thing. 

(i forget the name of)

Only know of AA working on these issues, blur also but it well blurs everything else also. 

Sadly taking "notes" of the processes of "up-scallings", perhaps that would help if it was not thought of to do, that would probably require a lot more things to do but might also; even if it blurs, be less costly, somehow? (honestly do not know.)

Alright look forward to hearing what you end up doing, perhaps you solved it already with the SSAA.

i tried. & would be great to hear if anyone found any of this useful.

just adding..

so.. Edit:  To get more ... "technical".

To add i for a time have dabbled into all this video editing filtering stuff while i took breaks from 3D stuff and came across these "programs", that utilized these "filters".

The program was called AviSynth & took some configuring to get going, cause it was created by users for users & gets really in-depth with all this video editing on a level i did not even know of.

Long story short they made these "plug-in/Scripts", that you would have to get your programmer or you, would have to look through and perhaps you could "add-in", one of these novice/advanced user created scripts.

I think they do come with "license and or free to use", stipulations you might have to address, they also have contact info along with their scripts. 

Only used these things to learn about stuff once upon a time, it was "fun" to clean things up and learn that side of things.

Now i can try to use what they do with scripts, with shaders. (hopefully)

Edit: i only mentioned avisynth and plugin cause they have plenty of information about AA and vast amount of "solutions" that might have the solution to the problem, but i am not that technical when it comes to coding things.

poopipe

TechArtDL said:

Thanks for the reply!
Unfortunately I can't use depth because it creates an outline around each blade of grass, which I don't want, and the stencil is buggy in stereo rendering on 4.27 so I can't mask it properly. 

That's why I set up the system to render the world normally, which allows me to avoid outlines on the grass or the UI. But I have this problem with AA, which makes the result unstable, with too many artifacts. Only temporal AA does the job, but it's too expensive. 

I wonder if SSAA would be suitable? 

TAA is really icky in VR in my experience so you want to avoid that.  SSAA should be stable but that will surely cost more? 
(tbf maybe they solved the TAA thing - ive not done unreal VR since the PS VR launch whih i think was 4.7)