Edge Detection Shader Deep Dive! Part 1 – Even or Thinner Edges?

It has been a while since the last update. I think this may actually be the longest gap between updates on this devlog in quite some time….

I spent the last week and a half deep diving into shaders, writing multiple edge detection algorithms, connecting them to MIDI controllers for better tweaking, so my mind was down that rabbit hole. I feel like I’ve finally emerged and am seeing sunglight for the first time in a while.

Anyway, I’m going to talk about what I did during this past week, and the different approaches I tried. What worked, and what didn’t.

Basic Shader Knowledge

Edge detection in Unity relies on using the depthNormal texture from the camera.

I’ve written about this before, so recommend checking it out just to get a basic understanding:

Unity Shaders – Depth and Normal Textures – Part 1

Even / Thinner Edges

The edge detection shader I had been using for most of the past year, relied on this algorithm:

- Sample surrounding pixels
- Compare the normal values of opposite surrounding pixels (if values are close, it’s not an edge, else, it is)
- Compare the depth values of opposite surrounding pixels (if values are close, it’s not an edge, else it is)

So basically we test for both normals and depth. The pixel only has to pass one of the tests to be considered an edge. Some edges the depth test miss, some the normal test miss.

This is what it looks like:


I describe this version as using uneven lines, as you can see that some of the lines are slightly thicker than the others. These are always at the place where there is both normal and depth difference, so it’ just that the edge there is being drawn by both the normal and depth test.

I decided to try another approach, inspired by Lucas Pope’s approach for the edge detection shader in Return of The Obra Dinn.

This time, here’s the algorithm:

- sample surrounding pixels
- combine depth and normal values to form a color
- compare the values of the new combined color in surrounding pixels (if values are close, it’s not an edge, else it is)

This is what it looks like:


The lines are definitely much more even (since we’re only doing one test now), but also a lot thinner.

I actually am not such a big fan of the thin lines. Some people said it looked cleaner, but I felt it looked too faded, and didn’t have a strong visual impact.

My last approach was combining the two approaches, using the first approach for lines that are closer (so that they are thicker), and using the second approach for lines farther away.

This basically just involved having both of the above algorithms together in one shader, and I would set a depth split threshold. If the depth values were below this (closer to camera), I’d use first approach, if depth values were above the depth split threshold (farther away), then we use second approach.

This is what it looks like:3

There are several problems with this. Having thick lines closer and thin lines farther doesn’t really help as much with depth perception as using color (lighter lines farther away, darker lines closer).

The width of the lines is due to sample distance, which is actually discrete, so that means the line width change is not continuous.

The farther lines are very faded, and I actually prefer how the far lines look in the uneven line version, as you can see better what’s in the distance.

Finally, and most importantly, this version looks awful in motion. You can see exactly where the algorithm change happens as the line width change is quite sudden. So it just looks like this bit of darkness that’s moving in front of you when you move.

So this whole exploration took about a week, as I was also writing a custom inspector to sort out variables (so that the inspector only showed the necessary variables for each edge detection method), and also connecting it to the MIDI controller for tweaking (which was a real pain to set up with all three methods as I had to write in ways to change the setup of the MIDI controller as well).

In the end, it turned out the method I had before was the best one. This was a bit of a disappointment, but such is game development.

Anyway, while doing all this shader stuff, I thought I might as well try to tackle the edge detection shader artifact issue that had been bugging me for the past 2 years.

Continuing in Part 2.


  • Facebook
  • Twitter
  • Tumblr
  • LinkedIn
  • del.icio.us
  • Google Bookmarks
  • Reddit
  • StumbleUpon
  • RSS

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>