Unreal Engine 4 Toon Outlines Tutorial

When people say "toon outlines", they are referring to any technique that render lines around objects. Like cel shading, outlines can help your game look more stylized. They can give the impression that objects are drawn or inked. You can see examples of this in games such as Okami, Borderlands and Dragon Ball FighterZ.

In this tutorial, you will learn how to:

• Create outlines using an inverted mesh
• Create outlines using post processing and convolution
• Create and use material functions
• Sample neighboring pixels

• Part 1: Cel Shading
• Part 2: Toon Outline (you are here!)
• Part 3: Custom Shaders Using HLSL
• Part 4: Paint Filter

Getting Started

Start by downloading the materials for this tutorial (you can find a link at the top or bottom of this tutorial). Unzip it and navigate to ToonOutlineStarter and open ToonOutline.uproject. You will see the following scene:

To start, you will create outlines by using an inverted mesh.

Inverted Mesh Outlines

The idea behind this method is to duplicate your target mesh. Then, make the duplicate a solid color (usually black) and expand it so that it is slightly larger than the original mesh. This will give you a silhouette.

If you use the duplicate as is, it will completely block the original.

To fix this, you can invert the normals of the duplicate. With backface culling enabled, you will see the inward faces instead of the outward faces.

This will allow the original to show through the duplicate. And because the duplicate is larger than the original, you will get an outline.

Advantages:

• You will always have clean lines since the outline is made up of polygons
• Appearance and thickness are easily adjustable by moving vertices
• Outlines shrink over distance. This can also be a disadvantage.

Disadvantages:

• Generally, does not outline details inside the mesh
• Since the outline consists of polygons, they are prone to clipping. You can see this in the example above where the duplicate overlaps the ground.
• Possibly bad for performance. This depends on how many polygons your mesh has. Since you are using duplicates, you are basically doubling your polygon count.
• Works better on smooth and convex meshes. Hard edges and concave areas will create holes in the outline. You can see this in the image below.

Generally, you should create the inverted mesh in a modelling program. This will give you more control over the silhouette. If working with skeletal meshes, it will also allow you to skin the duplicate to the original skeleton. This will allow the duplicate to move with the original mesh.

For this tutorial, you will create the mesh in Unreal rather than a modelling program. The method is slightly different but the concept remains the same.

First, you need to create the material for the duplicate.

Creating the Inverted Mesh Material

For this method, you will mask the outward-facing polygons. This will leave you with the inward-facing polygons.

Navigate to the Materials folder and open M_Inverted. Afterwards, go to the Details panel and adjust the following settings:

• Blend Mode: Set this to Masked. This will allow you to mark areas as visible or invisible. You can adjust the threshold by editing Opacity Mask Clip Value.
• Shading Model: Set this to Unlit. This will make it so lights do not affect the mesh.
• Two Sided: Set this to enabled. By default, Unreal culls backfaces. Enabling this option disables backface culling. If you leave backface culling enabled, you will not be able to see the inward-facing polygons.

Next, create a Vector Parameter and name it OutlineColor. This will control the color of the outline. Connect it to Emissive Color.

To mask the outward-facing polygons, create a TwoSidedSign and multiply it by -1. Connect the result to Opacity Mask.

TwoSidedSign will output 1 for frontfaces and -1 for backfaces. This means frontfaces will be visible and backfaces will be invisible. However, you want the opposite effect. To do this, you reverse the signs by multiplying by -1. Now frontfaces will output -1 and backfaces will output 1.

Finally, you need a way to control the outline thickness. To do this, add the highlighted nodes:

In Unreal, you can move the position of every vertex using World Position Offset. By multiplying the vertex normal by OutlineThickness, you are making the mesh thicker. Here is a demonstration using the original mesh:

At this point, the material is complete. Click Apply and then close M_Inverted.

Now, you need to duplicate the mesh and apply the material you just created.

Duplicating the Mesh

Navigate to the Blueprints folder and open BP_Viking. Add a Static Mesh component as a child of Mesh and name it Outline.

Make sure you have Outline selected and set its Static Mesh to SM_Viking. Afterwards, set its material to MI_Inverted.

MI_Inverted is an instance of M_Inverted. This will allow you to adjust the OutlineColor and OutlineThickness parameters without recompiling.

Click Compile and then close BP_Viking. The viking will now have an outline. You can control the color and thickness by opening MI_Inverted and adjusting the parameters.

That’s it for this method! See if you can create an inverted mesh in your modelling program and then bring it into Unreal.

If you want to create outlines in a different way, you can use post processing instead.

Post Process Outlines

You can create post process outlines by using edge detection. This is a technique which detects discontinuities across regions in an image. Here are a few types of discontinuities you can look for:

Advantages:

• Can apply to the entire scene easily
• Fixed performance cost since the shader always runs for every pixel
• Line width stays the same at various distances. This can also be a disadvantage.
• Lines don’t clip into geometry since it is a post process effect

Disadvantages:

• Usually requires multiple edge detectors to catch all edges. This has an impact on performance.
• Prone to noise. This means edges will show up in areas with a lot of variance.

A common way to do edge detection is to perform convolution on each pixel.