Unreal Engine 4 Custom Shaders Tutorial

The material editor is a great tool for artists to create shaders thanks to its node-based system. However, it does have its limitations. For example, you cannot create things such as loops and switch statements.

Luckily, you can get around these limitations by writing your own code. To do this, you can create a Custom node which will allow you to write HLSL code.

In this tutorial, you will learn how to:

• Create a Custom node and set up its inputs
• Convert material nodes to HLSL
• Edit shader files using an external text editor
• Create HLSL functions

To demonstrate all of this, you will use HLSL to desaturate the scene image, output different scene textures and create a Gaussian blur.

The tutorial also assumes you are familiar with a C-type language such as C++ or C#. If you know a syntactically similar language such as Java, you should still be able to follow along.

• Part 1: Cel Shading
• Part 2: Toon Outline
• Part 3: Custom Shaders Using HLSL (you are here!)
• Part 4: Paint Filter

Getting Started

Start by downloading the materials for this tutorial (you can find a link at the top or bottom of this tutorial). Unzip it and navigate to CustomShadersStarter and open CustomShaders.uproject. You will see the following scene:

First, you will use HLSL to desaturate the scene image. To do this, you need to create and use a Custom node in a post process material.

Creating a Custom Node

Navigate to the Materials folder and open PP_Desaturate. This is the material you will edit to create the desaturation effect.

First, create a Custom node. Just like other nodes, it can have multiple inputs but is limited to one output.

Next, make sure you have the Custom node selected and then go to the Details panel. You will see the following:

Here is what each property does:

• Code: This is where you will put your HLSL code
• Output Type: The output can range from a single value (CMOT Float 1) up to a four channel vector (CMOT Float 4).
• Description: The text that will display on the node itself. This is a good way to name your Custom nodes. Set this to Desaturate.
• Inputs: This is where you can add and name input pins. You can then reference the inputs in code using their names. Set the name for input 0 to SceneTexture.

To desaturate the image, replace the text inside Code with the following:

return dot(SceneTexture, float3(0.3,0.59,0.11));

Finally, connect everything like so:

Summary:

1. SceneTexture:PostProcessInput0 will output the color of the current pixel
2. Desaturate will take the color and desaturate it. It will then output the result to Emissive Color

Click Apply and then close PP_Desaturate. The scene image is now desaturated.

You might be wondering where the desaturation code came from. When you use a material node, it gets converted into HLSL. If you look through the generated code, you can find the appropriate section and copy-paste it. This is how I converted the Desaturation node into HLSL.

In the next section, you will learn how to convert a material node into HLSL.

Converting Material Nodes to HLSL

For this tutorial, you will convert the SceneTexture node into HLSL. This will be useful later on when you create a Gaussian blur.

First, navigate to the Maps folder and open GaussianBlur. Afterwards, go back to Materials and open PP_GaussianBlur.

Unreal will generate HLSL for any nodes that contribute to the final output. In this case, Unreal will generate HLSL for the SceneTexture node.

To view the HLSL code for the entire material, select Window\HLSL Code. This will open a separate window with the generated code.

Note: If the HLSL Code window is blank, you need to enable Live Preview in the Toolbar.

Since the generated code is a few thousand lines long, it’s quite difficult to navigate. To make searching easier, click the Copy button and paste it into a text editor (I use Notepad++). Afterwards, close the HLSL Code window.

Now, you need to find where the SceneTexture code is. The easiest way to do this is to find the definition for CalcPixelMaterialInputs(). This function is where the engine calculates all the material outputs. If you look at the bottom of the function, you will see the final values for each output:

PixelMaterialInputs.EmissiveColor = Local1;
PixelMaterialInputs.Opacity = 1.00000000;
PixelMaterialInputs.OpacityMask = 1.00000000;
PixelMaterialInputs.BaseColor = MaterialFloat3(0.00000000,0.00000000,0.00000000);
PixelMaterialInputs.Metallic = 0.00000000;
PixelMaterialInputs.Specular = 0.50000000;
PixelMaterialInputs.Roughness = 0.50000000;
PixelMaterialInputs.Subsurface = 0;
PixelMaterialInputs.AmbientOcclusion = 1.00000000;
PixelMaterialInputs.Refraction = 0;
PixelMaterialInputs.PixelDepthOffset = 0.00000000;

Since this is a post process material, you only need to worry about EmissiveColor. As you can see, its value is the value of Local1. The LocalX variables are local variables the function uses to store intermediate values. If you look right above the outputs, you will see how the engine calculates each local variable.

MaterialFloat4 Local0 = SceneTextureLookup(GetDefaultSceneTextureUV(Parameters, 14), 14, false);
MaterialFloat3 Local1 = (Local0.rgba.rgb + Material.VectorExpressions[1].rgb);

The final local variable (Local1 in this case) is usually a "dummy" calculation so you can ignore it. This means SceneTextureLookup() is the function for the SceneTexture node.

Now that you have the correct function, let’s test it out.

Using the SceneTextureLookup Function

First, what do the parameters do? This is the signature for SceneTextureLookup():

float4 SceneTextureLookup(float2 UV, int SceneTextureIndex, bool Filtered)

Here is what each parameter does:

• UV: The UV location to sample from. For example, a UV of (0.5, 0.5) will sample the middle pixel.
• SceneTextureIndex: This will determine which scene texture to sample from. You can find a table of each scene texture and their index below. For example, to sample Post Process Input 0, you would use 14 as the index.
• Filtered: Whether the scene texture should use bilinear filtering. Usually set to false.

To test, you will output the World Normal. Go to the material editor and create a Custom node named Gaussian Blur. Afterwards, put the following in the Code field:

return SceneTextureLookup(GetDefaultSceneTextureUV(Parameters, 8), 8, false);

This will output the World Normal for the current pixel. GetDefaultSceneTextureUV() will get the UV for the current pixel.

This is an example of how custom HLSL can break between versions of Unreal.

Next, disconnect the SceneTexture node. Afterwards, connect Gaussian Blur to Emissive Color and click Apply.

At this point, you will get the following error:

[SM5] /Engine/Generated/Material.ush(1410,8-76):  error X3004: undeclared identifier 'SceneTextureLookup'

This is telling you that SceneTextureLookup() does not exist in your material. So why does it work when using a SceneTexture node but not in a Custom node? When you use a SceneTexture, the compiler will include the definition for SceneTextureLookup(). Since you are not using one, you cannot use the function.

Luckily, the fix for this is easy. Set the SceneTexture node to the same texture as the one you are sampling. In this case, set it to WorldNormal.

Afterwards, connect it to the Gaussian Blur. Finally, you need to set the input pin’s name to anything besides None. For this tutorial, set it to SceneTexture.

Now the compiler will include the definition for SceneTextureLookup().

Click Apply and then go back to the main editor. You will now see the world normal for each pixel.

Right now, editing code in the Custom node isn’t too bad since you are working with little snippets. However, once your code starts getting longer, it becomes difficult to maintain.

To improve the workflow, Unreal allows you to include external shader files. With this, you can write code in your own text editor and then switch back to Unreal to compile.