11. Maps & Materials

Written by Marius Horga & Caroline Begbie

In the previous chapter, you set up a simple Phong lighting model. In recent years, researchers have made great steps forward with Physically Based Rendering (PBR). PBR attempts to accurately represent real-world shading, where the amount of light leaving a surface is less that the amount the surface receives. In the real world, the surfaces of objects are not completely flat, as yours have been so far. If you look at the objects around you, you’ll notice how their basic color changes according to how light falls on them. Some objects have a smooth surface, and some have a rough surface. Heck, some might even be shiny metal!

In this chapter, you’ll find out how to use material groups to describe a surface, and how to design textures for micro detail.

Normal Maps

The following example best describes normal maps:

On the left, there’s a lit cube with a color texture. On the right, there’s the same low-poly cube with the identical color texture and lighting. The only difference is that the cube on the right also has a second texture applied to it known as a normal map. This normal map makes it appear as if the cube is a high-poly model with lots of nooks and crannies. In truth, these high-end details are just an illusion.

For this illusion to work, the model needs a texture, like this:

All models have normals that stick out perpendicular to each face. A cube has six faces, and the normal for each face points in a different direction. Also, each face is flat. If you wanted to create the illusion of bumpiness, you’d need to change a normal in the fragment shader.

Look at the following image. On the left is a flat surface with normals in the fragment shader. On the right, you see perturbed normals. The texels in a normal map supply the direction vectors of these normals through the RGB channels.

Now, look at this single brick split out into the red, green and blue channels that make up an RGB image.

Each channel has a value between 0 and 1, and you generally visualize them in grayscale as it’s easier to read color values. For example, in the red channel, a value of 0 is no red at all, while a value of 1 is full red. When you convert 0 to an RGB color (0, 0, 0), the result is black. On the opposite spectrum, (1, 1, 1) is white. And in the middle, you have (0.5, 0.5, 0.5), which is mid-gray. In grayscale, all three RGB values are the same, so you only need to refer to a grayscale value by a single float.

Take a closer look at the edges of the red channel’s brick. Look at the left and right edges in the grayscale image. The red channel has the darkest color where the normal values of that fragment should point left (-X, 0, 0), and the lightest color where they should point right (+X, 0, 0).

Now look at the green channel. The left and right edges have equal value but are different for the top and bottom edges of the brick. The green channel in the grayscale image has darkest for pointing down (0, -Y, 0) and lightest for pointing up (0, +Y, 0).

Finally, the blue channel is mostly white in the grayscale image because the brick — except for a few irregularities in the texture — points outward. The edges of the brick are the only places where the normals should point away.

Note: Normal maps can be either right-handed or left-handed. Your renderer will expect positive y to be up, but some apps will generate normal maps with positive y down. To fix this, you can take the normal map into Photoshop and invert the green channel.

The base color of a normal map — where all normals are “normal” (orthogonal to the face) — is (0.5, 0.5, 1).

This is an attractive color but was not chosen arbitrarily. RGB colors have values between 0 and 1, whereas a model’s normal values are between -1 and 1. A color value of 0.5 in a normal map translates to a model normal of 0. The result of reading a flat texel from a normal map should be a z value of 1 and the x and y values as 0. Converting these values (0, 0, 1) into the colorspace of a normal map results in the color (0.5, 0.5, 1). This is why most normal maps appear bluish.

Creating Normal Maps

To create successful normal maps, you need a specialized app. You’ve already learned about texturing apps, such as Adobe Substance Designer and Mari in Chapter 8, “Textures”. Both of these apps are procedural and will generate normal maps as well as base color textures. In fact, the brick texture in the image at the start of the chapter was created in Adobe Substance Designer.

Sculpting programs, such as ZBrush, 3D-Coat, Mudbox and Blender will also generate normal maps from your sculpts. You first sculpt a detailed high-poly mesh. And then the app looks at the cavities and curvatures of your sculpt and bakes a normal map. Because high-poly meshes with tons of vertices aren’t resource-efficient in games, you should create a low-poly mesh and then apply the normal map to this mesh.

Photoshop CC (from 2015) and Adobe Substance 3D Sampler can generate a normal map from a photograph or diffuse texture. Because these apps look at the shading and calculate the values, they aren’t as good as the sculpting or procedural apps, but it can be quite amazing to take a photograph of a real-life, personal object, run it through one of these apps, and render out a shaded model.

Here’s a normal map that was created using Allegorithmic’s legacy app Bitmap2Material:

On the right, the normal map with a white color texture is rendered on to the same cube model as before, with minimal geometry.

Tangent Space

To render with a normal map texture, you send it to the fragment function in the same way as a color texture, and you extract the normal values using the same UVs. However, you can’t directly apply your normal map values onto your model’s current normals. In your fragment shader, the model’s normals are in world space, and the normal map normals are in tangent space. Tangent space is a little hard to wrap your head around. Think of the brick cube with all its six faces pointing in different directions. Now think of the normal map with all the bricks the same color on all the six faces.

If a cube face is pointing toward negative x, how does the normal map know to point in that direction?

Using a sphere as an example, every fragment has a tangent — that’s the line that touches the sphere at that point. The normal vector in this tangent space is thus relative to the surface. You can see that all of the arrows are at right angles to the tangent. So if you took all of the tangents and laid them out on a flat surface, the blue arrows would point upward in the same direction. That’s tangent space!

The following image shows a cube’s normals in world space.

To convert the cube’s normals to tangent space, you create a TBN matrix - that’s a Tangent Bitangent Normal matrix that’s calculated from the tangent, bitangent and normal value for each vertex.

In the TBN matrix, the normal is the perpendicular vector as usual; the tangent is the vector that points along the horizontal surface; and the bitangent is the vector — as calculated by the cross product — that is perpendicular to both the tangent and the normal.

Note: The cross product is an operation that gives you a vector perpendicular to two other vectors.

The tangent can be at right angles to the normal in any direction. However, to share normal maps across different parts of models, and even entirely different models, there are two standards:

1. The tangent and bitangent will represent the directions that u and v point, respectively, defined in model space.
2. The red channel will represent curvature along u, and the green channel, along v.

You could calculate these values when you load the model. However, with Model I/O, as long as you have data for both the position and texture coordinate attributes, Model I/O can calculate and store these tangent and bitangent values at each vertex for you.

Finally some code! :]

The Starter App

➤ In Xcode, open the starter project for this chapter.

There are different models in the project, with accompanying textures in Textures.xcassets. There are two lights in the scene - a sun light and a gentle directional fill light from the back. The code is the same as at the end of the previous chapter with the exception of GameScene and SceneLighting, which simply set up the different models and lighting. Pressing keys “1” and “2” take you to the front and default views respectively.

➤ Build and run the app, and you’ll see a quaint cartoon cottage.

It’s a bit plain, but you’re going to add a normal map to help give it some surface details.

Using Normal Maps

➤ In the Models ▸ Cottage group, open cottage1.mtl in a text editor.

There are two textures needed to render this cottage:

map_tangentSpaceNormal cottage-normal
map_Kd cottage-color

map_Kd defines the color map, and map_tangentSpaceNormal defines the normal map. cottage-color and cottage-normal are textures in Textures.xcassets.

The normal map holds data instead of color. Later, you’ll use other texture maps for other surface qualities. Looking at textures like cottage-normal in a photo editor, you’d think they are color, but the trick is to regard the RGB values as numerical data instead of color data.

➤ In the Geometry group, open Submesh.swift, and add a new property to Submesh.Textures:

let normal: MTLTexture?

➤ At the end of SubMesh.Textures.init(material:), read in this texture:

normal = property(with: .tangentSpaceNormal)

This is the normal map property that Model I/O expects to read in from the .mtl file.

➤ In the Shaders group, open Common.h, and add this to TextureIndices:

NormalTexture = 1

You’ll send the normal texture to the fragment shader using this index.

➤ Open Model.swift, and in render(encoder:uniforms:params:), locate where you set the base color texture inside for submesh in mesh.submeshes.

➤ Add this:

encoder.setFragmentTexture(
  submesh.textures.normal,
  index: NormalTexture.index)

Here, you send the normal texture to the GPU.

➤ Open Shaders.metal, and in fragment_main, add the normal texture to the list of parameters:

texture2d<float> normalTexture [[texture(NormalTexture)]]

Now that you’re transferring the normal texture map, the first step is to apply it to the cottage as if it were a color texture.

➤ In fragment_main, before calling phongLighting, add this:

float3 normal;
if (is_null_texture(normalTexture)) {
  normal = in.worldNormal;
} else {
  normal = normalTexture.sample(
  textureSampler,
  in.uv * params.tiling).rgb;
}
normal = normalize(normal);
return float4(normal, 1);

This reads in normalValue from the texture, if there is one. If there is no normal map texture for this model, set the default normal value. The return is only temporary to make sure the app is loading the normal map correctly, and that the normal map and UVs match.

➤ Build and run to verify the normal map is providing the fragment color.

You can see all the surface details the normal map will provide. There are scattered bricks on the wall, wood grain on the door and windows and a shingle-looking roof.

➤ Excellent! You tested that the normal map loads, so remove this from fragment_main:

return float4(normal, 1);

You may have noticed that in the normal map’s bricks, along the main surface of the house, the red seems to point along negative y, and the green seems to map to negative x.

You might expect that red (1, 0, 0) maps to x and green (0, 1, 0) maps to y. This happens because the UV island for the main part of the house is rotated 90 degrees counterclockwise.

Not to worry, the mesh’s stored tangents will map everything correctly. They take UV rotation into account.

Don’t celebrate just yet. You still have several tasks ahead of you. You still need to:

1. Load tangent and bitangent values using Model I/O.
2. Tell the render command encoder to send the newly created MTLBuffers containing the values to the GPU.
3. In the vertex shader, change the values to world space — just as you did normals — and pass the new values to the fragment shader.
4. Calculate the new normal based on these values.

1. Load Tangents and Bitangents

➤ Open VertexDescriptor.swift, and look at MDLVertexDescriptor’s defaultLayout. Here, you tell the vertex descriptor that there are normal values in the attribute named MDLVertexAttributeNormal.

So far, your models have normal values included with them, but you may come across odd files where you have to generate normals. You can also override how the modeler smoothed the model. For example, the house model has smoothing applied in Blender so that the roof, which has very few faces, does not appear too blocky.

Smoothing recalculates vertex normals so that they interpolate smoothly over a surface. Blender stores smoothing groups in the .obj file, which Model I/O reads in and understands. Notice in the above image, that although the surface of the sphere is smooth, the edges are unchanged and pointy. Smoothing only changes the way the renderer evaluates the surface. Smoothing does not change the geometry.

Try reloading vertex normals and overriding the smoothing.

➤ Open Model.swift, and in init(name:), replace:

let (mdlMeshes, mtkMeshes) = try! MTKMesh.newMeshes(
  asset: asset,
  device: Renderer.device)

With:

var mtkMeshes: [MTKMesh] = []
let mdlMeshes =
  asset.childObjects(of: MDLMesh.self) as? [MDLMesh] ?? []
_ = mdlMeshes.map { mdlMesh in
  mdlMesh.addNormals(
    withAttributeNamed: MDLVertexAttributeNormal,
    creaseThreshold: 1.0)
  mtkMeshes.append(
    try! MTKMesh(
      mesh: mdlMesh,
      device: Renderer.device))
}

You’re now loading the MDLMeshes first and changing them before initializing the MTKMeshes. You ask Model I/O to recalculate normals with a crease threshold of 1. This crease threshold, between 0 and 1, determines the smoothness, where 1.0 is unsmoothed.

➤ Build and run the app.

The cottage is now completely unsmoothed, and you can see all of its separate faces. If you were to try a creaseThreshold of zero, where everything is smoothed, you’d get some rendering artifacts because of the surfaces rounding too far. When dealing with smoothness remember this: Smoothness is good, but use it with caution. The artist needs to set up the model with smoothing in mind.

➤ Remove the line you just added that reads:

mdlMesh.addNormals(
  withAttributeNamed: MDLVertexAttributeNormal,
  creaseThreshold: 1.0)

➤ Replace it with this:

mdlMesh.addTangentBasis(
  forTextureCoordinateAttributeNamed:
    MDLVertexAttributeTextureCoordinate,
  tangentAttributeNamed: MDLVertexAttributeTangent,
  bitangentAttributeNamed: MDLVertexAttributeBitangent)

All the supplied models have normals provided by Blender, but not tangents and bitangents. This new code generates and loads the vertex tangent and bitangent values.

Model I/O does a few things behind the scenes:

• Add two named attributes to mdlMesh’s vertex descriptor: MDLVertexAttributeTangent and MDLVertexAttributeBitangent.
• Calculate the tangent and bitangent values.
• Create two new MTLBuffers to contain them.
• Update the layout strides on mdlMesh’s vertex descriptor to match the two new buffers.

With the addition of these two new attributes, you should update the default vertex descriptor so that the pipeline state in Renderer uses the same vertex descriptor.

First, define the new buffer attribute and buffer indices.

➤ Open Common.h and add this to Attributes:

Tangent = 4,
Bitangent = 5

➤ Add the indices to BufferIndices:

TangentBuffer = 3,
BitangentBuffer = 4,

➤ Open VertexDescriptor.swift, and add this to MDLVertexDescriptor’s defaultLayout before return:

vertexDescriptor.attributes[Tangent.index] =
  MDLVertexAttribute(
    name: MDLVertexAttributeTangent,
    format: .float3,
    offset: 0,
    bufferIndex: TangentBuffer.index)
vertexDescriptor.layouts[TangentBuffer.index]
  = MDLVertexBufferLayout(stride: MemoryLayout<float3>.stride)
vertexDescriptor.attributes[Bitangent.index] =
  MDLVertexAttribute(
    name: MDLVertexAttributeBitangent,
    format: .float3,
    offset: 0,
    bufferIndex: BitangentBuffer.index)
vertexDescriptor.layouts[BitangentBuffer.index]
  = MDLVertexBufferLayout(stride: MemoryLayout<float3>.stride)

When you create the pipeline state in Renderer, the pipeline descriptor will now notify the GPU that it needs to create space for these two extra buffers. It’s important that you remember that your model’s vertex descriptor layout must match the one in the render encoder’s pipeline state. In addition, your shader’s VertexIn attributes should also match the vertex descriptor.

Note: So far, you’ve only created one pipeline descriptor for all models. But often models will require different vertex layouts. Or if some of your models don’t contain normals, colors and tangents, you might wish to save on creating buffers for them. You can create multiple pipeline states for the different vertex descriptor layouts, and replace the render encoder’s pipeline state before drawing each model.

➤ Build and run the app to make sure your cottage still renders.

You’ve completed the necessary updates to the model’s vertex layouts, and now you’ll update the rendering code to match.

2. Send Tangent and Bitangent Values to the GPU

➤ Open Model.swift, and in render(encoder:uniforms:params:), locate for mesh in meshes.

For each mesh, you’re currently sending all the vertex buffers to the GPU:

for (index, vertexBuffer) in mesh.vertexBuffers.enumerated() {
  encoder.setVertexBuffer(
    vertexBuffer,
    offset: 0,
    index: index)
}

This code includes sending the tangent and bitangent buffers. You should be aware of the number of buffers that you send to the GPU. In Common.h, you’ve set up UniformsBuffer as index 11, but if you had defined that as index 4, you’d now have a conflict with the bitangent buffer.

3. Convert Tangent and Bitangent Values to World Space

Just as you converted the model’s normals to world space, you need to convert the tangents and bitangents to world space in the vertex function.

➤ Open Shaders.metal, and add these new attributes to VertexIn:

float3 tangent [[attribute(Tangent)]];
float3 bitangent [[attribute(Bitangent)]];

➤ Add new properties to VertexOut so that you can send the values to the fragment function:

float3 worldTangent;
float3 worldBitangent;

➤ In vertex_main after calculating out.worldNormal, add this:

.worldTangent = uniforms.normalMatrix * in.tangent,
.worldBitangent = uniforms.normalMatrix * in.bitangent

This code moves the tangent and bitangent values into world space.

4. Calculate the New Normal

Now that you have everything in place, it’ll be a simple matter to calculate the new normal.

Before doing the normal calculation, consider the normal color value that you’re reading. Colors are between 0 and 1, but normal values range from -1 to 1.

➤ Still in Shaders.metal, in fragment_main, locate where you read normalTexture. In the else part of the conditional, after reading the normal from the texture, add:

normal = normal * 2 - 1;

This code redistributes the normal value to be within the range -1 to 1.

➤ After the previous code, still inside the else part of the conditional, add this:

normal = float3x3(
  in.worldTangent,
  in.worldBitangent,
  in.worldNormal) * normal;

This code recalculates the normal direction into tangent space to match the tangent space of the normal texture.

➤ Replace the color definition with:

float3 color = phongLighting(
  normal,
  in.worldPosition,
  params,
  lights,
  baseColor
);

You send phongLighting your newly calculated normal.

➤ Remove normalDirection since you no longer need it:

float3 normalDirection = normalize(in.worldNormal);

➤ Build and run the app to see the normal map applied to the cottage.

As you rotate the cottage, notice how the lighting affects the small cavities on the model, especially on the door and roof where the specular light falls — it’s almost like you created new geometry, but you didn’t. That’s the magic of normal apps: Adding amazing detail to simple low-poly models.

Other Texture Map Types

Normal maps are not the only way of changing a model’s surface. There are other texture maps:

• Roughness: Describes the smoothness or roughness of a surface. You’ll add a roughness map shortly.
• Metallic: White for metal and black for dielectric. Metal is a conductor of electricity, whereas a dielectric material is a non-conductor.
• Ambient Occlusion: Describes areas that are occluded; in other words, areas that are hidden from light.
• Reflection: Identifies which part of the surface is reflective.
• Opacity: Describes the location of the transparent parts of the surface.

In fact, any value (thickness, curvature, etc.) that you can think of to describe a surface, can be stored in a texture. You just look up the relevant fragment in the texture using the UV coordinates and use the value recovered. That’s one of the bonuses of writing your own renderer. You can choose what maps to use and how to apply them.

You can use all of these textures in the fragment shader, and the geometry doesn’t change.

Note: A displacement or height map can change geometry. You’ll read about displacement in Chapter 19, “Tessellation & Terrains”.

Materials

Not all models have textures. For example, the train you rendered earlier in the book has different material groups that specify a color instead of using a texture.

In the Models ▸ Cottage group, open cottage1.mtl in a text editor. This is the file that describes the visual aspects of the cottage model. There are several material groups:

• glass
• roof
• wall
• wood

Each of these groups contain different material properties. Although you’re loading the same diffuse texture for all of them, you could use different textures for each group.

In the case of a property not having an associated texture, you can extract the values:

• Ns: Specular exponent (shininess)
• Kd: Diffuse color
• Ks: Specular color

Note: You can find a full list of the definitions at https://bit.ly/3HBVnGn.

The current .mtl file loads the color using map_Kd, but for experimentation, you’ll switch the rendered cottage file to one that gets its color from the material group and not a texture.

➤ Open cottage2.mtl, and see that none of the groups has a map_Kd property.

➤ Open GameScene.swift, and change cottage to use "cottage2.obj" instead of "cottage1.obj".

The diffuse color won’t be the only material property you’ll be reading.

➤ Open Common.h, and add a new structure to hold material values:

typedef struct {
  vector_float3 baseColor;
  vector_float3 specularColor;
  float roughness;
  float metallic;
  float ambientOcclusion;
  float shininess;
} Material;

There are more material properties available, but these are the most common. For now, you’ll read in baseColor, specularColor and shininess.

➤ Open Submesh.swift, and create a new property in Submesh under textures to hold the materials:

let material: Material

Your project won’t compile until you’ve initialized material.

➤ At the bottom of Submesh.swift, create a new Material extension with initializer:

private extension Material {
  init(material: MDLMaterial?) {
    self.init()
    if let baseColor = material?.property(with: .baseColor),
      baseColor.type == .float3 {
      self.baseColor = baseColor.float3Value
    }
  }
}

In Submesh.Textures, you read in string values for the textures’ file names from the submesh’s material properties. If there’s no texture available for a particular property, you can use the material base color. For example, if an object is solid red, you don’t have to go to the trouble of making a texture, you can just use the material’s base color of float3(1, 0, 0) to describe the color.

➤ Add the following code to the end of Material’s init(material:):

if let specular = material?.property(with: .specular),
  specular.type == .float3 {
  self.specularColor = specular.float3Value
}
if let shininess = material?.property(with: .specularExponent),
  shininess.type == .float {
  self.shininess = shininess.floatValue
}
self.ambientOcclusion = 1

Here, you read the specular and shininess values from the submesh’s materials. Currently you’re not loading or using ambient occlusion, but the default value should be 1.0 (white).

➤ In Submesh, in init(mdlSubmesh:mtkSubmesh:) and after initializing textures, initialize material:

material = Material(material: mdlSubmesh.material)

You’ll now send this material to the shader. This sequence of coding should be familiar to you by now.

➤ Open Common.h, and add another index to BufferIndices:

MaterialBuffer = 14

➤ Open Model.swift. In render(encoder:uniforms:params:), inside for submesh in mesh.submeshes where you call setFragmentTexture, add the following:

var material = submesh.material
encoder.setFragmentBytes(
  &material,
  length: MemoryLayout<Material>.stride,
  index: MaterialBuffer.index)

This code sends the material structure to the fragment shader. As long as your material structure stride is less than 4k bytes, then you don’t need to create and hold a special buffer.

➤ Open Shaders.metal, and add the following as a parameter of fragment_main:

constant Material &_material [[buffer(MaterialBuffer)]],

You pass the model’s material properties to the fragment shader. You use _ in front of the name, as _material is constant, and soon you’ll need to update the structure with the texture’s base color if there is one.

➤ At the top of fragment_main, add this:

Material material = _material;

➤ In fragment_main, replace:

float3 baseColor;
if (is_null_texture(baseColorTexture)) {
  baseColor = in.color;
} else {
  baseColor = baseColorTexture.sample(
  textureSampler,
  in.uv * params.tiling).rgb;
}

With:

if (!is_null_texture(baseColorTexture)) {
  material.baseColor = baseColorTexture.sample(
  textureSampler,
  in.uv * params.tiling).rgb;
}

If the texture exists, replace the material base color with the color extracted from the texture. Otherwise, you’ve already loaded the base color in material.

➤ Still in fragment_main, replace baseColor with material in phongLighting’s arguments:

float3 color = phongLighting(
  normal,
  in.worldPosition,
  params,
  lights,
  material
);

Your project won’t compile until you’ve updated phongLighting to match these parameters.

➤ Open Lighting.h, and replace float3 baseColor with:

Material material

➤ Open Lighting.metal, and in phongLighting’s parameters, replace the parameter float3 baseColor with:

Material material

You’re now sending material to phongLighting instead of just the base color, so you’ll be able to render the appropriate material properties for each submesh.

➤ Add the following code to the top of phongLighting:

float3 baseColor = material.baseColor;

➤ Replace the assignments of materialShininess and materialSpecularColor with:

float materialShininess = material.shininess;
float3 materialSpecularColor = material.specularColor;

➤ Build and run the app, and you’re now loading cottage2 with the colors coming from the material Kd values instead of a texture.

As you rotate the cottage, you can see the roof, door and window frames are shiny with strong specular highlights.

➤ In the Models ▸ Cottage group, open cottage2.mtl in a text editor, and in both the roof and wood groups, change:

• Ns: to 1.0; and
• Ks: to 0.2 0.2 0.2

These changes eliminate the specular highlights for those two groups.

➤ Build and run the app to see the difference:

You can now render models with or without textures, by reading in the values in the .mtl file. You’ve also found that it’s very easy to change material values by editing them in the .mtl file.

As you can see, models have various requirements. Some models need a color texture; some models need a roughness texture; and some models need normal maps. It’s up to you to check conditionally in the fragment function whether there are textures or constant material values.

Physically Based Rendering (PBR)

To achieve spectacular scenes, you need to have good textures, but shading plays an even more significant role. In recent years, the concept of PBR has replaced the simplistic Phong shading model. As its name suggests, PBR attempts physically realistic interaction of light with surfaces. Now that Augmented Reality has become part of our lives, it’s even more important to render your models to match their physical surroundings.

The general principles of PBR are:

• Surfaces should not reflect more light than they receive.
• Surfaces can be described with known, measured physical properties.

The Bidirectional Reflectance Distribution Function (BRDF) defines how a surface responds to light. There are various highly mathematical BRDF models for both diffuse and specular, but the most common are Lambertian diffuse; and for the specular, variations on the Cook-Torrance model (presented at SIGGRAPH 1981). This takes into account:

• micro-facet slope distribution: You learned about micro-facets and how light bounces off surfaces in many directions in Chapter 10, “Lighting Fundamentals”.
• Fresnel: If you look straight down into a clear lake, you can see through it to the bottom, however, if you look across the surface of the water, you only see a reflection like a mirror. This is the Fresnel effect, where the reflectivity of the surface depends upon the viewing angle.
• geometric attenuation: Self-shadowing of the micro-facets.

Each of these components have different approximations, or models written by many clever people. It’s a vast and complex topic. In the resources folder for this chapter, references.markdown contains a few places where you can learn more about physically based rendering and the calculations involved. You’ll also learn some more about BRDF and Fresnel in Chapter 21, “Image-Based Lighting”.

Artists generally provide some textures with their models that supply the BRDF values. These are the most common:

• Albedo: You already met the albedo map in the form of the base color map. Albedo is originally an astronomical term describing the measurement of diffuse reflection of solar radiation, but it has come to mean in computer graphics the surface color without any shading applied to it.
• Metallic: A surface is either a conductor of electricity — in which case it’s a metal; or it isn’t a conductor — in which case it’s a dielectric. Most metal textures consist of 0 (black) and 1 (white) values only: 0 for dielectric and 1 for metal.
• Roughness: A grayscale texture that indicates the shininess of a surface. White is rough, and black is smooth. If you have a scratched shiny surface, the texture might consist of mostly black or dark gray with light gray scratch marks.
• Ambient Occlusion: A grayscale texture that defines how much light reaches a surface. For example, less light will reach nooks and crannies.

Included in the starter project is a fragment function that uses a Cook-Torrance model for specular lighting. It takes as input the above textures, as well as the color and normal textures.

PBR Workflow

First, change the fragment function to use the PBR calculations.

➤ Open Renderer.swift, and in init(metalView:options:), change the name of the fragment function from "fragment_main" to "fragment_PBR".

➤ In the Shaders group, open PBR.metal. In the File inspector, add the file to the macOS and iOS targets.

➤ Examine fragment_PBR.

The function starts off similar to your previous fragment_main but with a few more texture parameters in the function header. The function extracts values from textures when available, and calculates the normals the same as previously. For simplicity, it only processes the first light in the lights array. This is the main sun light.

fragment_PBR calls computeSpecular that works through a Cook-Torrance shading model to calculate the specular highlight. Finally, it calls computeDiffuse to produce the diffuse color. The final color is the result of adding together the diffuse color and the specular highlight.

To add all of the PBR textures to your project is quite long-winded, so here you’ll only add roughness. You’ll add metallic and ambient occlusion in the challenge.

➤ Open Submesh.swift, and create a new property for roughness in Submesh.Textures:

let roughness: MTLTexture?

➤ In the Submesh.Textures extension, add the following code to the end of init(material:):

roughness = property(with: .roughness)

In addition to reading in a possible roughness texture, you need to read in the material value too.

➤ At the bottom of Material’s init(material:), add:

if let roughness = material?.property(with: .roughness),
  roughness.type == .float3 {
  self.roughness = roughness.floatValue
}

➤ Open Model.swift, and in render(encoder:uniforms:params:), locate where you send the base color and normal textures to the fragment function, then add this code afterward:

encoder.setFragmentTexture(
  submesh.textures.roughness,
  index: 2)

➤ Open GameScene.swift, and change the name of the cottage model to “cube.obj”.

➤ In init(), change the camera distance and target to:

camera.distance = 3.5
camera.target = .zero

These values fit viewing the shape and size of the cube better.

➤ Open cube.mtl in the Models ▸ Cube group in a text editor. The roughness and normal maps are commented out with a #. The default roughness value is 1.0, which is completely rough.

➤ Build and run the app to see a cube with only an albedo texture applied.

This texture has no lighting information baked into it. Textures altering the surface will change the lighting appropriately.

➤ In cube.mtl, remove the # in front of map_tangentSpaceNormal cube-normal.

➤ Build and run the app again to see the difference when the normal texture is applied.

➤ Still in cube.mtl, remove the # in front of map_roughness cube-roughness.

➤ In the Textures group, open Textures.xcassets, and select cube-roughness. Select the image and press the spacebar to preview it. The dark gray values will be smooth and shiny (exaggerated here for effect), and the white mortar between the bricks will be completely rough (not shiny).

Compare the roughness map to the cube’s color and normal maps to see how the model’s UV layout is used for all the textures.

➤ Build and run the app to see the PBR function in action. Admire how much you can affect how a model looks just by a few textures and a bit of fragment shading.

Channel Packing

Later, you’ll be using the PBR fragment function for rendering. Even if you don’t understand the mathematics, understand the layout of the function and the concepts used.

When loading models built by various artists, you’re likely going to come up against a variety of standards. Textures may be a different way up; normals might point in a different direction; sometimes you may even find three textures magically contained in a single file, a technique known as channel packing. Channel packing is an efficient way of managing external textures.

To understand how it works, open PBR.metal and look at the code where the fragment function reads single floats: roughness, metallic and ambient occlusion. When the function reads the texture for each of these values, it’s only reading the red channel. For example:

roughness = roughnessTexture.sample(textureSampler, in.uv).r;

Available within the roughness file are green and blue channels that are currently unused. As an example, you could use the green channel for metallic and the blue channel for ambient occlusion.

Included in the resources folder for this chapter is an image named channel-packed.png. If you have Photoshop or some other graphics application capable of reading individual channels, open this file and inspect the channels.

A different color channel contains each of the words. Similarly, you can load your different grayscale maps to each color channel. If you receive a file like this, you can split each channel into a different file by hiding channels and saving the new file. If you’re organizing your maps through an asset catalog, channel packing won’t impact the memory consumption and you won’t gain much advantage. However, some artists do use it for easy texture management.

Challenge

In the resources folder for this chapter is a fabulous helmet model from Malopolska’s Virtual Museums collection at sketchfab.com. Your challenge is to render this model. There are five textures that you’ll load into the asset catalog. Don’t forget to change Interpretation from Color to Data, so the textures don’t load as sRGB.

Just as you did with the roughness texture, you’ll add metallic and ambient occlusion textures to your code. You should also update TextureIndices with the correct buffer index numbers.

If you get stuck, you’ll find the finished project in the challenge folder.

The challenge project can also render USDZ files with textures. When Model I/O loads USDZ files, the textures are loaded as MDLTextures instead of string filenames. In the challenge project there is an additional method in TextureController to cope with this, as well as extra functionality in Submesh.Textures. To load these textures, you also have to preload the asset textures when you load the asset in Model, by using asset.loadTextures().

You can download USDZ samples from Apple’s AR Quick Look Gallery to try. The animated models, such as the toy robot, still won’t work properly until after you’ve completed the animation chapters, but the static models, such as the car and the teapot, should render with textures once you scale the model down to 0.1.

Where to Go From Here?

Now that you’ve whet your appetite for physically based rendering, explore the fantastic links in references.markdown, which you’ll find in the resources folder for this chapter. Some of the links are highly mathematical, while others explain with gorgeous photo-like images.

Apple’s sample code Using Function Specialization to Build Pipeline Variants is a fantastic piece of sample code to examine, complete with a gorgeous fire truck model. It uses function constants for creating different levels of detail depending on distance from the camera. As a further challenge, you can import the sample’s fire truck into your renderer to see how it looks. Remember, though, that you haven’t yet implemented great lighting and reflection. In Chapter 21, “Image-Based Lighting”, you’ll explore how to light your scene with reflection from a skycube texture. Metallic objects look much more realistic when they have something to reflect.