6. Textures

Written by Caroline Begbie

Now that you have light in your scene, the next step is to add color. The easiest way to add fine details to a model is to use an image texture. Using textures, you can make your models come to life!

In this chapter, you’ll learn about:

• UV coordinates: How to unwrap a mesh so that you can apply a texture to it.
• Texturing a model: How to read the texture in a fragment shader.
• Samplers: Different ways you can read (sample) a texture.
• Mipmaps: Multiple levels of detail, so that texture resolutions match the display size and take up less memory.
• The asset catalog: How to organize your textures.

Textures and UV maps

The following image shows a simple house model with twelve vertices. So you can experiment, the Blender and .obj files are included in the Resources ▸ LowPolyHouse folder for this chapter. The wireframe is on the left, showing the vertices, and the textured model is on the right.

The house started as a cube but has four extra vertices, two of which raise the roof.

To texture the model, you first have to flatten it using a process called UV unwrapping. With UV unwrapping, you create a UV map by unfolding the model; this unfolding can be done by marking and cutting seams using your modeling app. The following image is the result of UV unwrapping the house in Blender and exporting the UV map:

The walls of the house have been marked as seams so they can lie flat; the roof has also been separated out by marking seams as well. If you print this UV map on paper and cut it out, you can fold it back into the house model. In Blender, you have complete control of where the seams are, and how to cut up your mesh. Blender automatically unwraps the model by cutting the mesh at these seams, but if necessary, you can also move every vertex in the UV Unwrap window to suit your texture.

Now that you have a flattened map, you can paint on it by using the UV map exported from Blender as a guide. This is the house texture made in Photoshop. It was created by cutting up a photo of a real house.

Note how the edges of the texture aren’t perfect, and there’s a copyright message. In the spaces where there are no vertices in the map, you can put whatever you want, as it won’t show up on the model. It’s a good idea not to match the UV edges exactly, but to let the color bleed, as sometimes computers don’t accurately compute floating point numbers.

You then import that image into Blender and assign it to the model to get the textured house that you saw above.

When you export a UV mapped model to an .obj file, Blender adds the UV coordinates to the file. Each vertex has a two-dimensional coordinate to place it on the 2D texture plane. The top-left is (0, 1) and the bottom-right is (1, 0).

The following diagram indicates some of the house vertices, with the matching coordinates from the .obj file. You can look at the contents of the .obj file using TextEdit.

One of the advantages of mapping from 0 to 1 is that you can swap in lower or higher resolution textures. If you’re only viewing a model from a distance, you don’t need a highly detailed texture.

The house is easy to unwrap, but imagine how complex unwrapping curved surfaces might be. This is the UV map of the train (which is still a simple model):

Photoshop, naturally, is not the only solution for texturing a model. You can use any image editor for painting on a flat texture. In the last few years, several other apps that allow painting directly on the model have become mainstream:

• Blender (free)
• Substance Designer and Substance Painter by Adobe ($$): In Designer, you can create complex materials procedurally. Using Painter, you can paint these materials on the model. The yellow house you’ll encounter in the next chapter was textured in Substance Painter.
• 3DCoat by 3Dcoat.com ($$)
• Mudbox by Autodesk ($$)
• Mari by Foundry ($$$)

In addition to texturing, using Blender, 3DCoat or Mudbox, you can sculpt models in a similar fashion to ZBrush and create low poly models from the high poly sculpt.

As you’ll find out in the next chapter, color is not the only texture you can paint using these apps, so having a specialized texturing app is invaluable.

Texture the model

Open up the starter project for this chapter. The code is almost the same as the challenge project from the previous chapter, except that the scene lighting is refactored to a new Lighting struct, and the light debugging code is gone. The initial scene contains the house model that you’ve already been introduced to with a background color more appropriate to a pastoral scene.

Take a look at fragment_main() in Shaders.metal. Currently, you’re defining the color of the model using baseColor with a constant float3(1, 1, 1) (white). In this chapter, you’ll replace that constant with color from a texture. Initially, you’ll use lowpoly-house-color.png located in the group Models ▸ LowPolyHouse.

To read the image in the fragment function, these are the steps you’ll take:

1. Add texture UV coordinates to the model’s vertex descriptor.
2. Add a matching UV coordinates attribute to the shader’s VertexIn struct.
3. Load the image using a protocol extension method.
4. Pass the loaded texture to the fragment function before drawing the model.
5. Change the fragment function to read the appropriate pixel from the texture.

1. Add UV coordinates to the vertex descriptor

As you learned previously, when you unwrap a model in Blender (or the modeling app of your choice), it saves the UV coordinates with the model. To load these into your app, Model I/O needs to have a texture coordinate attribute set up in the vertex descriptor.

First, set up the index number for the texture coordinate attribute in Common.h.

Find the Attributes enum created for the Challenge sample in the previous chapter, and change it to:

typedef enum {
  Position = 0,
  Normal = 1,
  UV = 2
} Attributes;

Saving on typing, UV references TextureCoordinates.

In VertexDescriptor.swift, add a new attribute to MDLVertexDescriptor’s defaultVertexDescriptor:

vertexDescriptor.attributes[Int(UV.rawValue)] =
    MDLVertexAttribute(name: MDLVertexAttributeTextureCoordinate,
                       format: .float2,
                       offset: offset,
                       bufferIndex: Int(BufferIndexVertices.rawValue))
offset += MemoryLayout<float2>.stride

This specifies that you want to read in a float2 for the texture coordinates at an offset of 24 bytes (the position and normal attributes, being float3s, take up 12 bytes each).

The stride of the vertex descriptor layout will now include the size of the float2 UV attributes.

2. Update the shader attributes

In Shaders.metal, the vertex function vertexIn parameter uses the stage_in attribute which relies on the vertex descriptor layout. By simply updating the VertexIn struct with the new texture coordinate attribute, the vertex function will read in the texture coordinate data.

Add this to struct VertexIn:

float2 uv [[attribute(UV)]];

This matches your previous entry in the vertex descriptor attributes.

You’ll also need to pass along these coordinates to the fragment function. Add this to struct VertexOut:

float2 uv;

In vertex_main(), include setting this property when defining out:

.uv = vertexIn.uv

(Don’t forget to add a comma to the end of the line above this.)

As with all of the other values in VertexOut, the rasterizer will interpolate the correct UV coordinate for each fragment so the shader will read the appropriate value from the texture — that’s why you’re sending the UVs through the vertex shader and not straight to the fragment shader.

3. Load the image

Each submesh of a model’s mesh has a different material characteristic. In the next chapter, you’ll use a model that has a submesh for each unique color. For textured models, each submesh will contain a reference to a unique texture.

Note: For the sake of simplicity, this is a restriction in your app where you don’t hold the same texture over multiple submesh materials. To get around this, you could set up a texture controller and hold a list of textures, and point several submeshes to the one texture.

Create a new Swift file named Texturable.swift, and include it in both the macOS and iOS targets. Replace the code with:

import MetalKit

protocol Texturable {}

extension Texturable {
}

Inside the protocol extension, add a default method:

static func loadTexture(imageName: String) throws -> MTLTexture? {
  // 1
  let textureLoader = MTKTextureLoader(device: Renderer.device)
  
  // 2
  let textureLoaderOptions: [MTKTextureLoader.Option: Any] = 
              [.origin: MTKTextureLoader.Origin.bottomLeft]
  // 3
  let fileExtension =
    URL(fileURLWithPath: imageName).pathExtension.isEmpty ?
      "png" : nil
  // 4
  guard let url = Bundle.main.url(forResource: imageName,
                                  withExtension: fileExtension)
    else {
      print("Failed to load \(imageName)")
      return nil
  }
  
  let texture = 
    try textureLoader.newTexture(URL: url,
                                 options: textureLoaderOptions)
  print("loaded texture: \(url.lastPathComponent)")
  return texture
}

Going through the code:

1. Loading textures can get complicated. When Metal was first released, you had to specify everything about the image — including pixel format, dimensions and usage — using MTLTextureDescriptor. MetalKit introduced MTKTextureLoader which provides defaults you can optionally change using loading options.
2. Here, you change a loading option to ensure that the texture loads with the origin at the bottom-left. If you don’t specify this option, the texture will be flipped. Try it later: lowpoly-house-color.png is almost vertically symmetrical, but with the texture flipped, the model’s roof will be a plain color and show the copyright text.
3. Provide a default extension for the image name.
4. Finally, create a new MTLTexture using the provided image name and loader options, and return the newly created texture. Then, for debugging purposes, print the name of the loaded texture.

Now, conform Submesh to Texturable. In Submesh.swift, add this to the bottom of the file:

extension Submesh: Texturable {}

Submesh now has access to the texture loading method.

Conveniently, Model I/O loads a model complete with all the materials. Find lowpoly-house.mtl in the group Models ▸ LowPolyHouse. The Kd value holds the diffuse material color, in this case, a light gray. At the very bottom of the file, you’ll see map_Kd lowpoly-house-color.png. This gives Model I/O the diffuse color map file name.

You’ll have various textures, so, back in Submesh.swift, inside Submesh, create a struct and a property to hold the textures:

struct Textures {
  let baseColor: MTLTexture?
}

let textures: Textures

Your project won’t compile until you’ve initialized textures.

MDLSubmesh holds each submesh’s material in an MDLMaterial property. You provide the material with a semantic to retrieve the value for the relevant material. For example, the semantic for base color is MDLMaterialSemantic.baseColor.

At the end of Submesh.swift, add an initializer for Textures:

private extension Submesh.Textures {
  init(material: MDLMaterial?) {
    func property(with semantic: MDLMaterialSemantic) 
          -> MTLTexture? {
      guard let property = material?.property(with: semantic),
        property.type == .string,
        let filename = property.stringValue,
        let texture = 
            try? Submesh.loadTexture(imageName: filename) 
      else { return nil }
      return texture
    }
    baseColor = property(with: MDLMaterialSemantic.baseColor)
  }
}

property(with:) looks up the provided property in the submesh’s material, finds the filename string value of the property and returns a texture if there is one. You may remember that there was another material property in the file marked Kd. That was the base color using floats. As you’ll see in the next chapter, material properties can also be float values where there is no texture available for the submesh.

This loads the base color texture with the submesh’s material. Here, Base color means the same as diffuse. In the next chapter, you’ll load other textures for the submesh in the same way.

At the bottom of init(submesh:mdlSubmesh:) add:

textures = Textures(material: mdlSubmesh.material)

This completes the initialization and removes the compiler warning.

Build and run your app to check that everything’s working. Your model should look the same as in the initial screenshot, however, you should get a message in the console:

4. Pass the loaded texture to the fragment function

In the next chapter, you’ll learn about several other texture types and how to send them to the fragment function using different indices. So in Common.h, set up a new enum to keep track of these texture buffer index numbers:

typedef enum {
  BaseColorTexture = 0
} Textures;

In Renderer.swift, in draw(in:), where you’re processing the submeshes, add the following below the comment // set the fragment texture here:

renderEncoder.setFragmentTexture(submesh.textures.baseColor,
                       index: Int(BaseColorTexture.rawValue))

You’re now passing the texture to the fragment function in texture buffer 0.

Note: Buffers, textures and sampler states are held in argument tables and, as you’ve seen, you access them by index numbers. On iOS, you can hold up to 31 buffers and textures and 16 sampler states in the argument table; the number of textures on macOS increases to 128. You can find out features for your device at https://developer.apple.com/metal/Metal-Feature-Set-Tables.pdf

5. Update the fragment function

You’ll now change the fragment function to accept the texture and read from it.

In Shaders.metal, add the following new argument to fragment_main(), immediately after VertexOut in [[stage_in]],:

texture2d<float> baseColorTexture [[texture(BaseColorTexture)]],

When you read in, or sample the texture, you may not land precisely on a particular pixel. In texture space, the units that you sample are called texels, and you can decide how each texel is processed using a sampler. You’ll learn more about samplers shortly. Add the following to the start of fragment_main():

constexpr sampler textureSampler;

This will be used as a default sampler. Next, replace:

float3 baseColor = float3(1, 1, 1);

With:

float3 baseColor = baseColorTexture.sample(textureSampler, 
                                           in.uv).rgb;

Here, you’re sampling the texture using the interpolated UV coordinates sent from the vertex function and retrieving the RGB values. In Metal Shading Language, you can use rgb to address the float elements as an equivalent of xyz.

Temporarily, add this on the next line so that you can see the texture color before you process it with lighting code:

return float4(baseColor, 1);

Build and run the app to see your textured house!

sRGB color space

You’ll notice that the rendered texture looks much darker than the original image.

This is because lowpoly-house-color.png is a sRGB texture. sRGB is a standard color format that compromises between how cathode ray tube monitors worked and what colors the human eye sees. As you can see in the following example of grayscale values from 0 to 1, sRGB colors are not linear. Humans are more able to discern between lighter values than darker ones.

Unfortunately, it’s not easy to do the math on colors in a non-linear space. If you multiply a color by 0.5 to darken it, the difference in sRGB will vary along the scale.

You’re currently loading the texture as sRGB pixel data and rendering into a linear color space. So when you’re sampling a value of, say 0.2 which in sRGB space is mid-gray, the linear space will read that as dark-gray.

To approximately convert the color, you can use the inverse of gamma 2.2:

sRGBcolor = pow(linearColor, 1.0/2.2);

If you use this formula on baseColor before returning from the fragment function, your house texture will look about the same as the original sRGB texture. However, a better way of dealing with this is not to load the texture as sRGB at all. In Texturable.swift, find:

let textureLoaderOptions: [MTKTextureLoader.Option: Any] = 
  [.origin: MTKTextureLoader.Origin.bottomLeft]

And change it to:

let textureLoaderOptions: [MTKTextureLoader.Option: Any] = 
  [.origin: MTKTextureLoader.Origin.bottomLeft, .SRGB: false]

Build and run, and the texture will now load with the linear color pixel format bgra8Unorm.

Note: You’ll find further reading on chromaticity and color in references.markdown in the Resources directory for this chapter.

GPU frame capture

There’s an easy way to find out what format your texture is in on the GPU, and also to look at all the other Metal buffers currently residing there: the GPU frame capture tool (also called the GPU Debugger).

Run your app, and at the bottom of the Xcode window (or above the debug console if you have it open), click the camera icon:

This button captures the current GPU frame. On the left, in the Debug navigator, you’ll see the GPU trace:

You can see all the commands that you’ve given to the render command encoder in draw(in:), such as setFragmentBytes and setRenderPipelineState.

Later, when you have several command encoders, you’ll see each one of them listed, and you can select them to see what actions or textures they have produced from their encoding.

When you select drawIndexedPrimitives, the Vertex and Fragment resources show.

Double-click on each vertex resource to see what’s in the buffer:

• MDL_OBJ-Indices: The vertex indices.
• Buffer 0x000000: The vertex position, normal and texture coordinate data, matching the attributes of your VertexIn struct.
• Vertex Bytes: The uniform matrices.
• Vertex Attributes: The incoming data from VertexIn, and the VertexOut return data from the vertex function.
• vertex_main: The vertex function. When you have multiple vertex functions, this is very useful to make sure that you’ve got the correct pipeline state set.

Going through the fragment resources:

• lowpoly-house-color.png: The house texture in texture slot 0.

• Fragment Bytes: You have two of them: one holds the lights array, and the other holds the fragment uniforms light count.

• fragment_main: The fragment function.

• CAMetalLayer Drawable: The result of the encoding in color attachment 0. In this case, this is the view’s current drawable. Later, you’ll use multiple color attachments.

• MTKView Depth: The depth buffer. Black is closer and white is farther. The rasterizer uses the depth map.

You can see from this list that the GPU is holding the texture as BGRA8Unorm.

If you reverse the previous section’s texture loading options and comment out .SRGB: false, you will be able to see that the texture is now BGRA8Unorm_sRGB. (Make sure you restore the option .SRGB: false before continuing.)

If you’re ever uncertain as to what is happening in your app, capturing the GPU frame might give you the heads-up because you can examine every render encoder command and every buffer. It’s a good idea to use it throughout this book to examine what’s happening on the GPU. Chapter 23, “Debugging and Profiling” will give you a more thorough grounding in the GPU Frame Capture tool.

Samplers

When sampling your texture just now, you used a default sampler. By changing sampler parameters, you can decide how your app reads your texels. You’ll now add a ground plane to your scene to see how you can control the appearance of the ground texture.

In Renderer.swift, in init(metalView:), find where you add the house to the models array. After that, add:

let ground = Model(name: "plane.obj")
ground.scale = [40, 40, 40]
models.append(ground)

This adds a ground plane and scales it up to be huge.

The ground plane texture will stretch to fit. Build and run:

You can see every individual pixel of the ground texture, which doesn’t look good in this particular scene. By changing one of the sampler parameters, you can tell Metal how to process the texel where it’s smaller than the assigned fragments.

In Shaders.metal, in fragment_main(), change:

constexpr sampler textureSampler;

To:

constexpr sampler textureSampler(filter::linear);

This instructs the sampler to smooth the texture. Build and run, and see that the ground texture — although still stretched — is now smooth.

There will be times, such as when you make a retro game of Frogger, that you’ll want to keep the pixelation. In that case, use nearest filtering.

In this particular case, however, you want to tile the texture. That’s easy with sampling!

Change the sampler definition and the baseColor assignment to:

constexpr sampler textureSampler(filter::linear, 
                                 address::repeat);
float3 baseColor = baseColorTexture.sample(textureSampler, 
                                           in.uv * 16).rgb;

This multiples the UV coordinates by 16 and accesses the texture outside of the allowable limits of 0 to 1. address::repeat changes the sampler’s addressing mode, so here it will repeat the texture 16 times across the plane.

The following image illustrates the other address sampling options shown with a tiling value of 3. You can use s_address or t_address to change only the width or height coordinates respectively.

Build and run your app.

The ground looks great! The house… not so much. The shader has tiled the house texture as well. To overcome this, you’ll create a tiling property on the model and send it to the fragment function with fragmentUnforms.

In Model.swift, create a new property on Model:

var tiling: UInt32 = 1

In Common.h add this to struct FragmentUniforms:

uint tiling;

In Renderer.swift, where you set up the ground in init(metalView:), just before you append the ground to models, define the required tiling:

ground.tiling = 16

In draw(in:), just inside the for loop where you process the models, add this:

fragmentUniforms.tiling = model.tiling

Your assignment of fragmentUniforms to an MTLBuffer currently takes place before the for loop, so move:

renderEncoder.setFragmentBytes(&fragmentUniforms,
           length: MemoryLayout<FragmentUniforms>.stride,
           index: Int(BufferIndexFragmentUniforms.rawValue))

Inside the for loop, after your assignment of the tiling variable.

In Shaders.metal, you’re already receiving fragmentUniforms as a parameter into fragment_main(), so replace the constant tiling value with the variable:

float3 baseColor = baseColorTexture.sample(textureSampler, 
                         in.uv * fragmentUniforms.tiling).rgb;

Build and run. Both your ground and the house now tile correctly! :]

Metal API sampler states

Creating a sampler in the shader is not the only option. Instead, you’re going to create an MTLSamplerState in the Metal API and hold it with the model. You’ll then send the sampler state to the fragment function.

In Model.swift, add a new property to Model:

let samplerState: MTLSamplerState?

The compile error will go away after you’ve initialized samplerState.

Now add the following method:

private static func buildSamplerState() -> MTLSamplerState? {
  let descriptor = MTLSamplerDescriptor()
  descriptor.sAddressMode = .repeat
  descriptor.tAddressMode = .repeat
  let samplerState = 
       Renderer.device.makeSamplerState(descriptor: descriptor)
  return samplerState
}

This creates the sampler state and you set the sampler to repeat mode, just as you did in the fragment function.

In init(name:), call your new method before super.init():

samplerState = Model.buildSamplerState()

The compile error should now go away.

In Renderer.swift, in draw(in:), locate the for loop where you process the models. Inside the for loop, just after where you assigned the tiling variable, add this:

renderEncoder.setFragmentSamplerState(model.samplerState, 
                                      index: 0)

You’re now sending the sampler state to the fragment function using sampler state slot 0 in the argument table.

In Shaders.metal, add a new parameter to fragment_main, immediately after texture2d<float> baseColorTexture [[texture(BaseColorTexture)]],:

sampler textureSampler [[sampler(0)]],

Now that you’re receiving the sampler into the function, remove:

constexpr sampler textureSampler(filter::linear, 
                                 address::repeat);

Build and run, and the output should be the same as when you were creating the sampler within the fragment function. The main advantage is that it’s now easier to set individual parameters per model on the sampler state.

Zoom out of the scene quite a long way, and as you do, notice a moiré pattern is happening on the roof of the house.

Moiré is a rendering artifact that happens when you’re undersampling a texture.

As you rotate the scene, there’s also distracting noise on the grass toward the horizon, almost as if it’s sparkling. You can solve these artifact issues by sampling correctly using resized textures called mipmaps.

Mipmaps

Check out the relative sizes of the roof texture and how it appears on the screen:

The pattern occurs because you’re sampling more texels than you have pixels. The ideal would be to have the same number of texels to pixels, meaning that you’d require smaller and smaller textures the further away an object is. The solution is to use mipmaps which will let the GPU sample the texture at a suitable size.

MIP stands for multum in parvo — a Latin phrase meaning “much in small.”

Mipmaps are texture maps resized down by a power of 2 for each level, all the way down to 1 pixel in size. If you have a texture of 64 pixels by 64 pixels, then a complete mipmap set would consist of:

Level 0: 64 x 64, 1: 32 x 32, 2: 16 x 16, 3: 8 x 8, 4: 4 x 4, 5: 2 x 2, 6: 1 x 1.

In the following image, the top checkered texture has no mipmaps; but in the bottom image, every fragment is sampled from the appropriate MIP level. As the checkers recede, there’s much less noise, and the image is cleaner. At the horizon, you can see the solid color smaller gray mipmaps.

You can easily and automatically generate these mipmaps when first loading the texture.

In Texturable.swift, change the texture loading options to:

let textureLoaderOptions: [MTKTextureLoader.Option: Any] =
      [.origin: MTKTextureLoader.Origin.bottomLeft,
       .SRGB: false,
       .generateMipmaps: NSNumber(booleanLiteral: true)]

This will create mipmaps all the way down to the smallest pixel.

There’s one more thing to change: the sampler state. In Model.swift, in buildSamplerState(), add this before creating the sampler state:

descriptor.mipFilter = .linear

The default for mipFilter is .notMipmapped; however, if you provide either .linear or .nearest, then the GPU will sample the correct mipmap.

Build and run. The noise from both the building and the ground should be gone when you zoom and rotate.

Using the GPU Frame Capture tool, you can inspect the mipmaps. Double-click a texture, and at the bottom-left, you can choose the MIP level. This is MIP level 4 on the house texture:

Your rendered ground is looking a bit muddy and blurred in the background. This is due to anisotropy. Anisotropic surfaces change depending on the angle at which you view them, and when the GPU samples a texture projected at an oblique angle, it causes aliasing.

In Model.swift, in buildSamplerState(), add this before creating the sampler state:

descriptor.maxAnisotropy = 8

Metal will now take 8 samples from the texel to construct the fragment. You can specify up to 16 samples to improve quality. Use as few as you can to obtain the quality you need because the sampling can slow down rendering.

Build and run, and your render should be artifact-free.

When you write your full game, you’re likely to have many textures for the different models. Some models are likely to have several textures. Organizing all these textures and working out which ones need mipmaps can become labor intensive. Plus, you’ll also want to compress images where you can and send textures of varying sizes and color gamut to different devices. The asset catalog is where you’ll turn.

The asset catalog

As its name suggests, the asset catalog can hold all of your assets, whether they be data, images, textures or even colors. You’ve probably used the catalog for app icons and images. Textures differ from images in that the GPU uses them, and thus they have different attributes in the catalog. To create textures, you add a new texture set to the asset catalog.

You’ll now replace the textures for the low poly house and ground and use textures from a catalog. Create a new file using the Asset Catalog template under Resource, and name it Textures. Remember to check both the iOS and macOS targets. With Textures.xcassets open, choose Editor ▸ Add Assets ▸ New Texture Set (or click the + at the bottom of the panel and choose New Texture Set). Double-click the Texture name and rename it to grass.

Open the Models ▸ Textures group and drag barn-ground.png to the Universal slot in your catalog. With the Attributes inspector open, click on the grass to see all of the texture options.

Here, you can see that by default, all mipmaps are created automatically. If you change Mipmap Levels to Fixed, you can choose how many levels to make. If you don’t like the automatic mipmaps, you can replace them with your own custom mipmaps by dragging them to the correct slot. Asset catalogs give you complete control of your textures without having to write cumbersome code; although you can still write the code using the MTLTextureDescriptor API if you want.

Open Texturable.swift, and replace:

print("Failed to load \(imageName)")
return nil

With:

print(
  "Failed to load \(imageName)\n - loading from Assets Catalog")
return try textureLoader.newTexture(name: imageName, 
                                    scaleFactor: 1.0,
                                    bundle: Bundle.main, 
                                    options: nil)

This now searches the bundle for the named image. When loading from the asset catalog, the options that you set in the attributes inspector take the place of most of the texture loading options, so these options are now nil. However, .textureUsage and .textureStorageMode options still have effect.

Note: You can reverse the read order of the textures to make the asset catalog the default.

The last thing to do is to make sure the model points to the new texture. Open plane.mtl located in Models ▸ Ground.

Replace:

map_Kd ground.png

With:

#map_Kd ground.png
map_Kd grass

Here, you commented out the old texture and added the new one. Build and run, and you now have a new grass texture loading.

Repeat this for the low poly house to change it in to a barn:

1. Create a new texture set in the asset catalog and rename it barn.

2. Drag lowpoly-barn-color.png into the texture set from the Models ▸ Textures group.

3. Change the name of the diffuse texture in Models ▸ LowPolyHouse ▸ lowpoly-house.mtl to barn.

Note: Be careful to drop the images on the texture’s Universal slot. If you drag the images into the asset catalog, they are, by default, images and not textures. And you won’t be able to make mipmaps on images or change the pixel format.

Build and run and your app to see your new textures.

You can see that the textures have reverted to the sRGB space because you’re now loading them in their original format. You can confirm this using the GPU debugger. If you want to avoid converting sRGB to linear in your shader, you can instead set up the texture to be data. In Textures.xcassets, click on the barn texture, and in the Attributes inspector, change the Interpretation to Data:

When your app loads the sRGB texture to a non-sRGB buffer, it automatically converts from sRGB space to linear space. (See Apple’s Metal Shading Language document for the conversion rule.) By accessing as data instead of colors, your shader can treat the color data as linear.

You’ll also notice in the above image that the origin, unlike loading the .png texture manually, is Top Left. The asset catalog loads textures differently.

Repeat for the grass texture.

Build and run, and your colors should now be correct.

The right texture for the right job

Using asset catalogs gives you complete control over how to deliver your textures. Currently, you only have two color textures. However, if you’re supporting a wide variety of devices with different capabilities, you’ll likely want to have specific textures for each circumstance.

For example, here is a list of individual textures you can assign by checking the different options in the Attributes inspector, for the Apple watch, 1x, 2x and 3x sizes, and sRGB and P3 displays.

Texture compression

In recent years, people have put in much effort towards compressing textures to save both CPU and GPU memory. There are various formats you can use, such as ETC and PVRTC. Apple has embraced ASTC as being the most high-quality compressed format. ASTC is available on the A8 chip and newer.

Using texture sets within the asset catalog allows your app to determine for itself which is the best format to use.

With your app running on macOS, take a look at how much memory it’s consuming. Click on the Debug navigator and select Memory. This is the usage after 30 seconds — your app’s memory consumption will increase for about five minutes and then stabilize:

If you capture the frame with the GPU Capture button, you’ll see that the texture format on the GPU is BC7_RGBAUnorm. When you use asset catalogs, Apple will automatically determine the most appropriate format for your texture.

In Textures.xcassets, select each of your textures and in the Attributes inspector, change the Pixel Format from Automatic to ASTC 8×8 Compressed - Red Green Blue Alpha. This is a highly compressed format. Build and run your app, and check the memory usage again.

You’ll see that the memory footprint is slightly reduced. However, so is the quality of the render. For distant textures, this quality might be fine, but you have to balance memory usage with render quality.

Note: You may have to test the app on an iOS device to see the change in texture format in the GPU Debugger. On iOS, the automatic format will be ASTC 4×4, which is indistinguishable from the png render.

Where to go from here?

In this chapter, you found out how to wrap a model in a texture, how to sample that texture in a shader and how to enhance your renders using mipmaps. You also learned how to use the invaluable GPU Frame Capture tool. The GPU Frame Capture tool is great for looking at what’s happening on the GPU and analyzing whether or not the shaders are performing the proper steps.

Topics such as color and compression are huge. In the Resources folder for this chapter, in references.markdown, you’ll find some recommended articles to read further.

But you’re not done with textures just yet. In fact, the barn stone wall texture looks a little flat. In the next chapter, you’ll find out how to make those stones pop out and appear to have dimension using a normal texture. You’ll also find out some of the other uses for textures, and you’ll examine material properties and how they react to physically based lighting.