Chapters
Chapters

Chapters

Hide chapters

Metal by Tutorials

Fourth Edition · macOS 14, iOS 17 · Swift 5.9 · Xcode 15

Before You Begin

Section 0: 6 chapters
Show chapters Hide chapters

Section I: Beginning Metal

Section 1: 10 chapters
Show chapters Hide chapters

Section II: Intermediate Metal

Section 2: 8 chapters
Show chapters Hide chapters

Section III: Advanced Metal

Section 3: 8 chapters
Show chapters Hide chapters

Section IV: Ray Tracing

Section 4: 7 chapters
Show chapters Hide chapters

21. Image-Based Lighting
Written by Caroline Begbie & Marius Horga

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

In this chapter, you’ll add the finishing touches to rendering your environment. You’ll add a cube around the outside of the scene that displays a sky texture. You’ll then use that sky texture to shade the models within the scene, making them appear as if they belong there.

Look at the following comparison of two renders.

The final and challenge renders
The final and challenge renders

This comparison demonstrates how you can use the same shader code but change the sky image to create different lighting environments. The rendered models reflect the tinge of color from the sky.

The Starter Project

➤ In Xcode, open the starter project for this chapter and build and run the app.

The starter project
The starter project

The project contains the forward renderer with transparency from the previous chapter. The scene uses an arcball camera, and contains a ground plane and car. The scene lighting consists of one sunlight.

There are a few additional files that you’ll use throughout the chapter. Common.h provides some extra texture indices for textures that you’ll create later.

Aside from the darkness of the lighting, there are some glaring problems with the render:

  • All metals, such as the metallic wheel hubs, look dull. Pure metals reflect their surroundings, and there are currently no surroundings to reflect.
  • Where the light doesn’t directly hit the car, the color is pure black. This happens because the app doesn’t provide any ambient light. Later in this chapter, you’ll use the skylight as global ambient light.

The Skybox

Currently, the sky is a single color, which looks unrealistic. By adding a 360º image surrounding the scene, you can easily place the action in a desert or have snowy mountains as a backdrop. To do this, you’ll create a skybox cube that surrounds the entire scene.

import MetalKit

struct Skybox {
  let mesh: MTKMesh
  var skyTexture: MTLTexture?
  let pipelineState: MTLRenderPipelineState
  let depthStencilState: MTLDepthStencilState?
}

init(textureName: String?) {
  let allocator =
    MTKMeshBufferAllocator(device: Renderer.device)
  let cube = MDLMesh(
    boxWithExtent: [1, 1, 1],
    segments: [1, 1, 1],
    inwardNormals: true,
    geometryType: .triangles,
    allocator: allocator)
  do {
    mesh = try MTKMesh(
      mesh: cube, device: Renderer.device)
  } catch {
    fatalError("failed to create skybox mesh")
  }
}

static func createSkyboxPSO(
  vertexDescriptor: MTLVertexDescriptor?
) -> MTLRenderPipelineState {
  let vertexFunction =
    Renderer.library?.makeFunction(name: "vertex_skybox")
  let fragmentFunction =
    Renderer.library?.makeFunction(name: "fragment_skybox")
  let pipelineDescriptor = MTLRenderPipelineDescriptor()
  pipelineDescriptor.vertexFunction = vertexFunction
  pipelineDescriptor.fragmentFunction = fragmentFunction
  pipelineDescriptor.colorAttachments[0].pixelFormat =
    Renderer.viewColorPixelFormat
  pipelineDescriptor.depthAttachmentPixelFormat = .depth32Float
  pipelineDescriptor.vertexDescriptor = vertexDescriptor
  return createPSO(descriptor: pipelineDescriptor)
}

static func buildDepthStencilState() -> MTLDepthStencilState? {
  let descriptor = MTLDepthStencilDescriptor()
  descriptor.depthCompareFunction = .lessEqual
  descriptor.isDepthWriteEnabled = true
  return Renderer.device.makeDepthStencilState(
    descriptor: descriptor)
}

pipelineState = PipelineStates.createSkyboxPSO(
  vertexDescriptor: MTKMetalVertexDescriptorFromModelIO(
    cube.vertexDescriptor))
depthStencilState = Self.buildDepthStencilState()

Rendering the Skybox

➤ Still in Skybox.swift, create a new method to perform the skybox rendering:

func render(
  encoder: MTLRenderCommandEncoder,
  uniforms: Uniforms
) {
  encoder.pushDebugGroup("Skybox")
  encoder.setRenderPipelineState(pipelineState)
  // encoder.setDepthStencilState(depthStencilState)
  encoder.setVertexBuffer(
    mesh.vertexBuffers[0].buffer,
    offset: 0,
    index: 0)
}

var uniforms = uniforms
uniforms.viewMatrix.columns.3 = [0, 0, 0, 1]
encoder.setVertexBytes(
  &uniforms,
  length: MemoryLayout<Uniforms>.stride,
  index: UniformsBuffer.index)

let submesh = mesh.submeshes[0]
encoder.drawIndexedPrimitives(
  type: .triangle,
  indexCount: submesh.indexCount,
  indexType: submesh.indexType,
  indexBuffer: submesh.indexBuffer.buffer,
  indexBufferOffset: 0)
encoder.popDebugGroup()

The Skybox Shader Functions

In the Shaders group, create a new Metal file named Skybox.metal.

#import "Common.h"

struct VertexIn {
  float4 position [[attribute(Position)]];
};

struct VertexOut {
  float4 position [[position]];
};

vertex VertexOut vertex_skybox(
  const VertexIn in [[stage_in]],
  constant Uniforms &uniforms [[buffer(UniformsBuffer)]])
{
  VertexOut out;
  float4x4 vp = uniforms.projectionMatrix * uniforms.viewMatrix;
  out.position = (vp * in.position).xyww;
  return out;
}

fragment half4 fragment_skybox(
  VertexOut in [[stage_in]]) {
  return half4(1, 1, 0, 1);
}

Integrating the Skybox Into the Scene

➤ Open GameScene.swift, and add a new property to GameScene:

let skybox: Skybox?

skybox = Skybox(textureName: nil)

scene.skybox?.render(
  encoder: renderEncoder,
  uniforms: uniforms)
A flickering sky
A pbozgeqink gyj

A yellow sky
O neckah xsz

Procedural Skies

Yellow skies might be appropriate on a different planet, but how about a procedural sky? A procedural sky is one built out of various parameters such as weather conditions and time of day. Model I/O provides a procedural generator which creates physically realistic skies.

A sunrise
U fegdopa

Cube Textures

Cube textures are similar to the 2D textures that you’ve already been using. 2D textures map to a quad and have two texture coordinates, whereas cube textures consist of six 2D textures: one for each face of the cube. You sample the textures with a 3D vector.

The sky texture in the asset catalog
Fpi kkc mitliyu ew shi owqos soyisur

Adding the Procedural Sky

You’ll use these sky textures shortly, but for now, you’ll add a procedural sky to your scene.

struct SkySettings {
  var turbidity: Float = 0.15
  var sunElevation: Float = 0.56
  var upperAtmosphereScattering: Float = 0.66
  var groundAlbedo: Float = 0.8
}
var skySettings = SkySettings()

func loadGeneratedSkyboxTexture(dimensions: SIMD2<Int32>)
  -> MTLTexture? {
  var texture: MTLTexture?
  let skyTexture = MDLSkyCubeTexture(
    name: "sky",
    channelEncoding: .float16,
    textureDimensions: dimensions,
    turbidity: skySettings.turbidity,
    sunElevation: skySettings.sunElevation,
    upperAtmosphereScattering:
      skySettings.upperAtmosphereScattering,
    groundAlbedo: skySettings.groundAlbedo)
  do {
    let textureLoader =
      MTKTextureLoader(device: Renderer.device)
    texture = try textureLoader.newTexture(
      texture: skyTexture,
      options: nil)
  } catch {
    print(error.localizedDescription)
  }
  return texture
}

if let textureName {
  // load named texture here
} else {
  skyTexture = loadGeneratedSkyboxTexture(dimensions: [256, 256])
}

encoder.setFragmentTexture(
  skyTexture,
  index: SkyboxTexture.index)

float3 textureCoordinates;
Skybox coordinates
Nlvwuf tiabnezanis

out.textureCoordinates = in.position.xyz;

fragment half4 fragment_skybox(
  VertexOut in [[stage_in]],
  texturecube<half> cubeTexture [[texture(SkyboxTexture)]])
{
  constexpr sampler default_sampler(filter::linear);
  half4 color = cubeTexture.sample(
    default_sampler,
    in.textureCoordinates);
  return color;
}
A procedural sky
E dkihazulex wcw

Custom Sky Textures

As mentioned earlier, you can use your own 360º sky textures. The textures included in the starter project were downloaded from Poly Haven — a great place to find environment maps. Before adding the texture to the asset catalog, the HDRI was converted into six tone mapped sky cube textures.

skyTexture = TextureController.loadCubeTexture(
  imageName: textureName)

skybox = Skybox(textureName: "sky")
The skybox
Fqa kzxfaq

Reflection

Now that you have something to reflect, you can easily implement reflection of the sky onto the car. When rendering the car, all you have to do is take the camera view direction, reflect it about the surface normal, and sample the skycube along the reflected vector for the fragment color for the car.

Reflection
Noxfagdear

let fragmentFunction =
  Renderer.library?.makeFunction(name: "fragment_IBL")
Color textures only
Hugar qahgisop ohhr

func update(encoder: MTLRenderCommandEncoder) {
  encoder.setFragmentTexture(
    skyTexture,
    index: SkyboxTexture.index)
}

scene.skybox?.update(encoder: renderEncoder)

texturecube<float> skybox [[texture(SkyboxTexture)]]

float3 viewDirection =
  in.worldPosition.xyz - params.cameraPosition;
viewDirection = normalize(viewDirection);
float3 textureCoordinates =
  reflect(viewDirection, normal);

constexpr sampler defaultSampler(filter::linear);
color = skybox.sample(
  defaultSampler, textureCoordinates);
float4 copper = float4(0.86, 0.7, 0.48, 1);
color = color * copper;
Reflections
Tuzkepboivm

Image-Based Lighting

At the beginning of the chapter, there were two problems with the original car render. By adding reflection, you probably now have an inkling of how you’ll fix the metallic reflection problem. The other problem is rendering the car as if it belongs in the scene with environment lighting. IBL or Image-Based Lighting is one way of dealing with this problem.

Diffuse Reflection

Light comes from all around us. Sunlight bounces around and colors reflect. When rendering an object, you should take into account the color of the light coming from every direction.

Diffuse reflection
Wophune lixxetjoet

var diffuseTexture: MTLTexture?

mutating func loadIrradianceMap() {
  // 1
  guard let skyCube =
    MDLTexture(cubeWithImagesNamed: ["cube-sky.png"])
  else { return }
  // 2
  let irradiance =
    MDLTexture.irradianceTextureCube(
      with: skyCube,
      name: nil,
      dimensions: [64, 64],
      roughness: 0.6)
  // 3
  let loader = MTKTextureLoader(device: Renderer.device)
  do {
    diffuseTexture = try loader.newTexture(
    texture: irradiance,
    options: nil)
  } catch {
    fatalError(error.localizedDescription)
  }
}

loadIrradianceMap()

encoder.setFragmentTexture(
  diffuseTexture,
  index: SkyboxDiffuseTexture.index)

texturecube<float> skyboxDiffuse [[texture(SkyboxDiffuseTexture)]]

float4 diffuse = skyboxDiffuse.sample(textureSampler, normal);
color = diffuse * float4(material.baseColor, 1);
Diffuse from irradiance
Fejmahu jtug akrebiigsa

diffuseTexture =
  TextureController.loadCubeTexture(
    imageName: "irradiance.png")
Brighter irradiance
Nqikzjax inbadeusvu

Specular Reflection

The irradiance map provides the diffuse and ambient reflection, but the specular reflection is a bit more difficult.

Specular reflection
Sjanoxuj cunzexjiif

Pre-filtered environment maps
Lqo-wimlitiv opxugowtorg juqy

BRDF Look-Up Table

To calculate the final color, you use a Bidirectional Reflectance Distribution Function (BRDF) that takes in the actual roughness of the model and the current viewing angle and returns the scale and bias for the Fresnel and geometric attenuation contributions.

A BRDF LUT
A JNKS WAF

var brdfLut: MTLTexture?

brdfLut = Renderer.buildBRDF()

encoder.setFragmentTexture(
  brdfLut,
  index: BRDFLutTexture.index)
BRDF LUT is on the GPU
DCZD YIR oh ag wze CXO

texture2d<float> brdfLut [[texture(BRDFLutTexture)]]

// 1
constexpr sampler s(filter::linear, mip_filter::linear);
float3 prefilteredColor
  = skybox.sample(s,
                  textureCoordinates,
                  level(material.roughness * 10)).rgb;
// 2
float nDotV = saturate(dot(normal, -viewDirection));
float2 envBRDF
  = brdfLut.sample(s, float2(material.roughness, nDotV)).rg;
return float4(envBRDF, 0, 1);
The BRDF look up result
Dmi RQTK duiz of botohx

Fresnel Reflectance

When light hits an object straight on, some of the light is reflected. The amount of reflection is known as Fresnel zero, or F0, and you can calculate this from the material’s index of refraction, or IOR.

float3 f0 = mix(0.04, material.baseColor.rgb, material.metallic);
float3 specularIBL = f0 * envBRDF.r + envBRDF.g;

float3 specular = prefilteredColor * specularIBL;
color += float4(specular, 1);
return color;
Diffuse and specular
Zifkuwi ohp sjilezup

Tweaking

Being able to tweak shaders gives you complete power over how your renders look. Because you’re using low dynamic range lighting, the non-metal diffuse color looks a bit dark. You can tweak the color very easily.

diffuse = mix(pow(diffuse, 0.2), diffuse, material.metallic);
diffuse *= calculateShadow(in.shadowPosition, shadowTexture);
Tweaking the shader
Tceozuyw wzo ygavuy

Ambient Occlusion Maps

Ambient occlusion is a technique that approximates how much light should fall on a surface. If you look around you — even in a bright room — where surfaces are very close to each other, they’re darker than exposed surfaces. In Chapter 28, “Advanced Shadows”, you’ll learn how to generate global ambient occlusion using ray marching, but assigning pre-built local ambient occlusion maps to models is a fast and effective alternative.

color *= material.ambientOcclusion;

Challenge

On the first page of this chapter is a comparison of the car rendered in two different lighting situations. Your challenge is to create the red lighting scene.

Key Points

  • Using a cuboid skybox, you can surround your scene with a texture.
  • Model I/O has a feature to produce procedural skies which includes turbidity, sun elevation, upper atmosphere scattering and ground albedo.
  • Cube textures have six faces. Each of the faces can have mipmaps.
  • Simply by reflecting the view vector, you can sample the skybox texture and reflect it on your models.
  • Image-based lighting uses the sky texture for lighting. You derive the diffuse color from a convoluted irradiance map, and the specular from a Bidirectional Reflectance Distribution Function (BRDF) look-up table.

Where to Go From Here?

You’ve dipped a toe into the water of the great sea of realistic rendering. If you want to explore more about this fascinating topic, references.markdown in the resources folder for this chapter, contains links to interesting articles and videos.

20. Fragment Post-Processing
22. Reflection & Refraction
Have a technical question? Want to report a bug? You can ask questions and report bugs to the book authors in our official book forum here.
© 2024 Kodeco Inc.

You’re accessing parts of this content for free, with some sections shown as scrambled text. Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now