Chapters
Chapters

Chapters

Hide chapters

Metal by Tutorials

Second Edition · iOS 13 · Swift 5.1 · Xcode 11

Before You Begin

Section 0: 3 chapters
Show chapters Hide chapters

Section I: The Player

Section 1: 8 chapters
Show chapters Hide chapters

Section II: The Scene

Section 2: 7 chapters
Show chapters Hide chapters

Section III: The Effects

Section 3: 10 chapters
Show chapters Hide chapters

8. Character Animation
Written by Caroline Begbie

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

Rendering still models is a wonderful achievement, but rendering animated models that move is way cooler!

To animate means to bring to life. So what better way to play with animation than to render characters with personality and body movement. In this chapter, you’ll start off by bouncing a ball. Then, you’ll move on to rendering a friendly-looking skeleton.

In earlier chapters, you focused on shading in the fragment function. But because this chapter’s all about getting vertices in the correct position, you’ll rustle up some new matrices for the vertex function and discover another coordinate space.

The starter project

Open the starter project for this chapter, and review the changes from the previous chapter’s completed project:

  • To keep your attention on vertex shading, Shaders.metal contains only the vertex shader function. You won’t be dealing with materials or textures so PBR.metal holds the fragment shading.

  • Look at Renderer‘s draw(in:). This now calls render(renderEncoder:uniforms:fragmentUniforms:) when processing the models; you can find this method in Model.swift. Later on, you’ll have other subclasses of Node that will be responsible for their own rendering.

  • Two debugging statements: renderEncoder.pushDebugGroup(model.name) and renderEncoder.popDebugGroup() surround the new render method. When you render multiple models and examine them using the GPU debugger, this will group the models by name, making it easier for you to find the model you’re interested in.

  • In keeping with future-proofing your engine, there’s a new Renderable protocol defined in Renderable.swift. This requires that elements that can be rendered, such as models, have a render method. Model conforms to this protocol using the code that used to be in draw(in:).

  • There’s a new group called “Animation Support”. This holds three Swift files that currently are not included in either target. You’ll add these later when you come to animating a jointed skeleton.

Run the starter project, and you’ll see a beach ball. Notice how unnatural it looks just sitting there. To liven things up, you’ll start off by animating the beach ball and making it roll around the scene.

Procedural animation

Earlier, in Chapter 3, “The Rendering Pipeline,” you animated a cube and a train using sine. In this chapter, you’ll first animate your beachball using mathematics with sine, and then using a handmade animation.

var currentTime: Float = 0

func update(deltaTime: Float) {
  currentTime += deltaTime * 4
  let ball = models[0]
  ball.position.x = sin(currentTime)
}

let deltaTime = 1 / Float(view.preferredFramesPerSecond)
update(deltaTime: deltaTime)

Animation using physics

Instead of creating animation by hand using an animation app, using physics-based animation means that your models can simulate the real world. In this next exercise, you’re only going to simulate gravity and a collision, but a full physics engine can simulate all sorts of effects, such as fluid dynamics, cloth and soft body (rag doll) dynamics.

var ballVelocity: Float = 0

ball.position.x = sin(currentTime)

let gravity: Float = 9.8 // meter / sec2
let mass: Float = 0.05
let acceleration = gravity / mass
let airFriction: Float = 0.2
let bounciness: Float = 0.9
let timeStep: Float = 1 / 600

ballVelocity += (acceleration * timeStep) / airFriction
ball.position.y -= ballVelocity * timeStep
if ball.position.y <=  0.35 {     // collision with ground
  ball.position.y = 0.35
  ballVelocity = ballVelocity * -1 * bounciness
}

ball.position = [0, 3, 0]

Axis-aligned bounding box

You hard-coded the ball’s radius so that it collides with the ground, but collision systems generally require some kind of bounding box to test whether an object is impacted.

var boundingBox = MDLAxisAlignedBoundingBox()
var size: float3 {
  return boundingBox.maxBounds - boundingBox.minBounds
}

self.boundingBox = asset.boundingBox

if ball.position.y <= ball.size.y / 2 { // collision with ground
  ball.position.y = ball.size.y / 2
  ballVelocity = ballVelocity * -1 * bounciness
}

Keyframes

If you want to animate the ball being tossed around, you’ll need to input information about its position over time. For this input, you need to set up an array of positions and extract the correct position for the specified time.

func update(deltaTime: Float) {
  currentTime += deltaTime
  let ball = models[0]
  ball.position.y = ball.size.y

  let fps: Float = 60 
  let currentFrame = 
       Int(currentTime * fps) % (ballPositionXArray.count)
  ball.position.x = ballPositionXArray[currentFrame]
}

Interpolation

It’s a lot of work inputting a value for each frame. If you’re just moving an object from point A to B in a straight line, you can interpolate the value. Interpolation is where you can calculate a value given a range of values and a current location within the range. When animating, the current location is the current time as a percentage of the animation duration.

struct Keyframe {
  var time: Float = 0
  var value: float3 = [0, 0, 0]
}

struct Animation {
  var translations: [Keyframe] = []
  var repeatAnimation = true
}

func getTranslation(at time: Float) -> float3? {
  guard let lastKeyframe = translations.last else {
    return nil
  }
}

//1
var currentTime = time
if let first = translations.first, first.time >= currentTime {
  return first.value
}
//2
if currentTime >= lastKeyframe.time, !repeatAnimation {
  return lastKeyframe.value
}

// 1
currentTime = fmod(currentTime, lastKeyframe.time)
// 2
let keyFramePairs = translations.indices.dropFirst().map {
  (previous: translations[$0 - 1], next: translations[$0])
}
// 3
guard let (previousKey, nextKey) = ( keyFramePairs.first {
  currentTime < $0.next.time
})
else { return nil }
// 4
let interpolant = (currentTime - previousKey.time) / 
                  (nextKey.time - previousKey.time)
// 5
return simd_mix(previousKey.value, 
                nextKey.value,
                float3(repeating: interpolant))             

func update(deltaTime: Float) {
  currentTime += deltaTime
  let ball = models[0]
  var animation = Animation()
  animation.translations = generateBallTranslations()
  ball.position = animation.getTranslation(at: currentTime) 
                  ?? [0, 0, 0]
  ball.position.y += ball.size.y
}

Euler angle rotations

Now that you have the ball translating through the air, you probably want to rotate it as well. To express rotation of an object, you currently hold a float3 with rotation angles on x, y and z axes. These are called Euler angles after the mathematician Leonhard Euler. Euler is the one behind Euler’s rotation theorem, a theorem which states that any rotation can be described using three rotation angles. This is OK for a single rotation, but interpolating between these three values doesn’t work in a way that you may think.

init(rotation angle: float3) {
  let rotationX = float4x4(rotationX: angle.x)
  let rotationY = float4x4(rotationY: angle.y)
  let rotationZ = float4x4(rotationZ: angle.z)
  self = rotationX * rotationY * rotationZ
}

Quaternions

Multiplying x, y and z rotations without compelling a sequence on them is impossible unless you involve the fourth dimension. In 1843, Sir William Rowan Hamilton did just that! He inscribed his fundamental formula for quaternion multiplication on to a stone on a bridge in Dublin:

var quaternion = simd_quatf()

let rotateMatrix = float4x4(quaternion)

var rotation: float3 = [0, 0, 0] {
  didSet {
    let rotationMatrix = float4x4(rotation: rotation)
    quaternion = simd_quatf(rotationMatrix)
  }
}

struct KeyQuaternion {
  var time: Float = 0
  var value = simd_quatf()
}

var rotations: [KeyQuaternion] = []

func getRotation(at time: Float) -> simd_quatf? {
  guard let lastKeyframe = rotations.last else {
    return nil
  }
  var currentTime = time
  if let first = rotations.first, first.time >= currentTime {
    return first.value
  }
  if currentTime >= lastKeyframe.time, !repeatAnimation {
    return lastKeyframe.value
  }
  currentTime = fmod(currentTime, lastKeyframe.time)
  let keyFramePairs = rotations.indices.dropFirst().map {
    (previous: rotations[$0 - 1], next: rotations[$0])
  }
  guard let (previousKey, nextKey) = ( keyFramePairs.first {
    currentTime < $0.next.time
  })
  else { return nil }
  let interpolant = (currentTime - previousKey.time) /
                    (nextKey.time - previousKey.time)
  return simd_slerp(previousKey.value,
                    nextKey.value,
                    interpolant)
}

func update(deltaTime: Float) {
  currentTime += deltaTime
  let ball = models[0]
    
  var animation = Animation()
  animation.translations = generateBallTranslations()
  animation.rotations = generateBallRotations()
  ball.position = animation.getTranslation(at: currentTime) 
                  ?? float3(repeating: 0)
  ball.position.y += ball.size.y / 2
  ball.quaternion = animation.getRotation(at: currentTime) 
                  ?? simd_quatf()
}

USD and USDZ files

One major problem to overcome is how to import animation from 3D apps. Model I/O can import .obj files, but they only hold static information, not animation. USD is a format devised by Pixar, which can hold massive scenes with textures, animation and lighting information. There are various file extensions:

Animating meshes

The file beachball.usda holds translation and rotation animation, and Model I/O can extract this animation. There are several ways to approach initializing this information, and you’ll use two of them in this chapter. Model I/O transform components don’t allow you to access the rotation and translation values directly, but provides you with a method that returns a transform matrix at a particular time. So for mesh transform animation you’ll extract the animation data for every frame of the animation during the model loading process.

static var fps: Int!

Renderer.fps = metalView.preferredFramesPerSecond

let deltaTime = 1 / Float(view.preferredFramesPerSecond)
update(deltaTime: deltaTime)

let deltaTime = 1 / Float(Renderer.fps)
for model in models {
  model.update(deltaTime: deltaTime)
}

import ModelIO

class TransformComponent {
  let keyTransforms: [float4x4]
  let duration: Float
  var currentTransform: float4x4 = .identity()
}

init(transform: MDLTransformComponent,
     object: MDLObject,
     startTime: TimeInterval,
     endTime: TimeInterval) {
  duration = Float(endTime - startTime)
  let timeStride = stride(from: startTime,
                          to: endTime,
                          by: 1 / TimeInterval(Renderer.fps))
  keyTransforms = Array(timeStride).map { time in
    return MDLTransform.globalTransform(with: object, 
                                        atTime: time)
  }
}

func setCurrentTransform(at time: Float) {
  guard duration > 0 else {
    currentTransform = .identity()
    return
  }
  let frame = Int(fmod(time, duration) * Float(Renderer.fps))
  if frame < keyTransforms.count {
    currentTransform = keyTransforms[frame]
  } else {
    currentTransform = keyTransforms.last ?? .identity()
  }
}

init(mdlMesh: MDLMesh, mtkMesh: MTKMesh,
     startTime: TimeInterval,
     endTime: TimeInterval) 

Mesh(mdlMesh: $0.0, mtkMesh: $0.1,
     startTime: asset.startTime,
     endTime: asset.endTime)

let transform: TransformComponent?

if let mdlMeshTransform = mdlMesh.transform {
  transform = TransformComponent(transform: mdlMeshTransform,
                                 object: mdlMesh,
                                 startTime: startTime,
                                 endTime: endTime)
} else {
  transform = nil
}

var currentTime: Float = 0

override func update(deltaTime: Float) {
  currentTime += deltaTime
  for mesh in meshes {
    mesh.transform?.setCurrentTransform(at: currentTime)
  }
}

let currentLocalTransform = 
  mesh.transform?.currentTransform ?? .identity()
uniforms.modelMatrix = modelMatrix * currentLocalTransform

ball.position = [0, 3, 0]
ball.scale = [100, 100, 100]

Blender for animating

Imagine creating a walk cycle for a human figure by typing out keyframes! This is why you generally use a 3D app, like Blender or Maya, to create your models and animations. You then export those to your game or rendering engine of choice.

Weight painting in Blender

Left-click the skeleton’s head. At the bottom of the Blender window, click on the drop-down that currently reads Object Mode, and change it to Weight Paint.

Animation in Blender

Select the drop-down at the bottom of the window that currently reads Weight Paint, and go back into Object Mode. Press the space bar to start an animation. Your skeleton should now get friendly and wave at you. This wave animation is a 60 frame looping animation clip.

Skeletal Animation

Importing a skeletal animation into your app is a bit more difficult than importing a simple .obj file or a USDZ file with transform animation, because you have to deal with the joint hierarchy and joint weighting. You’ll read in the data from the USD file and restructure it to fit your rendering code. This is how the objects will fit together in your app:

let ball = Model(name: "beachball.usda")
models.append(ball)

let skeleton = Model(name: "skeletonWave.usda")
skeleton.rotation = [0, .pi, 0]
models.append(skeleton)

let animations: [String: AnimationClip]

// animations
let assetAnimations = asset.animations.objects.compactMap {
  $0 as? MDLPackedJointAnimation
}
let animations = Dictionary(uniqueKeysWithValues: assetAnimations.map {
  ($0.name, AnimationComponent.load(animation: $0))
})
self.animations = animations

for animation in animations {
  print("Animation: ", animation.key)
}


let skeleton: Skeleton?

let skeleton = 
      Skeleton(animationBindComponent: 
           (mdlMesh.componentConforming(to: MDLComponent.self) 
           as? MDLAnimationBindComponent))
self.skeleton = skeleton

skeleton?.jointPaths.map {
  print($0)
}

Loading the animation

To update the skeleton’s pose every frame, you’ll create a method that takes the animation clip and iterates through the joints to update each joint’s position for the frame. First you’ll create a method on AnimationClip that gets the pose for a joint at a particular time. This will use the interpolation methods that you’ve already created in Animation.

func getPose(at time: Float, jointPath: String) -> float4x4? {
  guard
    let jointAnimation = jointAnimation[jointPath] ?? nil
    else { return nil }
  let rotation = 
      jointAnimation.getRotation(at: time) ?? simd_quatf()
  let translation = 
      jointAnimation.getTranslation(at: time) 
               ?? float3(repeating: 0)
  let pose = float4x4(translation: translation) 
               * float4x4(rotation)
  return pose
}

Joint matrix palette

You’re now able to get the pose of a joint. However, each vertex is weighted to up to four joints. You saw this in the earlier elbow example, where some vertices belonging to the lower arm joint would get 50% of the upper arm joint`s rotation.

func updatePose(animationClip: AnimationClip,
                at time: Float) {
}

guard let paletteBuffer = jointMatrixPaletteBuffer 
    else { return }
var palettePointer = 
      paletteBuffer.contents().bindMemory(to: float4x4.self,
                                     capacity: jointPaths.count)
palettePointer.initialize(repeating: .identity(),
                          count: jointPaths.count)
var poses = [float4x4](repeatElement(.identity(), 
                       count: jointPaths.count))

for (jointIndex, jointPath) in jointPaths.enumerated() {
  // 1
  let pose = 
      animationClip.getPose(at: time * animationClip.speed,
                            jointPath: jointPath) 
                            ?? restTransforms[jointIndex]
  // 2
  let parentPose: float4x4
  if let parentIndex = parentIndices[jointIndex] {
    parentPose = poses[parentIndex]
  } else {
    parentPose = .identity()
  }
  poses[jointIndex] = parentPose * pose
}

The inverse bind matrix

Examine the properties held on Skeleton. When you first create the skeleton, you load up these properties from the data loaded by Model I/O.

palettePointer.pointee = 
    poses[jointIndex] * bindTransforms[jointIndex].inverse
palettePointer = palettePointer.advanced(by: 1)


for mesh in meshes {
  mesh.transform?.setCurrentTransform(at: currentTime)
}

for mesh in meshes {
  if let animationClip = animations.first?.value {
    mesh.skeleton?.updatePose(animationClip: animationClip,
                              at: currentTime)
    mesh.transform?.currentTransform = .identity()
  } else {
    mesh.transform?.setCurrentTransform(at: currentTime)
  }
}

if let paletteBuffer = mesh.skeleton?.jointMatrixPaletteBuffer {
  renderEncoder.setVertexBuffer(paletteBuffer, offset: 0, 
                                index: 22)
}

ushort4 joints [[attribute(Joints)]];
float4 weights [[attribute(Weights)]];

Vertex function constants

In Shaders.metal, add a new function constant after importing Common.h.

constant bool hasSkeleton [[function_constant(5)]];

vertex VertexOut 
       vertex_main(const VertexIn vertexIn [[stage_in]],
  constant float4x4 *jointMatrices [[buffer(22),
                             function_constant(hasSkeleton)]],
  constant Uniforms &uniforms [[buffer(BufferIndexUniforms)]])

float4 position = vertexIn.position;
float4 normal = float4(vertexIn.normal, 0);

if (hasSkeleton) {
  float4 weights = vertexIn.weights;
  ushort4 joints = vertexIn.joints;
  position =
      weights.x * (jointMatrices[joints.x] * position) +
      weights.y * (jointMatrices[joints.y] * position) +
      weights.z * (jointMatrices[joints.z] * position) +
      weights.w * (jointMatrices[joints.w] * position);
  normal =
      weights.x * (jointMatrices[joints.x] * normal) +
      weights.y * (jointMatrices[joints.y] * normal) +
      weights.z * (jointMatrices[joints.z] * normal) +
      weights.w * (jointMatrices[joints.w] * normal);
}

VertexOut out {
  .position = uniforms.projectionMatrix * uniforms.viewMatrix 
                * uniforms.modelMatrix * position,
  .worldPosition = (uniforms.modelMatrix * position).xyz,
  .worldNormal = uniforms.normalMatrix * normal.xyz,
  .worldTangent = 0,
  .worldBitangent = 0,
  .uv = vertexIn.uv
};

init(mdlSubmesh: MDLSubmesh, mtkSubmesh: MTKSubmesh, 
     hasSkeleton: Bool) {

static func makePipelineState(textures: Textures, 
                              hasSkeleton: Bool) 
  -> MTLRenderPipelineState {

pipelineState = 
    Submesh.makePipelineState(textures: textures, 
                              hasSkeleton: hasSkeleton)

Submesh(mdlSubmesh: mesh.0 as! MDLSubmesh, 
        mtkSubmesh: mesh.1,
        hasSkeleton: skeleton != nil)

let vertexFunction: MTLFunction?

let constantValues = 
    makeVertexFunctionConstants(hasSkeleton: hasSkeleton)
vertexFunction = 
    try library?.makeFunction(name: "vertex_main",
                              constantValues: constantValues)

skeleton.rotation = [0, .pi, 0]

skeleton.rotation = [.pi / 2, .pi, 0]
skeleton.scale = [100, 100, 100]

Where to go from here?

This chapter took you through the basics of character animation. But don’t stop there! There are so many different topics that you can investigate. For instance, you can:

7. Maps & Materials
9. The Scene Graph
Have a technical question? Want to report a bug? You can ask questions and report bugs to the book authors in our official book forum here.
© 2024 Kodeco Inc.

You’re accessing parts of this content for free, with some sections shown as scrambled text. Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now