8. Character Animation

Written by Caroline Begbie

Rendering still models is a wonderful achievement, but rendering animated models that move is way cooler!

To animate means to bring to life. So what better way to play with animation than to render characters with personality and body movement. In this chapter, you’ll start off by bouncing a ball. Then, you’ll move on to rendering a friendly-looking skeleton.

In earlier chapters, you focused on shading in the fragment function. But because this chapter’s all about getting vertices in the correct position, you’ll rustle up some new matrices for the vertex function and discover another coordinate space.

The starter project

Open the starter project for this chapter, and review the changes from the previous chapter’s completed project:

• To keep your attention on vertex shading, Shaders.metal contains only the vertex shader function. You won’t be dealing with materials or textures so PBR.metal holds the fragment shading.

• Look at Renderer‘s draw(in:). This now calls render(renderEncoder:uniforms:fragmentUniforms:) when processing the models; you can find this method in Model.swift. Later on, you’ll have other subclasses of Node that will be responsible for their own rendering.

• Two debugging statements: renderEncoder.pushDebugGroup(model.name) and renderEncoder.popDebugGroup() surround the new render method. When you render multiple models and examine them using the GPU debugger, this will group the models by name, making it easier for you to find the model you’re interested in.

• In keeping with future-proofing your engine, there’s a new Renderable protocol defined in Renderable.swift. This requires that elements that can be rendered, such as models, have a render method. Model conforms to this protocol using the code that used to be in draw(in:).

• There’s a new group called “Animation Support”. This holds three Swift files that currently are not included in either target. You’ll add these later when you come to animating a jointed skeleton.

Run the starter project, and you’ll see a beach ball. Notice how unnatural it looks just sitting there. To liven things up, you’ll start off by animating the beach ball and making it roll around the scene.

Procedural animation

Earlier, in Chapter 3, “The Rendering Pipeline,” you animated a cube and a train using sine. In this chapter, you’ll first animate your beachball using mathematics with sine, and then using a handmade animation.

Open Renderer.swift and create a new property in Renderer:

var currentTime: Float = 0

Add a new method to update objects every frame:

func update(deltaTime: Float) {
  currentTime += deltaTime * 4
  let ball = models[0]
  ball.position.x = sin(currentTime)
}

This updates the ball’s x position every frame by the sine of the accumulated current time. Sine is great for procedural animation; by changing the amplitude, period and frequency, you can create waves of motion, although, for a beach ball, that’s not too realistic. However, with some physics, you can add a little bounce to help.

Now, call this method at the top of draw(in:), after the guard statement:

let deltaTime = 1 / Float(view.preferredFramesPerSecond)
update(deltaTime: deltaTime)

The property view.preferredFramesPerSecond defaults to 60 frames per second (that’s 16.6 milliseconds per frame). Your app needs to ensure that your entire render for each frame fits within this time; otherwise, you should choose a lower frame rate. You’ll use deltaTime to update your animations for each frame.

Build and run, and the beach ball now moves from side-to-side.

Animation using physics

Instead of creating animation by hand using an animation app, using physics-based animation means that your models can simulate the real world. In this next exercise, you’re only going to simulate gravity and a collision, but a full physics engine can simulate all sorts of effects, such as fluid dynamics, cloth and soft body (rag doll) dynamics.

Create a new property in Renderer to track the ball’s velocity:

var ballVelocity: Float = 0

Remove this code from update(deltaTime:):

ball.position.x = sin(currentTime)

In update(deltaTime:), set some constants for the individual physics you’ll need for the simulation:

let gravity: Float = 9.8 // meter / sec2
let mass: Float = 0.05
let acceleration = gravity / mass
let airFriction: Float = 0.2
let bounciness: Float = 0.9
let timeStep: Float = 1 / 600

gravity represents the acceleration of an object falling to Earth. If you’re simulating gravity elsewhere in the universe, for example, Mars, this value would be different.

Newton’s Second Law of Motion is F = ma or force = mass * acceleration. Rearranging the equation gives acceleration = force (gravity) / mass.

The other constants describe the surroundings and the properties of the beach ball. If this were a bowling ball, it would have a higher mass and less bounce.

Add this code at the end of update(deltaTime:):

ballVelocity += (acceleration * timeStep) / airFriction
ball.position.y -= ballVelocity * timeStep
if ball.position.y <=  0.35 {     // collision with ground
  ball.position.y = 0.35
  ballVelocity = ballVelocity * -1 * bounciness
}

Here, you calculate the position of the ball based on the ball’s current velocity; you can read more about this in Chapter 16, “Particle Systems.” The ball’s origin is at its center, and it’s approximately 0.7 units in diameter, so when the ball’s center is 0.35 units above the ground, that’s when you reverse the velocity and travel upward.

In init(metalView:), change the ball’s initial position to be up in the air:

ball.position = [0, 3, 0]

Build and run, and watch the beach ball bounce around.

This is a simple physics animation, but it demonstrates the possibilities if you choose to take it further.

Axis-aligned bounding box

You hard-coded the ball’s radius so that it collides with the ground, but collision systems generally require some kind of bounding box to test whether an object is impacted.

Obviously, the ball would benefit from a spherical bounding volume, and you can investigate physics engines, such as the open source Bullet physics engine. However, as your ball is simply using the y-axis, you can determine the ball’s height using an axis-aligned bounding box that Model I/O calculates.

Open Node.swift, and add a bounding box property and a computed size property to Node:

var boundingBox = MDLAxisAlignedBoundingBox()
var size: float3 {
  return boundingBox.maxBounds - boundingBox.minBounds
}

In Model.swift, in init(name:), after calling super.init(), add the following line of code:

self.boundingBox = asset.boundingBox

This extracts the bounding box information from Model I/O.

In Renderer.swift, in update(deltaTime:), update the collision code with the correct size calculated from the bounding box value:

if ball.position.y <= ball.size.y / 2 { // collision with ground
  ball.position.y = ball.size.y / 2
  ballVelocity = ballVelocity * -1 * bounciness
}

Build and run, and your ball will collide with the ground, precisely at the edge of the ball.

Keyframes

If you want to animate the ball being tossed around, you’ll need to input information about its position over time. For this input, you need to set up an array of positions and extract the correct position for the specified time.

In BallAnimations.swift, in the Utility group, there’s already an array set up named ballPositionXArray. This array consists of 60 values ranging from -1 to 1, then back to -1. By calculating the current frame, you can grab the correct x position from the array.

In Renderer.swift, change update(deltaTime:) to:

func update(deltaTime: Float) {
  currentTime += deltaTime
  let ball = models[0]
  ball.position.y = ball.size.y

  let fps: Float = 60 
  let currentFrame = 
       Int(currentTime * fps) % (ballPositionXArray.count)
  ball.position.x = ballPositionXArray[currentFrame]
}

Here, you calculate the current frame working on a 60 fps (frames per seconds) basis, and you extract the correct value from the array.

Build and run. Watch as the ball moves around in a mechanical, backward and forward motion over 60 frames. This is almost the same result as the sine animation, but it’s now animated by an array of values that you can control and re-use.

Interpolation

It’s a lot of work inputting a value for each frame. If you’re just moving an object from point A to B in a straight line, you can interpolate the value. Interpolation is where you can calculate a value given a range of values and a current location within the range. When animating, the current location is the current time as a percentage of the animation duration.

To work out the time percentage, use this formula:

This results in a time value between 0 and 1.

For example, if you have a start value of 5, an end value of 10 and a duration of 2 seconds, after 1 second has passed, the interpolated value is 7.5.

That’s a linear interpolation. However, there are other ways of interpolating using different formulas.

In the above image, linear interpolation is on the left. The x-axis is time, and the y-axis is value. You sample the value at the appropriate time. The previous ball example was mechanical and can be improved using the ease in and out interpolation on the right.

With ease in and out, the ball gains speed slowly at the start and then slows down toward the end of the animation.

Instead of creating one value for every frame to animate your ball, you’ll hold only key positions. These will be the extreme of a pose. In the ball example, the extreme pose positions are -1 and 1. You’ll also hold key times that match the time of that key value. For example, if your animation is 2 seconds long, your extremes will be at 0 seconds for the starting pose on the left, 1 second for the ball to be on the right, and 2 seconds to take the ball back to the left again. All of the frames in between these times are interpolated.

Create a new Swift file named Animation.swift. Remember to add the file to both macOS and iOS targets. Here, you’ll hold your animation data, and create methods to return the interpolated value at a given time.

Add the following:

struct Keyframe {
  var time: Float = 0
  var value: float3 = [0, 0, 0]
}

This creates a struct to hold the animation key values and times. Now add this:

struct Animation {
  var translations: [Keyframe] = []
  var repeatAnimation = true
}

This struct holds an array of keyframes; repeatAnimation will be used to choose to repeat the animation clip forever or play it just once.

Within Animation, add this:

func getTranslation(at time: Float) -> float3? {
  guard let lastKeyframe = translations.last else {
    return nil
  }
}

This will return the interpolated keyframe. First, you ensure that there are translation keys in the array, otherwise, return a nil float value.

Continue adding to this method with the following:

//1
var currentTime = time
if let first = translations.first, first.time >= currentTime {
  return first.value
}
//2
if currentTime >= lastKeyframe.time, !repeatAnimation {
  return lastKeyframe.value
}

Here’s the breakdown:

1. If the first keyframe occurs on or after the time given, then return the first key value. The first frame of an animation clip should be at keyframe 0 to give a starting pose.

2. If the time given is greater than the last key time in the array, then check whether you should repeat the animation. If not, then return the last value.

Now add this after the previous code:

// 1
currentTime = fmod(currentTime, lastKeyframe.time)
// 2
let keyFramePairs = translations.indices.dropFirst().map {
  (previous: translations[$0 - 1], next: translations[$0])
}
// 3
guard let (previousKey, nextKey) = ( keyFramePairs.first {
  currentTime < $0.next.time
})
else { return nil }
// 4
let interpolant = (currentTime - previousKey.time) / 
                  (nextKey.time - previousKey.time)
// 5
return simd_mix(previousKey.value, 
                nextKey.value,
                float3(repeating: interpolant))             

Going through this code:

1. Use the modulo operation to get the current time within the clip.

2. Create a new array of tuples containing the previous and next keys for all keyframes, except the first one.

3. Find the first tuple of previous and next keyframes where the current time is less than the next keyframe time. The current time will, therefore, be between the previous and next keyframe times.

4. Use the interpolation formula to get a value between 0 and 1 for the progress percentage between the previous and next keyframe times.

5. Use the simd_mix function to interpolate between the two keyframes. (interpolant must be a value between 0 and 1.)

In BallAnimations.swift in the Utility group, uncomment generateBallTranslations(). This method creates an array of Keyframes with seven keys. The length of the clip is 2 seconds. You can see this by looking at the key time of the last keyframe.

In the x-axis, the ball will start off at position -1 and then move to position 1 at 0.35 seconds. It will hold its position until 1 second has passed, then return to -1 at 1.35 seconds. It will then hold its position until the end of the clip.

By changing the values in the array, you can speed up the throw and hold the ball for longer at either end. In Renderer.swift, change update(deltaTime:) to:

func update(deltaTime: Float) {
  currentTime += deltaTime
  let ball = models[0]
  var animation = Animation()
  animation.translations = generateBallTranslations()
  ball.position = animation.getTranslation(at: currentTime) 
                  ?? [0, 0, 0]
  ball.position.y += ball.size.y
}

Here, you load the animation clip with the generated keyframe translations. Generally, you’ll want to load the animation clip outside of the update, but for the sake of simplicity, in this example, handling things within update(deltaTime:) is fine. You also extract the ball’s position from the animation clip for the current time.

Build and run, and watch as creepy invisible hands toss your ball around.

Note: Notice the trajectory of the ball on the y-axis. It currently goes up and down in diagonal straight lines. Better keyframing can fix this.

Euler angle rotations

Now that you have the ball translating through the air, you probably want to rotate it as well. To express rotation of an object, you currently hold a float3 with rotation angles on x, y and z axes. These are called Euler angles after the mathematician Leonhard Euler. Euler is the one behind Euler’s rotation theorem, a theorem which states that any rotation can be described using three rotation angles. This is OK for a single rotation, but interpolating between these three values doesn’t work in a way that you may think.

To create a rotation matrix, you’ve been calling this function, hidden in the math library in MathLibrary.swift:

init(rotation angle: float3) {
  let rotationX = float4x4(rotationX: angle.x)
  let rotationY = float4x4(rotationY: angle.y)
  let rotationZ = float4x4(rotationZ: angle.z)
  self = rotationX * rotationY * rotationZ
}

Here, the final rotation matrix is made up of three rotation matrices multiplied in a particular order. This order is not set in stone and is one of six possible orders. Depending on the multiplication order, you’ll get a different rotation.

Note: Sometimes, for example in flight simulators, these rotations are referred to as Yaw-Pitch-Roll. Depending on your frame of reference, if you’re using the y-axis as up and down (remember that’s not universal), then Yawing is about the y-axis, Pitching is about the x-axis and Rolling is about the z-axis.

For static objects within one rendering engine, this is fine. The main problem comes with animation and interpolating these angles.

As you proceed through a rotation interpolation if two axes become aligned you get the terrifyingly named gimbal lock.

This means that you’ve lost one axis of rotation. Because the inner axis rotations build on the outer axis rotation, the two rotations overlap. This causes odd interpolation.

Quaternions

Multiplying x, y and z rotations without compelling a sequence on them is impossible unless you involve the fourth dimension. In 1843, Sir William Rowan Hamilton did just that! He inscribed his fundamental formula for quaternion multiplication on to a stone on a bridge in Dublin:

The formula uses four-dimensional vectors and complex numbers to describe rotations. The mathematics is complicated, but fortunately, you don’t have to understand how quaternions work to use them. The main benefit of quaternions are:

• They interpolate correctly when using spherical linear interpolation (or slerp).
• They never lose any axes of control.
• They always take the shortest path between two rotations unless you specifically ask for the longest path.

Note: If you’re interested in studying the internals of quaternions, references.markdown contains further reading.

You don’t have to write any complicated interpolation code, as simd has quaternion classes and methods that handle it all for you using simd_slerp(). The quaternions perform a spherical interpolation along the shortest path as described here:

Internally in simd, quaternions are vectors of four elements, but Apple suggests that you treat them as abstract mathematical objects rather than delving into internal storage. That lets you off the hook for learning that the last element of the quaternion is the real part, and the first three elements are the imaginary part.

You’ll switch from using Euler rotations to using quaternions for your rotations. Taking advantage of simd conversion of quaternions to and from rotation matrices, this switch is effortless. In Node.swift, add this property:

var quaternion = simd_quatf()

Where you define modelMatrix, change the definition of rotateMatrix to:

let rotateMatrix = float4x4(quaternion)

Now your Nodes will support quaternions instead of Euler angles. You should also change rotation so that it updates the quaternion. Change the definition of rotation to:

var rotation: float3 = [0, 0, 0] {
  didSet {
    let rotationMatrix = float4x4(rotation: rotation)
    quaternion = simd_quatf(rotationMatrix)
  }
}

This keeps the quaternion value in sync when you set a model’s rotation.

To animate using quaternion rotation, you’ll duplicate what you did for translations. In Animation.swift, add this struct (at top level) to hold the keyframe:

struct KeyQuaternion {
  var time: Float = 0
  var value = simd_quatf()
}

Add a new property to Animation:

var rotations: [KeyQuaternion] = []

Duplicate getTranslation(at:), but replace all translations and floats with rotations and quaternions:

func getRotation(at time: Float) -> simd_quatf? {
  guard let lastKeyframe = rotations.last else {
    return nil
  }
  var currentTime = time
  if let first = rotations.first, first.time >= currentTime {
    return first.value
  }
  if currentTime >= lastKeyframe.time, !repeatAnimation {
    return lastKeyframe.value
  }
  currentTime = fmod(currentTime, lastKeyframe.time)
  let keyFramePairs = rotations.indices.dropFirst().map {
    (previous: rotations[$0 - 1], next: rotations[$0])
  }
  guard let (previousKey, nextKey) = ( keyFramePairs.first {
    currentTime < $0.next.time
  })
  else { return nil }
  let interpolant = (currentTime - previousKey.time) /
                    (nextKey.time - previousKey.time)
  return simd_slerp(previousKey.value,
                    nextKey.value,
                    interpolant)
}

Note that here you change the interpolation function to use simd_slerp instead of simd_mix. This does the necessary spherical interpolation.

In BallAnimations.swift, uncomment the method generateBallRotations(). This holds an array of rotation keyframes. The rotation starts out at 0, then rotates by 90º on the z-axis over several keyframes to a rotation of 0 at 0.35 seconds.

The reason for rotating several times by 90º is because if you rotate from 0º to 360º, the shortest distance between those is 0º so that the ball won’t rotate at all.

For final rotation of the ball, in Renderer.swift, change update(deltaTime:) to this:

func update(deltaTime: Float) {
  currentTime += deltaTime
  let ball = models[0]
    
  var animation = Animation()
  animation.translations = generateBallTranslations()
  animation.rotations = generateBallRotations()
  ball.position = animation.getTranslation(at: currentTime) 
                  ?? float3(repeating: 0)
  ball.position.y += ball.size.y / 2
  ball.quaternion = animation.getRotation(at: currentTime) 
                  ?? simd_quatf()
}

Build and run, and your ball moves back and forth with rotation.

If you’re doing more complex animating, you’ll probably want to do it in a 3D app. The beach ball actually holds some hidden transformation animation in its USD file.

USD and USDZ files

One major problem to overcome is how to import animation from 3D apps. Model I/O can import .obj files, but they only hold static information, not animation. USD is a format devised by Pixar, which can hold massive scenes with textures, animation and lighting information. There are various file extensions:

• .usd: A Universal Scene Description (USD) file consists of assets or links to assets which allows multiple artists to work on the same scene. The file can contain mesh geometry, shading information, models, cameras and lighting.
• .usdz: A single archive file that contains all the files - not just links - necessary for rendering a model.
• .usda: This file is the USD file in text format. The models included in this chapter’s project are in .usda format so that you can open them with TextEdit and inspect the contents.
• .usdc: This file is the USD file in binary format.

Apple has adopted USDZ, the archive derivation of the USD format, as their preferred augmented reality 3D format. However, as yet, there aren’t many 3D apps that export to the USD format. Maya and Houdini do, but Blender doesn’t.

Apple have provided a set of USD Python tools, including an app named usdconvert that will convert to USD from supported formats. Currently the supported formats are .obj, .fbx, .abc and .glTF.

Note: You can download the tools from the bottom of the AR Quick Look Gallery page: https://developer.apple.com/augmented-reality/quick-look/. This is a set of USDZ tools that will convert, generate, validate and inspect .usdz files.

glTF was developed by the Khronos Group and they have open sourced the format so that anyone can use it in their engines. http://sketchfab.com is a major provider and showcase of 3D models; all of their downloadable models are available in the glTF format, many of which you can convert to USD using Apple’s usdconvert.

Animating meshes

The file beachball.usda holds translation and rotation animation, and Model I/O can extract this animation. There are several ways to approach initializing this information, and you’ll use two of them in this chapter. Model I/O transform components don’t allow you to access the rotation and translation values directly, but provides you with a method that returns a transform matrix at a particular time. So for mesh transform animation you’ll extract the animation data for every frame of the animation during the model loading process.

Later, when you come to define skeletal animation, you’ll have access to joint rotation and translation, so you’ll load data only where there are keyframes, and use your interpolation methods to interpolate each frame.

Note: When writing your own engine, you will have the choice to load this animation data up front for every frame, to match the transfomation animation. You should consider the requirements of your game and what information your models hold. Generally it is more efficient to extract the loading code to a separate app which loads models and saves materials, textures and animation data into a more efficient format that matches your game engine. A good example of this asset pipeline is Apple’s video and sample code From Art to Engine with Model I/O at https://developer.apple.com/videos/play/wwdc2017/610/.

You’ll be running your game at a fixed fps - generally 60, and you’ll hold a transform matrix for every frame of animation.

In Renderer.swift, add a new static variable to Renderer to hold this fps centrally:

static var fps: Int!

In init(metalView:), update fps with the other static variables:

Renderer.fps = metalView.preferredFramesPerSecond

In draw(in:), replace:

let deltaTime = 1 / Float(view.preferredFramesPerSecond)
update(deltaTime: deltaTime)

with:

let deltaTime = 1 / Float(Renderer.fps)
for model in models {
  model.update(deltaTime: deltaTime)
}

Currently update(deltaTime:) is a method in Node. You’ll override this in Model to set the correct pose for the frame. You can remove Renderer’s update method now if you wish.

Model I/O can hold transform information on all objects within the MDLAsset. For simplicity, you’ll hold a transform component on each Mesh, and just animate the transforms for the duration given by the asset.

Create a new file called TransformComponent.swift to hold this transformation information, remembering to add the file to both the macOS and iOS targets.

Replace the code with:

import ModelIO

class TransformComponent {
  let keyTransforms: [float4x4]
  let duration: Float
  var currentTransform: float4x4 = .identity()
}

You’ll hold all the transform matrices for each frame for the duration of the animation. For example, if the animation has a duration of 2.5 seconds at 60 frames per second, keyTransforms will have 150 elements. You’ll later update all the Meshs’ currentTransform every frame with the transform for the current frame taken from keyTransforms.

Now add the following to the class:

init(transform: MDLTransformComponent,
     object: MDLObject,
     startTime: TimeInterval,
     endTime: TimeInterval) {
  duration = Float(endTime - startTime)
  let timeStride = stride(from: startTime,
                          to: endTime,
                          by: 1 / TimeInterval(Renderer.fps))
  keyTransforms = Array(timeStride).map { time in
    return MDLTransform.globalTransform(with: object, 
                                        atTime: time)
  }
}

This initializer will receive an MDLTransformComponent from either an asset or a mesh and then creates all the transform matrices for every frame for the duration of the animation.

Now add the following:

func setCurrentTransform(at time: Float) {
  guard duration > 0 else {
    currentTransform = .identity()
    return
  }
  let frame = Int(fmod(time, duration) * Float(Renderer.fps))
  if frame < keyTransforms.count {
    currentTransform = keyTransforms[frame]
  } else {
    currentTransform = keyTransforms.last ?? .identity()
  }
}

This retrieves a transform matrix at a particular, given, time. You calculate the current frame of the animation from the time. Using the floating point modulo operation fmod function, you can loop the animation. For example, if the animation is 2.5 seconds long, at 60 frames per second, that would mean there are 150 frames in the animation. If the current time is 5 seconds, that will be the last frame of the animation looped for a second time, and the current frame will be 150.

You save the current transform on the transform component. You’ll use this to update the position of the mesh vertices shortly.

You’ll need the start and end time from the asset, so, in Mesh.swift, change the init parameters to:

init(mdlMesh: MDLMesh, mtkMesh: MTKMesh,
     startTime: TimeInterval,
     endTime: TimeInterval) 

Open Model.swift and, where you initialize meshes, you should have a compile error. Change the Mesh initialization to:

Mesh(mdlMesh: $0.0, mtkMesh: $0.1,
     startTime: asset.startTime,
     endTime: asset.endTime)

The compile error should now go away.

Back in Mesh.swift, add a new transform component property to Mesh:

let transform: TransformComponent?

Add this at the end of init(mdlMesh:mtkMesh:startTime:endTime:):

if let mdlMeshTransform = mdlMesh.transform {
  transform = TransformComponent(transform: mdlMeshTransform,
                                 object: mdlMesh,
                                 startTime: startTime,
                                 endTime: endTime)
} else {
  transform = nil
}

Now that you’ve set up the transform component with animation, you’ll be able to use it when rendering each frame.

In Model.swift, add a new property to keep track of elapsed game time:

var currentTime: Float = 0

Create a new override method update(deltaTime:) with this code:

override func update(deltaTime: Float) {
  currentTime += deltaTime
  for mesh in meshes {
    mesh.transform?.setCurrentTransform(at: currentTime)
  }
}

Here you update all the transforms in the model ready for rendering.

In render(renderEncoder:uniforms:fragmentUniforms:), at the top of the for mesh in meshes loop, replace the assignment to uniforms.modelMatrix with:

let currentLocalTransform = 
  mesh.transform?.currentTransform ?? .identity()
uniforms.modelMatrix = modelMatrix * currentLocalTransform

Here you combine the model’s world transform with the mesh’s transform.

Build and run and the beachball is huge and mostly out of frame. This is because of the ball’s initial scale. When you weren’t using the transform information from the file, you needed to set the initial scale and position in Renderer, but now you can remove this. In Renderer.swift, in init(metalView:) remove:

ball.position = [0, 3, 0]
ball.scale = [100, 100, 100]

Build and run and see a wild beachball animation.

Note: Try downloading and rendering some of Apple’s animated USDZ samples from https://developer.apple.com/augmented-reality/quick-look. The objects with animation currently are the robot, the drummer and the biplane. The scale is too big for your scene, so you’ll need to set the scale to [0.1, 0.1, 0.1].

Now that you’ve learned about simple mesh animation, you’re ready to move on to animating a jointed figure.

Blender for animating

Imagine creating a walk cycle for a human figure by typing out keyframes! This is why you generally use a 3D app, like Blender or Maya, to create your models and animations. You then export those to your game or rendering engine of choice.

In the Resources folder for this chapter, you’ll find skeleton.blend. Open this file in Blender.

You’ll see something like this:

Before examining the bones further, left-click on the skeleton’s head to select the skeleton object, and press the Tab key to go into Edit Mode:

Note: If you are using a version of Blender earlier than 2.8, your interface will be different, and you should right-click instead of left-click to select.

Here, you can see all of the skeleton’s vertices. This is the original model which you can export as a static .obj file. It’s in what’s called the bind pose with the arms stretched out. This is a standard pose for figures as it makes it easy to add animation bones to the figure.

Press the Tab key to go back to Object Mode. To animate the figure, you need to have control of groups of vertices. For example, to rotate the head, you’d need to rotate all of the head’s vertices. Rigging a figure means creating an Armature with a hierarchy of Joints. Joints and bones are generally used synonymously, but a bone is just a visual cue to see which joint affects which vertices.

The general process of creating a figure for animation goes like this:

1. Create the model.
2. Create an armature with a hierarchy of joints.
3. Apply the armature to the model with automatic weights.
4. Use weight painting to change which vertices go with each joint.

Just as in the song Dem Bones: “the toe bone’s connected to the foot bone,” this is how a typical rigged figure’s joint hierarchy might look:

In character animation, it’s (usually) all about rotation — your bones don’t translate unless you have some kind of disjointing skeleton. With this hierarchy of joints, when you rotate one joint, all the child joints follow.

Try bending your elbow without moving your wrist. Because your wrist is lower in the hierarchy, even though you haven’t actively changed the wrist’s position and rotation, it still follows the movement of your elbow.

This movement is called forward kinematics and is what you’ll be using in this chapter. It’s a fancy name for making all child joints follow.

Note: Inverse kinematics allows the animator to make actions, such as walk cycles, more easily. Place your hand on a table or in a fixed position. Now, rotate your elbow and shoulder joint with your hand fixed. The hierarchical chain no longer moves your hand as in forward kinematics. As opposed to forward kinematics, the mathematics of inverse kinematics is quite complicated.

This Blender skeleton has a limited rig for simplicity. It only has four bones: the body, left upper arm, left forearm and left hand. Each of these joints controls a group of vertices.

Weight painting in Blender

Left-click the skeleton’s head. At the bottom of the Blender window, click on the drop-down that currently reads Object Mode, and change it to Weight Paint.

This shows you how each bone affects the vertices. Currently the body vertex group is selected, which is attached to the body bone. All vertices affected by the body bone are shown in red.

The process of weight painting and binding each bone to the vertices is called skinning. Unlike human arms, the skeleton’s arm bones here have space between them, so all mesh is assigned to only one bone. However, if you’re rigging a human arm, you would typically weight the vertices to multiple bones.

Here’s a typically weighted arm with the forearm selected to show gradual blending of weights at the elbow and the wrist.

This is a side-by-side example of blended and non-blended weights at the elbow joint with the forearm selected:

The blue area indicates no weighting; the red area indicates total weighting. You can see in the right image, the forearm vertices dig uncomfortably into the upper arm vertices, whereas in the left image, the vertices move more evenly around the elbow joint.

At the elbow, where the vertices are green, the vertex weighting would be 50% to the upper arm, and 50% to the forearm; when the forearm rotates, the green vertices will rotate at 50% of the forearm’s rotation. By blending the weights gradually, you can achieve an even deformation of vertices over the joint.

Animation in Blender

Select the drop-down at the bottom of the window that currently reads Weight Paint, and go back into Object Mode. Press the space bar to start an animation. Your skeleton should now get friendly and wave at you. This wave animation is a 60 frame looping animation clip.

At the top of Blender’s window, click the Animation tab to show the Animation workspace.

You can now see the animation keys at the top left in the Dope Sheet. This is a summary of the keyframes in the scene. The joints are listed on the left, and each circle in the dope sheet means there’s a keyframe at that frame.

Note: Although animated transformations are generally rotations, the keyframe can be a translation or a scale; you can click the arrow on the left of the joint name to see the specific channel the key is set on.

Press space bar to stop the animation if it’s still going. Scrub through the animation by dragging the playhead at the top of the pane (the blue rectangle with 0 in it in the above image). Pause the playhead at each set of keyframes. Notice the position of the arm; at each keyframe, it’s in an extreme position. Blender interpolates all the frames between the extremes.

Now that you’ve had a whirlwind tour of how to create a rigged figure and animate it in Blender, you’ll move on to learning how to render it in your rendering engine.

Note: You’ve only skimmed the surface of creating animated models. If you’re interested in creating your own, you’ll find some additional resources in references.markdown.

Skeletal Animation

Importing a skeletal animation into your app is a bit more difficult than importing a simple .obj file or a USDZ file with transform animation, because you have to deal with the joint hierarchy and joint weighting. You’ll read in the data from the USD file and restructure it to fit your rendering code. This is how the objects will fit together in your app:

Each model can have a number of animation clips, such as walk and wave. Each animation clip has a list of animations for a particular joint. Each mesh can have a skeleton that holds a list of joint names, and, using the joint name as a key, you’ll be able to access the correct animation for that joint.

The starter project has several classes to aid with importing the animation.

To create the Mesh‘s skeleton, in Skeleton.swift, you’ll use the MDLAnimationBindComponent from the mdlMesh, if there is one. Skeleton holds the joint names in an array, and also the joints’ parent indices in another array.

To load the animations for the asset, in AnimationComponent.swift, load(animation:) iterates through the joints and loads up Animations for each joint. These are all combined into an AnimationClip. Model will hold a dictionary of these AnimationClips keyed on the animation’s name.

In the File inspector, add these three files to both macOS and iOS targets:

• AnimationClip.swift
• AnimationComponent.swift
• Skeleton.swift

Build the project to ensure that it compiles.

In Renderer.swift, change the model that you’ll render. Replace:

let ball = Model(name: "beachball.usda")
models.append(ball)

…with:

let skeleton = Model(name: "skeletonWave.usda")
skeleton.rotation = [0, .pi, 0]
models.append(skeleton)

This is the skeleton model that you examined in Blender converted to USD format, so that you can load it in your app. You rotate him to look at the camera.

In Model.swift, add a new property to Model to hold the animation clips:

let animations: [String: AnimationClip]

In init(name:), just before calling super.init(), add the following to load the animations:

// animations
let assetAnimations = asset.animations.objects.compactMap {
  $0 as? MDLPackedJointAnimation
}
let animations = Dictionary(uniqueKeysWithValues: assetAnimations.map {
  ($0.name, AnimationComponent.load(animation: $0))
})
self.animations = animations

Here you extract all the MDLPackedJointAnimation objects from the asset and load them using the provided loading code. This will create a dictionary of animation clips keyed by animation name.

After the previous code, add this:

for animation in animations {
  print("Animation: ", animation.key)
}

This is to check that you’re loading the animation correctly.

Build and run and you should see the skeleton in his bind pose, with Animation: /skeletonWave/Animations/wave in the debug console. You can remove the previous for loop and print statement now.

You’ve just loaded a set of animations listing all rotations and translations on joints. Now to set up the meshes’ skeletons.

In Mesh, create a new property:

let skeleton: Skeleton?

At the top of init(mdlMesh:mtkMesh:), initialize the skeleton using mdlMesh’s MDLAnimationBindComponent:

let skeleton = 
      Skeleton(animationBindComponent: 
           (mdlMesh.componentConforming(to: MDLComponent.self) 
           as? MDLAnimationBindComponent))
self.skeleton = skeleton

You’ve now loaded up a skeleton with joints. When rendering the skeleton, you’ll be able to access the model’s current animation and apply it to the mesh’s skeleton joints.

Add a print statement to show the joints:

skeleton?.jointPaths.map {
  print($0)
}

Build and run, and in the debug console you should see a listing of the four skeleton model’s joints:

These correspond to the bones that that you previously saw in Blender. You can remove that print code now.

Loading the animation

To update the skeleton’s pose every frame, you’ll create a method that takes the animation clip and iterates through the joints to update each joint’s position for the frame. First you’ll create a method on AnimationClip that gets the pose for a joint at a particular time. This will use the interpolation methods that you’ve already created in Animation.

The main difference is that these poses will be in joint space. For example, in this animation, the forearm swings by 45º. All the other joints’ rotations and translations will be 0.

Open AnimationClip.swift and create this method:

func getPose(at time: Float, jointPath: String) -> float4x4? {
  guard
    let jointAnimation = jointAnimation[jointPath] ?? nil
    else { return nil }
  let rotation = 
      jointAnimation.getRotation(at: time) ?? simd_quatf()
  let translation = 
      jointAnimation.getTranslation(at: time) 
               ?? float3(repeating: 0)
  let pose = float4x4(translation: translation) 
               * float4x4(rotation)
  return pose
}

Here you retrieve the interpolated rotation and translation for a given joint. You then create a transformation matrix and return it as the pose. This is much the same code as you used earlier for retrieving a transform at a particular time. You return a matrix that combines the translation and rotation for a joint for the current time.

Note: A full transform should include scale as well. The starter code for the following chapter will have scale keys included.

Joint matrix palette

You’re now able to get the pose of a joint. However, each vertex is weighted to up to four joints. You saw this in the earlier elbow example, where some vertices belonging to the lower arm joint would get 50% of the upper arm joint`s rotation.

Shortly, you’ll change the default vertex descriptor to load vertex buffers with four joints and four weights for each vertex. This set of joints and weights is called the joint matrix palette.

The vertex function will sample from each of these joint matrices and, using the weights, will apply the transformation matrix to each vertex.

The following image shows a vertex that is assigned 50% to joint 2 and 50% to joint 3. The other two joint indices are unused.

In the vertex function, after multiplying the vertex by the projection, view and model matrices, the vertex function will multiply the vertex by a weighting of each of the joint transforms. Using the example in the image above, the vertex is multiplied with a weighting of 50% of Bone 2’s joint matrix and 50% of Bone 3’s joint matrix.

In Skeleton.swift, in Skeleton, create a new method:

func updatePose(animationClip: AnimationClip,
                at time: Float) {
}

This method, when you’ve completed it, will iterate through the joints and fill a joint matrix palette buffer with the current pose for each joint.

You’ll send this buffer to the GPU’s vertex function, so that each vertex will be able to access all of the joint matrices that it is weighted to.

Add the following to the method:

guard let paletteBuffer = jointMatrixPaletteBuffer 
    else { return }
var palettePointer = 
      paletteBuffer.contents().bindMemory(to: float4x4.self,
                                     capacity: jointPaths.count)
palettePointer.initialize(repeating: .identity(),
                          count: jointPaths.count)
var poses = [float4x4](repeatElement(.identity(), 
                       count: jointPaths.count))

This initializes the buffer pointer and a matrix array containing the current poses for each joint.

Now, to iterate through the skeleton’s joints, add the following:

for (jointIndex, jointPath) in jointPaths.enumerated() {
  // 1
  let pose = 
      animationClip.getPose(at: time * animationClip.speed,
                            jointPath: jointPath) 
                            ?? restTransforms[jointIndex]
  // 2
  let parentPose: float4x4
  if let parentIndex = parentIndices[jointIndex] {
    parentPose = poses[parentIndex]
  } else {
    parentPose = .identity()
  }
  poses[jointIndex] = parentPose * pose
}

Going through this code:

1. You retrieve the transformation pose, if there is one, for the joint for this frame. restTransform gives a default pose for the joint.

2. The poses array is in flattened hierarchical order, so you can be sure that the parent of any joint has already had its pose updated. You retrieve the current joint’s parent pose, concatenate the pose with the current joint’s pose and save it in the poses array.

The inverse bind matrix

Examine the properties held on Skeleton. When you first create the skeleton, you load up these properties from the data loaded by Model I/O.

One of the properties on Skeleton is bindTransforms. This is an array of matrices, one element for each joint, that transforms vertices into the local joint space.

When all the joint transforms are set to identity, that’s when you’ll get the bind pose. If you apply the inverse bind matrix to each joint, it will move to the origin. The following image shows the skeleton’s joints all multiplied by the inverse bind transform matrix:

Why is this useful? Each joint should rotate around its base. To rotate an object around a particular point, you first need to translate the point to the origin, then do the rotation, then translate back again. (Review Chapter 4, “Coordinate Spaces” if you’re unsure of this rotation sequence.)

In the following image, the vertex is located at (4, 1) and bound 100% to Bone 2. With rotations 10º on Bone 1 and 40º on Bone 2, the vertex should end up at about (3.2, 0) as shown in the right-hand image.

When you currently render your vertices, you multiply each vertex position by the projection, view and model matrices in the vertex function. To get this example vertex in the correct position for the right-hand image, you’ll also have to multiply the vertex position by both Bone 1’s transform and Bone 2’s transform.

Add this to the end of the previous for loop:

palettePointer.pointee = 
    poses[jointIndex] * bindTransforms[jointIndex].inverse
palettePointer = palettePointer.advanced(by: 1)

Here you translate the pose back to the origin with the the inverse bind transform, and combine it with the current pose into the final joint palette matrix.

With all the frame data set up, you can now set the pose. In Model.swift, in update(deltaTime), replace the existing animation code:

for mesh in meshes {
  mesh.transform?.setCurrentTransform(at: currentTime)
}

…with:

for mesh in meshes {
  if let animationClip = animations.first?.value {
    mesh.skeleton?.updatePose(animationClip: animationClip,
                              at: currentTime)
    mesh.transform?.currentTransform = .identity()
  } else {
    mesh.transform?.setCurrentTransform(at: currentTime)
  }
}

Here you take the first animation in the list of animations and, if there is an animation, update the pose for the current time. If there is no animation, do the transform animation as you were doing before.

Note: You’re using the first animation for simplicity. The starter code for the following chapter will refactor the animation code so that you can send a named animation to the model.

All the meshes are now in position and ready to render.

In render(renderEncoder:uniforms:fragmentUniforms:), at the top of the loop for mesh in meshes, add this:

if let paletteBuffer = mesh.skeleton?.jointMatrixPaletteBuffer {
  renderEncoder.setVertexBuffer(paletteBuffer, offset: 0, 
                                index: 22)
}

Here you set up the joint matrix palette buffer so that the GPU can read it. The vertex shader function will take in this palette and apply the matrices to the vertices.

In Shaders.metal, add two attributes to VertexIn:

ushort4 joints [[attribute(Joints)]];
float4 weights [[attribute(Weights)]];

The attribute constants Joints and Weights were set up for you in the starter project in Common.h.

To match VertexIn, you’ll need to update Model‘s vertex descriptor. In VertexDescriptor.swift, uncomment the three extra vertex attributes. Model I/O will now save color, joint index and joint weight information in the model’s vertex buffers.

However, now you have the problem that you saw in the previous chapter - some models will have skeletons and some won’t. You can solve this with function constants.

Vertex function constants

In Shaders.metal, add a new function constant after importing Common.h.

constant bool hasSkeleton [[function_constant(5)]];

The function constant is number 5, as 1 through 4 are taken by the fragment textures. You could set these up as named constants in Common.h. Just as you did for textures in the previous chapter, add the joint matrix palette as a conditional parameter to the vertex function:

vertex VertexOut 
       vertex_main(const VertexIn vertexIn [[stage_in]],
  constant float4x4 *jointMatrices [[buffer(22),
                             function_constant(hasSkeleton)]],
  constant Uniforms &uniforms [[buffer(BufferIndexUniforms)]])

At the top of the function, add this:

float4 position = vertexIn.position;
float4 normal = float4(vertexIn.normal, 0);

This is to save the vertex position and vertex normal.

After this, add the following to combine the joint matrix and weight data with the position and normal:

if (hasSkeleton) {
  float4 weights = vertexIn.weights;
  ushort4 joints = vertexIn.joints;
  position =
      weights.x * (jointMatrices[joints.x] * position) +
      weights.y * (jointMatrices[joints.y] * position) +
      weights.z * (jointMatrices[joints.z] * position) +
      weights.w * (jointMatrices[joints.w] * position);
  normal =
      weights.x * (jointMatrices[joints.x] * normal) +
      weights.y * (jointMatrices[joints.y] * normal) +
      weights.z * (jointMatrices[joints.z] * normal) +
      weights.w * (jointMatrices[joints.w] * normal);
}

Here you take each joint to which the vertex is bound, calculate the final position and normal, and then take the weighted part of that calculation.

If the function constant hasSkeleton is false, you’ll just use the original position and normal.

Change VertexOut out assignment so that it looks like this:

VertexOut out {
  .position = uniforms.projectionMatrix * uniforms.viewMatrix 
                * uniforms.modelMatrix * position,
  .worldPosition = (uniforms.modelMatrix * position).xyz,
  .worldNormal = uniforms.normalMatrix * normal.xyz,
  .worldTangent = 0,
  .worldBitangent = 0,
  .uv = vertexIn.uv
};

This uses position and normal instead of vertexIn.position and vertexIn.normal. You should also pre-multiply by the tangent and bitangent as well here, but for brevity, you’ve set worldTangent and worldBitangent properties to zero.

In Swift, you’ll need to tell the pipeline that it has to conditionally prepare two different vertex functions, depending on whether the mesh has a skeleton or not.

In Submesh.swift, add a boolean parameter to init(mdlSubmesh:mtkSubmesh:):

init(mdlSubmesh: MDLSubmesh, mtkSubmesh: MTKSubmesh, 
     hasSkeleton: Bool) {

Add a parameter to makePipelineState(textures:):

static func makePipelineState(textures: Textures, 
                              hasSkeleton: Bool) 
  -> MTLRenderPipelineState {

In init(mdlSubmesh:mtkSubmesh:hasSkeleton:), update the pipelineState assignment:

pipelineState = 
    Submesh.makePipelineState(textures: textures, 
                              hasSkeleton: hasSkeleton)

In Mesh.swift, in init(mdlMesh:mtkMesh:), add the new parameter when initializing Submesh:

Submesh(mdlSubmesh: mesh.0 as! MDLSubmesh, 
        mtkSubmesh: mesh.1,
        hasSkeleton: skeleton != nil)

In Submesh.swift, you’ll find an existing function called makeVertexFunctionConstants(hasSkeleton:) to create the vertex function constants for the skeleton option. This is the same procedure as you did for the fragment function constants in Chapter 7, “Maps and Materials”.

In makePipelineState, change the vertexFunction definition to:

let vertexFunction: MTLFunction?

Inside the do where you assign fragmentFunction, add this:

let constantValues = 
    makeVertexFunctionConstants(hasSkeleton: hasSkeleton)
vertexFunction = 
    try library?.makeFunction(name: "vertex_main",
                              constantValues: constantValues)

Here you create Metal shaders for the two possibilities - whether the model has a skeleton or not.

Phew! You’ve now set up all the matrices with animated poses and read in the joint and weight painted data. There’s just one more thing to change.

Now that you’re taking into account animation data, the skeleton’s poses depend upon the transformation matrices derived from each frame’s pose. These don’t take into account scaling, and there is currently scaling information in the USDZ file’s animation data.

In Renderer.swift, change the initial rotation of the skeleton:

skeleton.rotation = [0, .pi, 0]

To:

skeleton.rotation = [.pi / 2, .pi, 0]
skeleton.scale = [100, 100, 100]

Build and run, and you’ll see your friendly skeleton waving.

Note: Depending on the power of your device, your animation may glitch. This is because you’re taking too long to render a frame, and they sometimes overlap. You can temporarily fix this in Renderer‘s draw(in:), by adding commandBuffer.waitUntilCompleted() after committing the command buffer. Later, you’ll find out how to optimize your CPU / GPU synchronization.

Where to go from here?

This chapter took you through the basics of character animation. But don’t stop there! There are so many different topics that you can investigate. For instance, you can:

• Learn how to animate your own characters in Blender and import them into your renderer. Start off with a simple robot arm, and work upward from there.
• Download models from http://sketchfab.com, convert them to USD and see what works and what doesn’t.
• Watch Disney and Pixar movies… call it research. No, seriously! Animation is a skill all of its own. Watch how people move; good animators can capture personality in a simple walk cycle.

Congratulations, you reached the end of the first section, and you now have a rendering engine in which you can render both simple props and complicated rigged characters. Sweet!

In the next section, you’ll move on to creating a game engine where you can build game scenes with their own logic. You’ll also discover how to improve your scenes’ environments with terrains and skyboxes. Then, you’ll examine alternative lighting methods and how to improve performance and use every millisecond available.