Chapters
Chapters

Chapters

Hide chapters

Metal by Tutorials

Fourth Edition · macOS 14, iOS 17 · Swift 5.9 · Xcode 15

Before You Begin

Section 0: 6 chapters
Show chapters Hide chapters

Section I: Beginning Metal

Section 1: 10 chapters
Show chapters Hide chapters

Section II: Intermediate Metal

Section 2: 8 chapters
Show chapters Hide chapters

Section III: Advanced Metal

Section 3: 8 chapters
Show chapters Hide chapters

Section IV: Ray Tracing

Section 4: 7 chapters
Show chapters Hide chapters

24. Character Animation
Written by Caroline Begbie & Marius Horga

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

In the previous chapter, you learned how to move objects over time using keyframes. Imagine how long it would take to create a walk cycle for a human figure by typing out keyframes. This is the reason why you generally use a 3D app, like Blender or Maya, to create your models and animations. You then export those animations to your game or rendering engine of choice.

Skeletal Animation

Rarely will you move the entire character when you’re animating it. Even a walk cycle is animated in place. Instead, you’ll move parts of the mesh, such as an arm, rather than the whole thing. Using a 3D app, the rigger creates a skeleton — in Blender, this is known as an armature. The rigger assigns bones and other controls to parts of the mesh so that the animator can transform the bones and record the movement into an animation clip.

You’ll use Blender 3.6 to examine an animated model and understand the principles and concepts behind 3D animation.

Note: If you haven’t installed Blender 3.6 yet, it’s free, and you can download it from https://www.blender.org.

➤ Go to the resources folder for this chapter, and open skeleton.blend in Blender.

You’ll see something like this:

The skeleton model in Blender 3.6
The skeleton model in Blender 3.6

Your Blender theme may have different colors.

➤ Before examining the bones further, left-click on the skeleton’s head to select the skeleton object. Press the Tab key to switch to Edit Mode:

The skeleton mesh
The skeleton mesh

Here, you can see all of the skeleton’s vertices as they were modeled. The skeleton has its arms stretched out in what’s known as the bind pose. Arms stretched out is a standard pose for figures as it makes it easier to add animation bones to the figure.

➤ Press the Tab key to go back to Object Mode.

Blender binds the vertices to the skeleton’s bones, with the arm down.

To animate the figure, you need to control groups of vertices. For example, to rotate the head, you’d rotate all of the head’s vertices.

Rigging a figure in Blender means creating an armature with a hierarchy of joints. Joints and bones are generally used synonymously, but a bone is simply a visual cue to see which joint affects which vertices.

The general process of creating a figure for animation goes like this:

  1. Create the model.
  2. Create an armature with a hierarchy of joints.
  3. Apply the armature to the model with automatic weights.
  4. Use weight painting to change which vertices go with each joint.

Just as in the song Dem Bones, “The toe bone’s connected to the foot bone,” this is how a typical rigged figure’s joint hierarchy might look:

A joint hierarchy
A joint hierarchy

In character animation, it’s (usually) all about rotation — your bones don’t translate unless you have some kind of disjointing skeleton. With this hierarchy of joints, when you rotate one joint, all the child joints follow. Try bending your elbow without moving your wrist. Because your wrist is lower in the hierarchy, even though you haven’t actively changed the wrist’s position and rotation, it still follows the movement of your elbow. This type of movement is known as forward kinematics and is what you’ll be using in this chapter. It’s a fancy name for making all child joints follow.

Note: Inverse kinematics allows the animator to make actions, such as walk cycles, more easily. Place your hand on a table or in a fixed position. Now, rotate your elbow and shoulder joint with your hand fixed. The hierarchical chain no longer moves your hand as in forward kinematics. As opposed to forward kinematics, the mathematics of inverse kinematics is quite complicated.

The skeleton model that you’re looking at in Blender has a limited rig for simplicity. It has four bones: the body, left upper arm, left forearm and left hand. Each of these joints controls a group of vertices.

Weight Painting in Blender

➤ Left-click the skeleton’s head.

Weight Paint Dropdown
Diaczr Buotb Fbowyevr

The skeleton's body bone weights
Cfu lpadeqig'k ting zodu zoivygz

A weighted arm
E kaahwbub iqn

Blended and non-blended weights
Wbuwpal end cek-rkohpex keuzdfw

Animation in Blender

➤ Select the drop-down at the bottom of the window that currently reads Weight Paint, and go back into Object Mode.

The dope sheet
Dva heto tqais

The Starter App

➤ In Xcode, open the starter project and build and run the app.

Implementing Skeletal Animation

Importing a skeletal animation into your app is a bit more difficult than importing a simple USD file with transform animation, because you have to deal with the joint hierarchy and joint weighting. You’ll read in the data from the USD file and restructure it to fit your rendering code.

The code architecture
Cmi musa iqcgowebhale

The Skeleton Map

A Skeleton will hold the joint names in a String array. You’ll convert the skeleton’s hierarchy of joints to an array of parent indices.

The skeleton map
Nla mmaveyuk suj

Skin Data

Each mesh will contain joint paths which bind the mesh to the skeleton. This is called skinning. On loading the mesh, you’ll load these joint paths in a Skin structure and map them to the skeleton’s joint paths.

Skin to skeleton map
Nhol ke qsepumay mul

Initializing Data

You’ll start by loading in the skeleton, the skinning data and lastly, the animations.

Loading the Skeleton

➤ In the Geometry group, open Model.swift, and add two new properties to Model:

var skeleton: Skeleton?
var animationClips: [String: AnimationClip] = [:]

func loadSkeleton(asset: MDLAsset) {
  let skeletons =
    asset.childObjects(of: MDLSkeleton.self) as? [MDLSkeleton] ?? []
  skeleton = Skeleton(mdlSkeleton: skeletons.first)
}

Loading the Skinning Data

Open Mesh.swift and add a new property to Mesh:

var skin: Skin?

func loadSkins(mdlMeshes: [MDLMesh]) {
  for index in 0..<mdlMeshes.count {
    let animationBindComponent =
        mdlMeshes[index].componentConforming(to: MDLComponent.self)
        as? MDLAnimationBindComponent
      guard let skeleton else { continue }
      let skin = Skin(
        animationBindComponent: animationBindComponent, 
        skeleton: skeleton)
      meshes[index].skin = skin
  }
}

Loading the Animation

➤ In Model, add a new method to load the animations from the asset:

func loadAnimations(asset: MDLAsset) {
  let assetAnimations = asset.animations.objects.compactMap {
    $0 as? MDLPackedJointAnimation
  }
  for assetAnimation in assetAnimations {
    let animationClip = AnimationClip(animation: assetAnimation)
    animationClips[assetAnimation.name] = animationClip
  }
}

// load animated characters
loadSkeleton(asset: asset)
loadSkins(mdlMeshes: mdlMeshes)
loadAnimations(asset: asset)

animationClips.forEach {
  print("Animations:", $0.key)
}
print(skeleton?.jointPaths)

Animations: /skeleton/Animations/Wave
Optional(["/body", "/body/upperarm_L", "/body/upperarm_L/forearm_L", "/body/upperarm_L/forearm_L/hand_L"])
nil

The Math of Striking a Pose

To update the skeleton’s pose every frame, you’ll create a method that takes the animation clip and iterates through the joints to update each joint’s position for the frame.

1. Load Animation Transform Data

First set up a method to extract the current frame animation.

func getPose(at time: Float, jointPath: String) -> float4x4? {
  guard let jointAnimation = jointAnimation[jointPath],
    let jointAnimation = jointAnimation
    else { return nil }
  let rotation =
  jointAnimation.getRotation(at: time) ?? simd_quatf(.identity)
  let translation =
    jointAnimation.getTranslation(at: time) ?? float3(repeating: 0)
  let pose = float4x4(translation: translation) * float4x4(rotation)
  return pose
}

func updatePose(
  at currentTime: Float,
  animationClip: AnimationClip) {
  // 1
  let time = fmod(currentTime, animationClip.duration)
  // 2
  var localPose = [float4x4](
    repeating: .identity, 
    count: jointPaths.count)
  // 3
  for index in 0..<jointPaths.count {
    let pose = animationClip.getPose(
      at: time * animationClip.speed,
      jointPath: jointPaths[index])
    ?? restTransforms[index]
    localPose[index] = pose
  }
}

2. Calculate the World Pose

In the following image, the forearm swings by 45º. All the other joint rotations are 0º. However, the rotation of the forearm affects the position (but not the rotation) of the hand.

var worldPose: [float4x4] = []
for index in 0..<parentIndices.count {
  let parentIndex = parentIndices[index]
  let localMatrix = localPose[index]
  if let parentIndex {
    worldPose.append(worldPose[parentIndex] * localMatrix)
  } else {
    worldPose.append(localMatrix)
  }
}
Array of transforms
Ilsex og wyugvqigjh

3. The Inverse Bind Matrix

➤ Examine the properties held on Skeleton.

Bind matrix applied
Hutz luvguq anfreuq

Inverse bind pose
Obqifhi zucl sese

for index in 0..<worldPose.count {
  worldPose[index] *= bindTransforms[index].inverse
}
currentPose = worldPose

if let skeleton,
   let animation = animationClips.first {
  let animationClip = animation.value
  skeleton.updatePose(
    at: currentTime,
    animationClip: animationClip)
}
The final pose
Ski bodiq dowo

4. Update the Joint Matrix Palette Buffer

➤ Open Skin.swift and examine the struct.

func updatePalette(skeleton: Skeleton?) {
  guard let skeletonPose = skeleton?.currentPose
    else { return }
  var palettePointer = jointMatrixPaletteBuffer.contents().bindMemory(
    to: float4x4.self,
    capacity: jointPaths.count)
  for index in 0..<jointPaths.count {
    let skinIndex = skinToSkeletonMap[index]
    palettePointer.pointee = skeletonPose[skinIndex]
    palettePointer = palettePointer.advanced(by: 1)
  }
}

for index in 0..<meshes.count {
  meshes[index].transform?.getCurrentTransform(at: currentTime)
}

for index in 0..<meshes.count {
  var mesh = meshes[index]
  mesh.transform?.getCurrentTransform(at: currentTime)
  mesh.skin?.updatePalette(skeleton: skeleton)
  meshes[index] = mesh
}

Joints and Weights Vertex Data

Each vertex is weighted to up to four joints. You saw this in the earlier elbow example, where some vertices belonging to the lower arm joint would get 50% of the upper arm joint’s rotation. Soon, you’ll change the default vertex descriptor to load vertex buffers with four joints and four weights for each vertex.

Transfer the Data to the GPU

All of the meshes are now in position and ready to render.

if let paletteBuffer = mesh.skin?.jointMatrixPaletteBuffer {
  encoder.setVertexBuffer(
    paletteBuffer,
    offset: 0,
    index: JointBuffer.index)
}

ushort4 joints [[attribute(Joints)]];
float4 weights [[attribute(Weights)]];
Vertex Buffer 0
Wucxel Jazrop 5

No obvious changes yet
He evxeaif xmijhid poc

Updating the Vertex Shader

➤ In the Shaders group, open Vertex.metal, and add a new parameter to vertex_main:

constant float4x4 *jointMatrices [[buffer(JointBuffer)]]

bool hasSkeleton = true;
float4 position = in.position;
float4 normal = float4(in.normal, 0);

if (hasSkeleton) {
  float4 weights = in.weights;
  ushort4 joints = in.joints;
  position =
      weights.x * (jointMatrices[joints.x] * position) +
      weights.y * (jointMatrices[joints.y] * position) +
      weights.z * (jointMatrices[joints.z] * position) +
      weights.w * (jointMatrices[joints.w] * position);
  normal =
      weights.x * (jointMatrices[joints.x] * normal) +
      weights.y * (jointMatrices[joints.y] * normal) +
      weights.z * (jointMatrices[joints.z] * normal) +
      weights.w * (jointMatrices[joints.w] * normal);
}

float4 worldPosition = uniforms.modelMatrix * position;
VertexOut out {
  .position = uniforms.projectionMatrix * uniforms.viewMatrix
                * uniforms.modelMatrix * position,
  .uv = in.uv,
  .worldPosition = worldPosition.xyz / worldPosition.w,
  .worldNormal = uniforms.normalMatrix * normal.xyz,
  .worldTangent = 0,
  .worldBitangent = 0
};
return out;
Skelly waving
Nrehjd fevann

Function Specialization

Over the years there has been much discussion about how to render conditionally. For example, in your fragment shaders when rendering textures, you use the Metal Shading Language function is_null_texture(textureName) to determine whether to use the value from the material or a texture.

Function constants
Xodvmueq legfcudmw

static func makeFunctionConstants(hasSkeleton: Bool)
  -> MTLFunctionConstantValues {
  let functionConstants = MTLFunctionConstantValues()
  var property = hasSkeleton
  functionConstants.setConstantValue(
    &property,
    type: .bool,
    index: 0)
  return functionConstants
}

static func createForwardPSO(hasSkeleton: Bool = false)
  -> MTLRenderPipelineState {

let functionConstants =
  makeFunctionConstants(hasSkeleton: hasSkeleton)
let vertexFunction = try? Renderer.library?.makeFunction(
  name: "vertex_main",
  constantValues: functionConstants)

var pipelineState: MTLRenderPipelineState!

let hasSkeleton = skeleton != nil
pipelineState =
  PipelineStates.createForwardPSO(hasSkeleton: hasSkeleton)

encoder.setRenderPipelineState(pipelineState)

constant bool hasSkeleton [[function_constant(0)]];

constant float4x4 *jointMatrices [[
  buffer(JointBuffer),
  function_constant(hasSkeleton)]]

bool hasSkeleton = true;

commandBuffer.waitUntilCompleted()

Key Points

  • Character animation differs from transform animation. With transform animation, you deform the mesh directly. When animating characters, you use a skeleton with joints. The geometry mesh is attached to these joints and deforms when you rotate a joint.
  • The skeleton consists of a hierarchy of joints. When you rotate one joint, all the child joints move appropriately.
  • You attach the mesh to joints by weight painting in a 3D app. Up to four joints can influence each vertex (this is a limitation in your app, but generally weighting four joints is ample).
  • Animation clips contain transformation data for keyframes. The app interpolates the transformations between keyframes.
  • Each joint has a bind matrix, which, when applied, moves the joint to the origin.
  • When your shaders have different requirements depending on different situations, you can use function specialization. You indicate the different requirements in the pipeline state, and the compiler creates multiple versions of the shader function.

Where to Go From Here?

This chapter took you through the basics of character animation. But don’t stop there! There are so many different topics that you can investigate. For instance, you can:

23. Animation
25. Managing Resources
Have a technical question? Want to report a bug? You can ask questions and report bugs to the book authors in our official book forum here.
© 2024 Kodeco Inc.

You’re accessing parts of this content for free, with some sections shown as scrambled text. Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now