Chapters
Chapters

Chapters

Hide chapters

Metal by Tutorials

Third Edition · macOS 12 · iOS 15 · Swift 5.5 · Xcode 13

Before You Begin

Section 0: 6 chapters
Show chapters Hide chapters

Section I: Beginning Metal

Section 1: 10 chapters
Show chapters Hide chapters

Section II: Intermediate Metal

Section 2: 8 chapters
Show chapters Hide chapters

Section III: Advanced Metal

Section 3: 8 chapters
Show chapters Hide chapters

Section IV: Ray Tracing

Section 4: 7 chapters
Show chapters Hide chapters

24. Character Animation
Written by Marius Horga & Caroline Begbie

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

In the previous chapter, you learned how to move objects over time using keyframes. Imagine how long it would take to create a walk cycle for a human figure by typing out keyframes. This is the reason why you generally use a 3D app, like Blender or Maya, to create your models and animations. You then export those animations to your game or rendering engine of choice.

Skeletal Animation

Rarely, will you move the entire character when you’re animating it. Instead, you’ll move parts of the mesh, such as an arm, rather than the whole thing. Using a 3D app, the rigger creates a skeleton — in Blender, this is known as an armature. She assigns bones and other controls to parts of the mesh so that the animator can transform the bones and record the movement into an animation clip.

You’ll use Blender 3.0 to examine an animated model and understand the principles and concepts behind 3D animation.

Note: If you haven’t installed Blender 3.0 yet, it’s free, and you can download it from https://www.blender.org.

➤ Go to the resources folder for this chapter, and open skeleton.blend in Blender 3.0.

You’ll see something like this:

The skeleton model in Blender 3.0
The skeleton model in Blender 3.0

➤ Before examining the bones further, left-click on the skeleton’s head to select the skeleton object. Press the Tab key to switch to Edit Mode:

The skeleton mesh
The skeleton mesh

Here, you can see all of the skeleton’s vertices. This is the original model, which you can export as a static .obj file. The skeleton has its arms stretched out in what’s known as the bind pose. This is a standard pose for figures as it makes it easier to add animation bones to the figure.

➤ Press the Tab key to go back to Object Mode.

To animate the figure, you need to control groups of vertices. For example, to rotate the head, you’d rotate all of the head’s vertices.

Rigging a figure in Blender means creating an armature with a hierarchy of joints. Joints and bones are generally used synonymously, but a bone is simply a visual cue to see which joint affects which vertices.

The general process of creating a figure for animation goes like this:

  1. Create the model.
  2. Create an armature with a hierarchy of joints.
  3. Apply the armature to the model with automatic weights.
  4. Use weight painting to change which vertices go with each joint.

Just as in the song Dem Bones, “The toe bone’s connected to the foot bone,” this is how a typical rigged figure’s joint hierarchy might look:

A joint hierarchy
A joint hierarchy

In character animation, it’s (usually) all about rotation — your bones don’t translate unless you have some kind of disjointing skeleton. With this hierarchy of joints, when you rotate one joint, all the child joints follow. Try bending your elbow without moving your wrist. Because your wrist is lower in the hierarchy, even though you haven’t actively changed the wrist’s position and rotation, it still follows the movement of your elbow. This type of movement is known as forward kinematics and is what you’ll be using in this chapter. It’s a fancy name for making all child joints follow.

Note: Inverse kinematics allows the animator to make actions, such as walk cycles, more easily. Place your hand on a table or in a fixed position. Now, rotate your elbow and shoulder joint with your hand fixed. The hierarchical chain no longer moves your hand as in forward kinematics. As opposed to forward kinematics, the mathematics of inverse kinematics is quite complicated.

The skeleton model that you’re looking at in Blender has a limited rig for simplicity. It has four bones: the body, left upper arm, left forearm and left hand. Each of these joints controls a group of vertices.

Weight Painting in Blender

➤ Left-click the skeleton’s head.

Weight Paint Dropdown
Xuusrt Kaojd Dxicwimr

The skeleton's body bone weights
Dbe zjuqolux't tusj nela niejwkg

A weighted arm
E luanmjan adw

Blended and non-blended weights
Cyonlez idl rek-sgohxul yuucpkp

Animation in Blender

➤ Select the drop-down at the bottom of the window that currently reads Weight Paint, and go back into Object Mode.

The dope sheet
Swi hipe yjuid

The Starter App

➤ In Xcode, open the starter project. The scene is a skeleton character rendered with the forward renderer, using the PBR shader, without shadows.

Implementing Skeletal Animation

Importing a skeletal animation into your app is a bit more difficult than importing a simple .obj file or a USDZ file with transform animation, because you have to deal with the joint hierarchy and joint weighting. You’ll read in the data from the USD file and restructure it to fit your rendering code. This is how the objects will fit together in your app:

The code architecture
Fve giqa apqzohosyave

let animations: [String: AnimationClip]

// animations
let assetAnimations = asset.animations.objects.compactMap {
  $0 as? MDLPackedJointAnimation
}
let animations
  = Dictionary(uniqueKeysWithValues: assetAnimations.map {
  ($0.name, AnimationComponent.load(animation: $0))
  })
self.animations = animations

animations.forEach {
  print("Animation:", $0.key)
}
The debug console
Vze yisis mocpovu

let skeleton: Skeleton?

let skeleton =
  Skeleton(animationBindComponent:
    (mdlMesh.componentConforming(to: MDLComponent.self)
    as? MDLAnimationBindComponent))
self.skeleton = skeleton

skeleton?.jointPaths.forEach {
  print($0)
}

Loading the Animation

To update the skeleton’s pose every frame, you’ll create a method that takes the animation clip and iterates through the joints to update each joint’s position for the frame.

func getPose(at time: Float, jointPath: String) -> float4x4? {
  guard let jointAnimation = jointAnimation[jointPath],
    let jointAnimation = jointAnimation
    else { return nil }
  let rotation =
    jointAnimation.getRotation(at: time) ?? simd_quatf()
  let translation =
    jointAnimation.getTranslation(at: time) ?? float3(repeating: 0)
  let scale =
    jointAnimation.getScale(at: time) ?? float3(repeating: 0)
  let pose = float4x4(translation: translation) * float4x4(rotation)
    * float4x4(scaling: scale)
  return pose
}

The Joint Matrix Palette

You’re now able to get the pose of a joint. However, each vertex is weighted to up to four joints. You saw this in the earlier elbow example, where some vertices belonging to the lower arm joint would get 50% of the upper arm joint’s rotation. Soon, you’ll change the default vertex descriptor to load vertex buffers with four joints and four weights for each vertex. This set of joints and weights is known as the joint matrix palette.

func updatePose(
  animationClip: AnimationClip?,
  at time: Float
) {
}

guard let paletteBuffer = jointMatrixPaletteBuffer
  else { return }
var palettePointer = paletteBuffer.contents().bindMemory(
  to: float4x4.self,
  capacity: jointPaths.count)
guard let animationClip = animationClip else {
  palettePointer.initialize(
    repeating: .identity,
    count: jointPaths.count)
  return
}

var poses =
  [float4x4](repeatElement(.identity, count: jointPaths.count))
for (jointIndex, jointPath) in jointPaths.enumerated() {
  // 1
  let pose = animationClip.getPose(
    at: time * animationClip.speed,
    jointPath: jointPath) ?? restTransforms[jointIndex]
  // 2
  let parentPose: float4x4
  if let parentIndex = parentIndices[jointIndex] {
    parentPose = poses[parentIndex]
  } else {
    parentPose = .identity
  }
  poses[jointIndex] = parentPose * pose
}

The Inverse Bind Matrix

➤ Examine the properties held on Skeleton.

The bind pose
Mfe musj yilu

The inverse bind matrix applied to all joints
Shu uzzixtu feqr xedsos arvluaf gi uyp piatmt

palettePointer.pointee =
  poses[jointIndex] * bindTransforms[jointIndex].inverse
palettePointer = palettePointer.advanced(by: 1)


for i in 0..<meshes.count {
  meshes[i].transform?.getCurrentTransform(at: currentTime)
}

for i in 0..<meshes.count {
  var mesh = meshes[i]
  if let animationClip = animations.first?.value {
    mesh.skeleton?.updatePose(
      animationClip: animationClip,
      at: currentTime)
  }
  mesh.transform?.getCurrentTransform(at: currentTime)
  meshes[i] = mesh
}

if let paletteBuffer = mesh.skeleton?.jointMatrixPaletteBuffer {
  encoder.setVertexBuffer(
    paletteBuffer,
    offset: 0,
    index: JointBuffer.index)
}

ushort4 joints [[attribute(Joints)]];
float4 weights [[attribute(Weights)]];
Vertex Buffer 0
Bekkum Kaykiy 7

No obvious changes yet
Fo uwxuoay tvulciw has

Updating the Vertex Shader

➤ In the Shaders group, open Shaders.metal, and add a new parameter to vertex_main:

constant float4x4 *jointMatrices [[buffer(JointBuffer)]]

bool hasSkeleton = true;
float4 position = in.position;
float4 normal = float4(in.normal, 0);

if (hasSkeleton) {
  float4 weights = in.weights;
  ushort4 joints = in.joints;
  position =
      weights.x * (jointMatrices[joints.x] * position) +
      weights.y * (jointMatrices[joints.y] * position) +
      weights.z * (jointMatrices[joints.z] * position) +
      weights.w * (jointMatrices[joints.w] * position);
  normal =
      weights.x * (jointMatrices[joints.x] * normal) +
      weights.y * (jointMatrices[joints.y] * normal) +
      weights.z * (jointMatrices[joints.z] * normal) +
      weights.w * (jointMatrices[joints.w] * normal);
}

VertexOut out {
  .position = uniforms.projectionMatrix * uniforms.viewMatrix
                * uniforms.modelMatrix * position,
  .uv = in.uv,
  .color = in.color,
  .worldPosition = (uniforms.modelMatrix * position).xyz,
  .worldNormal = uniforms.normalMatrix * normal.xyz,
  .worldTangent = 0,
  .worldBitangent = 0,
  .shadowPosition =
    uniforms.shadowProjectionMatrix * uniforms.shadowViewMatrix
    * uniforms.modelMatrix * position
};

models = [skeleton]
Skeleton waving
Byexizoc tajoln

Function Specialization

Over the years there has been much discussion about how to render conditionally. For example, in your fragment shaders when rendering textures, you use the Metal Shading Language function is_null_texture(textureName) to determine whether to use the value from the material or a texture.

Function constants
Racfloif bixvyampw

static func makeFunctionConstants(hasSkeleton: Bool)
-> MTLFunctionConstantValues {
  let functionConstants = MTLFunctionConstantValues()
  var property = hasSkeleton
  functionConstants.setConstantValue(
    &property,
    type: .bool,
    index: 0)
  return functionConstants
}

static func createForwardPSO(hasSkeleton: Bool = false)
-> MTLRenderPipelineState {

let functionConstants =
  makeFunctionConstants(hasSkeleton: hasSkeleton)
let vertexFunction = try? Renderer.library?.makeFunction(
  name: "vertex_main",
  constantValues: functionConstants)

static func createForwardTransparentPSO(hasSkeleton: Bool = false)
-> MTLRenderPipelineState {

let functionConstants =
  makeFunctionConstants(hasSkeleton: hasSkeleton)
let vertexFunction = try? Renderer.library?.makeFunction(
  name: "vertex_main",
  constantValues: functionConstants)

var pipelineState: MTLRenderPipelineState

let hasSkeleton = skeleton?.jointMatrixPaletteBuffer != nil
pipelineState =
  PipelineStates.createForwardPSO(hasSkeleton: hasSkeleton)

encoder.setRenderPipelineState(mesh.pipelineState)

constant bool hasSkeleton [[function_constant(0)]];

constant float4x4 *jointMatrices [[
  buffer(JointBuffer),
  function_constant(hasSkeleton)]]

bool hasSkeleton = true;

commandBuffer.waitUntilCompleted()

Key Points

  • Character animation differs from transform animation. With transform animation, you deform the mesh directly. When animating characters, you use a skeleton with joints. The geometry mesh is attached to these joints and deforms when you rotate a joint.
  • The skeleton consists of a hierarchy of joints. When you rotate one joint, all the child joints move appropriately.
  • You attach the mesh to joints by weight painting in a 3D app. Up to four joints can influence each vertex (this is a limitation in your app, but generally weighting four joints is ample).
  • Animation clips contain transformation data for keyframes. The app interpolates the transformations between keyframes.
  • Each joint has an inverse bind matrix, which, when applied, moves the joint to the origin.
  • When your shaders have different requirements depending on different situations, you can use function specialization. You indicate the different requirements in the pipeline state, and the compiler creates multiple versions of the shader function.

Where to Go From Here?

This chapter took you through the basics of character animation. But don’t stop there! There are so many different topics that you can investigate. For instance, you can:

23. Animation
25. Managing Resources
Have a technical question? Want to report a bug? You can ask questions and report bugs to the book authors in our official book forum here.
© 2024 Kodeco Inc.

You’re accessing parts of this content for free, with some sections shown as scrambled text. Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now