Pointer Interaction Tutorial for iOS: Supporting the Mouse and Trackpad

Apple has been steering its iPad product range toward productivity and professional use for several years. This includes support for hardware keyboards. And iOS APIs like UIKeyCommand reinforce keyboard support by allowing you, as a developer, to add keyboard shortcuts to your app.

What about the mouse, though? How do you handle a screen pointer on a device designed for touch input? This is where pointer interactions come in. Ever since iOS 13.4, you can connect a trackpad or mouse to your iPad and use an on-screen pointer with your controls.

But pointer interactions are not limited to the presence of an on-screen pointer. Many standard UIKit controls will react to the presence of the pointer with shapes or animations — with little or no work from you! In this tutorial, you’ll add pointer interactions to a simple game app. By doing this, you’ll learn how to:

• Enable built-in pointer interactions
• Customize UIButton interactions
• Apply pointer interactions to other views
• Configure custom pointer shapes
• Perform coordinated animations for pointer movement
• Respond to hover events

And you’ll get to work with pointer interactions in both UIKit and SwiftUI!

To do this tutorial you’ll need Xcode 11.5 or higher. You don’t need hardware or a physical mouse, as the simulator provides a pointer widget for your use.

Getting Started

Use the Download Materials link at the top or bottom of the tutorial to download the starter project. This project is a simple pattern matching game called RaySays. You’ll be adding pointer interactions in multiple places in the code. The project is ready to run, so you can focus on the task of enhancing it with pointer interactions.

Select the iPad Pro (9.7-inch) iPad simulator in the target selector.

Open the Xcode project in the folder RaySays-Starter, then build and run. The UI has implementations in both SwiftUI and UIKit. You’ll be modifying both UI versions during the tutorial.

Now try out the game. When you tap UIKit or SwiftUI then Play, a sequence will flash at you. You’ll repeat the same sequence back. Be prepared for a challenge — the sequence gets longer at every level!

Enabling UIKit Interactions

In this section, you’ll add pointer interactions to the UIKit version of your user interface. Later, you’ll add similar interactions to the SwiftUI version.

Switching on Built-in Behaviors

In this section, you’ll switch on the built-in pointer behavior for UIButton, UIBarButton and UISegmentedControl.

Run the app again, this time staying on the start screen. It has two buttons to start the game and a segmented control to choose a difficulty level.

Capture the pointer by using the simulator’s Capture Cursor button.

You’ll need to remember to do that with every build and run step in this tutorial, or you won’t see any pointer interactions!

When you’re finished, press the Escape key to release the pointer from the simulator.

Move the pointer over the three controls. The two buttons do nothing, but the segmented control responds with a little wriggle. It must be ticklish!

Tap the UIKit button to enter the game view. The back button in the navigation bar also responds to the pointer. These pointer interactions are built in.

Examine the Project navigator. Locate and open the RaySays folder. Open the UIKit View Controllers folder, and then open Main.storyboard. Find the RaySays Scene, which is the root of the initial navigation controller.

In the View hierarchy, expand the RaySays Scene. Continue to expand the view until you see the buttons UIKit Selector and SwiftUI Selector. Select both the green buttons by clicking on them while holding down the Shift key. Open the Attributes inspector and locate the control to enable pointer interaction. Switch it on by clicking the checkbox.

Build and run and capture the pointer. Now the two buttons respond with movement when you move the pointer over them!

Customizing UIButton Interaction

iOS provides a default visual effect you get when you move the pointer over the buttons. In this section, you’ll find out how to choose your own style.

In the Project navigator, find the UIKit View Controllers folder. Locate and open RootViewController.swift. Add this code at the end of the main class:

func configureButtons() {
  uikitSelector.pointerStyleProvider = { button, effect, shape in
    let preview = UITargetedPreview(view: button)
    return UIPointerStyle(effect: .highlight(preview))
  }

  swiftuiSelector.pointerStyleProvider = { button, effect, shape in
    let preview = UITargetedPreview(view: button)
    return UIPointerStyle(effect: .lift(preview))
  }
}

In this code, you set the pointerStyleProvider on each button to give a custom UIPointerStyle. Pointer styles affect the shape of the pointer and the visual appearance of the button. One way to create a pointer style is to use a UIPointerEffect, which is what you are using here. .highlight and .lift are two different effects. Notice how you create a UITargetedPreview to receive the effect. That allows UIKit to mess around with the appearance of your button during pointer interactions without changing the underlying views.

To apply your styles, add this line at the end of viewDidLoad():

configureButtons()

Build and run and use the pointer to observe the differences between the two effects. .highlight is quite subtle, while .lift is more flamboyant.

Another effect you can experiment with is .hover. Have fun!

Applying Pointer Interactions to Other Views

You’ve learned how to apply a pointer effect to a button, but what about other views? In this section, you’ll apply a UIPointerInteraction to any UIView.

Reviewing the Game Controls

First, before making any changes, look at how the existing code works. In the Project navigator, open the folder UIKit View Controllers. In GameViewController.swift, locate configureGameButtons():

func configureGameButtons() {
  for (index, item) in zip(allColors, allButtons).enumerated() {
    let button = item.1
    let color = item.0
    button.color = color
    button.tag = index

    let tapGesture = UITapGestureRecognizer(
      target: self,
      action: #selector(gameButtonAction(_:))
    )
    button.addGestureRecognizer(tapGesture)
  }
}

This is where you set up the four large color tiles that make up the game controls. You apply a color, a tag and a UITapGestureRecognizer to each of the four controls.

Each of the tiles is an instance of GameButton.

Now return to the Project navigator. In UIKit View Controllers, open GameButton.swift. GameButton is a subclass of UIView. You’ll now add a pointer effect to instances of this view.

Adding Tracking Variables

First, you need to add some tracking variables to GameButton. Add these two properties inside the body of the main class:

var pointerLocation: CGPoint = .zero {
  didSet {
    setNeedsDisplay()
  }
}
  
var pointerInside = false {
  didSet {
    setNeedsDisplay()
  }
}

These properties allow you to keep track of the location of the pointer and whether the pointer is inside the view. Each time they change, you mark the view as needing a redraw. This will come in handy soon!

Adding a Delegate Extension

Next, add this delegate extension to the end of the file:

extension GameButton: UIPointerInteractionDelegate {
  //1
  func pointerInteraction(
    _ interaction: UIPointerInteraction,
    regionFor request: UIPointerRegionRequest,
    defaultRegion: UIPointerRegion
  ) -> UIPointerRegion? {
    pointerLocation = request.location
    return defaultRegion
  }

  //2
  func pointerInteraction(
    _ interaction: UIPointerInteraction,
    styleFor region: UIPointerRegion
  ) -> UIPointerStyle? {
    return nil
  }

  //3
  func pointerInteraction(
    _ interaction: UIPointerInteraction,
    willEnter region: UIPointerRegion,
    animator: UIPointerInteractionAnimating
  ) {
    pointerInside = true
  }

  //4
  func pointerInteraction(
    _ interaction: UIPointerInteraction,
    willExit region: UIPointerRegion,
    animator: UIPointerInteractionAnimating
  ) {
    pointerInside = false
  }
}

In this code, you define four delegate methods in an extension to GameButton. The game button object is the view and also the delegate. The app calls these four methods during the various parts of the lifecycle:

1. The pointer has moved, or is about to move, within the view. You, as the delegate, can return a UIPointerRegion. This is a sub-rectangle relative to the view’s coordinate space. You record the current location and return defaultRegion, i.e. bounds.
2. What kind of pointer style do you want the system to apply to this rectangular region you returned? For now, it’s nothing!
3. The pointer is about to enter a region.
4. The pointer is about to leave a region.

These four methods implement UIPointerInteractionDelegate. You use this protocol to define pointer behavior within the pointer interaction lifecycle. This lifecycle has three participants:

• The view receives support for pointer interaction.
• The system calls delegate methods as the pointer interacts with the view.
• The delegate responds to those delegate method calls.

As you can see, UIPointerInteractionDelegate offers very fine-grained control over the behavior of the pointer when it’s within a view.

Adding a Pointer Effect

Finally, locate awakeFromNib(). The system calls this method when you create a view from a XIB or Storyboard. At this point, everything in the storyboard or XIB has been created.

Add these lines to the start of awakeFromNib():

let interaction = UIPointerInteraction(delegate: self)
addInteraction(interaction)

You’ll now use this information to apply some super-fancy and useful drawing to the view. Add this stored property and function to the body of the main class:

let blobSize = CGFloat(60)

override func draw(_ rect: CGRect) {
  if pointerInside {
    let rect = CGRect(center: pointerLocation, size: blobSize)
    let blob = UIBezierPath(ovalIn: rect)
    UIColor.white.set()
    blob.fill()
  }
}

Build and run. Go to the UIKit section of the app and capture the pointer. When the cursor enters any of the four game buttons, you can see that you draw a circle at the cursor point — which is neither useful nor fancy! :]

OK, even though this code isn’t useful and fancy, it does demonstrate the potential to react to the position of the cursor within a view. You’ll have some fun with this circle later in the tutorial.