Introduction to Using OpenCV With Unity

Installing Additional Software

Setting Up Python and OpenCV

The process of setting up OpenCV varies a lot across the two major operating systems supported by Unity (Windows and macOS).

Follow the instructions below to set up OpenCV and Python according to your operating system.

Windows

– Register Anaconda as my default Python.
– Add Anaconda to my PATH environment variable.

You will be asked Proceed ([y]/n)?
Type ‘y’ and press Enter

1. Download and install Anaconda for Python 3.6 from https://www.continuum.io/downloads.
2. Make sure to check both these options while installing:

– Register Anaconda as my default Python.
– Add Anaconda to my PATH environment variable.

3. After Anaconda is installed, open command prompt as Administrator and execute the following command to install the required packages.

conda install -c menpo opencv

You will be asked Proceed ([y]/n)?
Type ‘y’ and press Enter

4. Once all packages have successfully installed, test your install by executing the following commands in the command prompt.

python

>>> import cv2

conda install -c menpo opencv

python

>>> import cv2

If you do not see any error, it means OpenCV has been successfully installed.

macOS

You will be asked Proceed ([y]/n)?
Type ‘y’ and press Enter.

1. Download and install Anaconda for Python 3 from https://www.continuum.io/downloads.
2. After Anaconda is installed, open Terminal (Applications ▸ Utilities ▸ Terminal) and execute the following command to install the required packages.

conda install -c menpo opencv

You will be asked Proceed ([y]/n)?
Type ‘y’ and press Enter.

3. Once all packages have successfully installed, test your install by executing the following command.

python

>>> import cv2

conda install -c menpo opencv

python

>>> import cv2

If you do not see any error, it means OpenCV has been successfully installed.

Getting the Python Server Running

Now that you have the theoretical part out of the way, start with getting your Hand Gesture Recognition working.

Open a command line utility, such as terminal Terminal (if you’re using MacOS) or CMD (if you’re using Windows), and change the directory to Starter ▸ Python.

Now, run the command python Recognition.py

Your webcam should become active and two new windows will open: “Full Frame” and “Recognition.”

The Full Frame window shows the complete frame of what your webcam is capturing. However, you will process only a small part of that (the frames within the green outline box) where the user’s hand is supposed to be.

Now, align your hand so that it is completely inside the green box. You should be able to see it in effect in the “Detection” window, with two previews of your hand:

1. The outline of your hand.
2. Your hand as seen by the webcam, with the outline overlaid on it as detected by OpenCV.

If everything works as expected, you should be able to see something like this:

The red colored outline makes up a complete polygon area covering your entire hand, and the green colored outline joins from point to point around the tips of your fingers:

To trigger a Jump action, completely open your hand and then close it to make a fist. If you do this properly, you see “Jump Action Triggered!” in the command line logs:

When a fist is detected, the string “JUMP!” is sent via Sockets to port “5065.”

Check out line number 126 in Recognition.py to see the exact code that is being executed.

To stop the Python server, with one of the windows active, press the ‘Q’ key on your keyboard, or with your Terminal window active, press Ctrl + C on Windows or Command + C if you’re using a Mac.

Next, learn how to receive this data in Unity.

Receiving Data Via UDP in Unity

Now that you understand the basics of communications protocols and have Hand Detection working with OpenCV, the following steps will guide you to receive that data in your Unity instance. Open the Main scenes from the Scenes folder to get started.

In the Project Hierarchy, select Managers ▸ PlayerController. In the Inspector window, you’ll see that a script PlayerController.cs is attached to it.

Open PlayerController.cs in your favorite code editor. You should see six comments.

You’ll now add pieces of code snippet below these comments to add the necessary functionality to your project.

To start, add the following code below the comment // 1. Declare Variables.

// 1. Declare Variables

Thread receiveThread; //1
UdpClient client; //2
int port; //3

public GameObject Player; //4
AudioSource jumpSound; //5
bool jump; //6

Looking at each piece comment-by-comment:

1. Declare a variable of class Thread: This will be used to start a thread that will be continuously running in the background.
2. Declare a variable of class UdpClient: This will parse the pre-defined address for data, which will be used to call the necessary methods.
3. An integer type variable that stores the port number.
4. A reference to the Player.
5. An AudioSource type variable to store the reference to the ‘Jump’ sound that will be played whenever the Player jumps.
6. A Boolean variable. The value of this variable will be checked in each frame and, based on that, the Jump action will be triggered.

Save the file and go back to the editor.

From the Hierarchy window, select Managers ▸ PlayerController to make it active and see its properties in the Inspector. Now, select PlayerObject ▸ Player and drag-and-drop it onto the placeholder for the Player GameObject in PlayerController script:

Also, with the PlayerController selected, add an AudioSource component to it by selecting Add Component ▸ Audio ▸ Audio Source.

In the Inspector window, de-select Play On Awake.

From the Project window, open the Sounds folder, and drag-and-drop the JumpSound audio file, and drop it onto the placeholder for AudioClip in the Inspector window:

Now that you have all the necessary variables declared and the GameObject references set, you’ll add pieces of program logic.

Add the following method under // 2. Initialize variables.

// 2. Initialize variables

void Start () 
{
  port = 5065; //1 
  jump = false; //2 
  jumpSound = gameObject.GetComponent<AudioSource>(); //3

  InitUDP(); //4
}

This is fairly straightforward:

1. Variable port is initialized with value: 5065 (the same value we are using in our Python instance).
2. Variable jump is initialized as Boolean false. This variable will be set to true whenever you get the “Jump!” message from the Python instance.
3. A reference to the AudioSource component attached to the PlayerController GameObject is stored in variable jumpSound.
4. The InitUDP() method is called. Don’t worry about the error, you will the code for it in the next step.

In order to be able to read a given IP address via UDP, a thread has to be created and set to run in the background.

If you’re unfamiliar with threads, they are components of a process that can be used to achieve parallelism by executing concurrently in the running process, also sharing resources such as memory within this process. In simple terms a thread can be started to run work (such as polling for UDP data in this case) in the background whilst your Unity script code continues to run in your Unity game process.

When you are dealing with computationally expensive or long-term operations, threads are very useful. In addition to performing network communication, they are also commonly used for running AI sub-processes in the background, running path-finding algorithms, performing file operations and much more.

Now, add the following method under // 3. InitUDP.

// 3. InitUDP

private void InitUDP()
{
  print ("UDP Initialized");

  receiveThread = new Thread (new ThreadStart(ReceiveData)); //1 
  receiveThread.IsBackground = true; //2
  receiveThread.Start(); //3
}

This method creates a new thread, and gives the method ReceiveData() as an argument, which will be defined in the next step, for it to handle all the data.

Take a look at what each line of code means:

1. Variable receiveThread is initialized as a new Thread with the method ReceiveData() as an argument.
2. The thread type is set as “Background” so that it runs parallel to your game code.
3. Thread receiveThread is set to Start.

Once a separate thread has been initiated, you’ll need a method to actually read the data from the predefined IP Address that will be sent by the Python-OpenCV server.

Add the following lines of code below // 4. Receive Data:

// 4. Receive Data

private void ReceiveData()
{
  client = new UdpClient (port); //1
  while (true) //2
  {
    try
    {
      IPEndPoint anyIP = new IPEndPoint(IPAddress.Parse("0.0.0.0"), port); //3
      byte[] data = client.Receive(ref anyIP); //4

      string text = Encoding.UTF8.GetString(data); //5
      print (">> " + text);

      jump = true; //6

    } 
    catch(Exception e)
    {
      print (e.ToString()); //7
    }
  }
}

A comment-by-comment explanation is below:

1. Variable client is assigned port 5065.
2. A while loop is initiated. You can use a variable as an exit condition if you require, but for the purposes of this tutorial this will do just fine. Within the body, instead of just having the code for receiving the data using the UDP Client, adding a try/catch is always a good practise, so that in case of an error and if no data is received, the complete program does not crash. It also makes managing errors and bugs easy with the help of logs.
3. The IP Endpoint (where the value “Jump!” will be read from) is declared.
4. Data read from the IP Endpoint declared above stored in the variable “data” in binary form.
5. Data in binary form is encoded to a utf-8 string format and stored in the “text” variable.
6. Since the only data being sent from the Python instance is the string “Jump!”, the Boolean variable “jump” is set to true. Later on you will add the Update() method, which is called once every frame and will check for the value of this variable. If true it will trigger the Jump action on the player.
7. If an exception occurs it will be logged to the console.

Now that all the checks for getting the data are in place, add the functionality to trigger the Jump animation and play the Jump sound.

Add the following method below // 5. Make the Player Jump:

// 5. Make the Player Jump

public void Jump()
{
  Player.GetComponent<Animator>().SetTrigger ("Jump"); //1
  jumpSound.PlayDelayed(44100); // Play Jump Sound with a 1 second delay to match the animation
}

Save the file and return to the editor. From the Hierarchy window, select the player from PlayerObjects ▸ Player and Open the Animator window.

You will see that there are two animations baked within the player: “Idle” (the default Animation) and “Jump”, which is set to run when “Jump” Trigger is set.

1. The first line in the Jump() method sets the Trigger as “Jump” so that the player moves from “Idle” animation to the “Jump” animation and back.
2. The next line simply plays the Jump sound with a one-second delay, in order to be in sync with the Jump animation.

Now that you have added all the necessary logic for Initializing the required thread, and the checking and receiving of data via UDP, all that’s left is to add a check for the value of Boolean variable Jump in the Update() method.

Finally, add the following lines of code after the comment // 6. Check for variable value, and make the Player jump!.

// 6. Check for variable value, and make the Player jump!

void Update () 
{
  if(jump == true)
  {
    Jump ();
    jump = false;
  }
}

Here, the Update() method, which is called once every frame check if the value of Boolean variable is set to true.

When it does, it calls the Jump() Method and then sets the value of “jump” back to false so that our player is not continuously jumping.

And that’s it! That was all the code required to add to the project to work.

Finally, save the PlayerController.cs script and go back to the Unity editor.

As done previously, type python Detection.py and click Enter in your command line utility. Your Python OpenCV instance should now be running.

Simultaneously, click on the Play button in Unity to play the scene.

If you’ve done everything correctly, you should now be able to make a fist with your hand and it will make the player in Unity jump!

Here’s how it should look:

And that’s it! You now have a working example project of how to use OpenCV with Unity.

As mentioned earlier, if you got stuck at some point or have any errors, you can find the complete project using the “Download Materials” link at the top or the bottom of this tutorial.