Getting Started with Hand Pose Classifier in Block Coding

Tutorial Video

Introduction

The Hand Gesture Classifier of the PictoBlox Machine Learning Environment is used for classifying different hand poses into different classes. The model works by analyzing the position of your hand with the help of 21 data points. We’ll look at this in detail in this tutorial.

For this project, we’re going to make a “Beetle in the Maze” game with hand gesture control.

We will control the Beetle using hand gestures and make sure that it doesn’t touch the maze. For this task, we’ll need 5 classes:

1. Forward 
2. Backward
3. Left
4. Right
5. Stop

We’ll train our model to recognize different gestures and then map them to the Beetle Sprite. That way, the Beetle can execute specific functions when a specific gesture is recognized.

In this lesson, we’ll learn how to construct the ML model using the PictoBlox Hand Gesture Classifier. These are the steps involved in the procedure:

1. Setting up the environment
2. Gathering gestures(Data Collection)
3. Training the model
4. Testing the model
5. Exporting the model to PictoBlox
6. Creating a script in PictoBlox

Setting up the Environment

First, we need to set the ML environment for hand gesture classification.

Alert: The Machine Learning Environment for model creation is available in the only desktop version of PictoBlox for Windows, macOS, or Linux. It is not available in Web, Android, and iOS versions.

Follow the steps below:

1. Download the Beetle in the Maze game from here: Beetle in the Maze Game Basic
2. Open PictoBlox and open the downloaded project. The environment of the project is already set for us to code.
   
3. To access the ML Environment, select the “Open ML Environment” option under the “Files” tab.
   
4. You’ll be greeted with the following screen.
   Click on “Create New Project“.
5. A window will open. Type in a project name of your choice and select the “Hand Gesture Classifier” extension. Click the “Create Project” button to open the Hand Gesture Classifier window.
6. You shall see the Hand Gesture Classifier workflow with two classes already made for you. Your environment is all set. Now it’s time to upload the data.
   

Collecting and Uploading the Data

Class is the category in which the Machine Learning model classifies the images. Similar images are put in one class.

There are 2 things that you have to provide in a class:

1. Class Name
2. Image Data: This data can either be taken from the webcam or by uploading from local storage or from google drive.

In this project:

1. The face of the palm is set to “Forward”.
2. The back of the palm is set to “Backward”.
3. The palm tilted towards the left is set to “Left”.
4. The palm tilted towards the right is set to “Right”.
5. The closed fist is set to “Stop”.

Note: Feel free to choose your own gestures.

Follow the steps to upload the data for the classes:

1. Rename the first class name Forward.
2. Click the Webcam button.
   If you want to change your camera feed, you can do it from the webcam selector in the top right corner.
   
3. Next, click on the “Hold to Record” button to capture the for. Take 100+ hand gestures with different hand positions. If you want to delete any image, then hover over the image and click on the delete button. Once uploaded, you will be able to see the images in the class.
4. Rename Class 2 as Backward and take the samples from the webcam while showing the back of the palm.
5. Click the “Add Class” button, and you shall see a new class in your Environment. Rename the class name to Left. Take the samples from the webcam while showing the palm tilted towards the left.
   
6. Click the “Add Class” button and rename the class name to Right.  Take the samples from the webcam while showing the palm tilted towards the right.
7. Click the “Add Class” button and rename the class name to Stop.  Take the samples from the webcam while showing the closed fist.
   

Note: You must add at least 20 samples to each of your classes for your model to train. More samples will lead to better results.

As you can see, now each class has some data to derive patterns from. In order to extract and use these patterns, we must train our model.

Training the Model

Now that we have gathered the data, it’s time to teach our model how to classify new, unseen data into these three classes. In order to do this, we have to train the model.

By training the model, we extract meaningful information from the images, and that in turn updates the weights. Once these weights are saved, we can use our model to make predictions on data previously unseen.

However, before training the model, there are a few hyperparameters that you should be aware of. Click on the “Advanced” tab to view them.

Note: These hyperparameters can affect the accuracy of your model to a great extent. Experiment with them to find what works best for your data.

There are three hyperparameters you can play along with here:

1. Epochs– The total number of times your data will be fed through the training model. Therefore, in 10 epochs, the dataset will be fed through the training model 10 times. Increasing the number of epochs can often lead to better performance.
2. Batch Size– The size of the set of samples that will be used in one step. For example, if you have 160 data samples in your dataset, and you have a batch size of 16, each epoch will be completed in 160/16=10 steps. You’ll rarely need to alter this hyperparameter.
3. Learning Rate– It dictates the speed at which your model updates the weights after iterating through a step. Even small changes in this parameter can have a huge impact on the model performance. The usual range lies between 0.001 and 0.0001.

Note: Hover your mouse over the question mark next to the hyperparameters to see their description.

Let’s train the model with the standard hyperparameters and see how it performs. You can train the model in both JavaScript and Python. In order to choose between the two, click on the switch on top of the Training box. 

Note: You must download dependencies to train the model in Python, JavaScript will be chosen by default.

We’ll be training this model in Python. Click on the “Train Model” button to commence training. Let’s change the “Epochs” hyperparameter to 20.

The model shows great results! Remember, the higher the reading in the accuracy graph, the better the model. The x-axis of the graph shows the epochs, and the y-axis represents the corresponding accuracy. The range of the accuracy is 0 to 1.

Testing the Model

Now that the model is trained, let us see if it delivers the expected results. We can test the model by either using the device’s camera or by uploading an image from the device’s storage. Let’s use our webcam to start with.

Click on the “Webcam” option in the testing box and the model will start predicting based on the image in the window.

Great! The model is able to recognize gestures in real-time. Now close the window by clicking on the left arrow on the top right of the testing box. Time to export the model into PictoBlox!

Exporting the Model to the Block Coding Environment

Click on the “Export Model” button on the top right of the Testing box, and PictoBlox will load your model into the Block Coding Environment.

Observe how we have blocks pertaining to the model we just trained on the left side panel.

Each block here has a specific function:

1. open recognition window:  Opens a window that uses the device’s camera to make predictions.
   
2. turn on/off video one stage with x transparency: This shows the video captured by the camera on the stage.
3. analyse image from web camera/stage/costume/backdrop: Analyze the image input.
4. show/hide key points: This shows how the key points are mapped.
5. is identify class in “class”: Identifies if the analyzed image belongs to a certain class.
6. get confidence of class “class”: returns the confidence level of gesture classification. Can be used as a threshold value.

The script in the Block Coding

You already have the following code available in the PictoBlox code:

We will edit the following code:

1. Add the is identified class () block in the if condition and set the appropriate direction.
   
2. Add the analyse image from () block and select stage as the option.

Let’s see the results. Click on the Green flag to start the execution.

And we’re done! We have made a Gesture-controlled Beetle in a Maze game.