Object Counting Bot in Block Coding using Object Detection

Introduction

In this example project we are going to create a Machine Learning Model which can count number of nuts and bolts from the camera feed or images.

Object Detection in Machine Learning Environment

Object Detection is an extension of the ML environment that allows users to detect images and making bounding box into different classes. This feature is available only in the desktop version of PictoBlox for Windows, macOS, or Linux. As part of the Object Detection workflow, users can add classes, upload data, train the model, test the model, and export the model to the Block Coding Environment.

 Opening Image Detection Workflow

Follow the steps below:

1. Open PictoBlox and create a new file.
   
2.  Select the Block Coding Environment.
   
3. Select the “Open ML Environment” option under the “Files” tab to access the ML Environment.
   
4. You’ll be greeted with the following screen.
   Click on “Create New Project”.
   
5. A window will open. Type in a project name of your choice and select the “Object Detection” extension. Click the “Create Project” button to open the Object Detection window.

You shall see the Object Detection workflow. Your environment is all set.
   

Collecting and Uploading the Data

Uploading images from your device’s hard drive

1. Now it’s time to upload the images which you downloaded from another source or captured from your camera. Click on the “Select from device” option from the Import Images block.
2. Now click on the “Choose images from your computer” and go to the folder where you downloaded your images.
3. Select all images which you want to upload then click on “open” option.
4. Now page of PictoBlox looks like:

Making Bounding Box – Labelling Images

1. Labeling is essential for Object Detection. Click on the “Bbox” tab to make the labels.
   
   

   Notes: Notice how the targets are marked with a bounding box. The labels appear in the “Label List” column on the right.

2. To create the bounding box in the images, click on the “Create Box” button, to create a bounding box. After the box is drawn, go to the “Label List” column and click on the edit button, and type in a name for the object under the bounding box. This name will become a class. Once you’ve entered the name, click on the tick mark to label the object.
   
3. File List: It shows the list of images available for labeling in the project.
4. Label List: It shows the list of Labels created for the selected image.
5. Class Info: It shows the summary of the classes with the total number of bounding boxes created for each class.
6.   You can view all the images under the “Image” tab.

Training the Model

In Object Detection, the model must locate and identify all the targets in the given image. This makes Object Detection a complex task to execute. Hence, the hyperparameters work differently in the Object Detection Extension.

1. Go to the “Train” tab. You should see the following screen:
   
2. Click on the “Train New Model” button.
   
3. Select all the classes, and click on “Generate Dataset”.
   
4.  Once the dataset is generated, click “Next”. You shall see the training configurations.
5. Specify your hyperparameters. If the numbers go out of range, PictoBlox will show a message.
6. Click “Create”, It creates new model according to inserting value of hyperparameter.
   
7. Click “Start Training”, If desired performance is reached, click on the “Stop” 
   1. Total Loss
   2. Regularization Loss
   3. Localization Loss
   4. Classification Loss
8. After the training is completed, you’ll see four loss graphs: 
   
   Note: Training an Object Detection model is a time taking task. It might take a couple of hours to complete training

    

9. You’ll be able to see the graphs under the “Graphs” panel. Click on the buttons to view the graph.
   
   1. Graph between “Total loss” and “Number of steps”. 
      
   2. Graph between “Regularization loss” and “Number of steps”.
      
   3. Graph between “Localization” and “Number of steps”.
      
   4. Graph between “Classification loss” and “Number of steps”.
      

Evaluating the Model

Now, let’s move to the “Evaluate” tab. You can view True Positives, False Negatives, and False Positives for each class here along with metrics like Precision and Recall.

Testing the Model

The model will be tested by uploading an Image from device:

Export in Block Coding

Click on the “PictoBlox” button, and PictoBlox will load your model into the Block Coding Environment if you have opened the ML Environment in the Block Coding.

Code

The idea is simple, we’ll add image samples in the “Backdrops” column. We’ll keep cycling through the backdrops and keep predicting the image on the stage.

1. Add testing images in the backdrop and delete default backdrop.
2. Now, come back to the coding tab and select the Tobi sprite.
3. We’ll start by adding a when flag clicked block from the Events palette.
4. Add () bounding box block from the Machine Learning palette. Select the “hide” option.
5. Follow it up with a set detection threshold to () block from the Machine Learning palette and set the drop-down to 0.5.
6. Add switch backdrop to () block from the Looks palette. Select any image.
   
7. Add a forever block from the  Control palette
8. Add analyse image from() block from the Machine Learning palette. Select the “stage” option.
   
9. Add () bounding box block from the Machine Learning palette. Select the “show” option.
10. Add two blocks of say () for () seconds from the Looks palette.
11. Inside the say block add join () () block from operator palette.
12. Inside the join block write statement at first empty place and at second empty place add get number of () detected? from the Machine Learning palette.
13. Select the “Nut” option for first get number of () detected and for second choose “Bolt” option.
14. Add the () bounding box block from the Looks palette. Select the “hide” option.
15. Finally, add the next backdrop block from the Looks palette below the () bounding box block.
    

Final Result