Gesture Controlled Quadruped With Machine Learning

Introduction

This project demonstrates how to use Machine Learning Environment to make a machine–learning model that identifies hand gestures and makes the quadruped move accordingly.

To test the model, simply enter the input values in the “Testing” panel and click on the “Predict” button.

The model will return the probability of the input belonging to the classes.

Export in Python Coding

Click on the “Export Model” button on the top right of the Testing box, and PictoBlox will load your model into the Python Coding Environment if you have opened the ML Environment in Python Coding.

code

The following code appears in the Python Editor of the selected sprite.

####################imports####################
# Do not change

import numpy as np
import tensorflow as tf
import time
quarky=Quarky

quad=Quadruped(4,1,8,5,3,2,7,6)

# Do not change
####################imports####################

#Following are the model and video capture configurations
# Do not change

model=tf.keras.models.load_model(
    "num_model.h5",
    custom_objects=None,
    compile=True,
    options=None)
pose = Posenet()                                                    # Initializing Posenet
pose.enablebox()                                                    # Enabling video capture box
pose.video("on",0)                                                  # Taking video input
class_list=['Forward','Backward','Left','Right','Stop']                  # List of all the classes

# Do not change
###############################################

#This is the while loop block, computations happen here
# Do not change

while True:
  pose.analysehand()                                             # Using Posenet to analyse hand pose
  coordinate_xy=[]
    
    # for loop to iterate through 21 points of recognition
  for i in range(21):
    if(pose.gethandposition(1,i,0)!="NULL"  or pose.gethandposition(2,i,0)!="NULL"):
      coordinate_xy.append(int(240+float(pose.gethandposition(1,i,0))))
      coordinate_xy.append(int(180-float(pose.gethandposition(2,i,0))))
    else:
      coordinate_xy.append(0)
      coordinate_xy.append(0)
            
  coordinate_xy_tensor = tf.expand_dims(coordinate_xy, 0)        # Expanding the dimension of the coordinate list
  predict=model.predict(coordinate_xy_tensor)                    # Making an initial prediction using the model
  predict_index=np.argmax(predict[0], axis=0)                    # Generating index out of the prediction
  predicted_class=class_list[predict_index]                      # Tallying the index with class list
  print(predicted_class)

    
  # Do not change

Logic

1. If the identified class from the analyzed image is “forward,” the Quadruped will move forward at a specific speed.
2. If the identified class is “backward,” the Quadruped will move backward.
3. If the identified class is “left,” the Quadruped will move left.
4. If the identified class is “right,” the Quadruped will move right.
5. Otherwise, the Quadruped will be in the home position.

Add this code in

def runQuarky(predicted_class):
    if pose.ishanddetected():
      if predicted_class == "Forward":
        quad.move("forward",1000,1)
        
      elif predicted_class == "Backward":
        quad.move("backward",1000,1)
        
      elif predicted_class == "Left":
        quad.move("turn left",1000,1)
        
      elif predicted_class == "Right":
        quad.move("turn right",1000,1)
        
      elif predicted_class == "Stop":
        quad.home()

Final code

####################imports####################
# Do not change

import numpy as np
import tensorflow as tf
import time
quarky=Quarky

quad=Quadruped(4,1,8,5,3,2,7,6)

# Do not change
####################imports####################

#Following are the model and video capture configurations
# Do not change

model=tf.keras.models.load_model(
    "num_model.h5",
    custom_objects=None,
    compile=True,
    options=None)
pose = Posenet()                                                    # Initializing Posenet
pose.enablebox()                                                    # Enabling video capture box
pose.video("on",0)                                                  # Taking video input
class_list=['Forward','Backward','Left','Right','Stop']                  # List of all the classes

def runQuarky(predicted_class):
    if pose.ishanddetected():
      if predicted_class == "Forward":
        quad.move("forward",1000,1)
        
      elif predicted_class == "Backward":
        quad.move("backward",1000,1)
        
      elif predicted_class == "Left":
        quad.move("turn left",1000,1)
        
      elif predicted_class == "Right":
        quad.move("turn right",1000,1)
        
      elif predicted_class == "Stop":
        quad.home()
# Do not change
###############################################

#This is the while loop block, computations happen here
# Do not change

while True:
  pose.analysehand()                                             # Using Posenet to analyse hand pose
  coordinate_xy=[]
    
    # for loop to iterate through 21 points of recognition
  for i in range(21):
    if(pose.gethandposition(1,i,0)!="NULL"  or pose.gethandposition(2,i,0)!="NULL"):
      coordinate_xy.append(int(240+float(pose.gethandposition(1,i,0))))
      coordinate_xy.append(int(180-float(pose.gethandposition(2,i,0))))
    else:
      coordinate_xy.append(0)
      coordinate_xy.append(0)
            
  coordinate_xy_tensor = tf.expand_dims(coordinate_xy, 0)        # Expanding the dimension of the coordinate list
  predict=model.predict(coordinate_xy_tensor)                    # Making an initial prediction using the model
  predict_index=np.argmax(predict[0], axis=0)                    # Generating index out of the prediction
  predicted_class=class_list[predict_index]                      # Tallying the index with class list
  print(predicted_class)
  runQuarky(predicted_class)
    
  # Do not change

Output