analysehand()

Function Definition: analysehand()

Parameters

Description

This function is used to analyze the image received as input from the camera, for human hand detection.

Alert: This block processes the image input and updates the values in the other functions hence it needs to be put inside loops while making projects.

Example

Air Draw – Python

The example demonstrates how to use hand recognition and pen extensions to make an air draw game in the Python Coding Environment.

Code

sprite = Sprite('Pencil')

hand = Posenet()
pen = Pen()

hand.video("on", 0)
hand.disablebox()

pen.clear()
pen.setcolor([0, 255, 0])
pen.setsize(2)

while True:
  hand.analysehand()
  if hand.ishanddetected():
    if sprite.iskeypressed("space"):
      pen.down()
      sprite.setx(hand.gethandposition(1, 8))
      sprite.sety(hand.gethandposition(2, 8))
    else:
      pen.up()

Output

Finger Tracking – Python

The example demonstrates how to use hand recognition to track the different parts of the fingers in Python Coding Environment.

Code

thumb = Sprite('Thumb')
index = Sprite('Index')
middle = Sprite('Middle')
ring = Sprite('Ring')
pinky = Sprite('Pinky')

hand = Posenet()
hand.video("on", 0)
hand.enablebox()

thumb.switchcostume("ball-a")
thumb.setsize(50)
index.switchcostume("ball-b")
index.setsize(50)
middle.switchcostume("ball-c")
middle.setsize(50)
ring.switchcostume("ball-d")
ring.setsize(50)
pinky.switchcostume("ball-e")
pinky.setsize(50)

while True:
  hand.analysehand()
  
  if hand.ishanddetected():
    thumb.setx(hand.gethandposition(1, 4))
    thumb.sety(hand.gethandposition(2, 4))
    thumb.show()
    
    index.setx(hand.gethandposition(1, 8))
    index.sety(hand.gethandposition(2, 8))
    index.show()
    
    middle.setx(hand.gethandposition(1, 12))
    middle.sety(hand.gethandposition(2, 12))
    middle.show()
    
    ring.setx(hand.gethandposition(1, 16))
    ring.sety(hand.gethandposition(2, 16))
    ring.show()
    
    pinky.setx(hand.gethandposition(1, 20))
    pinky.sety(hand.gethandposition(2, 20))
    pinky.show()
  
  else:
    thumb.hide()
    index.hide()
    middle.hide()
    ring.hide()
    pinky.hide()

Output

Gesture Controlled Quadruped With Machine Learning

Learn to demonstrate how to use ML Environment to make a model that identifies the hand gestures and makes the Quadruped move accordingly.

Introduction

This project demonstrates how to use Machine Learning Environment to make a machine–learning model that identifies hand gestures and makes the quadruped move accordingly.

To test the model, simply enter the input values in the “Testing” panel and click on the “Predict” button.

The model will return the probability of the input belonging to the classes.

Export in Python Coding

Click on the “Export Model” button on the top right of the Testing box, and PictoBlox will load your model into the Python Coding Environment if you have opened the ML Environment in Python Coding.

code

The following code appears in the Python Editor of the selected sprite.

####################imports####################
# Do not change

import numpy as np
import tensorflow as tf
import time
quarky=Quarky

quad=Quadruped(4,1,8,5,3,2,7,6)

# Do not change
####################imports####################

#Following are the model and video capture configurations
# Do not change

model=tf.keras.models.load_model(
    "num_model.h5",
    custom_objects=None,
    compile=True,
    options=None)
pose = Posenet()                                                    # Initializing Posenet
pose.enablebox()                                                    # Enabling video capture box
pose.video("on",0)                                                  # Taking video input
class_list=['Forward','Backward','Left','Right','Stop']                  # List of all the classes

# Do not change
###############################################

#This is the while loop block, computations happen here
# Do not change

while True:
  pose.analysehand()                                             # Using Posenet to analyse hand pose
  coordinate_xy=[]
    
    # for loop to iterate through 21 points of recognition
  for i in range(21):
    if(pose.gethandposition(1,i,0)!="NULL"  or pose.gethandposition(2,i,0)!="NULL"):
      coordinate_xy.append(int(240+float(pose.gethandposition(1,i,0))))
      coordinate_xy.append(int(180-float(pose.gethandposition(2,i,0))))
    else:
      coordinate_xy.append(0)
      coordinate_xy.append(0)
            
  coordinate_xy_tensor = tf.expand_dims(coordinate_xy, 0)        # Expanding the dimension of the coordinate list
  predict=model.predict(coordinate_xy_tensor)                    # Making an initial prediction using the model
  predict_index=np.argmax(predict[0], axis=0)                    # Generating index out of the prediction
  predicted_class=class_list[predict_index]                      # Tallying the index with class list
  print(predicted_class)

    
  # Do not change

Logic

If the identified class from the analyzed image is “forward,” the Quadruped will move forward at a specific speed.
If the identified class is “backward,” the Quadruped will move backward.
If the identified class is “left,” the Quadruped will move left.
If the identified class is “right,” the Quadruped will move right.
Otherwise, the Quadruped will be in the home position.

Add this code in

def runQuarky(predicted_class):
    if pose.ishanddetected():
      if predicted_class == "Forward":
        quad.move("forward",1000,1)
        
      elif predicted_class == "Backward":
        quad.move("backward",1000,1)
        
      elif predicted_class == "Left":
        quad.move("turn left",1000,1)
        
      elif predicted_class == "Right":
        quad.move("turn right",1000,1)
        
      elif predicted_class == "Stop":
        quad.home()

Final code

####################imports####################
# Do not change

import numpy as np
import tensorflow as tf
import time
quarky=Quarky

quad=Quadruped(4,1,8,5,3,2,7,6)

# Do not change
####################imports####################

#Following are the model and video capture configurations
# Do not change

model=tf.keras.models.load_model(
    "num_model.h5",
    custom_objects=None,
    compile=True,
    options=None)
pose = Posenet()                                                    # Initializing Posenet
pose.enablebox()                                                    # Enabling video capture box
pose.video("on",0)                                                  # Taking video input
class_list=['Forward','Backward','Left','Right','Stop']                  # List of all the classes

def runQuarky(predicted_class):
    if pose.ishanddetected():
      if predicted_class == "Forward":
        quad.move("forward",1000,1)
        
      elif predicted_class == "Backward":
        quad.move("backward",1000,1)
        
      elif predicted_class == "Left":
        quad.move("turn left",1000,1)
        
      elif predicted_class == "Right":
        quad.move("turn right",1000,1)
        
      elif predicted_class == "Stop":
        quad.home()
# Do not change
###############################################

#This is the while loop block, computations happen here
# Do not change

while True:
  pose.analysehand()                                             # Using Posenet to analyse hand pose
  coordinate_xy=[]
    
    # for loop to iterate through 21 points of recognition
  for i in range(21):
    if(pose.gethandposition(1,i,0)!="NULL"  or pose.gethandposition(2,i,0)!="NULL"):
      coordinate_xy.append(int(240+float(pose.gethandposition(1,i,0))))
      coordinate_xy.append(int(180-float(pose.gethandposition(2,i,0))))
    else:
      coordinate_xy.append(0)
      coordinate_xy.append(0)
            
  coordinate_xy_tensor = tf.expand_dims(coordinate_xy, 0)        # Expanding the dimension of the coordinate list
  predict=model.predict(coordinate_xy_tensor)                    # Making an initial prediction using the model
  predict_index=np.argmax(predict[0], axis=0)                    # Generating index out of the prediction
  predicted_class=class_list[predict_index]                      # Tallying the index with class list
  print(predicted_class)
  runQuarky(predicted_class)
    
  # Do not change

Output

Python Based Hand Pose Controlled Mecanum

Learn how to use the Hand Gesture Classifier of the Machine Learning Environment to make a machine-learning model that identifies hand gestures and makes the Mecanum move accordingly.

This project demonstrates how to use Machine Learning Environment to make a machine–learning model that identifies the hand gestures and makes the Mecanum move accordingly.

We are going to use the Hand Classifier of the Machine Learning Environment. The model works by analyzing your hand position with the help of 21 data points. We will add in total 8 different classes to operate the different motions of the Mecanum Robot with the help of the ML Environment of the Pictoblox Software.

Hand Gesture Classifier Workflow

Follow the steps below:

Open PictoBlox and create a new file.
Select the coding environment as appropriate Coding Environment.
Select the “Open ML Environment” option under the “Files” tab to access the ML Environment.
Click on “Create New Project“.
A window will open. Type in a project name of your choice and select the “Hand Gesture Classifier” extension. Click the “Create Project” button to open the Hand Pose Classifier window.
You shall see the Classifier workflow with two classes already made for you. Your environment is all set. Now it’s time to upload the data.

Class in Hand Gesture Classifier

There are 2 things that you have to provide in a class:

Class Name: It’s the name to which the class will be referred as.
Hand Pose Data: This data can either be taken from the webcam or by uploading from local storage.

Note: You can add more classes to the projects using the Add Class button.

Adding Data to Class

You can perform the following operations to manipulate the data into a class.

Naming the Class: You can rename the class by clicking on the edit button.
Adding Data to the Class: You can add the data using the Webcam or by Uploading the files from the local folder.
1. Webcam:

Note: You must add at least 20 samples to each of your classes for your model to train. More samples will lead to better results.

Training the Model

After data is added, it’s fit to be used in model training. In order to do this, we have to train the model. By training the model, we extract meaningful information from the hand pose, and that in turn updates the weights. Once these weights are saved, we can use our model to make predictions on data previously unseen.

The accuracy of the model should increase over time. The x-axis of the graph shows the epochs, and the y-axis represents the accuracy at the corresponding epoch. Remember, the higher the reading in the accuracy graph, the better the model. The range of the accuracy is 0 to 1.

Testing the Model

To test the model, simply enter the input values in the “Testing” panel and click on the “Predict” button.

The model will return the probability of the input belonging to the classes.

Export in Python Coding

Logic

The mecanum will move according to the following logic:

When the forward gesture is detected – Mecanum will move forward.
When the backward gesture is detected – Mecanum will move backwards.
When the Lateral Left gesture is detected – Mecanum will move towards the left direction laterally with the help of its omnidirectional wheels.
When the Lateral Right gesture is detected – Mecanum will move towards the right direction laterally with the help of its omnidirectional wheels.
When the Stop gesture is detected – Mecanum will stop moving.
When the Normal Left gesture is detected – Mecanum will rotate in the left direction.
When the Normal Right gesture is detected – Mecanum will rotate in the right direction.
When the Circular Motion gesture is detected – Mecanum will move in a lateral arc.

Code

The following code appears in the Python Editor of the selected sprite.

####################imports####################
# Do not change

import numpy as np
import tensorflow as tf
import time

# Do not change
####################imports####################

#Following are the model and video capture configurations
# Do not change

model=tf.keras.models.load_model(
    "num_model.h5",
    custom_objects=None,
    compile=True,
    options=None)
pose = Posenet()                                                    # Initializing Posenet
pose.enablebox()                                                    # Enabling video capture box
pose.video("on",0)                                                  # Taking video input
class_list=['Forward','Backward','Stop','LateralRight','LateralLeft','NormalRight','NormalLeft','CircularMotion']                  # List of all the classes
meca=Mecanum(1,2,7,8)
def runmecanum(predicted_class):
  if pose.ishanddetected():
    if predicted_class=="Forward":
      meca.runtimedrobot("forward",100,2)
    if predicted_class=="Backward":
      meca.runtimedrobot("backward",100,2)
    if predicted_class=="Stop":
      meca.stoprobot()
    if predicted_class=="LateralRight":
      meca.runtimedrobot("lateral right",100,2)
    if predicted_class=="LateralLeft":
      meca.runtimedrobot("lateral left",100,2)
    if predicted_class=="NormalRight":
      meca.runtimedrobot("circular right",100,1)
    if predicted_class=="NormalLeft":
      meca.runtimedrobot("circular left",100,1)
    if predicted_class=="CircularMotion":
      meca.runtimedrobot("lateral arc",100,1)
    
# Do not change
###############################################

#This is the while loop block, computations happen here
# Do not change

while True:
  pose.analysehand()                                             # Using Posenet to analyse hand pose
  coordinate_xy=[]
    
    # for loop to iterate through 21 points of recognition
  for i in range(21):
    if(pose.gethandposition(1,i,0)!="NULL"  or pose.gethandposition(2,i,0)!="NULL"):
      coordinate_xy.append(int(240+float(pose.gethandposition(1,i,0))))
      coordinate_xy.append(int(180-float(pose.gethandposition(2,i,0))))
    else:
      coordinate_xy.append(0)
      coordinate_xy.append(0)
            
  coordinate_xy_tensor = tf.expand_dims(coordinate_xy, 0)        # Expanding the dimension of the coordinate list
  predict=model.predict(coordinate_xy_tensor)                    # Making an initial prediction using the model
  predict_index=np.argmax(predict[0], axis=0)                    # Generating index out of the prediction
  predicted_class=class_list[predict_index]                      # Tallying the index with class list
  print(predicted_class)
  runmecanum(predicted_class)
  # Do not change

Logical Code

def runmecanum(predicted_class):
  if pose.ishanddetected():
    if predicted_class=="Forward":
      meca.runtimedrobot("forward",100,2)
    if predicted_class=="Backward":
      meca.runtimedrobot("backward",100,2)
    if predicted_class=="Stop":
      meca.stoprobot()
    if predicted_class=="LateralRight":
      meca.runtimedrobot("lateral right",100,2)
    if predicted_class=="LateralLeft":
      meca.runtimedrobot("lateral left",100,2)
    if predicted_class=="NormalRight":
      meca.runtimedrobot("circular right",100,1)
    if predicted_class=="NormalLeft":
      meca.runtimedrobot("circular left",100,1)
    if predicted_class=="CircularMotion":
      meca.runtimedrobot("lateral arc",100,1)

Output

Forward-Backward Motions:

Lateral Right-Left Motions:

Circular Right-Left Motions:

Lateral Arc Motion:

Python Based Gesture Controlled Mars Rover

Learn how to use the Hand Gesture Classifier of the Machine Learning Environment to make a machine-learning model that identifies hand gestures and makes the Mars Rover move accordingly.

This project demonstrates how to use Machine Learning Environment to make a machine–learning model that identifies hand gestures and makes the Mars Rover move accordingly.

We are going to use the Hand Classifier of the Machine Learning Environment. The model works by analyzing your hand position with the help of 21 data points.

Hand Gesture Classifier Workflow

Follow the steps below:

Open PictoBlox and create a new file.
Select the coding environment as appropriate Coding Environment.
Select the “Open ML Environment” option under the “Files” tab to access the ML Environment.
Click on “Create New Project“.
A window will open. Type in a project name of your choice and select the “Hand Gesture Classifier” extension. Click the “Create Project” button to open the Hand Pose Classifier window.
You shall see the Classifier workflow with two classes already made for you. Your environment is all set. Now it’s time to upload the data.

Class in Hand Gesture Classifier

There are 2 things that you have to provide in a class:

Class Name: It’s the name to which the class will be referred as.
Hand Pose Data: This data can either be taken from the webcam or by uploading from local storage.

Note: You can add more classes to the projects using the Add Class button.

Adding Data to Class

You can perform the following operations to manipulate the data into a class.

Naming the Class: You can rename the class by clicking on the edit button.
Adding Data to the Class: You can add the data using the Webcam or by Uploading the files from the local folder.
1. Webcam:

Note: You must add at least 20 samples to each of your classes for your model to train. More samples will lead to better results.

Training the Model

Testing the Model

To test the model, simply enter the input values in the “Testing” panel and click on the “Predict” button.

The model will return the probability of the input belonging to the classes.

Export in Python Coding

Code

The following code appears in the Python Editor of the selected sprite.

####################imports####################
# Do not change

import numpy as np
import tensorflow as tf
import time
sprite=Sprite('Tobi')
import time
quarky = Quarky()
rover = MarsRover(4, 1, 7, 2, 6)
# Do not change
####################imports####################

#Following are the model and video capture configurations
# Do not change

model=tf.keras.models.load_model(
    "num_model.h5",
    custom_objects=None,
    compile=True,
    options=None)
pose = Posenet()                                                    # Initializing Posenet
pose.enablebox()                                                    # Enabling video capture box
pose.video("on",0)                                                  # Taking video input
class_list=['forward','backward','left','right','stop']                  # List of all the classes
def runQuarky(predicted_class):
    if pose.ishanddetected():
      if predicted_class == "forward":
        rover.home()
        rover.setinangle(0)
        quarky.runtimedrobot("F",100,3)
      if predicted_class == "backward":
        rover.home()
        rover.setinangle(0)
        quarky.runtimedrobot("B",100,3)
      if predicted_class == "left":
        rover.home()
        rover.setinangle(40)
        quarky.runtimedrobot("L",100,3)
      if predicted_class == "right":
        rover.home()
        rover.setinangle(40)
        quarky.runtimedrobot("R",100,3)
      if predicted_class == "stop":
        quarky.stoprobot()
    else:
      quarky.stoprobot()

# Do not change
###############################################

#This is the while loop block, computations happen here
# Do not change

while True:
  pose.analysehand()                                             # Using Posenet to analyse hand pose
  coordinate_xy=[]
    
    # for loop to iterate through 21 points of recognition
  for i in range(21):
    if(pose.gethandposition(1,i,0)!="NULL"  or pose.gethandposition(2,i,0)!="NULL"):
      coordinate_xy.append(int(240+float(pose.gethandposition(1,i,0))))
      coordinate_xy.append(int(180-float(pose.gethandposition(2,i,0))))
    else:
      coordinate_xy.append(0)
      coordinate_xy.append(0)
            
  coordinate_xy_tensor = tf.expand_dims(coordinate_xy, 0)        # Expanding the dimension of the coordinate list
  predict=model.predict(coordinate_xy_tensor)                    # Making an initial prediction using the model
  predict_index=np.argmax(predict[0], axis=0)                    # Generating index out of the prediction
  predicted_class=class_list[predict_index]                      # Tallying the index with class list
  print(predicted_class)
  runQuarky(predicted_class)
    
  # Do not change

Logic

If the identified class from the analyzed image is “forward,” the Mars Rover will move forward at a specific speed.
If the identified class is “backward,” the Mars Rover will move backward.
If the identified class is “left,” the Mars Rover will move left.
If the identified class is “right,” the Mars Rover will move right.
Otherwise, the Mars Rover will be in the home position , stopped.

def runQuarky(predicted_class):
    if pose.ishanddetected():
      if predicted_class == "forward":
        rover.home()
        rover.setinangle(0)
        quarky.runtimedrobot("F",100,3)
      if predicted_class == "backward":
        rover.home()
        rover.setinangle(0)
        quarky.runtimedrobot("B",100,3)
      if predicted_class == "left":
        rover.home()
        rover.setinangle(40)
        quarky.runtimedrobot("L",100,3)
      if predicted_class == "right":
        rover.home()
        rover.setinangle(40)
        quarky.runtimedrobot("R",100,3)
      if predicted_class == "stop":
        quarky.stoprobot()
    else:
      quarky.stoprobot()

Output

Python Based Gesture Controlled Gripper Mecanum

In this activity, learn how to create a new Machine Learning model that will be able to identify and detect different types of hand poses and that can help us to control the Mecanum Gripper Robot.

In this activity, we will try to create a new Machine Learning model that will be able to identify and detect different types of hand poses and that can help us to control the Mecanum Gripper Robot. This activity can be quite fun and by knowing the process, you can develop your own customized hand pose classifier model easily!

We will use the same model that we have created in the previous Hand Pose Controlled Mecanum model to avoid any misdirection and confusion.

Note: You can always create your own model and use it to perform any type of functions as per your choice. This example proves the same point and helps you understand well the concept of Machine Learning models and environment.

Hand Gesture Classifier Workflow

Follow the steps below:

Open PictoBlox and create a new file.
Select the Python coding environment as appropriate Coding Environment.
Select the “Open ML Environment” option under the “Files” tab to access the ML Environment.
Click on “Create New Project“.
A window will open. Type in a project name of your choice and select the “Hand Gesture Classifier” extension. Click the “Create Project” button to open the Hand Pose Classifier window.
You shall see the Classifier workflow with two classes already made for you. Your environment is all set. Now it’s time to upload the data.

Class in Hand Gesture Classifier

There are 2 things that you have to provide in a class:

Class Name: It’s the name to which the class will be referred as.
Hand Pose Data: This data can either be taken from the webcam or by uploading from local storage.

Note: You can add more classes to the projects using the Add Class button.

Adding Data to Class

You can perform the following operations to manipulate the data into a class.

Naming the Class: You can rename the class by clicking on the edit button.
Adding Data to the Class: You can add the data using the Webcam or by Uploading the files from the local folder.
1. Webcam:

Note: You must add at least 20 samples to each of your classes for your model to train. More samples will lead to better results.

Training the Model

Testing the Model

To test the model, simply enter the input values in the “Testing” panel and click on the “Predict” button.

The model will return the probability of the input belonging to the classes.

Export in Python Coding

Logic

The mecanum will move according to the following logic:

If the detected class is “forward”, we will make the Mecanum move forward.
When the backward gesture is detected – Mecanum will move backwards.
When the Lateral Left gesture is detected – Mecanum will move towards the left direction laterally with the help of its omnidirectional wheels.
When the Lateral Right gesture is detected – Mecanum will move towards the right direction laterally with the help of its omnidirectional wheels.
When the Normal Right gesture is detected – Mecanum will rotate on a single point towards the right direction.
When the Normal Left gesture is detected – Mecanum will rotate on a single point towards the left direction.
When the “Stop” class gesture is detected, we will use the gripper functions of the Mecanum and Pick the object.
When the “Circular Motion ” class is detected, we will use the gripper functions of the Mecanum and Drop the object by opening the arms of the gripper robot.

Code

Logical Code:

meca=Mecanum(1,2,7,8)
meca.initialisegripper(5)
meca.setcloseangle(90)
meca.setopenangle(0)

def runmecanum(predicted_class):
  if pose.ishanddetected():
    if predicted_class=="Forward":
      meca.runtimedrobot("forward",100,2)
    if predicted_class=="Backward":
      meca.runtimedrobot("backward",100,2)
    if predicted_class=="Stop":
      meca.closearm()
    if predicted_class=="LateralRight":
      meca.runtimedrobot("lateral right",100,2)
    if predicted_class=="LateralLeft":
      meca.runtimedrobot("lateral left",100,2)
    if predicted_class=="NormalRight":
      meca.runtimedrobot("circular right",100,1)
    if predicted_class=="NormalLeft":
      meca.runtimedrobot("circular left",100,1)
    if predicted_class=="CircularMotion":
      meca.openarm()

Final Code

We will create a custom function as shown above that will help us to control the Gripper Mecanum robot easily with the help of the Machine Learning model that we created.
We will also initialize the gripper pins and the mecanum pins before writing the main function. We can also set the gripper angle for closing and opening of the gripper with the help of “setgripperangle()” function.
We will also have to call the function at the last line of the code where we will call the function that we have created for the code to run properly.

####################imports####################
# Do not change

import numpy as np
import tensorflow as tf
import time

# Do not change
####################imports####################

#Following are the model and video capture configurations
# Do not change

model=tf.keras.models.load_model(
    "num_model.h5",
    custom_objects=None,
    compile=True,
    options=None)
pose = Posenet()                                                    # Initializing Posenet
pose.enablebox()                                                    # Enabling video capture box
pose.video("on",0)                                                  # Taking video input
class_list=['Forward','Backward','Stop','LateralRight','LateralLeft','NormalRight','NormalLeft','CircularMotion']                  # List of all the classes
meca=Mecanum(1,2,7,8)
meca.initialisegripper(5)
meca.setcloseangle(90)
meca.setopenangle(0)

def runmecanum(predicted_class):
  if pose.ishanddetected():
    if predicted_class=="Forward":
      meca.runtimedrobot("forward",100,2)
    if predicted_class=="Backward":
      meca.runtimedrobot("backward",100,2)
    if predicted_class=="Stop":
      meca.closearm()
    if predicted_class=="LateralRight":
      meca.runtimedrobot("lateral right",100,2)
    if predicted_class=="LateralLeft":
      meca.runtimedrobot("lateral left",100,2)
    if predicted_class=="NormalRight":
      meca.runtimedrobot("circular right",100,1)
    if predicted_class=="NormalLeft":
      meca.runtimedrobot("circular left",100,1)
    if predicted_class=="CircularMotion":
      meca.openarm()
# Do not change
###############################################

#This is the while loop block, computations happen here
# Do not change

while True:
  pose.analysehand()                                             # Using Posenet to analyse hand pose
  coordinate_xy=[]
    
    # for loop to iterate through 21 points of recognition
  for i in range(21):
    if(pose.gethandposition(1,i,0)!="NULL"  or pose.gethandposition(2,i,0)!="NULL"):
      coordinate_xy.append(int(240+float(pose.gethandposition(1,i,0))))
      coordinate_xy.append(int(180-float(pose.gethandposition(2,i,0))))
    else:
      coordinate_xy.append(0)
      coordinate_xy.append(0)
            
  coordinate_xy_tensor = tf.expand_dims(coordinate_xy, 0)        # Expanding the dimension of the coordinate list
  predict=model.predict(coordinate_xy_tensor)                    # Making an initial prediction using the model
  predict_index=np.argmax(predict[0], axis=0)                    # Generating index out of the prediction
  predicted_class=class_list[predict_index]                      # Tallying the index with class list
  print(predicted_class)
  runmecanum(predicted_class)
  # Do not change

Output

Forward-Backward Motion:

Circular Right-Left Motion:

Lateral Right-Left Motions:

Gripper Mechanism with Hand Gestures:

Gesture Controlled Humanoid With Machine Learning

Learn to demonstrate how to use ML Environment to make a model that identifies the hand gestures and makes the humanoid move accordingly.

Introduction

This project demonstrates how to use Machine Learning Environment to make a machine–learning model that identifies hand gestures and makes the humanoid move accordingly.

We are going to use the Hand Classifier of the Machine Learning Environment. The model works by analyzing your hand position with the help of 21 data points.

Hand Gesture Classifier Workflow

Follow the steps below:

Open PictoBlox and create a new file.
You can click on “Machine Learning Environment” to open it.
Click on “Create New Project“.
A window will open. Type in a project name of your choice and select the “Hand Gesture Classifier” extension. Click the “Create Project” button to open the Hand Pose Classifier window.
You shall see the Classifier workflow with two classes already made for you. Your environment is all set. Now it’s time to upload the data.

Class in Hand Gesture Classifier

There are 2 things that you have to provide in a class:

Class Name: It’s the name to which the class will be referred as.
Hand Pose Data: This data can either be taken from the webcam or by uploading from local storage.

Note: You can add more classes to the projects using the Add Class button.

Adding Data to Class

You can perform the following operations to manipulate the data into a class.

Naming the Class: You can rename the class by clicking on the edit button.
Adding Data to the Class: You can add the data using the Webcam or by Uploading the files from the local folder.
1. Webcam:

Note: You must add at least 20 samples to each of your classes for your model to train. More samples will lead to better results.

Training the Model

Testing the Model

To test the model, simply enter the input values in the “Testing” panel and click on the “Predict” button.

The model will return the probability of the input belonging to the classes.

Export in Python Coding

code

The following code appears in the Python Editor of the selected sprite.

####################imports####################
# Do not change

import numpy as np
import tensorflow as tf

sprite=Sprite('Tobi')
import time
quarky = Quarky()
import time

humanoid = Humanoid(7, 2, 6, 3, 8, 1)

# Do not change
####################imports####################

#Following are the model and video capture configurations
# Do not change

model=tf.keras.models.load_model(
    "num_model.h5",
    custom_objects=None,
    compile=True,
    options=None)
pose = Posenet()                                                    # Initializing Posenet
pose.enablebox()                                                    # Enabling video capture box
pose.video("on",0)                                                  # Taking video input
class_list=['forward','backward','left','right','stop']                  # List of all the classes
def runQuarky(predicted_class):
    if pose.ishanddetected():
      if predicted_class == "forward":
        humanoid.move("forward", 1000, 1)
        
      if predicted_class == "backward":
        humanoid.move("backward", 1000, 1)
        
      if predicted_class == "left":
        humanoid.move("left", 1000, 1)
        
      if predicted_class == "right":
        humanoid.move("right", 1000, 1)
        
      if predicted_class == "stop":
        humanoid.home()
    else:
      quarky.stoprobot()

# Do not change
###############################################

#This is the while loop block, computations happen here
# Do not change

while True:
  pose.analysehand()                                             # Using Posenet to analyse hand pose
  coordinate_xy=[]
    
    # for loop to iterate through 21 points of recognition
  for i in range(21):
    if(pose.gethandposition(1,i,0)!="NULL"  or pose.gethandposition(2,i,0)!="NULL"):
      coordinate_xy.append(int(240+float(pose.gethandposition(1,i,0))))
      coordinate_xy.append(int(180-float(pose.gethandposition(2,i,0))))
    else:
      coordinate_xy.append(0)
      coordinate_xy.append(0)
            
  coordinate_xy_tensor = tf.expand_dims(coordinate_xy, 0)        # Expanding the dimension of the coordinate list
  predict=model.predict(coordinate_xy_tensor)                    # Making an initial prediction using the model
  predict_index=np.argmax(predict[0], axis=0)                    # Generating index out of the prediction
  predicted_class=class_list[predict_index]                      # Tallying the index with class list
  print(predicted_class)
  runQuarky(predicted_class)

Note: You can edit the code to add custom code according to your requirement.

Logic

If the identified class from the analyzed image is “forward,” the humanoid will move forward at a specific speed.
If the identified class is “backward,” the humanoid will move backward.
If the identified class is “left,” the humanoid will move left.
If the identified class is “right,” the humanoid will move right.
Otherwise, the humanoid will be in the home position.

if pose.ishanddetected():
      if predicted_class == "forward":
        humanoid.move("forward", 1000, 1)
        
      if predicted_class == "backward":
        humanoid.move("backward", 1000, 1)
        
      if predicted_class == "left":
        humanoid.move("left", 1000, 1)
        
      if predicted_class == "right":
        humanoid.move("right", 1000, 1)
        
      if predicted_class == "stop":
        humanoid.home()
    else:
      quarky.stoprobot()

Output

Python Based Hand Pose Controlled Mecanum Pick and Place Robot

In this activity, learn how to create a new Machine Learning model that will be able to identify and detect different types of hand poses and that can help us to control the Mecanum Pick and Place Robot.

In this activity, we will try to create a new Machine Learning model that will be able to identify and detect different types of hand poses and that can help us to control the Mecanum Pick and Place Robot. This activity can be quite fun and by knowing the process, you can develop your own customized hand pose classifier model easily!

We will use the same model that we have created in the previous Hand Pose Controlled Mecanum model to avoid any misdirection and confusion.

Note: You can always create your own model and use it to perform any type of functions as per your choice. This example proves the same point and helps you understand well the concept of Machine Learning models and environment.

Hand Gesture Classifier Workflow

Follow the steps below:

Open PictoBlox and create a new file.
Select the Python coding environment as appropriate Coding Environment.
Select the “Open ML Environment” option under the “Files” tab to access the ML Environment.
Click on “Create New Project“.
A window will open. Type in a project name of your choice and select the “Hand Gesture Classifier” extension. Click the “Create Project” button to open the Hand Pose Classifier window.
You shall see the Classifier workflow with two classes already made for you. Your environment is all set. Now it’s time to upload the data.

Class in Hand Gesture Classifier

There are 2 things that you have to provide in a class:

Class Name: It’s the name to which the class will be referred as.
Hand Pose Data: This data can either be taken from the webcam or by uploading from local storage.

Note: You can add more classes to the projects using the Add Class button.

Adding Data to Class

You can perform the following operations to manipulate the data into a class.

Naming the Class: You can rename the class by clicking on the edit button.
Adding Data to the Class: You can add the data using the Webcam or by Uploading the files from the local folder.
1. Webcam:

Note: You must add at least 20 samples to each of your classes for your model to train. More samples will lead to better results.

Training the Model

Testing the Model

To test the model, simply enter the input values in the “Testing” panel and click on the “Predict” button.

The model will return the probability of the input belonging to the classes.

Export in Python Coding

Logic

The mecanum will move according to the following logic:

If the detected class is “forward”, we will make the Mecanum move forward.
When the backward gesture is detected – Mecanum will move backwards.
When the Lateral Left gesture is detected – Mecanum will move towards the left direction laterally with the help of its omnidirectional wheels.
When the Lateral Right gesture is detected – Mecanum will move towards the right direction laterally with the help of its omnidirectional wheels.
When the Normal Right gesture is detected – Mecanum will rotate on a single point towards the right direction.
When the Normal Left gesture is detected – Mecanum will rotate on a single point towards the left direction.
When the Stop gesture is detected – Mecanum will stop and initiate the Pick Mechanism using the Pick function.
When the Circular Motion gesture is detected – Mecanum will initiate the Place Mechanism using the Place function.

Code

Logical Code

meca=Mecanum(1,2,7,8)
meca.initialisepickplace(5,4)
meca.setpickangle(40)
meca.setarmangle(90)

def runmecanum(predicted_class):
  if pose.ishanddetected():
    if predicted_class=="Forward":
      meca.runtimedrobot("forward",100,2)
    if predicted_class=="Backward":
      meca.runtimedrobot("backward",100,2)
    if predicted_class=="Stop":
      meca.pick()
    if predicted_class=="LateralRight":
      meca.runtimedrobot("lateral right",100,2)
    if predicted_class=="LateralLeft":
      meca.runtimedrobot("lateral left",100,2)
    if predicted_class=="NormalRight":
      meca.runtimedrobot("circular right",100,1)
    if predicted_class=="NormalLeft":
      meca.runtimedrobot("circular left",100,1)
    if predicted_class=="CircularMotion":
      meca.place()

Final Code

We will create a custom function as shown above that will help us to control the Pick and Place Mecanum robot easily with the help of the Machine Learning model that we created.
We will also initialize the Pick and Place servo pins and the mecanum pins before writing the main function. We can also set the servo angle for Picking and placing functions with the help of “setpickangle()” function.
We will also have to call the function at the last line of the code where we will call the function that we have created for the code to run properly.

####################imports####################
# Do not change

import numpy as np
import tensorflow as tf
import time

# Do not change
####################imports####################

#Following are the model and video capture configurations
# Do not change

model=tf.keras.models.load_model(
    "num_model.h5",
    custom_objects=None,
    compile=True,
    options=None)
pose = Posenet()                                                    # Initializing Posenet
pose.enablebox()                                                    # Enabling video capture box
pose.video("on",0)                                                  # Taking video input
class_list=['Forward','Backward','Stop','LateralRight','LateralLeft','NormalRight','NormalLeft','CircularMotion']                  # List of all the classes
meca=Mecanum(1,2,7,8)
meca.initialisepickplace(5,4)
meca.setpickangle(40)
meca.setarmangle(90)

def runmecanum(predicted_class):
  if pose.ishanddetected():
    if predicted_class=="Forward":
      meca.runtimedrobot("forward",100,2)
    if predicted_class=="Backward":
      meca.runtimedrobot("backward",100,2)
    if predicted_class=="Stop":
      meca.pick()
    if predicted_class=="LateralRight":
      meca.runtimedrobot("lateral right",100,2)
    if predicted_class=="LateralLeft":
      meca.runtimedrobot("lateral left",100,2)
    if predicted_class=="NormalRight":
      meca.runtimedrobot("circular right",100,1)
    if predicted_class=="NormalLeft":
      meca.runtimedrobot("circular left",100,1)
    if predicted_class=="CircularMotion":
      meca.place()

# Do not change
###############################################

#This is the while loop block, computations happen here
# Do not change

while True:
  pose.analysehand()                                             # Using Posenet to analyse hand pose
  coordinate_xy=[]
    
    # for loop to iterate through 21 points of recognition
  for i in range(21):
    if(pose.gethandposition(1,i,0)!="NULL"  or pose.gethandposition(2,i,0)!="NULL"):
      coordinate_xy.append(int(240+float(pose.gethandposition(1,i,0))))
      coordinate_xy.append(int(180-float(pose.gethandposition(2,i,0))))
    else:
      coordinate_xy.append(0)
      coordinate_xy.append(0)
            
  coordinate_xy_tensor = tf.expand_dims(coordinate_xy, 0)        # Expanding the dimension of the coordinate list
  predict=model.predict(coordinate_xy_tensor)                    # Making an initial prediction using the model
  predict_index=np.argmax(predict[0], axis=0)                    # Generating index out of the prediction
  predicted_class=class_list[predict_index]                      # Tallying the index with class list
  print(predicted_class)
  runmecanum(predicted_class)
  # Do not change

Output

Forward-Backward Motion:

Circular Right-Left Motion:

Lateral Right-Left Motion:

Pick and Place Mechanism with Hand Pose:

All articles loaded

No more articles to load

PictoBlox Extensions

Parameters

Description

Example

Code

Output

Code

Output

Introduction

Export in Python Coding

code

Logic

Final code

Output

Hand Gesture Classifier Workflow

Class in Hand Gesture Classifier

Adding Data to Class

Training the Model

Testing the Model

Export in Python Coding

Logic

Code

Logical Code

Output

Hand Gesture Classifier Workflow

Class in Hand Gesture Classifier

Adding Data to Class

Training the Model

Testing the Model

Export in Python Coding

Code

Logic

Output

Hand Gesture Classifier Workflow

Class in Hand Gesture Classifier

Adding Data to Class

Training the Model

Testing the Model

Export in Python Coding

Logic

Code

Output

Introduction

Hand Gesture Classifier Workflow

Class in Hand Gesture Classifier

Adding Data to Class

Training the Model

Testing the Model

Export in Python Coding

code

Logic

Output

Hand Gesture Classifier Workflow

Class in Hand Gesture Classifier

Adding Data to Class

Training the Model

Testing the Model

Export in Python Coding

Logic

Code

Output

Company

Community

Get in Touch

Follow Us

Dabble App - One App. Infinite Control.