Lab F: Train an Image Classifier and Program Turtlebot3 to React to Different Images

Today's Lab

Learning more about how to train an image classifier
Knowing how to leverage outputs from neural networks to control robots
Gaining experience with pair programming

Lab Exercise: Training an Image Classifier to allow a Turtlebot3 to React to Different Images

You will complete this exercise in pairs (ideally with your Q-learning project partner).

You will be programming a Turtlebot3 to respond to different images it "sees". The robot's behavior will change based on whether it recognizes a cat, a dog, or neither in its camera. You will need to pass the robot's camera footage to a trained neural network and use the output to command the robot to perform different behaviors described in the following narrative.

Getting Started

To get started on this exercise, you'll need to clone the following package: lab_f_image_classifier

Please run the following commands:

$ cd ~/intro_robo_ws/src/
$ git clone https://github.com/Intro-Robotics-UChicago-Fall-2025/lab_f_image_classifier.git
$ cd ~/intro_robo_ws
$ colcon build --symlink-install --packages-select lab_f_image_classifier
$ source install/setup.bash

Additionally, you'll need to make sure you have PyTorch installed (e.g., by executing pip3 show torch ). If you don't have PyTorch installed, run the following command to install it:

$ pip3 install torch torchvision torchaudio

Part 1: Train an Image Classifier

In order for the robot to respond to different images (e.g., cat, dog, something else) it sees, we'll first need to train an image classifier that can distinguish these different classes. We'll do this work in the Jupiter Notebook defined in cifar10_tutorial.ipynb.

You can open the Jupiter Notebook by running:

$ jupyter notebook cifar10_tutorial.ipynb

Next, follow the instructions in the Jupiter Notebook to train an image classifier. Your classifier will be able to distinguish several different types of objects within an image, see below.

Part 2: Program Turtlebot3 to React to Different Images

Now that you've trained a neural network to identify different objects within an image, we'll use that trained network to allow our turtlebot to respond uniquely to seeing a cat, a dog, or neither from it's RGB camera. Since there aren't any real cats or dogs in JCL, we recommend that you bring up an images of cats, dogs, and/or other items on your phone or computer and place your phone/computer so that the robot can see it.

Robot Narrative: Imagine you are programming a curious robot named Scout. Scout is equipped with a camera that allows it to observe its surroundings in the real world. Scout is designed to be adaptive and responsive, capable of reacting to visual cues to navigate its environment safely. One day, Scout is exploring a park when its camera captures various images. Each time Scout's camera processes an image, it analyzes it to determine whether it depicts a cat, a dog, or something else entirely.

Encountering a Cat
- Scout's sensors detect an image of a cat. Instantly, Scout feels a surge of caution. To express its fear, Scout decides to move backward slowly, trying to create distance from the perceived threat.
Spotting a Dog
- Another image appears, this time showing a dog. Scout recognizes it's a different animal but isn't sure whether to be excited or wary. To show its excitement, Scout decides to spin in a joyful circle.
Seeing Something Else
- Sometimes, Scout's camera captures images that don't resemble cats or dogs. When this happens, Scout simply pauses, observing its surroundings but choosing not to move until the next image is processed.

Programming the Robot Scout

Now, let's get to programming. Open up robot_scout.py to see your TODOs for implemeting your robot Scout.

Running your Code

Connect with a Turtlebot4, remembering to run:

$ set_robot_num [robot-number]

Then, to run the code, you can run:

$ ros2 run lab_f_image_classifier robot-scout

The starter code implements a helpful debugging window to help visualize the camera view.

Note: If you're experiencing some opencv issues, try commenting out line 15 cv2.namedWindow("window", 1)

Tips:

robot_scout.py imports a class object called Net from image_classifier.py. See how the neural network is defined there.

Acknowledgments

The code in this lab was created by Ting-Han (Timmy) Lin, Tewodros (Teddy) Ayalew, and Haley Breslin. Part of the code is taken and modified from Training a Classifier .