Okay, class, welcome.
Let's outline Module 3 for our "Machine Vision for Robotics" course,
focusing on the essential framework that often ties our vision algorithms to the physical robot:
the Robot Operating System (ROS).
Course: ECE/CS 5XXX - Machine Vision for Robotics
Module 3: Introduction to ROS for Machine Vision Applications
Instructor: Professor [Your Name/Default AI Persona]
Module Overview:
While understanding machine vision algorithms is crucial, deploying them effectively on a
robotic platform requires a robust software framework. This module introduces the Robot
Operating System (ROS), the de facto standard middleware for robotics research and
development. We will focus specifically on how ROS facilitates the integration of cameras,
image processing pipelines, and vision-based decision-making within a larger robotic system.
This is not an exhaustive ROS course, but rather a targeted introduction to the concepts and tools
most relevant to machine vision engineers working in robotics.
Prerequisites:
Basic familiarity with Linux command-line interface (Ubuntu recommended).
Proficiency in either Python or C++. (Examples will be provided primarily in Python,
with C++ concepts highlighted).
Conceptual understanding of robotics (sensors, actuators, coordinate frames).
Basic knowledge of image representation (pixels, color spaces - covered in Module 1).
Learning Objectives:
Upon successful completion of this module, students will be able to:
1. Understand the motivation for using ROS in robotics, particularly for vision tasks.
2. Explain the core concepts of ROS: Nodes, Topics, Messages, Services, Actions, Master,
and Parameter Server.
3. Navigate the ROS file system and utilize essential command-line tools (roscore, rosrun,
roslaunch, rostopic, rosnode, rosmsg, rosservice, rqt_graph).
4. Understand how image data is represented and transmitted in ROS (sensor_msgs/Image,
sensor_msgs/CameraInfo).
5. Utilize cv_bridge to convert between ROS Image messages and OpenCV image formats
(NumPy/cv::Mat).
6. Write simple ROS nodes (publishers/subscribers) to acquire image data from simulated
or real cameras.
7. Implement basic image processing tasks within a ROS node.
8. Use ROS tools (rqt_image_view, rviz) to visualize image streams and related data.
9. Appreciate the role of tf (transformations) in relating camera data to robot coordinate
frames (introduction).
10. Understand the concept of image_transport for efficient image topic handling.
Module Duration: Approx. 2.5 Weeks (Assuming 2 lectures + 1 lab session per week)
Key Topics Covered:
1. Week 1: ROS Fundamentals & Vision Context
o Lecture 1: Why ROS? The Middleware Concept.
Challenges in robot software development (hardware abstraction,
concurrency, distributed systems).
ROS as a solution: Architecture overview (graph-based, peer-to-peer).
Relevance to Machine Vision: Handling sensor data streams, modular
processing pipelines, integration with control.
o Lecture 2: Core ROS Concepts.
Nodes: Executable processes (e.g., camera driver, object detector, robot
controller).
Master: Facilitating node discovery.
Parameter Server: Storing configuration.
Topics & Messages: Asynchronous, one-to-many communication (ideal
for sensor streams like images). Message types (std_msgs, sensor_msgs).
Services: Synchronous, request-reply communication (e.g., trigger image
capture).
Actions: Asynchronous, long-running goal-based tasks with feedback
(e.g., "track this object").
o Lab 1: ROS Environment Setup & Basic Tools.
Installing ROS (e.g., Noetic Ninjemys or Humble Hawksbill).
Creating a Catkin Workspace / Colcon Workspace.
Using roscore, rosrun (turtlesim example).
Exploring nodes, topics, messages using command-line tools (rosnode list,
rostopic list/echo/hz/info, rosmsg show).
Visualizing the node graph with rqt_graph.
2. Week 2: Handling Image Data in ROS
o Lecture 3: Images and Camera Info in ROS.
The sensor_msgs/Image message: structure (header, height, width,
encoding, data). Common encodings (mono8, bgr8, rgb8).
The sensor_msgs/CameraInfo message: intrinsic/extrinsic parameters,
distortion model. Importance for 3D vision.
image_transport: Why direct topic publishing can be inefficient for
images. Publisher/subscriber plugins for compression (theora,
compressed).
o Lecture 4: Interfacing with OpenCV: cv_bridge.
The need for conversion between ROS messages and standard vision
library formats.
Introduction to cv_bridge: CvBridge class.
Converting sensor_msgs/Image to OpenCV format (imgmsg_to_cv2 /
toCvShare, toCvCopy).
Converting OpenCV format back to sensor_msgs/Image
(cv2_to_imgmsg / cvToImgMsg).
Handling different image encodings during conversion.
o Lab 2: Image Publisher & Subscriber.
Using a standard ROS camera driver package (e.g., usb_cam, cv_camera,
or a simulator like Gazebo) to publish image topics.
Visualizing image topics using rqt_image_view.
Writing a Python/C++ subscriber node that uses cv_bridge to receive
images.
Performing a simple OpenCV operation (e.g., grayscale conversion,
thresholding, edge detection) on the received image.
(Optional) Publishing the processed image back onto a new topic.
3. Week 3: Integration and Visualization
o Lecture 5: Coordinate Frames (tf) and Visualization.
Why coordinate frames matter: Relating pixel coordinates to the real
world / robot base.
Introduction to tf2: The ROS transformation library. Frames, Transforms,
Listeners, Broadcasters.
Visualizing transforms and sensor data in RViz. Adding Image displays,
Camera views, TF trees.
Brief overview of roslaunch for starting multiple nodes.
o Lab 3: Basic Visualization and TF Awareness.
Using roslaunch to start a camera node and an image processing node.
Visualizing raw and processed image streams in rqt_image_view and
RViz.
(If simulator/robot available) Inspecting the tf tree using view_frames and
rqt_tf_tree. Visualizing the camera frame relative to the robot base in
RViz.
Tools & Software:
Ubuntu Linux (LTS version recommended, e.g., 20.04 for Noetic, 22.04 for Humble)
ROS Distribution (Noetic Ninjemys or Humble Hawksbill recommended)
Python 3 / C++ Compiler (GCC)
OpenCV (usually installed with ROS)
cv_bridge package
Standard ROS tools (rqt, rviz, etc.)
Text Editor or IDE (e.g., VS Code with ROS extensions)
(Optional) Gazebo simulator for more complex scenarios.
(Optional) Physical camera compatible with ROS drivers (e.g., USB webcam).
Assessment:
Lab 1 Submission: Basic ROS commands and workspace setup verification. (10%)
Lab 2 Submission: Working image subscriber node performing basic processing. (45%)
Lab 3 Submission: Launch file setup and visualization screenshots/report. (25%)
Module Quiz: Covering core ROS concepts and their application to vision. (20%)
Further Reading/Resources:
ROS Wiki: [Link] (Especially Tutorials, sensor_msgs, cv_bridge, tf2,
image_transport pages)
cv_bridge Tutorial: [Link]
tf2 Tutorials: [Link]
Official ROS Documentation for your chosen distribution (Noetic/Humble).
Okay class, that's the plan for Module 3. It's a practical, hands-on module designed to give you
the foundational ROS skills needed to connect your vision algorithms to robotic systems later in
the course. Make sure your Linux environment and ROS installation are ready before Lab 1. Any
initial questions?
