D. Topics in Computer Science · Topics in Computer Science: Sustainability, Emerging Research and Careers

Robotics, Computer Vision and Image Processing

D2.1About 70 minutes

Learning goals

I can describe a robot as a system of sensors, a controller and actuators in a sense–think–act loop.
I can explain feedback control and simulate a simple controller in Python.
I can explain how a computer stores an image and apply thresholding and simple edge detection in Python.
I can describe current research and Canadian contributions in robotics and computer vision.

Words to know

robot
A machine that senses its environment, makes decisions with a computer, and acts on the world, often without direct human control.

sensor
A device that measures something (light, distance, touch, temperature) and sends the measurement to the controller.

actuator
A part that makes the robot move or act, such as a motor, wheel or gripper.

feedback control
Using sensor measurements to adjust actions continually so the system reaches its goal.

pixel
One tiny dot of a digital image; in a greyscale image it is a brightness number, often 0 (black) to 255 (white).

computer vision
The field that helps computers understand the content of images and video.

image processing
Changing or analysing images with algorithms, for example sharpening, thresholding or finding edges.

A Canadian arm in space

In 1981 a robotic arm built in Canada flew in space for the first time on the U.S. space shuttle. It became famous as the Canadarm. Its successor, Canadarm2, has worked on the International Space Station since 2001, catching visiting spacecraft and moving equipment, together with the two-armed robot Dextre. Canada is also building Canadarm3 for a planned station near the Moon.

Robots like these, along with self-driving cars, warehouse robots and robotic surgery tools, depend on two research areas named in the curriculum: robotics and computer vision (with its partner, image processing).

What makes a robot a robot?

A robot has three kinds of parts:

Sensors measure the world: cameras, distance sensors (ultrasonic, infrared, lidar), touch switches, wheel encoders that count rotations, gyroscopes.
A controller, a computer running a program, decides what to do.
Actuators carry out the decision: motors, wheels, joints, grippers, speakers.

The robot runs a sense–think–act loop over and over, many times a second.

Every robot repeats a sense, think, act loop. This feedback lets it correct itself as the world changes.

Feedback control

Imagine a robot that must stop exactly 100 cm along a hallway. If it just runs its motors for a fixed time, small differences (a low battery, a slippery floor) make it miss. Instead it uses feedback: it measures where it is, computes the error (target − position), and moves by an amount based on that error. A proportional controller moves by a fixed fraction of the error, so it moves fast when far away and slowly when close.

position = 0.0

target = 100.0

for step in range(1, 7):

error = target - position

position = position + 0.5 * error

print(step, round(position, 1))

A proportional controller: each step the robot moves half of the remaining distance to the target (cm). Output (one line each): 1 50.0 / 2 75.0 / 3 87.5 / 4 93.8 / 5 96.9 / 6 98.4

Worked example

Example 1: Tracing the controller

Trace the first three steps of the controller by hand, and explain why the robot never quite overshoots.

Step 1: error = 100 − 0 = 100; position = 0 + 0.5 × 100 = 50.
Step 2: error = 100 − 50 = 50; position = 50 + 25 = 75.
Step 3: error = 100 − 75 = 25; position = 75 + 12.5 = 87.5.
Each step covers half of what is left, so the remaining error halves: 100, 50, 25, 12.5, ... It gets closer every step but never passes 100.

50, 75, 87.5. The error halves each step, so the robot approaches the target without overshooting.

How a computer sees

To a computer, a digital image is a grid of numbers. In a greyscale image each pixel is a brightness, usually from 0 (black) to 255 (white). A colour image stores three numbers per pixel, for red, green and blue. A phone photo can have more than 10 million pixels.

Image processing changes or measures these numbers. Computer vision goes further and tries to understand what is in the image: is that a stop sign? a pedestrian? a tumour on a scan?

Two basic image-processing operations:

Thresholding: turn each pixel into on or off depending on whether it is brighter than a cut-off. This separates objects from the background.
Edge detection: find places where brightness changes sharply, which are usually the outlines of objects.

A tiny greyscale image is just a grid of numbers. Thresholding marks every pixel brighter than 128, and a ring shape appears.

image = [

[12, 15, 200, 210, 14, 10],

[11, 190, 220, 215, 205, 13],

[10, 205, 30, 25, 198, 12],

[14, 199, 215, 220, 201, 11],

[13, 16, 208, 212, 15, 10],

]

for row in image:

line = ""

for v in row:

if v > 128:

line += "#"

else:

line += "."

print(line)

Thresholding an image stored as a list of lists. Output (one line each): ..##.. / .####. / .#..#. / .####. / ..##..

image = [

[12, 15, 200, 210, 14, 10],

[11, 190, 220, 215, 205, 13],

[10, 205, 30, 25, 198, 12],

[14, 199, 215, 220, 201, 11],

[13, 16, 208, 212, 15, 10],

]

counts = []

for row in image:

edges = 0

for i in range(len(row) - 1):

if abs(row[i + 1] - row[i]) > 100:

edges += 1

counts.append(edges)

print(counts)

A simple edge detector: an edge is where brightness jumps by more than 100 between neighbouring pixels. Output: [2, 2, 4, 2, 2]

Research frontiers

Robots that learn: using machine learning (previous lesson) so robots can grasp unfamiliar objects or walk on rough ground.
Self-driving vehicles combine cameras, lidar and radar with computer vision to detect lanes, signs, cyclists and pedestrians. Making them safe in snow, rain and darkness is a hard research problem, which matters a lot in Canada.
Medical imaging: computer vision helps detect disease in X-rays and scans; a doctor still makes the decision.
Robots that work with people: "collaborative robots" that can work safely beside humans in factories and hospitals.
Space and exploration robotics: robotic arms, rovers and drones for places people cannot easily go.

Just for fun

Why is a robot vacuum such a determined problem solver?

Every time it hits a wall, it just turns around and tries a new direction. (That really is how many simple robots navigate: sense a bump, turn, try again.)

Hands-on activity

Be the robot

You need: a partner, a tape measure, masking tape, a blindfold or closed eyes, and a clear, safe hallway (move anything you could trip on).

Mark a start line and a target 4 m away with tape.
No feedback: the "robot" (wearing a blindfold, with the partner walking beside them for safety) takes the number of steps they think will reach the target. Measure the miss.
Feedback: repeat, but now the partner is the sensor and calls out the remaining distance after each step. The robot takes a step about half that distance each time, and stops when the partner says "less than 10 cm".
Compare the two misses, and count how many steps the feedback version took.
Then, on a computer: change 0.5 in the controller to 0.9 and to 1.5. What happens? Describe it in one sentence each.

What to look for: the feedback run should finish much closer to the target. With 0.9 the program reaches the target faster; with 1.5 it overshoots (150, then 75, then 112.5, ...) and swings back and forth around the target while getting closer.

Your turn

Name the three main kinds of parts in a robot and give an example of each.

What is feedback control?

In a greyscale image, what do the values 0 and 255 usually mean?

Apply a threshold of 128 to the row [90, 140, 200, 60]. Write the result with # and . as in the program.

How many edges (jumps greater than 100) are in the row [10, 15, 180, 190, 20]?

What is the difference between image processing and computer vision?

What Canadian robot has worked on the International Space Station since 2001?

More practice: warm-up

What does a sensor do?

What does an actuator do?

What is a pixel?

How many numbers does an RGB colour pixel store?

Name two kinds of robots used today.

More practice: on your own

A proportional controller moves 0.25 of the error each step. Starting at 0 with target 80, find the positions after steps 1, 2 and 3.

Why does a robot that runs its motors for a fixed time often miss its target?

Change the threshold in the thresholding program to v > 220. What does the middle row become?

A self-driving car must work in a Canadian winter. Name two problems snow causes for computer vision.

Explain why the edge detector uses abs().

Challenge

Using the controller with factor 0.5, after how many steps is the robot first within 1 cm of the target 100 cm? Show the reasoning.

Write Python that counts how many pixels in image are brighter than 128. What does it print?

A robot uses a factor of 1.5 in its proportional controller. Trace three steps from 0 to target 100 and describe the behaviour. Why might this be dangerous for a real robot arm?

Check your understanding

Quiz

Which part of a robot is a motor?

What is the correct order of the robot control loop?

A proportional controller with factor 0.5 starts at 0 with target 100. Where is it after two steps?

Using a threshold of 128, which pixel values are marked "on"?

An edge in an image is usually a place where:

Which is a computer vision task?

For parents and teachers

Big idea: Expectation D2.1 names robotics, computer vision and image processing as emerging research areas. Students learn the sense–think–act model, feedback control, and how images are stored and processed as numbers.

How to help: Do the "Be the robot" activity together; stay beside the blindfolded walker and clear the hallway first. Point out robots and sensors at home and around town: automatic doors, robot vacuums, self-checkout scanners.

Common difficulties:

Confusing sensors (input) with actuators (output). Connect to the input-process-output model from the unit Software Development.
Forgetting that the controller's error changes every step.
Mixing up rows and columns in the image grid: image[r][c] is row r, column c.