This page describe's Nasa Tjan's final-year project. He wrote some computer software to steer a
radio-controlled car towards a goal. The software sees the car through an image captured from a webcam.
Nasa Tjan completed this under my supervision during the final year of his computer science degree at the
Malaysia Campus of the University of Nottingham. The custom hardware was
built by Ellias Ismail.
Video
Here's some video footage of the car in motion, with a voiceover that explains briefly how it worked.
Details
There are five basic 'hardware' components to the system, 'connected' in a loop, described below.
The most important is the software, written by Nasa Tjan. Written in java, there are classes and methods
to send the car commands such as "move forwards", "turn left", and so on.
These commands are actually sent to the interface board, from phidgets.com. "Go forwards" actually
means "switch digital output 0 on for 250 milliseconds". Phidgets.com provides code that
allows their boards to be easily programmed from many languages, including java. They claim
it makes it easy (at last) to connect a computer to hardware. We found this claim to be mstly true.
The digital outputs of the 8/8/8 PhidgetInterfaceKit are connected via transistors and relays to the
car's remote control. Thanks, Ellias, for doing the electronics for us! Without you, this project
would have been doomed!
The remote control drives the car in the normal way, the relays taking place of human thumbs.
The car is a cheap made-in-china job, off the shelf of a department store. It cost about 30 or 40 ringgit (equivalent
to about 10 US dollars). It's been equipped, as you can see, with bright orange stickers.
The 640x480 Logitech Webcam provides the next link in the chain. Pointed
at the car, it allows the software to see.
Yes, closing the loop is the software. The captured image from the webcam is processed,
and based on the results of the processing, the software decides how the car will be instructed to move.
That's the 'hardware' loop. The 'software' loop is described below.
Capture Image: The software captures the image from the webcam using classes and methods
available in the java media framework.
Filter Colour: Then, the captured image is put through a simple colour filter - the orange stickers remain
orange, everything else is changed to black.
Dilate and Shrink: In order to coalesce the edges of the image of each sticker, the orange part of the image is dilated (black pixels next to orange ones are changed to orange),
then shrunk (orange pixels near black ones are changed to black). This ensures that a stray pixel near the
edge of an image of a sticker is not counted as a very small sticker.
Detect Regions: Using a simple region-filling algorithm, the orange pixels are collected
into groups, each group (hopefully) representing one sticker.
Apply Geometry: The centre and size of each group is measured. The large one is assumed to be the goal,
the other five to be the four corners and the front of the car. From this, the software finds
the distance and angle from car to goal, and the orientation of the car. The software doesn't bother with
real-world coordinates, it measures everything in pixels.
Decide Strategy: Knowing the relative positions of the car and goal, the software applies
a set of rules to decide what to do. An example might be if distance is LARGE and angle is between -5 to 5, then move straight forward.
Drive! Once the decision is made, the appropriate commands are sent to the interface board. The software
waits for a reasonable amount of time, captures another webcam image, and the software loop begins again.
Strengths and Weaknesses
The system works.
It doesn't depend on carefully calibrating the camera. In fact, in the video clip above, the webcam was hand-held, about a metre above the car.
It doesn't depend on careful mechanical engineering. We used a cheap toy. Even on a slippery floor, where the effect of each steering command
is unpredictable, the car is able to find the goal.
It doesn't depend on careful path planning. No long-term plan to reach the goal is formed. Instead,
the car formulates only the next step in the plan in each cycle. This allows it to adapt well to slippery tyres or
researchers who keep moving the goal.
There are limitations however.
If the system detects too many regions of orange colour, the loop halts. For example, if there
are 6 or 7 'car stickers' detected, the software doesn't know what to do. This could be overcome via a 'best fit'
approach that tried each set of 5 regions out of the 6 or 7 in turn, and seeing which set was closest to the configuration
of stickers on the car.
Partly because of this, the system is quite sensitive to lighting conditions. Because the thresholds used in the colour filtering step
were decided under bright overhead fluorescent light, the system performs poorly under other lighting conditions. This could likely be overcome
using histogram equalisation, more thorough experimentation with lighting, or more forgiving thresholds.
Also partly because of the first issue, the system is somewhat sensitive to the height at which the
webcam is held. Too high, and the stickers are not properly resolved. (Too low, and the car drives out of view!)
Personally, I feel these limitations could be overcome with a one or two more months of work. However, like any
undergraduate project student, Nasa had to meet his deadline...
