Computer vision

Computer vision: color detection and face tracking

Time: 11:35 AM to 12:15 PM

Your robot’s camera can detect specific colors and track faces. In this section you will build two programs from scratch: a color detector with a live web stream, and a face tracker that moves the camera servos to follow you. Both use classical computer vision — fast, offline, but only able to detect what you explicitly program.

Program 1: color detection

Build a script that detects colored objects in the camera feed and streams the annotated video to your browser.

nano ~/camp/day2/color_detect.py

Step 1 — Imports

You need four libraries: picamera2 for the camera, cv2 (OpenCV) for image processing, numpy for arrays, and Python’s built-in http.server and threading for the web stream:

#!/usr/bin/env python3
import sys
import time
import argparse
import threading
from http.server import HTTPServer, BaseHTTPRequestHandler

from picamera2 import Picamera2
import cv2
import numpy as np

Step 2 — Define color ranges

Colors are defined in HSV (hue/saturation/value) color space. HSV separates the color itself (hue) from brightness, which makes detection more reliable under different lighting:

COLOR_RANGES = {
    "red":    [((0, 100, 100),   (10, 255, 255)),
               ((160, 100, 100), (180, 255, 255))],
    "blue":   [((100, 100, 80),  (130, 255, 255))],
    "green":  [((40, 70, 70),    (80, 255, 255))],
    "yellow": [((20, 100, 100),  (40, 255, 255))],
}

COLOR_BGR = {
    "red": (0, 0, 255), "blue": (255, 0, 0),
    "green": (0, 255, 0), "yellow": (0, 255, 255),
}

Red has two ranges because red wraps around in HSV — hue 0 and hue 180 are both red.

Step 3 — Detection function

Convert the frame to HSV, create a mask of matching pixels, find the largest group, and draw a bounding box:

def detect_color(frame, color_name):
    hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
    ranges = COLOR_RANGES.get(color_name, [])
    draw_color = COLOR_BGR.get(color_name, (255, 255, 255))

    mask = np.zeros(hsv.shape[:2], dtype=np.uint8)
    for lower, upper in ranges:
        partial = cv2.inRange(hsv, np.array(lower), np.array(upper))
        mask = cv2.bitwise_or(mask, partial)

    kernel = np.ones((5, 5), np.uint8)
    mask = cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel)
    mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, kernel)

    contours, _ = cv2.findContours(
        mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE
    )

    found = False
    if contours:
        largest = max(contours, key=cv2.contourArea)
        if cv2.contourArea(largest) > 500:
            x, y, w, h = cv2.boundingRect(largest)
            found = True
            cv2.rectangle(frame, (x, y), (x+w, y+h), draw_color, 2)
            cv2.putText(frame, f"{color_name} ({w}x{h})",
                        (x, y-10), cv2.FONT_HERSHEY_SIMPLEX,
                        0.6, draw_color, 2)

    return frame, found

cv2.inRange checks every pixel: is it within the HSV range? cv2.findContours groups the matching pixels into shapes. cv2.contourArea measures the size — we ignore anything smaller than 500 pixels to filter out noise.

Step 4 — MJPEG web stream

Serve the annotated frames as a live video stream in any browser:

latest_frame = None
frame_lock = threading.Lock()

class MJPGHandler(BaseHTTPRequestHandler):
    def log_message(self, format, *args):
        pass

    def do_GET(self):
        if self.path in ('/', '/mjpg'):
            self.send_response(200)
            self.send_header(
                'Content-Type',
                'multipart/x-mixed-replace; boundary=frame'
            )
            self.end_headers()
            while True:
                with frame_lock:
                    frame = latest_frame
                if frame is not None:
                    _, jpeg = cv2.imencode('.jpg', frame,
                        [cv2.IMWRITE_JPEG_QUALITY, 70])
                    data = jpeg.tobytes()
                    try:
                        self.wfile.write(b'--frame\r\n')
                        self.wfile.write(
                            b'Content-Type: image/jpeg\r\n\r\n'
                        )
                        self.wfile.write(data)
                        self.wfile.write(b'\r\n')
                    except (BrokenPipeError, ConnectionResetError):
                        break
                time.sleep(0.05)

This uses the MJPEG (Motion JPEG) format — each frame is sent as a separate JPEG image, and the browser displays them in sequence like a video. The threading.Lock prevents the camera loop and the web server from accessing the frame at the same time.

Step 5 — Main loop

Start the camera, start the web server, and run the detection loop:

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--color", default="red",
                        choices=list(COLOR_RANGES.keys()))
    parser.add_argument("--port", type=int, default=9000)
    args = parser.parse_args()

    cam = Picamera2()
    config = cam.create_preview_configuration(
        main={"size": (640, 480), "format": "RGB888"}
    )
    cam.configure(config)
    cam.start()
    time.sleep(1)

    server = HTTPServer(("0.0.0.0", args.port), MJPGHandler)
    threading.Thread(target=server.serve_forever, daemon=True).start()
    print(f"  Detecting {args.color} — stream at http://picar.local:{args.port}")

    try:
        while True:
            frame = cam.capture_array()
            frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
            annotated, found = detect_color(frame, args.color)
            with frame_lock:
                latest_frame = annotated
            time.sleep(0.03)
    except KeyboardInterrupt:
        pass
    finally:
        cam.stop(); cam.close(); server.shutdown()
        print("\n  Done.\n")

Run it

sudo python3 ~/camp/day2/color_detect.py --color red

Open your browser to:

http://picar.local:9000

Hold a red object in front of the camera. You should see a bounding box appear. Try other colors:

sudo python3 ~/camp/day2/color_detect.py --color blue
sudo python3 ~/camp/day2/color_detect.py --color green

Detection works best in consistent lighting. If the robot does not detect your object, try a larger or more saturated colored item, or adjust the room lighting.

Final code: color_detect.py

Click to see the complete program

#!/usr/bin/env python3
import sys
import time
import argparse
import threading
from http.server import HTTPServer, BaseHTTPRequestHandler

from picamera2 import Picamera2
import cv2
import numpy as np

COLOR_RANGES = {
    "red":    [((0, 100, 100),   (10, 255, 255)),
               ((160, 100, 100), (180, 255, 255))],
    "blue":   [((100, 100, 80),  (130, 255, 255))],
    "green":  [((40, 70, 70),    (80, 255, 255))],
    "yellow": [((20, 100, 100),  (40, 255, 255))],
}

COLOR_BGR = {
    "red": (0, 0, 255), "blue": (255, 0, 0),
    "green": (0, 255, 0), "yellow": (0, 255, 255),
}

latest_frame = None
frame_lock = threading.Lock()


class MJPGHandler(BaseHTTPRequestHandler):
    def log_message(self, format, *args):
        pass

    def do_GET(self):
        if self.path in ('/', '/mjpg'):
            self.send_response(200)
            self.send_header('Content-Type',
                             'multipart/x-mixed-replace; boundary=frame')
            self.end_headers()
            while True:
                with frame_lock:
                    frame = latest_frame
                if frame is not None:
                    _, jpeg = cv2.imencode('.jpg', frame,
                        [cv2.IMWRITE_JPEG_QUALITY, 70])
                    data = jpeg.tobytes()
                    try:
                        self.wfile.write(b'--frame\r\n')
                        self.wfile.write(
                            b'Content-Type: image/jpeg\r\n\r\n')
                        self.wfile.write(data)
                        self.wfile.write(b'\r\n')
                    except (BrokenPipeError, ConnectionResetError):
                        break
                time.sleep(0.05)
        else:
            self.send_response(404)
            self.end_headers()


def detect_color(frame, color_name):
    hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
    ranges = COLOR_RANGES.get(color_name, [])
    draw_color = COLOR_BGR.get(color_name, (255, 255, 255))

    mask = np.zeros(hsv.shape[:2], dtype=np.uint8)
    for lower, upper in ranges:
        partial = cv2.inRange(hsv, np.array(lower), np.array(upper))
        mask = cv2.bitwise_or(mask, partial)

    kernel = np.ones((5, 5), np.uint8)
    mask = cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel)
    mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, kernel)

    contours, _ = cv2.findContours(
        mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

    found = False
    if contours:
        largest = max(contours, key=cv2.contourArea)
        if cv2.contourArea(largest) > 500:
            x, y, w, h = cv2.boundingRect(largest)
            found = True
            cv2.rectangle(frame, (x, y), (x+w, y+h), draw_color, 2)
            cv2.putText(frame, f"{color_name} ({w}x{h})",
                        (x, y-10), cv2.FONT_HERSHEY_SIMPLEX,
                        0.6, draw_color, 2)

    h_frame, w_frame = frame.shape[:2]
    status = f"Tracking: {color_name.upper()}" + (
        " - DETECTED" if found else " - not found")
    cv2.rectangle(frame, (0, h_frame-30), (w_frame, h_frame), (0,0,0), -1)
    cv2.putText(frame, status, (10, h_frame-8),
                cv2.FONT_HERSHEY_SIMPLEX, 0.6, (255,255,255), 1)

    return frame, found


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--color", default="red",
                        choices=list(COLOR_RANGES.keys()))
    parser.add_argument("--port", type=int, default=9000)
    args = parser.parse_args()

    cam = Picamera2()
    config = cam.create_preview_configuration(
        main={"size": (640, 480), "format": "RGB888"})
    cam.configure(config)
    cam.start()
    time.sleep(1)

    server = HTTPServer(("0.0.0.0", args.port), MJPGHandler)
    threading.Thread(target=server.serve_forever, daemon=True).start()
    print(f"  Detecting {args.color} at http://picar.local:{args.port}")

    try:
        while True:
            frame = cam.capture_array()
            frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
            annotated, found = detect_color(frame, args.color)
            with frame_lock:
                latest_frame = annotated
            time.sleep(0.03)
    except KeyboardInterrupt:
        pass
    finally:
        cam.stop(); cam.close(); server.shutdown()
        print("\n  Done.\n")

Program 2: face tracker

Build a script that detects faces and moves the pan/tilt servos to keep the face centered in the frame.

nano ~/camp/day2/face_tracker.py

Step 1 — Face detection with Haar cascades

OpenCV includes a pre-trained Haar cascade classifier for face detection. Load it at the top of the script:

#!/usr/bin/env python3
import sys
import time
import threading
from http.server import HTTPServer, BaseHTTPRequestHandler

from picamera2 import Picamera2
import cv2
import numpy as np
from picarx import Picarx

CASCADE_PATH = "/usr/share/opencv4/haarcascades/haarcascade_frontalface_default.xml"
face_cascade = cv2.CascadeClassifier(CASCADE_PATH)

A Haar cascade is a series of simple image features (edges, lines, rectangles) trained to distinguish “face” from “not face”. It is not a neural network — it is a decades-old technique that runs fast on low-power hardware.

Step 2 — Servo helper functions

The PiCar-X API for camera servos varies between library versions. These helpers handle both:

def set_pan(px, angle):
    if hasattr(px, 'set_camera_servo1_angle'):
        px.set_camera_servo1_angle(angle)
    elif hasattr(px, 'set_cam_pan_angle'):
        px.set_cam_pan_angle(angle)

def set_tilt(px, angle):
    if hasattr(px, 'set_camera_servo2_angle'):
        px.set_camera_servo2_angle(angle)
    elif hasattr(px, 'set_cam_tilt_angle'):
        px.set_cam_tilt_angle(angle)

Step 3 — Proportional tracking

The tracking algorithm uses proportional control: measure how far the face is from the center of the frame, multiply by a small gain, and adjust the servo by that amount.

px = Picarx()
pan_angle = 0.0
tilt_angle = 0.0

PAN_GAIN = 0.035
TILT_GAIN = 0.025
PAN_LIMIT = 35
TILT_LIMIT = 30
dead_zone = 60

Parameter	Purpose
`PAN_GAIN` / `TILT_GAIN`	How aggressively the servo reacts. Lower = smoother but slower.
`dead_zone`	Ignore errors smaller than this (in pixels). Prevents jitter when the face is near center.
`PAN_LIMIT` / `TILT_LIMIT`	Maximum servo angle in either direction.

Step 4 — The tracking loop

Detect faces, find the largest one, calculate the error from frame center, and adjust servos:

frame_w, frame_h = 640, 480
center_x, center_y = frame_w // 2, frame_h // 2

while True:
    frame = cam.capture_array()
    frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    faces = face_cascade.detectMultiScale(
        gray, scaleFactor=1.1, minNeighbors=3, minSize=(40, 40)
    )

    if len(faces) > 0:
        largest = max(faces, key=lambda f: f[2] * f[3])
        x, y, w, h = largest
        face_cx = x + w // 2
        face_cy = y + h // 2

        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 255, 0), 2)

        err_x = face_cx - center_x
        err_y = face_cy - center_y

        if abs(err_x) > dead_zone:
            pan_angle += err_x * PAN_GAIN
            pan_angle = max(-PAN_LIMIT, min(PAN_LIMIT, pan_angle))
            set_pan(px, int(pan_angle))

        if abs(err_y) > dead_zone:
            tilt_angle -= err_y * TILT_GAIN
            tilt_angle = max(-TILT_LIMIT, min(TILT_LIMIT, tilt_angle))
            set_tilt(px, int(tilt_angle))

If the face is 100 pixels to the right of center and the gain is 0.035, the servo moves 3.5°. If the face is within 60 pixels of center (the dead zone), the servo does not move at all. This prevents the jittery back-and-forth motion that happens with no dead zone.

Run it

sudo python3 ~/camp/day2/face_tracker.py

Open http://picar.local:9000 in your browser. Stand in front of the camera and move around — the camera should physically follow your face.

Final code: face_tracker.py

Click to see the complete program

#!/usr/bin/env python3
import sys
import time
import threading
from http.server import HTTPServer, BaseHTTPRequestHandler

from picamera2 import Picamera2
import cv2
import numpy as np
from picarx import Picarx

CASCADE_PATH = "/usr/share/opencv4/haarcascades/haarcascade_frontalface_default.xml"

latest_frame = None
frame_lock = threading.Lock()


class MJPGHandler(BaseHTTPRequestHandler):
    def log_message(self, format, *args):
        pass

    def do_GET(self):
        if self.path in ('/', '/mjpg'):
            self.send_response(200)
            self.send_header('Content-Type',
                             'multipart/x-mixed-replace; boundary=frame')
            self.end_headers()
            while True:
                with frame_lock:
                    frame = latest_frame
                if frame is not None:
                    _, jpeg = cv2.imencode('.jpg', frame,
                        [cv2.IMWRITE_JPEG_QUALITY, 70])
                    data = jpeg.tobytes()
                    try:
                        self.wfile.write(b'--frame\r\n')
                        self.wfile.write(
                            b'Content-Type: image/jpeg\r\n\r\n')
                        self.wfile.write(data)
                        self.wfile.write(b'\r\n')
                    except (BrokenPipeError, ConnectionResetError):
                        break
                time.sleep(0.05)
        else:
            self.send_response(404)
            self.end_headers()


def set_pan(px, angle):
    if hasattr(px, 'set_camera_servo1_angle'):
        px.set_camera_servo1_angle(angle)
    elif hasattr(px, 'set_cam_pan_angle'):
        px.set_cam_pan_angle(angle)


def set_tilt(px, angle):
    if hasattr(px, 'set_camera_servo2_angle'):
        px.set_camera_servo2_angle(angle)
    elif hasattr(px, 'set_cam_tilt_angle'):
        px.set_cam_tilt_angle(angle)


def main():
    global latest_frame

    face_cascade = cv2.CascadeClassifier(CASCADE_PATH)
    if face_cascade.empty():
        print(f"  ERROR: Could not load {CASCADE_PATH}")
        sys.exit(1)

    px = Picarx()
    pan_angle = 0.0
    tilt_angle = 0.0
    set_pan(px, 0)
    set_tilt(px, 0)

    PAN_GAIN = 0.035
    TILT_GAIN = 0.025
    PAN_LIMIT = 35
    TILT_LIMIT = 30
    dead_zone = 60

    cam = Picamera2()
    config = cam.create_preview_configuration(
        main={"size": (640, 480), "format": "RGB888"})
    cam.configure(config)
    cam.start()
    time.sleep(1)

    server = HTTPServer(("0.0.0.0", 9000), MJPGHandler)
    threading.Thread(target=server.serve_forever, daemon=True).start()
    print("  Stream at http://picar.local:9000")

    frame_w, frame_h = 640, 480
    center_x, center_y = frame_w // 2, frame_h // 2

    try:
        while True:
            frame = cam.capture_array()
            frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
            gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

            faces = face_cascade.detectMultiScale(
                gray, scaleFactor=1.1, minNeighbors=3,
                minSize=(40, 40))

            if len(faces) > 0:
                largest = max(faces, key=lambda f: f[2] * f[3])
                x, y, w, h = largest
                face_cx = x + w // 2
                face_cy = y + h // 2

                cv2.rectangle(frame, (x, y), (x+w, y+h),
                              (0, 255, 0), 2)
                cv2.circle(frame, (face_cx, face_cy), 5,
                           (0, 255, 0), -1)

                err_x = face_cx - center_x
                err_y = face_cy - center_y

                if abs(err_x) > dead_zone:
                    pan_angle += err_x * PAN_GAIN
                    pan_angle = max(-PAN_LIMIT,
                                    min(PAN_LIMIT, pan_angle))
                    set_pan(px, int(pan_angle))

                if abs(err_y) > dead_zone:
                    tilt_angle -= err_y * TILT_GAIN
                    tilt_angle = max(-TILT_LIMIT,
                                     min(TILT_LIMIT, tilt_angle))
                    set_tilt(px, int(tilt_angle))

                status = (f"FACE ({face_cx},{face_cy}) "
                          f"pan={pan_angle:+.0f} tilt={tilt_angle:+.0f}")
            else:
                status = "No face detected"

            cv2.line(frame, (center_x-15, center_y),
                     (center_x+15, center_y), (100,100,100), 1)
            cv2.line(frame, (center_x, center_y-15),
                     (center_x, center_y+15), (100,100,100), 1)

            cv2.rectangle(frame, (0, frame_h-30),
                          (frame_w, frame_h), (0,0,0), -1)
            cv2.putText(frame, status, (10, frame_h-8),
                        cv2.FONT_HERSHEY_SIMPLEX, 0.5,
                        (255,255,255), 1)

            with frame_lock:
                latest_frame = frame

            sys.stdout.write(f"\r  {status}                    ")
            sys.stdout.flush()
            time.sleep(0.03)

    except KeyboardInterrupt:
        print("\n\n  Stopping...")
    finally:
        set_pan(px, 0)
        set_tilt(px, 0)
        cam.stop(); cam.close(); server.shutdown()
        print("  Done.\n")


if __name__ == "__main__":
    print("=" * 50)
    print("  PiCar-X — Face Tracker")
    print("=" * 50)
    print("  Stream: http://picar.local:9000")
    print("  Press Ctrl-C to stop.\n")
    main()

Classical CV vs. AI

	Classical computer vision (today)	AI vision (Day 4)
How it works	Hand-written rules and thresholds	Learned patterns from training data
What it detects	Only what you programmed	Thousands of object categories
Speed	Very fast	Slower (needs more compute)
Internet	Not needed	Usually needed (cloud API)
Flexibility	Brittle — breaks with lighting changes	Robust — handles many conditions

Everything today used explicit rules. The color detector only finds colors you defined in COLOR_RANGES. The face tracker only recognizes the pattern “face” using a pre-built classifier. Neither system can learn, adapt, or recognize new objects.On Day 4, you will replace these rules with an AI that can recognize any object and describe what it sees in natural language.

Day 2 wrap-up

Recap: your robot now moves, speaks, follows lines, detects colors, and tracks faces — all without any AI. Every program today used explicit rules and thresholds that you wrote. What’s coming tomorrow: connect a large language model, give the robot a microphone, and have a real conversation with it. The LLM will control the robot using tool calls — the same mechanism that powers ChatGPT plugins.

Welcome

Class Recordings

Day 1: Setup and Calibration

Day 2: Code & Computer Vision

Day 3: GenAI and Cloud LLMs

Day 4: Vision AI

Day 5: AI Ethics & Final Project

Computer vision: color detection and face tracking

Program 1: color detection

Step 1 — Imports

Step 2 — Define color ranges

Step 3 — Detection function

Step 4 — MJPEG web stream

Step 5 — Main loop

Run it

Final code: color_detect.py

Program 2: face tracker

Step 1 — Face detection with Haar cascades

Step 2 — Servo helper functions

Step 3 — Proportional tracking

Step 4 — The tracking loop

Run it

Final code: face_tracker.py

Classical CV vs. AI

Day 2 wrap-up

​Computer vision: color detection and face tracking

​Program 1: color detection

​Step 1 — Imports

​Step 2 — Define color ranges

​Step 3 — Detection function

​Step 4 — MJPEG web stream

​Step 5 — Main loop

​Run it

​Final code: color_detect.py

​Program 2: face tracker

​Step 1 — Face detection with Haar cascades

​Step 2 — Servo helper functions

​Step 3 — Proportional tracking

​Step 4 — The tracking loop

​Run it

​Final code: face_tracker.py

​Classical CV vs. AI

​Day 2 wrap-up

Computer vision: color detection and face tracking

Program 1: color detection

Step 1 — Imports

Step 2 — Define color ranges

Step 3 — Detection function

Step 4 — MJPEG web stream

Step 5 — Main loop

Run it

Final code: color_detect.py

Program 2: face tracker

Step 1 — Face detection with Haar cascades

Step 2 — Servo helper functions

Step 3 — Proportional tracking

Step 4 — The tracking loop

Run it

Final code: face_tracker.py

Classical CV vs. AI

Day 2 wrap-up