> ## Documentation Index
> Fetch the complete documentation index at: https://stem-docs.intellectualpoint.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Computer vision

> Build a color detector and a face tracker from scratch using OpenCV, stream video to your browser, and understand the difference between classical CV and AI.

## Computer vision: color detection and face tracking

<Info>
  **Time**: 11:35 AM to 12:15 PM
</Info>

Your robot's camera can detect specific colors and track faces. In this section you will build two programs from scratch: a color detector with a live web stream, and a face tracker that moves the camera servos to follow you.

Both use **classical computer vision** — fast, offline, but only able to detect what you explicitly program.

***

## Program 1: color detection

Build a script that detects colored objects in the camera feed and streams the annotated video to your browser.

```bash theme={null}
nano ~/camp/day2/color_detect.py
```

### Step 1 — Imports

You need four libraries: `picamera2` for the camera, `cv2` (OpenCV) for image processing, `numpy` for arrays, and Python's built-in `http.server` and `threading` for the web stream:

```python theme={null}
#!/usr/bin/env python3
import sys
import time
import argparse
import threading
from http.server import HTTPServer, BaseHTTPRequestHandler

from picamera2 import Picamera2
import cv2
import numpy as np
```

### Step 2 — Define color ranges

Colors are defined in **HSV** (hue/saturation/value) color space. HSV separates the color itself (hue) from brightness, which makes detection more reliable under different lighting:

```python theme={null}
COLOR_RANGES = {
    "red":    [((0, 100, 100),   (10, 255, 255)),
               ((160, 100, 100), (180, 255, 255))],
    "blue":   [((100, 100, 80),  (130, 255, 255))],
    "green":  [((40, 70, 70),    (80, 255, 255))],
    "yellow": [((20, 100, 100),  (40, 255, 255))],
}

COLOR_BGR = {
    "red": (0, 0, 255), "blue": (255, 0, 0),
    "green": (0, 255, 0), "yellow": (0, 255, 255),
}
```

Red has two ranges because red wraps around in HSV — hue 0 and hue 180 are both red.

### Step 3 — Detection function

Convert the frame to HSV, create a mask of matching pixels, find the largest group, and draw a bounding box:

```python theme={null}
def detect_color(frame, color_name):
    hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
    ranges = COLOR_RANGES.get(color_name, [])
    draw_color = COLOR_BGR.get(color_name, (255, 255, 255))

    mask = np.zeros(hsv.shape[:2], dtype=np.uint8)
    for lower, upper in ranges:
        partial = cv2.inRange(hsv, np.array(lower), np.array(upper))
        mask = cv2.bitwise_or(mask, partial)

    kernel = np.ones((5, 5), np.uint8)
    mask = cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel)
    mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, kernel)

    contours, _ = cv2.findContours(
        mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE
    )

    found = False
    if contours:
        largest = max(contours, key=cv2.contourArea)
        if cv2.contourArea(largest) > 500:
            x, y, w, h = cv2.boundingRect(largest)
            found = True
            cv2.rectangle(frame, (x, y), (x+w, y+h), draw_color, 2)
            cv2.putText(frame, f"{color_name} ({w}x{h})",
                        (x, y-10), cv2.FONT_HERSHEY_SIMPLEX,
                        0.6, draw_color, 2)

    return frame, found
```

`cv2.inRange` checks every pixel: is it within the HSV range? `cv2.findContours` groups the matching pixels into shapes. `cv2.contourArea` measures the size — we ignore anything smaller than 500 pixels to filter out noise.

### Step 4 — MJPEG web stream

Serve the annotated frames as a live video stream in any browser:

```python theme={null}
latest_frame = None
frame_lock = threading.Lock()

class MJPGHandler(BaseHTTPRequestHandler):
    def log_message(self, format, *args):
        pass

    def do_GET(self):
        if self.path in ('/', '/mjpg'):
            self.send_response(200)
            self.send_header(
                'Content-Type',
                'multipart/x-mixed-replace; boundary=frame'
            )
            self.end_headers()
            while True:
                with frame_lock:
                    frame = latest_frame
                if frame is not None:
                    _, jpeg = cv2.imencode('.jpg', frame,
                        [cv2.IMWRITE_JPEG_QUALITY, 70])
                    data = jpeg.tobytes()
                    try:
                        self.wfile.write(b'--frame\r\n')
                        self.wfile.write(
                            b'Content-Type: image/jpeg\r\n\r\n'
                        )
                        self.wfile.write(data)
                        self.wfile.write(b'\r\n')
                    except (BrokenPipeError, ConnectionResetError):
                        break
                time.sleep(0.05)
```

This uses the MJPEG (Motion JPEG) format — each frame is sent as a separate JPEG image, and the browser displays them in sequence like a video. The `threading.Lock` prevents the camera loop and the web server from accessing the frame at the same time.

### Step 5 — Main loop

Start the camera, start the web server, and run the detection loop:

```python theme={null}
if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--color", default="red",
                        choices=list(COLOR_RANGES.keys()))
    parser.add_argument("--port", type=int, default=9000)
    args = parser.parse_args()

    cam = Picamera2()
    config = cam.create_preview_configuration(
        main={"size": (640, 480), "format": "RGB888"}
    )
    cam.configure(config)
    cam.start()
    time.sleep(1)

    server = HTTPServer(("0.0.0.0", args.port), MJPGHandler)
    threading.Thread(target=server.serve_forever, daemon=True).start()
    print(f"  Detecting {args.color} — stream at http://picar.local:{args.port}")

    try:
        while True:
            frame = cam.capture_array()
            frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
            annotated, found = detect_color(frame, args.color)
            with frame_lock:
                latest_frame = annotated
            time.sleep(0.03)
    except KeyboardInterrupt:
        pass
    finally:
        cam.stop(); cam.close(); server.shutdown()
        print("\n  Done.\n")
```

### Run it

```bash theme={null}
sudo python3 ~/camp/day2/color_detect.py --color red
```

Open your browser to:

```
http://picar.local:9000
```

Hold a red object in front of the camera. You should see a bounding box appear. Try other colors:

```bash theme={null}
sudo python3 ~/camp/day2/color_detect.py --color blue
sudo python3 ~/camp/day2/color_detect.py --color green
```

<Tip>
  Detection works best in consistent lighting. If the robot does not detect your object, try a larger or more saturated colored item, or adjust the room lighting.
</Tip>

### Final code: color\_detect.py

<Accordion title="Click to see the complete program">
  ```python theme={null}
  #!/usr/bin/env python3
  import sys
  import time
  import argparse
  import threading
  from http.server import HTTPServer, BaseHTTPRequestHandler

  from picamera2 import Picamera2
  import cv2
  import numpy as np

  COLOR_RANGES = {
      "red":    [((0, 100, 100),   (10, 255, 255)),
                 ((160, 100, 100), (180, 255, 255))],
      "blue":   [((100, 100, 80),  (130, 255, 255))],
      "green":  [((40, 70, 70),    (80, 255, 255))],
      "yellow": [((20, 100, 100),  (40, 255, 255))],
  }

  COLOR_BGR = {
      "red": (0, 0, 255), "blue": (255, 0, 0),
      "green": (0, 255, 0), "yellow": (0, 255, 255),
  }

  latest_frame = None
  frame_lock = threading.Lock()


  class MJPGHandler(BaseHTTPRequestHandler):
      def log_message(self, format, *args):
          pass

      def do_GET(self):
          if self.path in ('/', '/mjpg'):
              self.send_response(200)
              self.send_header('Content-Type',
                               'multipart/x-mixed-replace; boundary=frame')
              self.end_headers()
              while True:
                  with frame_lock:
                      frame = latest_frame
                  if frame is not None:
                      _, jpeg = cv2.imencode('.jpg', frame,
                          [cv2.IMWRITE_JPEG_QUALITY, 70])
                      data = jpeg.tobytes()
                      try:
                          self.wfile.write(b'--frame\r\n')
                          self.wfile.write(
                              b'Content-Type: image/jpeg\r\n\r\n')
                          self.wfile.write(data)
                          self.wfile.write(b'\r\n')
                      except (BrokenPipeError, ConnectionResetError):
                          break
                  time.sleep(0.05)
          else:
              self.send_response(404)
              self.end_headers()


  def detect_color(frame, color_name):
      hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
      ranges = COLOR_RANGES.get(color_name, [])
      draw_color = COLOR_BGR.get(color_name, (255, 255, 255))

      mask = np.zeros(hsv.shape[:2], dtype=np.uint8)
      for lower, upper in ranges:
          partial = cv2.inRange(hsv, np.array(lower), np.array(upper))
          mask = cv2.bitwise_or(mask, partial)

      kernel = np.ones((5, 5), np.uint8)
      mask = cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel)
      mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, kernel)

      contours, _ = cv2.findContours(
          mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

      found = False
      if contours:
          largest = max(contours, key=cv2.contourArea)
          if cv2.contourArea(largest) > 500:
              x, y, w, h = cv2.boundingRect(largest)
              found = True
              cv2.rectangle(frame, (x, y), (x+w, y+h), draw_color, 2)
              cv2.putText(frame, f"{color_name} ({w}x{h})",
                          (x, y-10), cv2.FONT_HERSHEY_SIMPLEX,
                          0.6, draw_color, 2)

      h_frame, w_frame = frame.shape[:2]
      status = f"Tracking: {color_name.upper()}" + (
          " - DETECTED" if found else " - not found")
      cv2.rectangle(frame, (0, h_frame-30), (w_frame, h_frame), (0,0,0), -1)
      cv2.putText(frame, status, (10, h_frame-8),
                  cv2.FONT_HERSHEY_SIMPLEX, 0.6, (255,255,255), 1)

      return frame, found


  if __name__ == "__main__":
      parser = argparse.ArgumentParser()
      parser.add_argument("--color", default="red",
                          choices=list(COLOR_RANGES.keys()))
      parser.add_argument("--port", type=int, default=9000)
      args = parser.parse_args()

      cam = Picamera2()
      config = cam.create_preview_configuration(
          main={"size": (640, 480), "format": "RGB888"})
      cam.configure(config)
      cam.start()
      time.sleep(1)

      server = HTTPServer(("0.0.0.0", args.port), MJPGHandler)
      threading.Thread(target=server.serve_forever, daemon=True).start()
      print(f"  Detecting {args.color} at http://picar.local:{args.port}")

      try:
          while True:
              frame = cam.capture_array()
              frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
              annotated, found = detect_color(frame, args.color)
              with frame_lock:
                  latest_frame = annotated
              time.sleep(0.03)
      except KeyboardInterrupt:
          pass
      finally:
          cam.stop(); cam.close(); server.shutdown()
          print("\n  Done.\n")
  ```
</Accordion>

***

## Program 2: face tracker

Build a script that detects faces and moves the pan/tilt servos to keep the face centered in the frame.

```bash theme={null}
nano ~/camp/day2/face_tracker.py
```

### Step 1 — Face detection with Haar cascades

OpenCV includes a pre-trained **Haar cascade** classifier for face detection. Load it at the top of the script:

```python theme={null}
#!/usr/bin/env python3
import sys
import time
import threading
from http.server import HTTPServer, BaseHTTPRequestHandler

from picamera2 import Picamera2
import cv2
import numpy as np
from picarx import Picarx

CASCADE_PATH = "/usr/share/opencv4/haarcascades/haarcascade_frontalface_default.xml"
face_cascade = cv2.CascadeClassifier(CASCADE_PATH)
```

A Haar cascade is a series of simple image features (edges, lines, rectangles) trained to distinguish "face" from "not face". It is not a neural network — it is a decades-old technique that runs fast on low-power hardware.

### Step 2 — Servo helper functions

The PiCar-X API for camera servos varies between library versions. These helpers handle both:

```python theme={null}
def set_pan(px, angle):
    if hasattr(px, 'set_camera_servo1_angle'):
        px.set_camera_servo1_angle(angle)
    elif hasattr(px, 'set_cam_pan_angle'):
        px.set_cam_pan_angle(angle)

def set_tilt(px, angle):
    if hasattr(px, 'set_camera_servo2_angle'):
        px.set_camera_servo2_angle(angle)
    elif hasattr(px, 'set_cam_tilt_angle'):
        px.set_cam_tilt_angle(angle)
```

### Step 3 — Proportional tracking

The tracking algorithm uses **proportional control**: measure how far the face is from the center of the frame, multiply by a small gain, and adjust the servo by that amount.

```python theme={null}
px = Picarx()
pan_angle = 0.0
tilt_angle = 0.0

PAN_GAIN = 0.035
TILT_GAIN = 0.025
PAN_LIMIT = 35
TILT_LIMIT = 30
dead_zone = 60
```

| Parameter                  | Purpose                                                                                    |
| -------------------------- | ------------------------------------------------------------------------------------------ |
| `PAN_GAIN` / `TILT_GAIN`   | How aggressively the servo reacts. Lower = smoother but slower.                            |
| `dead_zone`                | Ignore errors smaller than this (in pixels). Prevents jitter when the face is near center. |
| `PAN_LIMIT` / `TILT_LIMIT` | Maximum servo angle in either direction.                                                   |

### Step 4 — The tracking loop

Detect faces, find the largest one, calculate the error from frame center, and adjust servos:

```python theme={null}
frame_w, frame_h = 640, 480
center_x, center_y = frame_w // 2, frame_h // 2

while True:
    frame = cam.capture_array()
    frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    faces = face_cascade.detectMultiScale(
        gray, scaleFactor=1.1, minNeighbors=3, minSize=(40, 40)
    )

    if len(faces) > 0:
        largest = max(faces, key=lambda f: f[2] * f[3])
        x, y, w, h = largest
        face_cx = x + w // 2
        face_cy = y + h // 2

        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 255, 0), 2)

        err_x = face_cx - center_x
        err_y = face_cy - center_y

        if abs(err_x) > dead_zone:
            pan_angle += err_x * PAN_GAIN
            pan_angle = max(-PAN_LIMIT, min(PAN_LIMIT, pan_angle))
            set_pan(px, int(pan_angle))

        if abs(err_y) > dead_zone:
            tilt_angle -= err_y * TILT_GAIN
            tilt_angle = max(-TILT_LIMIT, min(TILT_LIMIT, tilt_angle))
            set_tilt(px, int(tilt_angle))
```

If the face is 100 pixels to the right of center and the gain is 0.035, the servo moves 3.5°. If the face is within 60 pixels of center (the dead zone), the servo does not move at all. This prevents the jittery back-and-forth motion that happens with no dead zone.

### Run it

```bash theme={null}
sudo python3 ~/camp/day2/face_tracker.py
```

Open `http://picar.local:9000` in your browser. Stand in front of the camera and move around — the camera should physically follow your face.

***

### Final code: face\_tracker.py

<Accordion title="Click to see the complete program">
  ```python theme={null}
  #!/usr/bin/env python3
  import sys
  import time
  import threading
  from http.server import HTTPServer, BaseHTTPRequestHandler

  from picamera2 import Picamera2
  import cv2
  import numpy as np
  from picarx import Picarx

  CASCADE_PATH = "/usr/share/opencv4/haarcascades/haarcascade_frontalface_default.xml"

  latest_frame = None
  frame_lock = threading.Lock()


  class MJPGHandler(BaseHTTPRequestHandler):
      def log_message(self, format, *args):
          pass

      def do_GET(self):
          if self.path in ('/', '/mjpg'):
              self.send_response(200)
              self.send_header('Content-Type',
                               'multipart/x-mixed-replace; boundary=frame')
              self.end_headers()
              while True:
                  with frame_lock:
                      frame = latest_frame
                  if frame is not None:
                      _, jpeg = cv2.imencode('.jpg', frame,
                          [cv2.IMWRITE_JPEG_QUALITY, 70])
                      data = jpeg.tobytes()
                      try:
                          self.wfile.write(b'--frame\r\n')
                          self.wfile.write(
                              b'Content-Type: image/jpeg\r\n\r\n')
                          self.wfile.write(data)
                          self.wfile.write(b'\r\n')
                      except (BrokenPipeError, ConnectionResetError):
                          break
                  time.sleep(0.05)
          else:
              self.send_response(404)
              self.end_headers()


  def set_pan(px, angle):
      if hasattr(px, 'set_camera_servo1_angle'):
          px.set_camera_servo1_angle(angle)
      elif hasattr(px, 'set_cam_pan_angle'):
          px.set_cam_pan_angle(angle)


  def set_tilt(px, angle):
      if hasattr(px, 'set_camera_servo2_angle'):
          px.set_camera_servo2_angle(angle)
      elif hasattr(px, 'set_cam_tilt_angle'):
          px.set_cam_tilt_angle(angle)


  def main():
      global latest_frame

      face_cascade = cv2.CascadeClassifier(CASCADE_PATH)
      if face_cascade.empty():
          print(f"  ERROR: Could not load {CASCADE_PATH}")
          sys.exit(1)

      px = Picarx()
      pan_angle = 0.0
      tilt_angle = 0.0
      set_pan(px, 0)
      set_tilt(px, 0)

      PAN_GAIN = 0.035
      TILT_GAIN = 0.025
      PAN_LIMIT = 35
      TILT_LIMIT = 30
      dead_zone = 60

      cam = Picamera2()
      config = cam.create_preview_configuration(
          main={"size": (640, 480), "format": "RGB888"})
      cam.configure(config)
      cam.start()
      time.sleep(1)

      server = HTTPServer(("0.0.0.0", 9000), MJPGHandler)
      threading.Thread(target=server.serve_forever, daemon=True).start()
      print("  Stream at http://picar.local:9000")

      frame_w, frame_h = 640, 480
      center_x, center_y = frame_w // 2, frame_h // 2

      try:
          while True:
              frame = cam.capture_array()
              frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
              gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

              faces = face_cascade.detectMultiScale(
                  gray, scaleFactor=1.1, minNeighbors=3,
                  minSize=(40, 40))

              if len(faces) > 0:
                  largest = max(faces, key=lambda f: f[2] * f[3])
                  x, y, w, h = largest
                  face_cx = x + w // 2
                  face_cy = y + h // 2

                  cv2.rectangle(frame, (x, y), (x+w, y+h),
                                (0, 255, 0), 2)
                  cv2.circle(frame, (face_cx, face_cy), 5,
                             (0, 255, 0), -1)

                  err_x = face_cx - center_x
                  err_y = face_cy - center_y

                  if abs(err_x) > dead_zone:
                      pan_angle += err_x * PAN_GAIN
                      pan_angle = max(-PAN_LIMIT,
                                      min(PAN_LIMIT, pan_angle))
                      set_pan(px, int(pan_angle))

                  if abs(err_y) > dead_zone:
                      tilt_angle -= err_y * TILT_GAIN
                      tilt_angle = max(-TILT_LIMIT,
                                       min(TILT_LIMIT, tilt_angle))
                      set_tilt(px, int(tilt_angle))

                  status = (f"FACE ({face_cx},{face_cy}) "
                            f"pan={pan_angle:+.0f} tilt={tilt_angle:+.0f}")
              else:
                  status = "No face detected"

              cv2.line(frame, (center_x-15, center_y),
                       (center_x+15, center_y), (100,100,100), 1)
              cv2.line(frame, (center_x, center_y-15),
                       (center_x, center_y+15), (100,100,100), 1)

              cv2.rectangle(frame, (0, frame_h-30),
                            (frame_w, frame_h), (0,0,0), -1)
              cv2.putText(frame, status, (10, frame_h-8),
                          cv2.FONT_HERSHEY_SIMPLEX, 0.5,
                          (255,255,255), 1)

              with frame_lock:
                  latest_frame = frame

              sys.stdout.write(f"\r  {status}                    ")
              sys.stdout.flush()
              time.sleep(0.03)

      except KeyboardInterrupt:
          print("\n\n  Stopping...")
      finally:
          set_pan(px, 0)
          set_tilt(px, 0)
          cam.stop(); cam.close(); server.shutdown()
          print("  Done.\n")


  if __name__ == "__main__":
      print("=" * 50)
      print("  PiCar-X — Face Tracker")
      print("=" * 50)
      print("  Stream: http://picar.local:9000")
      print("  Press Ctrl-C to stop.\n")
      main()
  ```
</Accordion>

***

## Classical CV vs. AI

|                     | Classical computer vision (today)      | AI vision (Day 4)                   |
| ------------------- | -------------------------------------- | ----------------------------------- |
| **How it works**    | Hand-written rules and thresholds      | Learned patterns from training data |
| **What it detects** | Only what you programmed               | Thousands of object categories      |
| **Speed**           | Very fast                              | Slower (needs more compute)         |
| **Internet**        | Not needed                             | Usually needed (cloud API)          |
| **Flexibility**     | Brittle — breaks with lighting changes | Robust — handles many conditions    |

<Note>
  Everything today used **explicit rules**. The color detector only finds colors you defined in `COLOR_RANGES`. The face tracker only recognizes the pattern "face" using a pre-built classifier. Neither system can learn, adapt, or recognize new objects.

  On Day 4, you will replace these rules with an AI that can recognize **any object** and describe what it sees in natural language.
</Note>

***

## Day 2 wrap-up

**Recap**: your robot now moves, speaks, follows lines, detects colors, and tracks faces — all without any AI. Every program today used explicit rules and thresholds that you wrote.

**What's coming tomorrow**: connect a large language model, give the robot a microphone, and have a real conversation with it. The LLM will control the robot using **tool calls** — the same mechanism that powers ChatGPT plugins.