Embedded Systems + Computer Vision + Projection
A real-time fighting game where your shadow is the controller. Stand in front of a wall, throw punches and kicks with your body, and fight a projected AI opponent that reacts, blocks, and counters. Built on dual Raspberry Pis running QNX RTOS. View source →
01 Overview
A projector casts light onto a wall. You stand between the projector and the wall, creating a real shadow. On the other half, the projector renders a dark silhouette: the AI enemy. The projector projects light everywhere except where the enemy should appear, making it look like a second shadow. Two shadows fight.
01.5 Live Demo
Real-time shadow detection, bounding box tracking, and extremity extraction running on a live camera feed. Left panel shows the raw camera with overlay markers. Right panel shows the segmented binary silhouette.
02 How the Illusion Works
An overhead camera on Pi B tracks your shadow's silhouette using background subtraction, contour detection, and extremity extraction. Head, hands, and feet are tracked as normalized coordinates every frame.
Pi A drives the projector via HDMI. Arena, enemy skeleton, health bars, and danger zones rendered in SDL2 at 1280x720. The enemy appears dark against bright projection, matching your real shadow.
The camera also sees the projected enemy. A rolling buffer of PiShadowState packets lets Pi B subtract the projected silhouette, isolating only the user's real shadow via timestamp matching.
Both Pis communicate wirelessly over UDP at 30fps. ~50 bytes per packet, under 12 kbps total. Non-blocking sockets with stale packet rejection and rolling circular buffer of 16 states.
03 System Architecture
Pi A handles game logic, enemy AI, and projection rendering. Pi B handles camera capture, shadow segmentation, feature extraction, and punch detection. They communicate over WiFi UDP with timestamp-matched frame selection.
04 Technology Stack
Hard real-time OS on both Pis. Deterministic scheduling ensures vision and rendering loops never miss frames under load.
Background subtraction, morphological cleanup, contour extraction, extremity detection. Homography calibration with 4-point auto-detect or manual click mode.
Rendering engine on Pi A. Arena, skeleton, health bars, danger zones drawn with SDL_RenderDrawLine and SDL_RenderFillRect. Zero external assets or textures.
Wide-angle camera on Pi B captures the wall via CSI ribbon cable. V4L2 capture with GStreamer fallback for format compatibility.
POSIX sockets, non-blocking recv, rolling circular buffer of 16 states, timestamp-matched frame selection, stale packet rejection, mutex-protected latest state.
Entire codebase in modern C++. snake_case functions, PascalCase structs, no exceptions, no external dependencies beyond OpenCV and SDL2. CMake cross-compilation.
05 Enemy AI
The AI opponent is a geometric skeleton driven by a finite state machine. It reacts to player movements, executes telegraphed combo patterns, and scales difficulty across three rounds.
06 Frame Pipeline
Pi Camera grabs a frame of the wall. Timestamped with steady_clock for UDP matching.
→Background subtraction isolates shadows. Pi shadow mask from UDP buffer subtracted to keep only the user.
→Contour analysis finds extremities: head, hands, feet, centroid, bounding box. All normalized to [0,1].
→Velocity-based punch/kick detection with history window and cooldown. Events tagged with position and type.
→VisionResult sent to Pi A over UDP. Enemy AI transitions state and renders the response within the same frame.
07 Difficulty Scaling
The AI gets faster, more aggressive, and more likely to counter-attack as rounds progress. Reaction time drops from 450ms to 180ms.