docs: Add comprehensive build guide for all environments

Covers quick start verification, Python/Rust pipelines, Docker deployment (dev/prod/swarm), ESP32 hardware setup, WASM edge builds, ARM cross-compilation, and Three.js visualization. https://claude.ai/code/session_01Ki7pvEZtJDvqJkmyn6B714
2026-02-28 06:53:37 +00:00
parent 195f7150ac
commit 45f8a0d3e7
1 changed files with 684 additions and 0 deletions
--- a/docs/build-guide.md
+++ b/docs/build-guide.md
@@ -0,0 +1,684 @@
+# WiFi-DensePose Build and Run Guide
+
+Covers every way to build, run, and deploy the system -- from a zero-hardware verification to a full ESP32 mesh with 3D visualization.
+
+---
+
+## Table of Contents
+
+1. [Quick Start (Verification Only -- No Hardware)](#1-quick-start-verification-only----no-hardware)
+2. [Python Pipeline (v1/)](#2-python-pipeline-v1)
+3. [Rust Pipeline (v2)](#3-rust-pipeline-v2)
+4. [Three.js Visualization](#4-threejs-visualization)
+5. [Docker Deployment](#5-docker-deployment)
+6. [ESP32 Hardware Setup](#6-esp32-hardware-setup)
+7. [Environment-Specific Builds](#7-environment-specific-builds)
+
+---
+
+## 1. Quick Start (Verification Only -- No Hardware)
+
+The fastest way to confirm the signal processing pipeline is real and deterministic. Requires only Python 3.8+, numpy, and scipy. No WiFi hardware, no GPU, no Docker.
+
+```bash
+# From the repository root:
+./verify
+```
+
+This runs three phases:
+
+1. **Environment checks** -- confirms Python, numpy, scipy, and proof files are present.
+2. **Proof pipeline replay** -- feeds a published reference signal through the full signal processing chain (noise filtering, Hamming windowing, amplitude normalization, FFT-based Doppler extraction, power spectral density via scipy.fft) and computes a SHA-256 hash of the output.
+3. **Production code integrity scan** -- scans `v1/src/` for `np.random.rand` / `np.random.randn` calls in production code (test helpers are excluded).
+
+Exit codes:
+- `0` PASS -- pipeline hash matches the published expected hash
+- `1` FAIL -- hash mismatch or error
+- `2` SKIP -- no expected hash file to compare against
+
+Additional flags:
+
+```bash
+./verify --verbose         # Detailed feature statistics and Doppler spectrum
+./verify --verbose --audit # Full verification + codebase audit
+
+# Or via make:
+make verify
+make verify-verbose
+make verify-audit
+```
+
+If the expected hash file is missing, regenerate it:
+
+```bash
+python3 v1/data/proof/verify.py --generate-hash
+```
+
+### Minimal dependencies for verification only
+
+```bash
+pip install numpy==1.26.4 scipy==1.14.1
+```
+
+Or install the pinned set that guarantees hash reproducibility:
+
+```bash
+pip install -r v1/requirements-lock.txt
+```
+
+The lock file pins: `numpy==1.26.4`, `scipy==1.14.1`, `pydantic==2.10.4`, `pydantic-settings==2.7.1`.
+
+---
+
+## 2. Python Pipeline (v1/)
+
+The Python pipeline lives under `v1/` and provides the full API server, signal processing, sensing modules, and WebSocket streaming.
+
+### Prerequisites
+
+- Python 3.8+
+- pip
+
+### Install (verification-only -- lightweight)
+
+```bash
+pip install -r v1/requirements-lock.txt
+```
+
+This installs only the four packages needed for deterministic pipeline verification.
+
+### Install (full pipeline with API server)
+
+```bash
+pip install -r requirements.txt
+```
+
+This pulls in FastAPI, uvicorn, torch, OpenCV, SQLAlchemy, Redis client, and all other runtime dependencies.
+
+### Verify the pipeline
+
+```bash
+python3 v1/data/proof/verify.py
+```
+
+Same as `./verify` but calls the Python script directly, skipping the bash wrapper's codebase scan phase.
+
+### Run the API server
+
+```bash
+uvicorn v1.src.api.main:app --host 0.0.0.0 --port 8000
+```
+
+The server exposes:
+- REST API docs: http://localhost:8000/docs
+- Health check: http://localhost:8000/health
+- Latest poses: http://localhost:8000/api/v1/pose/latest
+- WebSocket pose stream: ws://localhost:8000/ws/pose/stream
+- WebSocket analytics: ws://localhost:8000/ws/analytics/events
+
+For development with auto-reload:
+
+```bash
+uvicorn v1.src.api.main:app --host 0.0.0.0 --port 8000 --reload
+```
+
+### Run with commodity WiFi (RSSI sensing -- no custom hardware)
+
+The commodity sensing module (`v1/src/sensing/`) extracts presence and motion features from standard Linux WiFi metrics (RSSI, noise floor, link quality) without any hardware modification. See [ADR-013](adr/ADR-013-feature-level-sensing-commodity-gear.md) for full design details.
+
+Requirements:
+- Any Linux machine with a WiFi interface (laptop, Raspberry Pi, etc.)
+- Connected to a WiFi access point (the AP is the signal source)
+- No root required for basic RSSI reading via `/proc/net/wireless`
+
+The module provides:
+- `LinuxWifiCollector` -- reads real RSSI from `/proc/net/wireless` and `iw` commands
+- `RssiFeatureExtractor` -- computes rolling statistics, FFT spectral features, CUSUM change-point detection
+- `PresenceClassifier` -- rule-based presence/motion classification
+
+What it can detect:
+| Capability | Single Receiver | 3+ Receivers |
+|-----------|----------------|-------------|
+| Binary presence | Yes (90-95%) | Yes (90-95%) |
+| Coarse motion (still/moving) | Yes (85-90%) | Yes (85-90%) |
+| Room-level location | No | Marginal (70-80%) |
+
+What it cannot detect: body pose, heartbeat, reliable respiration. See ADR-013 for the honest capability matrix.
+
+### Python project structure
+
+```
+v1/
+  src/
+    api/
+      main.py              # FastAPI application entry point
+      routers/             # REST endpoint routers (pose, stream, health)
+      middleware/           # Auth, rate limiting
+      websocket/           # WebSocket connection manager, pose stream
+    config/                # Settings, domain configs
+    sensing/
+      rssi_collector.py    # LinuxWifiCollector + SimulatedCollector
+      feature_extractor.py # RssiFeatureExtractor (FFT, CUSUM, spectral)
+      classifier.py        # PresenceClassifier (rule-based)
+      backend.py           # SensingBackend protocol
+  data/
+    proof/
+      sample_csi_data.json       # Deterministic reference signal
+      expected_features.sha256   # Published expected hash
+      verify.py                  # One-command verification script
+  requirements-lock.txt          # Pinned deps for hash reproducibility
+```
+
+---
+
+## 3. Rust Pipeline (v2)
+
+A high-performance Rust port with ~810x speedup over the Python pipeline for the full signal processing chain.
+
+### Prerequisites
+
+- Rust 1.70+ (install via [rustup](https://rustup.rs/))
+- cargo (included with Rust)
+- System dependencies for OpenBLAS (used by ndarray-linalg):
+  ```bash
+  # Ubuntu/Debian
+  sudo apt-get install build-essential gfortran libopenblas-dev pkg-config
+
+  # macOS
+  brew install openblas
+  ```
+
+### Build
+
+```bash
+cd rust-port/wifi-densepose-rs
+cargo build --release
+```
+
+Release profile is configured with LTO, single codegen unit, and `-O3` for maximum performance.
+
+### Test
+
+```bash
+cd rust-port/wifi-densepose-rs
+cargo test --workspace
+```
+
+Runs 107 tests across all workspace crates.
+
+### Benchmark
+
+```bash
+cd rust-port/wifi-densepose-rs
+cargo bench --package wifi-densepose-signal
+```
+
+Expected throughput:
+| Operation | Latency | Throughput |
+|-----------|---------|------------|
+| CSI Preprocessing (4x64) | ~5.19 us | 49-66 Melem/s |
+| Phase Sanitization (4x64) | ~3.84 us | 67-85 Melem/s |
+| Feature Extraction (4x64) | ~9.03 us | 7-11 Melem/s |
+| Motion Detection | ~186 ns | -- |
+| Full Pipeline | ~18.47 us | ~54,000 fps |
+
+### Workspace crates
+
+The Rust workspace contains 10 crates under `crates/`:
+
+| Crate | Description |
+|-------|-------------|
+| `wifi-densepose-core` | Core types, traits, and domain models |
+| `wifi-densepose-signal` | Signal processing (FFT, phase unwrapping, Doppler, correlation) |
+| `wifi-densepose-nn` | Neural network inference (ONNX Runtime, candle, tch) |
+| `wifi-densepose-api` | Axum-based HTTP/WebSocket API server |
+| `wifi-densepose-db` | Database layer (SQLx, PostgreSQL, SQLite, Redis) |
+| `wifi-densepose-config` | Configuration loading (env vars, YAML, TOML) |
+| `wifi-densepose-hardware` | Hardware adapters (ESP32, Intel 5300, Atheros, UDP, PCAP) |
+| `wifi-densepose-wasm` | WebAssembly bindings for browser deployment |
+| `wifi-densepose-cli` | Command-line interface |
+| `wifi-densepose-mat` | WiFi-Mat disaster response module (search and rescue) |
+
+Build individual crates:
+
+```bash
+# Signal processing only
+cargo build --release --package wifi-densepose-signal
+
+# API server
+cargo build --release --package wifi-densepose-api
+
+# Disaster response module
+cargo build --release --package wifi-densepose-mat
+
+# WASM target (see Section 7 for full instructions)
+cargo build --release --package wifi-densepose-wasm --target wasm32-unknown-unknown
+```
+
+---
+
+## 4. Three.js Visualization
+
+A browser-based 3D visualization dashboard that renders DensePose body models with 24 body parts, signal visualization, and environment rendering.
+
+### Run
+
+Open `ui/viz.html` directly in a browser:
+
+```bash
+# macOS
+open ui/viz.html
+
+# Linux
+xdg-open ui/viz.html
+
+# Or serve it locally
+python3 -m http.server 3000 --directory ui
+# Then open http://localhost:3000/viz.html
+```
+
+### WebSocket connection
+
+The visualization connects to `ws://localhost:8000/ws/pose` for real-time pose data. If no server is running, it falls back to a demo mode with simulated data so you can still see the 3D rendering.
+
+To see live data:
+
+1. Start the API server (Python or Rust)
+2. Open `ui/viz.html`
+3. The dashboard will connect automatically
+
+---
+
+## 5. Docker Deployment
+
+### Development (with hot-reload, Postgres, Redis, Prometheus, Grafana)
+
+```bash
+docker compose up
+```
+
+This starts:
+- `wifi-densepose-dev` -- API server with `--reload`, debug logging, auth disabled (port 8000)
+- `postgres` -- PostgreSQL 15 (port 5432)
+- `redis` -- Redis 7 with AOF persistence (port 6379)
+- `prometheus` -- metrics scraping (port 9090)
+- `grafana` -- dashboards (port 3000, login: admin/admin)
+- `nginx` -- reverse proxy (ports 80, 443)
+
+```bash
+# View logs
+docker compose logs -f wifi-densepose
+
+# Run tests inside the container
+docker compose exec wifi-densepose pytest tests/ -v
+
+# Stop everything
+docker compose down
+
+# Stop and remove volumes
+docker compose down -v
+```
+
+### Production
+
+Uses the production Dockerfile stage with 4 uvicorn workers, auth enabled, rate limiting, and resource limits.
+
+```bash
+# Build production image
+docker build --target production -t wifi-densepose:latest .
+
+# Run standalone
+docker run -d \
+  --name wifi-densepose \
+  -p 8000:8000 \
+  -e ENVIRONMENT=production \
+  -e SECRET_KEY=your-secret-key \
+  wifi-densepose:latest
+```
+
+For the full production stack with Docker Swarm secrets:
+
+```bash
+# Create required secrets first
+echo "db_password_here" | docker secret create db_password -
+echo "redis_password_here" | docker secret create redis_password -
+echo "jwt_secret_here" | docker secret create jwt_secret -
+echo "api_key_here" | docker secret create api_key -
+echo "grafana_password_here" | docker secret create grafana_password -
+
+# Set required environment variables
+export DATABASE_URL=postgresql://wifi_user:db_password_here@postgres:5432/wifi_densepose
+export REDIS_URL=redis://redis:6379/0
+export SECRET_KEY=your-secret-key
+export JWT_SECRET=your-jwt-secret
+export ALLOWED_HOSTS=your-domain.com
+export POSTGRES_DB=wifi_densepose
+export POSTGRES_USER=wifi_user
+
+# Deploy with Docker Swarm
+docker stack deploy -c docker-compose.prod.yml wifi-densepose
+```
+
+Production compose includes:
+- 3 API server replicas with rolling updates and rollback
+- Resource limits (2 CPU, 4GB RAM per replica)
+- Health checks on all services
+- JSON file logging with rotation
+- Separate monitoring network (overlay)
+- Prometheus with alerting rules and 15-day retention
+- Grafana with provisioned datasources and dashboards
+
+### Dockerfile stages
+
+The multi-stage `Dockerfile` provides four targets:
+
+| Target | Use | Command |
+|--------|-----|---------|
+| `development` | Local dev with hot-reload | `docker build --target development .` |
+| `production` | Optimized production image | `docker build --target production .` |
+| `testing` | Runs pytest during build | `docker build --target testing .` |
+| `security` | Runs safety + bandit scans | `docker build --target security .` |
+
+---
+
+## 6. ESP32 Hardware Setup
+
+Uses ESP32-S3 boards as WiFi CSI sensor nodes. See [ADR-012](adr/ADR-012-esp32-csi-sensor-mesh.md) for the full specification.
+
+### Bill of Materials (Starter Kit -- $54)
+
+| Item | Qty | Unit Cost | Total |
+|------|-----|-----------|-------|
+| ESP32-S3-DevKitC-1 | 3 | $10 | $30 |
+| USB-A to USB-C cables | 3 | $3 | $9 |
+| USB power adapter (multi-port) | 1 | $15 | $15 |
+| Consumer WiFi router (any) | 1 | $0 (existing) | $0 |
+| Aggregator (laptop or Pi 4) | 1 | $0 (existing) | $0 |
+| **Total** | | | **$54** |
+
+### Prerequisites
+
+Install ESP-IDF (Espressif's official development framework):
+
+```bash
+# Clone ESP-IDF
+mkdir -p ~/esp
+cd ~/esp
+git clone --recursive https://github.com/espressif/esp-idf.git
+cd esp-idf
+git checkout v5.2  # Pin to tested version
+
+# Install tools
+./install.sh esp32s3
+
+# Activate environment (run each session)
+. ./export.sh
+```
+
+### Flash a node
+
+```bash
+cd firmware/esp32-csi-node
+
+# Set target chip
+idf.py set-target esp32s3
+
+# Configure WiFi SSID/password and aggregator IP
+idf.py menuconfig
+# Navigate to: Component config > WiFi-DensePose CSI Node
+#   - Set WiFi SSID
+#   - Set WiFi password
+#   - Set aggregator IP address
+#   - Set node ID (1, 2, 3, ...)
+#   - Set sampling rate (10-100 Hz)
+
+# Build and flash (with USB cable connected)
+idf.py build flash monitor
+```
+
+`idf.py monitor` shows live serial output including CSI callback data. Press `Ctrl+]` to exit.
+
+Repeat for each node, incrementing the node ID.
+
+### Firmware project structure
+
+```
+firmware/esp32-csi-node/
+  CMakeLists.txt
+  sdkconfig.defaults          # Menuconfig defaults with CSI enabled
+  main/
+    main.c                    # Entry point, WiFi init, CSI callback
+    csi_collector.c           # CSI data collection and buffering
+    feature_extract.c         # On-device FFT and feature extraction
+    stream_sender.c           # UDP stream to aggregator
+    config.h                  # Node configuration
+    Kconfig.projbuild         # Menuconfig options
+  components/
+    esp_dsp/                  # Espressif DSP library for FFT
+```
+
+Each node does on-device feature extraction (raw I/Q to amplitude + phase + spectral bands), reducing bandwidth from ~11 KB/frame to ~470 bytes/frame. Nodes stream features via UDP to the aggregator.
+
+### Run the aggregator
+
+The aggregator collects UDP streams from all ESP32 nodes, performs feature-level fusion (not signal-level -- see ADR-012 for why), and feeds the fused data into the Rust or Python pipeline.
+
+```bash
+# Start the aggregator and pipeline via Docker
+docker compose -f docker-compose.esp32.yml up
+
+# Or run the Rust aggregator directly
+cd rust-port/wifi-densepose-rs
+cargo run --release --package wifi-densepose-hardware -- --mode esp32-aggregator --port 5000
+```
+
+### Verify with real hardware
+
+```bash
+docker exec aggregator python verify_esp32.py
+```
+
+This captures 10 seconds of data, produces feature JSON, and verifies the hash against the proof bundle.
+
+### What the ESP32 mesh can and cannot detect
+
+| Capability | 1 Node | 3 Nodes | 6 Nodes |
+|-----------|--------|---------|---------|
+| Presence detection | Good | Excellent | Excellent |
+| Coarse motion | Good | Excellent | Excellent |
+| Room-level location | None | Good | Excellent |
+| Respiration | Marginal | Good | Good |
+| Heartbeat | Poor | Poor | Marginal |
+| Multi-person count | None | Marginal | Good |
+| Pose estimation | None | Poor | Marginal |
+
+---
+
+## 7. Environment-Specific Builds
+
+### Browser (WASM)
+
+Compiles the Rust pipeline to WebAssembly for in-browser execution. See [ADR-009](adr/ADR-009-rvf-wasm-runtime-edge-deployment.md) for the edge deployment architecture.
+
+Prerequisites:
+
+```bash
+# Install wasm-pack
+curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
+
+# Or via cargo
+cargo install wasm-pack
+
+# Add the WASM target
+rustup target add wasm32-unknown-unknown
+```
+
+Build:
+
+```bash
+cd rust-port/wifi-densepose-rs
+
+# Build WASM package (outputs to pkg/)
+wasm-pack build crates/wifi-densepose-wasm --target web --release
+
+# Build with disaster response module included
+wasm-pack build crates/wifi-densepose-wasm --target web --release -- --features mat
+```
+
+The output `pkg/` directory contains `.wasm`, `.js` glue, and TypeScript definitions. Import in a web project:
+
+```javascript
+import init, { WifiDensePoseWasm } from './pkg/wifi_densepose_wasm.js';
+
+async function main() {
+  await init();
+  const processor = new WifiDensePoseWasm();
+  const result = processor.process_frame(csiJsonString);
+  console.log(JSON.parse(result));
+}
+main();
+```
+
+Run WASM tests:
+
+```bash
+wasm-pack test --headless --chrome crates/wifi-densepose-wasm
+```
+
+Container size targets by deployment profile:
+
+| Profile | Size | Suitable For |
+|---------|------|-------------|
+| Browser (int8 quantization) | ~10 MB | Chrome/Firefox dashboard |
+| IoT (int4 quantization) | ~0.7 MB | ESP32, constrained devices |
+| Mobile (int8 quantization) | ~6 MB | iOS/Android WebView |
+| Field (fp16 quantization) | ~62 MB | Offline disaster tablets |
+
+### IoT (ESP32)
+
+See [Section 6](#6-esp32-hardware-setup) for full ESP32 setup. The firmware runs on the device itself (C, compiled with ESP-IDF). The Rust aggregator runs on a host machine.
+
+For deploying the WASM runtime to a Raspberry Pi or similar:
+
+```bash
+# Cross-compile for ARM
+rustup target add aarch64-unknown-linux-gnu
+cargo build --release --package wifi-densepose-cli --target aarch64-unknown-linux-gnu
+```
+
+### Server (Docker)
+
+See [Section 5](#5-docker-deployment).
+
+Quick reference:
+
+```bash
+# Development
+docker compose up
+
+# Production standalone
+docker build --target production -t wifi-densepose:latest .
+docker run -d -p 8000:8000 wifi-densepose:latest
+
+# Production stack (Swarm)
+docker stack deploy -c docker-compose.prod.yml wifi-densepose
+```
+
+### Server (Direct -- no Docker)
+
+```bash
+# 1. Install Python dependencies
+pip install -r requirements.txt
+
+# 2. Set environment variables (copy from example.env)
+cp example.env .env
+# Edit .env with your settings
+
+# 3. Run with uvicorn (production)
+uvicorn v1.src.api.main:app \
+  --host 0.0.0.0 \
+  --port 8000 \
+  --workers 4
+
+# Or run the Rust API server
+cd rust-port/wifi-densepose-rs
+cargo run --release --package wifi-densepose-api
+```
+
+### Development (local with hot-reload)
+
+Python:
+
+```bash
+# Create virtual environment
+python3 -m venv venv
+source venv/bin/activate
+
+# Install all dependencies including dev tools
+pip install -r requirements.txt
+
+# Run with auto-reload
+uvicorn v1.src.api.main:app --host 0.0.0.0 --port 8000 --reload
+
+# Run verification in another terminal
+./verify --verbose
+
+# Run tests
+pytest tests/ -v
+pytest --cov=wifi_densepose --cov-report=html
+```
+
+Rust:
+
+```bash
+cd rust-port/wifi-densepose-rs
+
+# Build in debug mode (faster compilation)
+cargo build
+
+# Run tests with output
+cargo test --workspace -- --nocapture
+
+# Watch mode (requires cargo-watch)
+cargo install cargo-watch
+cargo watch -x 'test --workspace' -x 'build --release'
+
+# Run benchmarks
+cargo bench --package wifi-densepose-signal
+```
+
+Both (visualization + API):
+
+```bash
+# Terminal 1: Start API server
+uvicorn v1.src.api.main:app --host 0.0.0.0 --port 8000 --reload
+
+# Terminal 2: Serve visualization
+python3 -m http.server 3000 --directory ui
+
+# Open http://localhost:3000/viz.html -- it connects to ws://localhost:8000/ws/pose
+```
+
+---
+
+## Appendix: Key File Locations
+
+| File | Purpose |
+|------|---------|
+| `./verify` | Trust kill switch -- one-command pipeline proof |
+| `Makefile` | `make verify`, `make verify-verbose`, `make verify-audit` |
+| `v1/requirements-lock.txt` | Pinned Python deps for hash reproducibility |
+| `requirements.txt` | Full Python deps (API server, torch, etc.) |
+| `v1/data/proof/verify.py` | Python verification script |
+| `v1/data/proof/sample_csi_data.json` | Deterministic reference signal |
+| `v1/data/proof/expected_features.sha256` | Published expected hash |
+| `v1/src/api/main.py` | FastAPI application entry point |
+| `v1/src/sensing/` | Commodity WiFi sensing module (RSSI) |
+| `rust-port/wifi-densepose-rs/Cargo.toml` | Rust workspace root |
+| `ui/viz.html` | Three.js 3D visualization |
+| `Dockerfile` | Multi-stage Docker build (dev/prod/test/security) |
+| `docker-compose.yml` | Development stack (Postgres, Redis, Prometheus, Grafana) |
+| `docker-compose.prod.yml` | Production stack (Swarm, secrets, resource limits) |
+| `docs/adr/ADR-009-rvf-wasm-runtime-edge-deployment.md` | WASM edge deployment architecture |
+| `docs/adr/ADR-012-esp32-csi-sensor-mesh.md` | ESP32 firmware and mesh specification |
+| `docs/adr/ADR-013-feature-level-sensing-commodity-gear.md` | Commodity WiFi (RSSI) sensing |