docs(adr-018): Add ESP32 development implementation ADR

Documents the concrete 4-layer development sequence for closing the hardware gap: firmware (ESP-IDF C), UDP aggregator (Rust), CsiFrame→CsiData bridge, and Python _read_raw_data() UDP socket replacement. Builds on ADR-012 architecture and existing wifi-densepose-hardware parser crate. Includes testability path for all layers before hardware acquisition. https://claude.ai/code/session_01BSBAQJ34SLkiJy4A8SoiL4
2026-02-28 17:11:51 +00:00
parent 5cc21987c5
commit c6ad6746e3
1 changed files with 312 additions and 0 deletions
--- a/docs/adr/ADR-018-esp32-dev-implementation.md
+++ b/docs/adr/ADR-018-esp32-dev-implementation.md
@@ -0,0 +1,312 @@
+# ADR-018: ESP32 Development Implementation Path
+
+## Status
+Proposed
+
+## Date
+2026-02-28
+
+## Context
+
+ADR-012 established the ESP32 CSI Sensor Mesh architecture: hardware rationale, firmware file structure, `csi_feature_frame_t` C struct, aggregator design, clock-drift handling via feature-level fusion, and a $54 starter BOM. That ADR answers *what* to build and *why*.
+
+This ADR answers *how* to build it — the concrete development sequence, the specific integration points in existing code, and how to test each layer before hardware is in hand.
+
+### Current State
+
+**Already implemented:**
+
+| Component | Location | Status |
+|-----------|----------|--------|
+| Binary frame parser | `wifi-densepose-hardware/src/esp32_parser.rs` | Complete — `Esp32CsiParser::parse_frame()`, `parse_stream()`, 7 passing tests |
+| Frame types | `wifi-densepose-hardware/src/csi_frame.rs` | Complete — `CsiFrame`, `CsiMetadata`, `SubcarrierData`, `to_amplitude_phase()` |
+| Parse error types | `wifi-densepose-hardware/src/error.rs` | Complete — `ParseError` enum with 6 variants |
+| Signal processing pipeline | `wifi-densepose-signal` crate | Complete — Hampel, Fresnel, BVP, Doppler, spectrogram |
+| CSI extractor (Python) | `v1/src/hardware/csi_extractor.py` | Stub — `_read_raw_data()` raises `NotImplementedError` |
+| Router interface (Python) | `v1/src/hardware/router_interface.py` | Stub — `_parse_csi_response()` raises `RouterConnectionError` |
+
+**Not yet implemented:**
+
+- ESP-IDF C firmware (`firmware/esp32-csi-node/`)
+- UDP aggregator binary (`crates/wifi-densepose-hardware/src/aggregator/`)
+- `CsiFrame` → `wifi_densepose_signal::CsiData` bridge
+- Python `_read_raw_data()` real UDP socket implementation
+- Proof capture tooling for real hardware
+
+### Binary Frame Format (implemented in `esp32_parser.rs`)
+
+```
+Offset  Size  Field
+0       4     Magic: 0xC5110001 (LE)
+4       1     Node ID (0-255)
+5       1     Number of antennas
+6       2     Number of subcarriers (LE u16)
+8       4     Frequency Hz (LE u32, e.g. 2412 for 2.4 GHz ch1)
+12      4     Sequence number (LE u32)
+16      1     RSSI (i8, dBm)
+17      1     Noise floor (i8, dBm)
+18      2     Reserved (zero)
+20      N*2   I/Q pairs: (i8, i8) per subcarrier, repeated per antenna
+```
+
+Total frame size: 20 + (n_antennas × n_subcarriers × 2) bytes.
+
+For 3 antennas, 56 subcarriers: 20 + 336 = 356 bytes per frame.
+
+The firmware must write frames in this exact format. The parser already validates magic, bounds-checks `n_subcarriers` (≤512), and resyncs the stream on magic search for `parse_stream()`.
+
+## Decision
+
+We will implement the ESP32 development stack in four sequential layers, each independently testable before hardware is available.
+
+### Layer 1 — ESP-IDF Firmware (`firmware/esp32-csi-node/`)
+
+Implement the C firmware project per the file structure in ADR-012. Key design decisions deferred from ADR-012:
+
+**CSI callback → frame serializer:**
+
+```c
+// main/csi_collector.c
+static void csi_data_callback(void *ctx, wifi_csi_info_t *info) {
+    if (!info || !info->buf) return;
+
+    // Write binary frame header (20 bytes, little-endian)
+    uint8_t frame[FRAME_MAX_BYTES];
+    uint32_t magic = 0xC5110001;
+    memcpy(frame + 0,  &magic,              4);
+    frame[4] = g_node_id;
+    frame[5] = info->rx_ctrl.ant;           // antenna index (1 for ESP32 single-antenna)
+    uint16_t n_sub = info->len / 2;         // len = n_subcarriers * 2 (I + Q bytes)
+    memcpy(frame + 6,  &n_sub,              2);
+    uint32_t freq_mhz = g_channel_freq_mhz;
+    memcpy(frame + 8,  &freq_mhz,           4);
+    memcpy(frame + 12, &g_seq_num,          4);
+    frame[16] = (int8_t)info->rx_ctrl.rssi;
+    frame[17] = (int8_t)info->rx_ctrl.noise_floor;
+    frame[18] = 0; frame[19] = 0;
+
+    // Write I/Q payload directly from info->buf
+    memcpy(frame + 20, info->buf, info->len);
+
+    // Send over UDP to aggregator
+    stream_sender_write(frame, 20 + info->len);
+    g_seq_num++;
+}
+```
+
+**No on-device FFT** (contradicting ADR-012's optional feature extraction path): The Rust aggregator will do feature extraction using the SOTA `wifi-densepose-signal` pipeline. Raw I/Q is cheaper to stream at ESP32 sampling rates (~100 Hz at 56 subcarriers = ~35 KB/s per node).
+
+**`sdkconfig.defaults`** must enable:
+
+```
+CONFIG_ESP_WIFI_CSI_ENABLED=y
+CONFIG_LWIP_SO_RCVBUF=y
+CONFIG_FREERTOS_HZ=1000
+```
+
+**Build toolchain**: ESP-IDF v5.2+ (pinned). Docker image: `espressif/idf:v5.2` for reproducible CI.
+
+### Layer 2 — UDP Aggregator (`crates/wifi-densepose-hardware/src/aggregator/`)
+
+New module within the hardware crate. Entry point: `aggregator_main()` callable as a binary target.
+
+```rust
+// crates/wifi-densepose-hardware/src/aggregator/mod.rs
+
+pub struct Esp32Aggregator {
+    socket: UdpSocket,
+    nodes: HashMap<u8, NodeState>,       // keyed by node_id from frame header
+    tx: mpsc::SyncSender<CsiFrame>,      // outbound to bridge
+}
+
+struct NodeState {
+    last_seq: u32,
+    drop_count: u64,
+    last_recv: Instant,
+}
+
+impl Esp32Aggregator {
+    /// Bind UDP socket and start blocking receive loop.
+    /// Each valid frame is forwarded on `tx`.
+    pub fn run(&mut self) -> Result<(), AggregatorError> {
+        let mut buf = vec![0u8; 4096];
+        loop {
+            let (n, _addr) = self.socket.recv_from(&mut buf)?;
+            match Esp32CsiParser::parse_frame(&buf[..n]) {
+                Ok((frame, _consumed)) => {
+                    let state = self.nodes.entry(frame.metadata.node_id)
+                        .or_insert_with(NodeState::default);
+                    // Track drops via sequence number gaps
+                    if frame.metadata.seq_num != state.last_seq + 1 {
+                        state.drop_count += (frame.metadata.seq_num
+                            .wrapping_sub(state.last_seq + 1)) as u64;
+                    }
+                    state.last_seq = frame.metadata.seq_num;
+                    state.last_recv = Instant::now();
+                    let _ = self.tx.try_send(frame); // drop if pipeline is full
+                }
+                Err(e) => {
+                    // Log and continue — never crash on bad UDP packet
+                    eprintln!("aggregator: parse error: {e}");
+                }
+            }
+        }
+    }
+}
+```
+
+**Testable without hardware**: The test suite generates frames using `build_test_frame()` (same helper pattern as `esp32_parser.rs` tests) and sends them over a loopback UDP socket. The aggregator receives and forwards them identically to real hardware frames.
+
+### Layer 3 — CsiFrame → CsiData Bridge
+
+Bridge from `wifi-densepose-hardware::CsiFrame` to the signal processing type `wifi_densepose_signal::CsiData` (or a compatible intermediate type consumed by the Rust pipeline).
+
+```rust
+// crates/wifi-densepose-hardware/src/bridge.rs
+
+use crate::{CsiFrame};
+
+/// Intermediate type compatible with the signal processing pipeline.
+/// Maps directly from CsiFrame without cloning the I/Q storage.
+pub struct CsiData {
+    pub timestamp_unix_ms: u64,
+    pub node_id: u8,
+    pub n_antennas: usize,
+    pub n_subcarriers: usize,
+    pub amplitude: Vec<f64>,   // length: n_antennas * n_subcarriers
+    pub phase: Vec<f64>,       // length: n_antennas * n_subcarriers
+    pub rssi_dbm: i8,
+    pub noise_floor_dbm: i8,
+    pub channel_freq_mhz: u32,
+}
+
+impl From<CsiFrame> for CsiData {
+    fn from(frame: CsiFrame) -> Self {
+        let n_ant = frame.metadata.n_antennas as usize;
+        let n_sub = frame.metadata.n_subcarriers as usize;
+        let (amplitude, phase) = frame.to_amplitude_phase();
+        CsiData {
+            timestamp_unix_ms: frame.metadata.timestamp_unix_ms,
+            node_id: frame.metadata.node_id,
+            n_antennas: n_ant,
+            n_subcarriers: n_sub,
+            amplitude,
+            phase,
+            rssi_dbm: frame.metadata.rssi_dbm,
+            noise_floor_dbm: frame.metadata.noise_floor_dbm,
+            channel_freq_mhz: frame.metadata.channel_freq_mhz,
+        }
+    }
+}
+```
+
+The bridge test: parse a known binary frame, convert to `CsiData`, assert `amplitude[0]` = √(I₀² + Q₀²) to within f64 precision.
+
+### Layer 4 — Python `_read_raw_data()` Real Implementation
+
+Replace the `NotImplementedError` stub in `v1/src/hardware/csi_extractor.py` with a UDP socket reader. This allows the Python pipeline to receive real CSI from the aggregator while the Rust pipeline is being integrated.
+
+```python
+# v1/src/hardware/csi_extractor.py
+# Replace _read_raw_data() stub:
+
+import socket as _socket
+
+class CSIExtractor:
+    ...
+    def _read_raw_data(self) -> bytes:
+        """Read one raw CSI frame from the UDP aggregator.
+
+        Expects binary frames in the ESP32 format (magic 0xC5110001 header).
+        Aggregator address configured via AGGREGATOR_HOST / AGGREGATOR_PORT
+        environment variables (defaults: 127.0.0.1:5005).
+        """
+        if not hasattr(self, '_udp_socket'):
+            host = self.config.get('aggregator_host', '127.0.0.1')
+            port = int(self.config.get('aggregator_port', 5005))
+            sock = _socket.socket(_socket.AF_INET, _socket.SOCK_DGRAM)
+            sock.bind((host, port))
+            sock.settimeout(1.0)
+            self._udp_socket = sock
+        try:
+            data, _ = self._udp_socket.recvfrom(4096)
+            return data
+        except _socket.timeout:
+            raise CSIExtractionError(
+                "No CSI data received within timeout — "
+                "is the ESP32 aggregator running?"
+            )
+```
+
+This is tested with a mock UDP server in the unit tests (existing `test_csi_extractor_tdd.py` pattern) and with the real aggregator in integration.
+
+## Development Sequence
+
+```
+Phase 1 (Firmware + Aggregator — no pipeline integration needed):
+  1. Write firmware/esp32-csi-node/ C project (ESP-IDF v5.2)
+  2. Flash to one ESP32-S3-DevKitC board
+  3. Verify binary frames arrive on laptop UDP socket using Wireshark
+  4. Write aggregator crate + loopback test
+
+Phase 2 (Bridge + Python stub):
+  5. Implement CsiFrame → CsiData bridge
+  6. Replace Python _read_raw_data() with UDP socket
+  7. Run Python pipeline end-to-end against loopback aggregator (synthetic frames)
+
+Phase 3 (Real hardware integration):
+  8. Run Python pipeline against live ESP32 frames
+  9. Capture 10-second real CSI bundle (firmware/esp32-csi-node/proof/)
+  10. Verify proof bundle hash (ADR-011 pattern)
+  11. Mark ADR-012 Accepted, mark this ADR Accepted
+```
+
+## Testing Without Hardware
+
+All four layers are testable before a single ESP32 is purchased:
+
+| Layer | Test Method |
+|-------|-------------|
+| Firmware binary format | Build a `build_test_frame()` helper in Rust, compare its output byte-for-byte against a hand-computed reference frame |
+| Aggregator | Loopback UDP: test sends synthetic frames to 127.0.0.1:5005, aggregator receives and forwards on channel |
+| Bridge | `assert_eq!(csi_data.amplitude[0], f64::sqrt((iq[0].i as f64).powi(2) + (iq[0].q as f64).powi(2)))` |
+| Python UDP reader | Mock UDP server in pytest using `socket.socket` in a background thread |
+
+The existing `esp32_parser.rs` test suite already validates parsing of correctly-formatted binary frames. The aggregator and bridge tests build on top of the same test frame construction.
+
+## Consequences
+
+### Positive
+- **Layered testability**: Each layer can be validated independently before hardware acquisition.
+- **No new external dependencies**: UDP sockets are in stdlib (both Rust and Python). Firmware uses only ESP-IDF and esp-dsp component.
+- **Stub elimination**: Replaces the last two `NotImplementedError` stubs in the Python hardware layer with real code backed by real data.
+- **Proof of reality**: Phase 3 produces a captured CSI bundle hashed to a known value, satisfying ADR-011 for hardware-sourced data.
+- **Signal-crate reuse**: The SOTA Hampel/Fresnel/BVP/Doppler processing from ADR-014 applies unchanged to real ESP32 frames after the bridge converts them.
+
+### Negative
+- **Firmware requires ESP-IDF toolchain**: Not buildable without a 2+ GB ESP-IDF installation. CI must use the official Docker image or skip firmware compilation.
+- **Raw I/Q bandwidth**: Streaming raw I/Q (not features) at 100 Hz × 3 antennas × 56 subcarriers = ~35 KB/s/node. At 6 nodes = ~210 KB/s. Fine for LAN; not suitable for WAN.
+- **Single-antenna real-world**: Most ESP32-S3-DevKitC boards have one on-board antenna. Multi-antenna data requires external antenna + board with U.FL connector or purpose-built multi-radio setup.
+
+### Deferred
+- **Multi-node clock drift compensation**: ADR-012 specifies feature-level fusion. The aggregator in this ADR passes raw `CsiFrame` per-node. Drift compensation lives in a future `FeatureFuser` layer (not scoped here).
+- **ESP-IDF firmware CI**: Firmware compilation in GitHub Actions requires the ESP-IDF Docker image. CI integration is deferred until Phase 3 hardware validation.
+
+## Interaction with Other ADRs
+
+| ADR | Interaction |
+|-----|-------------|
+| ADR-011 | Phase 3 produces a real CSI proof bundle satisfying mock elimination |
+| ADR-012 | This ADR implements the development path for ADR-012's architecture |
+| ADR-014 | SOTA signal processing applies unchanged after bridge layer |
+| ADR-008 | Aggregator handles multi-node; distributed consensus is a later concern |
+
+## References
+
+- [Espressif ESP-CSI Repository](https://github.com/espressif/esp-csi)
+- [ESP-IDF WiFi CSI API Reference](https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-guides/wifi.html#wi-fi-channel-state-information)
+- `wifi-densepose-hardware/src/esp32_parser.rs` — binary frame parser implementation
+- `wifi-densepose-hardware/src/csi_frame.rs` — `CsiFrame`, `to_amplitude_phase()`
+- ADR-012: ESP32 CSI Sensor Mesh (architecture)
+- ADR-011: Python Proof-of-Reality and Mock Elimination
+- ADR-014: SOTA Signal Processing