From c6ad6746e389829b161e18860be854b33ff019ea Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 28 Feb 2026 17:11:51 +0000 Subject: [PATCH] docs(adr-018): Add ESP32 development implementation ADR MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents the concrete 4-layer development sequence for closing the hardware gap: firmware (ESP-IDF C), UDP aggregator (Rust), CsiFrame→CsiData bridge, and Python _read_raw_data() UDP socket replacement. Builds on ADR-012 architecture and existing wifi-densepose-hardware parser crate. Includes testability path for all layers before hardware acquisition. https://claude.ai/code/session_01BSBAQJ34SLkiJy4A8SoiL4 --- docs/adr/ADR-018-esp32-dev-implementation.md | 312 +++++++++++++++++++ 1 file changed, 312 insertions(+) create mode 100644 docs/adr/ADR-018-esp32-dev-implementation.md diff --git a/docs/adr/ADR-018-esp32-dev-implementation.md b/docs/adr/ADR-018-esp32-dev-implementation.md new file mode 100644 index 0000000..26a3dd4 --- /dev/null +++ b/docs/adr/ADR-018-esp32-dev-implementation.md @@ -0,0 +1,312 @@ +# ADR-018: ESP32 Development Implementation Path + +## Status +Proposed + +## Date +2026-02-28 + +## Context + +ADR-012 established the ESP32 CSI Sensor Mesh architecture: hardware rationale, firmware file structure, `csi_feature_frame_t` C struct, aggregator design, clock-drift handling via feature-level fusion, and a $54 starter BOM. That ADR answers *what* to build and *why*. + +This ADR answers *how* to build it — the concrete development sequence, the specific integration points in existing code, and how to test each layer before hardware is in hand. + +### Current State + +**Already implemented:** + +| Component | Location | Status | +|-----------|----------|--------| +| Binary frame parser | `wifi-densepose-hardware/src/esp32_parser.rs` | Complete — `Esp32CsiParser::parse_frame()`, `parse_stream()`, 7 passing tests | +| Frame types | `wifi-densepose-hardware/src/csi_frame.rs` | Complete — `CsiFrame`, `CsiMetadata`, `SubcarrierData`, `to_amplitude_phase()` | +| Parse error types | `wifi-densepose-hardware/src/error.rs` | Complete — `ParseError` enum with 6 variants | +| Signal processing pipeline | `wifi-densepose-signal` crate | Complete — Hampel, Fresnel, BVP, Doppler, spectrogram | +| CSI extractor (Python) | `v1/src/hardware/csi_extractor.py` | Stub — `_read_raw_data()` raises `NotImplementedError` | +| Router interface (Python) | `v1/src/hardware/router_interface.py` | Stub — `_parse_csi_response()` raises `RouterConnectionError` | + +**Not yet implemented:** + +- ESP-IDF C firmware (`firmware/esp32-csi-node/`) +- UDP aggregator binary (`crates/wifi-densepose-hardware/src/aggregator/`) +- `CsiFrame` → `wifi_densepose_signal::CsiData` bridge +- Python `_read_raw_data()` real UDP socket implementation +- Proof capture tooling for real hardware + +### Binary Frame Format (implemented in `esp32_parser.rs`) + +``` +Offset Size Field +0 4 Magic: 0xC5110001 (LE) +4 1 Node ID (0-255) +5 1 Number of antennas +6 2 Number of subcarriers (LE u16) +8 4 Frequency Hz (LE u32, e.g. 2412 for 2.4 GHz ch1) +12 4 Sequence number (LE u32) +16 1 RSSI (i8, dBm) +17 1 Noise floor (i8, dBm) +18 2 Reserved (zero) +20 N*2 I/Q pairs: (i8, i8) per subcarrier, repeated per antenna +``` + +Total frame size: 20 + (n_antennas × n_subcarriers × 2) bytes. + +For 3 antennas, 56 subcarriers: 20 + 336 = 356 bytes per frame. + +The firmware must write frames in this exact format. The parser already validates magic, bounds-checks `n_subcarriers` (≤512), and resyncs the stream on magic search for `parse_stream()`. + +## Decision + +We will implement the ESP32 development stack in four sequential layers, each independently testable before hardware is available. + +### Layer 1 — ESP-IDF Firmware (`firmware/esp32-csi-node/`) + +Implement the C firmware project per the file structure in ADR-012. Key design decisions deferred from ADR-012: + +**CSI callback → frame serializer:** + +```c +// main/csi_collector.c +static void csi_data_callback(void *ctx, wifi_csi_info_t *info) { + if (!info || !info->buf) return; + + // Write binary frame header (20 bytes, little-endian) + uint8_t frame[FRAME_MAX_BYTES]; + uint32_t magic = 0xC5110001; + memcpy(frame + 0, &magic, 4); + frame[4] = g_node_id; + frame[5] = info->rx_ctrl.ant; // antenna index (1 for ESP32 single-antenna) + uint16_t n_sub = info->len / 2; // len = n_subcarriers * 2 (I + Q bytes) + memcpy(frame + 6, &n_sub, 2); + uint32_t freq_mhz = g_channel_freq_mhz; + memcpy(frame + 8, &freq_mhz, 4); + memcpy(frame + 12, &g_seq_num, 4); + frame[16] = (int8_t)info->rx_ctrl.rssi; + frame[17] = (int8_t)info->rx_ctrl.noise_floor; + frame[18] = 0; frame[19] = 0; + + // Write I/Q payload directly from info->buf + memcpy(frame + 20, info->buf, info->len); + + // Send over UDP to aggregator + stream_sender_write(frame, 20 + info->len); + g_seq_num++; +} +``` + +**No on-device FFT** (contradicting ADR-012's optional feature extraction path): The Rust aggregator will do feature extraction using the SOTA `wifi-densepose-signal` pipeline. Raw I/Q is cheaper to stream at ESP32 sampling rates (~100 Hz at 56 subcarriers = ~35 KB/s per node). + +**`sdkconfig.defaults`** must enable: + +``` +CONFIG_ESP_WIFI_CSI_ENABLED=y +CONFIG_LWIP_SO_RCVBUF=y +CONFIG_FREERTOS_HZ=1000 +``` + +**Build toolchain**: ESP-IDF v5.2+ (pinned). Docker image: `espressif/idf:v5.2` for reproducible CI. + +### Layer 2 — UDP Aggregator (`crates/wifi-densepose-hardware/src/aggregator/`) + +New module within the hardware crate. Entry point: `aggregator_main()` callable as a binary target. + +```rust +// crates/wifi-densepose-hardware/src/aggregator/mod.rs + +pub struct Esp32Aggregator { + socket: UdpSocket, + nodes: HashMap, // keyed by node_id from frame header + tx: mpsc::SyncSender, // outbound to bridge +} + +struct NodeState { + last_seq: u32, + drop_count: u64, + last_recv: Instant, +} + +impl Esp32Aggregator { + /// Bind UDP socket and start blocking receive loop. + /// Each valid frame is forwarded on `tx`. + pub fn run(&mut self) -> Result<(), AggregatorError> { + let mut buf = vec![0u8; 4096]; + loop { + let (n, _addr) = self.socket.recv_from(&mut buf)?; + match Esp32CsiParser::parse_frame(&buf[..n]) { + Ok((frame, _consumed)) => { + let state = self.nodes.entry(frame.metadata.node_id) + .or_insert_with(NodeState::default); + // Track drops via sequence number gaps + if frame.metadata.seq_num != state.last_seq + 1 { + state.drop_count += (frame.metadata.seq_num + .wrapping_sub(state.last_seq + 1)) as u64; + } + state.last_seq = frame.metadata.seq_num; + state.last_recv = Instant::now(); + let _ = self.tx.try_send(frame); // drop if pipeline is full + } + Err(e) => { + // Log and continue — never crash on bad UDP packet + eprintln!("aggregator: parse error: {e}"); + } + } + } + } +} +``` + +**Testable without hardware**: The test suite generates frames using `build_test_frame()` (same helper pattern as `esp32_parser.rs` tests) and sends them over a loopback UDP socket. The aggregator receives and forwards them identically to real hardware frames. + +### Layer 3 — CsiFrame → CsiData Bridge + +Bridge from `wifi-densepose-hardware::CsiFrame` to the signal processing type `wifi_densepose_signal::CsiData` (or a compatible intermediate type consumed by the Rust pipeline). + +```rust +// crates/wifi-densepose-hardware/src/bridge.rs + +use crate::{CsiFrame}; + +/// Intermediate type compatible with the signal processing pipeline. +/// Maps directly from CsiFrame without cloning the I/Q storage. +pub struct CsiData { + pub timestamp_unix_ms: u64, + pub node_id: u8, + pub n_antennas: usize, + pub n_subcarriers: usize, + pub amplitude: Vec, // length: n_antennas * n_subcarriers + pub phase: Vec, // length: n_antennas * n_subcarriers + pub rssi_dbm: i8, + pub noise_floor_dbm: i8, + pub channel_freq_mhz: u32, +} + +impl From for CsiData { + fn from(frame: CsiFrame) -> Self { + let n_ant = frame.metadata.n_antennas as usize; + let n_sub = frame.metadata.n_subcarriers as usize; + let (amplitude, phase) = frame.to_amplitude_phase(); + CsiData { + timestamp_unix_ms: frame.metadata.timestamp_unix_ms, + node_id: frame.metadata.node_id, + n_antennas: n_ant, + n_subcarriers: n_sub, + amplitude, + phase, + rssi_dbm: frame.metadata.rssi_dbm, + noise_floor_dbm: frame.metadata.noise_floor_dbm, + channel_freq_mhz: frame.metadata.channel_freq_mhz, + } + } +} +``` + +The bridge test: parse a known binary frame, convert to `CsiData`, assert `amplitude[0]` = √(I₀² + Q₀²) to within f64 precision. + +### Layer 4 — Python `_read_raw_data()` Real Implementation + +Replace the `NotImplementedError` stub in `v1/src/hardware/csi_extractor.py` with a UDP socket reader. This allows the Python pipeline to receive real CSI from the aggregator while the Rust pipeline is being integrated. + +```python +# v1/src/hardware/csi_extractor.py +# Replace _read_raw_data() stub: + +import socket as _socket + +class CSIExtractor: + ... + def _read_raw_data(self) -> bytes: + """Read one raw CSI frame from the UDP aggregator. + + Expects binary frames in the ESP32 format (magic 0xC5110001 header). + Aggregator address configured via AGGREGATOR_HOST / AGGREGATOR_PORT + environment variables (defaults: 127.0.0.1:5005). + """ + if not hasattr(self, '_udp_socket'): + host = self.config.get('aggregator_host', '127.0.0.1') + port = int(self.config.get('aggregator_port', 5005)) + sock = _socket.socket(_socket.AF_INET, _socket.SOCK_DGRAM) + sock.bind((host, port)) + sock.settimeout(1.0) + self._udp_socket = sock + try: + data, _ = self._udp_socket.recvfrom(4096) + return data + except _socket.timeout: + raise CSIExtractionError( + "No CSI data received within timeout — " + "is the ESP32 aggregator running?" + ) +``` + +This is tested with a mock UDP server in the unit tests (existing `test_csi_extractor_tdd.py` pattern) and with the real aggregator in integration. + +## Development Sequence + +``` +Phase 1 (Firmware + Aggregator — no pipeline integration needed): + 1. Write firmware/esp32-csi-node/ C project (ESP-IDF v5.2) + 2. Flash to one ESP32-S3-DevKitC board + 3. Verify binary frames arrive on laptop UDP socket using Wireshark + 4. Write aggregator crate + loopback test + +Phase 2 (Bridge + Python stub): + 5. Implement CsiFrame → CsiData bridge + 6. Replace Python _read_raw_data() with UDP socket + 7. Run Python pipeline end-to-end against loopback aggregator (synthetic frames) + +Phase 3 (Real hardware integration): + 8. Run Python pipeline against live ESP32 frames + 9. Capture 10-second real CSI bundle (firmware/esp32-csi-node/proof/) + 10. Verify proof bundle hash (ADR-011 pattern) + 11. Mark ADR-012 Accepted, mark this ADR Accepted +``` + +## Testing Without Hardware + +All four layers are testable before a single ESP32 is purchased: + +| Layer | Test Method | +|-------|-------------| +| Firmware binary format | Build a `build_test_frame()` helper in Rust, compare its output byte-for-byte against a hand-computed reference frame | +| Aggregator | Loopback UDP: test sends synthetic frames to 127.0.0.1:5005, aggregator receives and forwards on channel | +| Bridge | `assert_eq!(csi_data.amplitude[0], f64::sqrt((iq[0].i as f64).powi(2) + (iq[0].q as f64).powi(2)))` | +| Python UDP reader | Mock UDP server in pytest using `socket.socket` in a background thread | + +The existing `esp32_parser.rs` test suite already validates parsing of correctly-formatted binary frames. The aggregator and bridge tests build on top of the same test frame construction. + +## Consequences + +### Positive +- **Layered testability**: Each layer can be validated independently before hardware acquisition. +- **No new external dependencies**: UDP sockets are in stdlib (both Rust and Python). Firmware uses only ESP-IDF and esp-dsp component. +- **Stub elimination**: Replaces the last two `NotImplementedError` stubs in the Python hardware layer with real code backed by real data. +- **Proof of reality**: Phase 3 produces a captured CSI bundle hashed to a known value, satisfying ADR-011 for hardware-sourced data. +- **Signal-crate reuse**: The SOTA Hampel/Fresnel/BVP/Doppler processing from ADR-014 applies unchanged to real ESP32 frames after the bridge converts them. + +### Negative +- **Firmware requires ESP-IDF toolchain**: Not buildable without a 2+ GB ESP-IDF installation. CI must use the official Docker image or skip firmware compilation. +- **Raw I/Q bandwidth**: Streaming raw I/Q (not features) at 100 Hz × 3 antennas × 56 subcarriers = ~35 KB/s/node. At 6 nodes = ~210 KB/s. Fine for LAN; not suitable for WAN. +- **Single-antenna real-world**: Most ESP32-S3-DevKitC boards have one on-board antenna. Multi-antenna data requires external antenna + board with U.FL connector or purpose-built multi-radio setup. + +### Deferred +- **Multi-node clock drift compensation**: ADR-012 specifies feature-level fusion. The aggregator in this ADR passes raw `CsiFrame` per-node. Drift compensation lives in a future `FeatureFuser` layer (not scoped here). +- **ESP-IDF firmware CI**: Firmware compilation in GitHub Actions requires the ESP-IDF Docker image. CI integration is deferred until Phase 3 hardware validation. + +## Interaction with Other ADRs + +| ADR | Interaction | +|-----|-------------| +| ADR-011 | Phase 3 produces a real CSI proof bundle satisfying mock elimination | +| ADR-012 | This ADR implements the development path for ADR-012's architecture | +| ADR-014 | SOTA signal processing applies unchanged after bridge layer | +| ADR-008 | Aggregator handles multi-node; distributed consensus is a later concern | + +## References + +- [Espressif ESP-CSI Repository](https://github.com/espressif/esp-csi) +- [ESP-IDF WiFi CSI API Reference](https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-guides/wifi.html#wi-fi-channel-state-information) +- `wifi-densepose-hardware/src/esp32_parser.rs` — binary frame parser implementation +- `wifi-densepose-hardware/src/csi_frame.rs` — `CsiFrame`, `to_amplitude_phase()` +- ADR-012: ESP32 CSI Sensor Mesh (architecture) +- ADR-011: Python Proof-of-Reality and Mock Elimination +- ADR-014: SOTA Signal Processing