diff --git a/README.md b/README.md index 2e24487..d2d2d0d 100644 --- a/README.md +++ b/README.md @@ -275,15 +275,15 @@ The signal processing stack transforms raw WiFi Channel State Information into a
-๐Ÿง  Models & Training โ€” DensePose pipeline, RVF containers, SONA adaptation +๐Ÿง  Models & Training โ€” DensePose pipeline, RVF containers, SONA adaptation, RuVector integration -The neural pipeline uses a graph transformer with cross-attention to map CSI feature matrices to 17 COCO body keypoints and DensePose UV coordinates. Models are packaged as single-file `.rvf` containers with progressive loading (Layer A instant, Layer B warm, Layer C full). SONA (Self-Optimizing Neural Architecture) enables continuous on-device adaptation via micro-LoRA + EWC++ without catastrophic forgetting. +The neural pipeline uses a graph transformer with cross-attention to map CSI feature matrices to 17 COCO body keypoints and DensePose UV coordinates. Models are packaged as single-file `.rvf` containers with progressive loading (Layer A instant, Layer B warm, Layer C full). SONA (Self-Optimizing Neural Architecture) enables continuous on-device adaptation via micro-LoRA + EWC++ without catastrophic forgetting. Signal processing is powered by 5 [RuVector](https://github.com/ruvnet/ruvector) crates (v2.0.4) with 7 integration points across the Rust workspace, plus 6 additional vendored crates for inference and graph intelligence. | Section | Description | Docs | |---------|-------------|------| | [RVF Model Container](#rvf-model-container) | Binary packaging with Ed25519 signing, progressive 3-layer loading, SIMD quantization | [ADR-023](docs/adr/ADR-023-trained-densepose-model-ruvector-pipeline.md) | -| [Training & Fine-Tuning](#training--fine-tuning) | MM-Fi/Wi-Pose pre-training, 6-term composite loss, cosine-scheduled SGD, SONA LoRA | [ADR-023](docs/adr/ADR-023-trained-densepose-model-ruvector-pipeline.md) | -| [RuVector Crates](#ruvector-crates) | 11 vendored Rust crates: HNSW, attention, GNN, temporal compression, min-cut, solver | [Source](vendor/ruvector/) | +| [Training & Fine-Tuning](#training--fine-tuning) | 8-phase pure Rust pipeline (7,832 lines), MM-Fi/Wi-Pose pre-training, 6-term composite loss, SONA LoRA | [ADR-023](docs/adr/ADR-023-trained-densepose-model-ruvector-pipeline.md) | +| [RuVector Crates](#ruvector-crates) | 11 vendored Rust crates from [ruvector](https://github.com/ruvnet/ruvector): attention, min-cut, solver, GNN, HNSW, temporal compression, sparse inference | [GitHub](https://github.com/ruvnet/ruvector) ยท [Source](vendor/ruvector/) |
@@ -522,7 +522,7 @@ See [ADR-014](docs/adr/ADR-014-sota-signal-processing.md) for full mathematical
๐Ÿ“ฆ RVF Model Container โ€” Single-file deployment with progressive loading -The RuVector Format (RVF) packages an entire trained model โ€” weights, HNSW indexes, quantization codebooks, SONA adaptation deltas, and WASM inference runtime โ€” into a single self-contained binary file. No external dependencies are needed at deployment time. +The [RuVector Format (RVF)](https://github.com/ruvnet/ruvector/tree/main/crates/rvf) packages an entire trained model โ€” weights, HNSW indexes, quantization codebooks, SONA adaptation deltas, and WASM inference runtime โ€” into a single self-contained binary file. No external dependencies are needed at deployment time. **Container structure:** @@ -548,14 +548,23 @@ The RuVector Format (RVF) packages an entire trained model โ€” weights, HNSW ind โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ ``` +**Deployment targets:** + +| Target | Quantization | Size | Load Time | Use Case | +|--------|-------------|------|-----------|----------| +| **ESP32 / IoT** | int4 | ~0.7 MB | <5ms (Layer A) | Presence + breathing only | +| **Mobile / WebView** | int8 | ~6 MB | ~200ms (Layer B) | Pose estimation on phone | +| **Browser (WASM)** | int8 | ~10 MB | ~500ms (Layer B) | In-browser demo | +| **Field (WiFi-Mat)** | fp16 | ~62 MB | ~2s (Layer C) | Full DensePose + disaster triage | +| **Server / Cloud** | f32 | ~50+ MB | ~3s (Layer C) | Training + full inference | + | Property | Detail | |----------|--------| | **Format** | Segment-based binary, 20+ segment types, CRC32 integrity per segment | | **Progressive Loading** | **Layer A** (<5ms): manifest + entry points โ†’ **Layer B** (100ms-1s): hot weights + adjacency โ†’ **Layer C** (seconds): full graph | | **Signing** | Ed25519 training proofs for verifiable provenance โ€” chain of custody from training data to deployed model | -| **Quantization** | Per-segment temperature-tiered: f32 (full), f16 (half), u8 (int8), binary โ€” with SIMD-accelerated distance computation | +| **Quantization** | Per-segment temperature-tiered: f32 (full), f16 (half), u8 (int8), int4 โ€” with SIMD-accelerated distance computation | | **CLI** | `--export-rvf` (generate), `--load-rvf` (config), `--save-rvf` (persist), `--model` (inference), `--progressive` (3-layer load) | -| **Size** | Typical: 13 KB (sensing-only) to 50+ MB (full DensePose) | ```bash # Export model package @@ -568,37 +577,44 @@ The RuVector Format (RVF) packages an entire trained model โ€” weights, HNSW ind docker run --rm -v $(pwd):/out ruvnet/wifi-densepose:latest --export-rvf /out/model.rvf ``` -See [ADR-023](docs/adr/ADR-023-trained-densepose-model-ruvector-pipeline.md). +Built on the [rvf](https://github.com/ruvnet/ruvector/tree/main/crates/rvf) crate family (rvf-types, rvf-wire, rvf-manifest, rvf-index, rvf-quant, rvf-crypto, rvf-runtime). See [ADR-023](docs/adr/ADR-023-trained-densepose-model-ruvector-pipeline.md).
๐Ÿงฌ Training & Fine-Tuning โ€” MM-Fi/Wi-Pose pre-training, SONA adaptation -The training pipeline implements 8 phases in pure Rust (7,832 lines, zero external ML dependencies). It trains a graph transformer with cross-attention to map CSI feature matrices to 17 COCO body keypoints and DensePose UV coordinates โ€” following the approach from the CMU "DensePose From WiFi" paper (arXiv:2301.00250). +The training pipeline implements 8 phases in pure Rust (7,832 lines, zero external ML dependencies). It trains a graph transformer with cross-attention to map CSI feature matrices to 17 COCO body keypoints and DensePose UV coordinates โ€” following the approach from the CMU "DensePose From WiFi" paper ([arXiv:2301.00250](https://arxiv.org/abs/2301.00250)). RuVector crates provide the core building blocks: [ruvector-attention](https://github.com/ruvnet/ruvector/tree/main/crates/ruvector-attention) for cross-attention layers, [ruvector-mincut](https://github.com/ruvnet/ruvector/tree/main/crates/ruvector-mincut) for multi-person matching, and [ruvector-temporal-tensor](https://github.com/ruvnet/ruvector/tree/main/crates/ruvector-temporal-tensor) for CSI buffer compression. **Three-tier data strategy:** -| Tier | Method | Purpose | -|------|--------|---------| -| **1. Pre-train** | MM-Fi + Wi-Pose public datasets | Cross-environment generalization (multi-subject, multi-room) | -| **2. Fine-tune** | ESP32 CSI + camera pseudo-labels | Environment-specific multipath adaptation | -| **3. SONA adapt** | Micro-LoRA (rank-4) + EWC++ | Continuous on-device learning without catastrophic forgetting | +| Tier | Method | Purpose | RuVector Integration | +|------|--------|---------|---------------------| +| **1. Pre-train** | MM-Fi + Wi-Pose public datasets | Cross-environment generalization (multi-subject, multi-room) | `ruvector-temporal-tensor` compresses CSI windows (114โ†’56 subcarrier resampling) | +| **2. Fine-tune** | ESP32 CSI + camera pseudo-labels | Environment-specific multipath adaptation | `ruvector-solver` for Fresnel geometry, `ruvector-attn-mincut` for subcarrier gating | +| **3. SONA adapt** | Micro-LoRA (rank-4) + EWC++ | Continuous on-device learning without catastrophic forgetting | [SONA](https://github.com/ruvnet/ruvector/tree/main/crates/sona) architecture (Self-Optimizing Neural Architecture) | **Training pipeline components:** -| Phase | Module | What It Does | -|-------|--------|-------------| -| 1 | `dataset.rs` (850 lines) | MM-Fi `.npy` + Wi-Pose `.mat` loaders, subcarrier resampling (114โ†’56, 30โ†’56), windowing, normalization | -| 2 | `graph_transformer.rs` (855 lines) | COCO BodyGraph (17 kp, 16 edges), AntennaGraph, multi-head CrossAttention, GCN message passing | -| 3 | `trainer.rs` (881 lines) | 6-term composite loss (MSE, cross-entropy, UV regression, temporal consistency, bone length, symmetry), SGD+momentum, cosine+warmup scheduler, PCK/OKS metrics | -| 4 | `sona.rs` (639 lines) | LoRA adapters (Aร—B delta), EWC++ Fisher regularization, EnvironmentDetector (3-sigma drift detection) | -| 5 | `sparse_inference.rs` (753 lines) | NeuronProfiler hot/cold partitioning, SparseLinear (skip cold rows), INT8/FP16 quantization with <0.01 MSE | -| 6 | `rvf_pipeline.rs` (1,027 lines) | Progressive 3-layer loader, HNSW index, OverlayGraph, `RvfModelBuilder` | -| 7 | `rvf_container.rs` (914 lines) | Binary container format, 6+ segment types, CRC32 integrity | -| 8 | `main.rs` integration | `--train`, `--model`, `--progressive` CLI flags, REST endpoints | +| Phase | Module | What It Does | RuVector Crate | +|-------|--------|-------------|----------------| +| 1 | `dataset.rs` (850 lines) | MM-Fi `.npy` + Wi-Pose `.mat` loaders, subcarrier resampling (114โ†’56, 30โ†’56), windowing | [ruvector-temporal-tensor](https://github.com/ruvnet/ruvector/tree/main/crates/ruvector-temporal-tensor) | +| 2 | `graph_transformer.rs` (855 lines) | COCO BodyGraph (17 kp, 16 edges), AntennaGraph, multi-head CrossAttention, GCN message passing | [ruvector-attention](https://github.com/ruvnet/ruvector/tree/main/crates/ruvector-attention) | +| 3 | `trainer.rs` (881 lines) | 6-term composite loss (MSE, CE, UV, temporal, bone, symmetry), SGD+momentum, cosine+warmup, PCK/OKS | [ruvector-mincut](https://github.com/ruvnet/ruvector/tree/main/crates/ruvector-mincut) (person matching) | +| 4 | `sona.rs` (639 lines) | LoRA adapters (Aร—B delta), EWC++ Fisher regularization, EnvironmentDetector (3-sigma drift) | [sona](https://github.com/ruvnet/ruvector/tree/main/crates/sona) | +| 5 | `sparse_inference.rs` (753 lines) | NeuronProfiler hot/cold partitioning, SparseLinear (skip cold rows), INT8/FP16 quantization | [ruvector-sparse-inference](https://github.com/ruvnet/ruvector/tree/main/crates/ruvector-sparse-inference) | +| 6 | `rvf_pipeline.rs` (1,027 lines) | Progressive 3-layer loader, HNSW index, OverlayGraph, `RvfModelBuilder` | [ruvector-core](https://github.com/ruvnet/ruvector/tree/main/crates/ruvector-core) (HNSW) | +| 7 | `rvf_container.rs` (914 lines) | Binary container format, 6+ segment types, CRC32 integrity | [rvf](https://github.com/ruvnet/ruvector/tree/main/crates/rvf) | +| 8 | `main.rs` integration | `--train`, `--model`, `--progressive` CLI flags, REST endpoints | โ€” | -**Best-epoch snapshotting**: the trainer saves the best validation loss weights and restores them before checkpoint/export โ€” prevents exporting overfit final-epoch parameters. +**SONA (Self-Optimizing Neural Architecture)** โ€” the continuous adaptation system: + +| Component | What It Does | Why It Matters | +|-----------|-------------|----------------| +| **Micro-LoRA (rank-4)** | Trains small Aร—B weight deltas instead of full weights | 100x fewer parameters to update โ†’ runs on ESP32 | +| **EWC++ (Fisher matrix)** | Penalizes changes to important weights from previous environments | Prevents catastrophic forgetting when moving between rooms | +| **EnvironmentDetector** | Monitors CSI feature drift with 3-sigma threshold | Auto-triggers adaptation when the model is moved to a new space | +| **Best-epoch snapshot** | Saves best validation loss weights, restores before export | Prevents shipping overfit final-epoch parameters | ```bash # Pre-train on MM-Fi dataset @@ -612,28 +628,35 @@ docker run --rm -v $(pwd)/data:/data ruvnet/wifi-densepose:latest \ --train --dataset /data --epochs 100 --export-rvf /data/model.rvf ``` -See [ADR-023](docs/adr/ADR-023-trained-densepose-model-ruvector-pipeline.md). +See [ADR-023](docs/adr/ADR-023-trained-densepose-model-ruvector-pipeline.md) ยท [SONA crate](https://github.com/ruvnet/ruvector/tree/main/crates/sona) ยท [arXiv:2301.00250](https://arxiv.org/abs/2301.00250)
-๐Ÿ”ฉ RuVector Crates โ€” 11 vendored signal intelligence crates +๐Ÿ”ฉ RuVector Crates โ€” 11 vendored signal intelligence crates from github.com/ruvnet/ruvector -| Crate | Purpose | -|-------|---------| -| `ruvector-core` | VectorDB, HNSW index, SIMD distance, quantization | -| `ruvector-attention` | Scaled dot-product, MoE, sparse attention | -| `ruvector-gnn` | Graph neural network, graph attention, EWC training | -| `ruvector-nervous-system` | PredictiveLayer, OscillatoryRouter, Hopfield | -| `ruvector-coherence` | Spectral coherence, HNSW health, Fiedler value | -| `ruvector-temporal-tensor` | Tiered temporal compression (8/7/5/3-bit) | -| `ruvector-mincut` | Subpolynomial dynamic min-cut | -| `ruvector-attn-mincut` | Attention-gated min-cut | -| `ruvector-solver` | Sparse Neumann solver O(sqrt(n)) | -| `ruvector-graph-transformer` | Proof-gated graph transformer | -| `ruvector-sparse-inference` | PowerInfer-style sparse execution | +**5 directly-used crates** (v2.0.4, declared in `Cargo.toml`, 7 integration points): -See `vendor/ruvector/` for full source. +| Crate | What It Does | Where It's Used in WiFi-DensePose | Source | +|-------|-------------|-----------------------------------|--------| +| [`ruvector-attention`](https://github.com/ruvnet/ruvector/tree/main/crates/ruvector-attention) | Scaled dot-product attention, MoE routing, sparse attention | `model.rs` (spatial attention), `bvp.rs` (sensitivity-weighted velocity profiles) | [crate](https://crates.io/crates/ruvector-attention) | +| [`ruvector-mincut`](https://github.com/ruvnet/ruvector/tree/main/crates/ruvector-mincut) | Subpolynomial dynamic min-cut O(n^1.5 log n) | `metrics.rs` (DynamicPersonMatcher โ€” multi-person assignment), `subcarrier_selection.rs` (sensitive/insensitive split) | [crate](https://crates.io/crates/ruvector-mincut) | +| [`ruvector-attn-mincut`](https://github.com/ruvnet/ruvector/tree/main/crates/ruvector-attn-mincut) | Attention-gated spectrogram noise suppression | `model.rs` (antenna attention gating), `spectrogram.rs` (gate noisy time-frequency bins) | [crate](https://crates.io/crates/ruvector-attn-mincut) | +| [`ruvector-solver`](https://github.com/ruvnet/ruvector/tree/main/crates/ruvector-solver) | Sparse Neumann series solver O(sqrt(n)) | `fresnel.rs` (TX-body-RX geometry), `triangulation.rs` (3D localization), `subcarrier.rs` (sparse interpolation 114โ†’56) | [crate](https://crates.io/crates/ruvector-solver) | +| [`ruvector-temporal-tensor`](https://github.com/ruvnet/ruvector/tree/main/crates/ruvector-temporal-tensor) | Tiered temporal compression (8/7/5/3-bit) | `dataset.rs` (CSI buffer compression), `breathing.rs` + `heartbeat.rs` (compressed vital sign spectrograms) | [crate](https://crates.io/crates/ruvector-temporal-tensor) | + +**6 additional vendored crates** (used by training pipeline and inference): + +| Crate | What It Does | Source | +|-------|-------------|--------| +| [`ruvector-core`](https://github.com/ruvnet/ruvector/tree/main/crates/ruvector-core) | VectorDB engine, HNSW index, SIMD distance functions, quantization codebooks | [crate](https://crates.io/crates/ruvector-core) | +| [`ruvector-gnn`](https://github.com/ruvnet/ruvector/tree/main/crates/ruvector-gnn) | Graph neural network layers, graph attention, EWC-regularized training | [crate](https://crates.io/crates/ruvector-gnn) | +| [`ruvector-graph-transformer`](https://github.com/ruvnet/ruvector/tree/main/crates/ruvector-graph-transformer) | Proof-gated graph transformer with cross-attention | [crate](https://crates.io/crates/ruvector-graph-transformer) | +| [`ruvector-sparse-inference`](https://github.com/ruvnet/ruvector/tree/main/crates/ruvector-sparse-inference) | PowerInfer-style hot/cold neuron partitioning, skip cold rows at runtime | [crate](https://crates.io/crates/ruvector-sparse-inference) | +| [`ruvector-nervous-system`](https://github.com/ruvnet/ruvector/tree/main/crates/ruvector-nervous-system) | PredictiveLayer, OscillatoryRouter, Hopfield associative memory | [crate](https://crates.io/crates/ruvector-nervous-system) | +| [`ruvector-coherence`](https://github.com/ruvnet/ruvector/tree/main/crates/ruvector-coherence) | Spectral coherence monitoring, HNSW graph health, Fiedler connectivity | [crate](https://crates.io/crates/ruvector-coherence) | + +The full RuVector ecosystem includes 90+ crates. See [github.com/ruvnet/ruvector](https://github.com/ruvnet/ruvector) for the complete library, and [`vendor/ruvector/`](vendor/ruvector/) for the vendored source in this project.