Squashed 'vendor/ruvector/' content from commit b64c2172

git-subtree-dir: vendor/ruvector
git-subtree-split: b64c21726f2bb37286d9ee36a7869fef60cc6900
This commit is contained in:
ruv
2026-02-28 14:39:40 -05:00
commit d803bfe2b1
7854 changed files with 3522914 additions and 0 deletions

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,513 @@
# ADR-002 Addendum: BMSSP WASM Integration
**Status**: Proposed
**Date**: 2026-01-25
**Extends**: ADR-002, ADR-002-addendum-sota-optimizations
---
## Executive Summary
Integrate `@ruvnet/bmssp` (Bounded Multi-Source Shortest Path) WASM module to accelerate j-tree operations:
- **O(m·log^(2/3) n)** complexity (beats O(n log n) all-pairs)
- **Multi-source queries** for terminal-based j-tree operations
- **Neural embeddings** via WasmNeuralBMSSP for learned sparsification
- **27KB WASM** enables browser/edge deployment
- **10-15x speedup** over JavaScript fallbacks
---
## The Path-Cut Duality
### Key Insight
In many graph classes, shortest paths and minimum cuts are dual:
```
Shortest Path in G* (dual) ←→ Minimum Cut in G
Where:
- G* has vertices = faces of G
- Edge weight in G* = cut capacity crossing that edge
```
For j-tree hierarchies specifically:
```
j-Tree Level Query:
┌─────────────────────────────────────────────────────────┐
│ Find min-cut between vertex sets S and T │
│ │
│ ≡ Find shortest S-T path in contracted auxiliary graph │
│ │
│ BMSSP complexity: O(m·log^(2/3) n) │
│ vs. direct cut: O(n log n) │
│ │
│ Speedup: ~log^(1/3) n factor │
└─────────────────────────────────────────────────────────┘
```
---
## Architecture Integration
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ J-TREE + BMSSP INTEGRATED ARCHITECTURE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌────────────────────────────────────────────────────────────────────────┐ │
│ │ LAYER 0: WASM ACCELERATION │ │
│ │ │ │
│ │ ┌─────────────────┐ ┌─────────────────┐ │ │
│ │ │ WasmGraph │ │ WasmNeuralBMSSP │ │ │
│ │ │ (27KB WASM) │ │ (embeddings) │ │ │
│ │ ├─────────────────┤ ├─────────────────┤ │ │
│ │ │ • add_edge │ │ • set_embedding │ │ │
│ │ │ • shortest_paths│ │ • semantic_dist │ │ │
│ │ │ • vertex_count │ │ • neural_paths │ │ │
│ │ │ • edge_count │ │ • update_embed │ │ │
│ │ └─────────────────┘ └─────────────────┘ │ │
│ │ │ │ │ │
│ │ └────────────┬───────────────────┘ │ │
│ │ ▼ │ │
│ └────────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────────────────┐ │
│ │ LAYER 1: HYBRID CUT COMPUTATION │ │
│ │ │ │
│ │ Query Type │ Method │ Complexity │ │
│ │ ────────────────────┼───────────────────────┼─────────────────────── │ │
│ │ Point-to-point cut │ BMSSP path → cut │ O(m·log^(2/3) n) │ │
│ │ Multi-terminal cut │ BMSSP multi-source │ O(k·m·log^(2/3) n) │ │
│ │ All-pairs cuts │ BMSSP batch + cache │ O(n·m·log^(2/3) n) │ │
│ │ Sparsest cut │ Neural semantic dist │ O(n²) → O(n·d) │ │
│ │ │ │
│ └────────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────────────────┐ │
│ │ LAYER 2: J-TREE HIERARCHY │ │
│ │ │ │
│ │ Each j-tree level maintains: │ │
│ │ • WasmGraph for contracted graph at that level │ │
│ │ • WasmNeuralBMSSP for learned edge importance │ │
│ │ • Cached shortest-path distances (cut values) │ │
│ │ │ │
│ │ Level L: WasmGraph(O(1) vertices) │ │
│ │ Level L-1: WasmGraph(O(α) vertices) │ │
│ │ ... │ │
│ │ Level 0: WasmGraph(n vertices) │ │
│ │ │ │
│ └────────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
```
---
## API Integration
### 1. BMSSP-Accelerated Cut Queries
```rust
/// J-tree level backed by BMSSP WASM
pub struct BmsspJTreeLevel {
/// WASM graph for this level
wasm_graph: WasmGraph,
/// Neural BMSSP for learned operations
neural_bmssp: Option<WasmNeuralBMSSP>,
/// Cached path distances (= cut values in dual)
path_cache: HashMap<(VertexId, VertexId), f64>,
/// Level index
level: usize,
}
impl BmsspJTreeLevel {
/// Create from contracted graph
pub fn from_contracted(contracted: &ContractedGraph, level: usize) -> Self {
let n = contracted.vertex_count();
let mut wasm_graph = WasmGraph::new(n as u32, false); // undirected
// Add edges with weights = capacities
for edge in contracted.edges() {
wasm_graph.add_edge(
edge.source as u32,
edge.target as u32,
edge.capacity,
);
}
Self {
wasm_graph,
neural_bmssp: None,
path_cache: HashMap::new(),
level,
}
}
/// Min-cut between s and t via path-cut duality
/// Complexity: O(m·log^(2/3) n) vs O(n log n) direct
pub fn min_cut(&mut self, s: VertexId, t: VertexId) -> f64 {
// Check cache first
if let Some(&cached) = self.path_cache.get(&(s, t)) {
return cached;
}
// Compute shortest paths from s
let distances = self.wasm_graph.compute_shortest_paths(s as u32);
// Distance to t = min-cut value (in dual representation)
let cut_value = distances[t as usize];
// Cache for future queries
self.path_cache.insert((s, t), cut_value);
self.path_cache.insert((t, s), cut_value); // symmetric
cut_value
}
/// Multi-terminal cut using BMSSP multi-source
pub fn multi_terminal_cut(&mut self, terminals: &[VertexId]) -> f64 {
// BMSSP handles multi-source natively
let sources: Vec<u32> = terminals.iter().map(|&v| v as u32).collect();
// Compute shortest paths from all terminals simultaneously
// This amortizes the cost across terminals
let mut min_cut = f64::INFINITY;
for (i, &s) in terminals.iter().enumerate() {
let distances = self.wasm_graph.compute_shortest_paths(s as u32);
for (j, &t) in terminals.iter().enumerate() {
if i < j {
let cut = distances[t as usize];
min_cut = min_cut.min(cut);
}
}
}
min_cut
}
}
```
### 2. Neural Sparsification via WasmNeuralBMSSP
```rust
/// Neural sparsifier using BMSSP embeddings
pub struct BmsspNeuralSparsifier {
/// Neural BMSSP instance
neural: WasmNeuralBMSSP,
/// Embedding dimension
embedding_dim: usize,
/// Learning rate for gradient updates
learning_rate: f64,
/// Alpha for semantic edge weighting
semantic_alpha: f64,
}
impl BmsspNeuralSparsifier {
/// Initialize with node embeddings
pub fn new(graph: &DynamicGraph, embedding_dim: usize) -> Self {
let n = graph.vertex_count();
let mut neural = WasmNeuralBMSSP::new(n as u32, embedding_dim as u32);
// Initialize embeddings (could use pre-trained or random)
for v in 0..n {
let embedding = Self::initial_embedding(v, embedding_dim);
neural.set_embedding(v as u32, &embedding);
}
// Add semantic edges based on graph structure
for edge in graph.edges() {
neural.add_semantic_edge(
edge.source as u32,
edge.target as u32,
0.5, // alpha parameter
);
}
Self {
neural,
embedding_dim,
learning_rate: 0.01,
semantic_alpha: 0.5,
}
}
/// Compute edge importance via semantic distance
pub fn edge_importance(&self, u: VertexId, v: VertexId) -> f64 {
// Semantic distance inversely correlates with importance
let distance = self.neural.semantic_distance(u as u32, v as u32);
// Convert to importance: closer = more important
1.0 / (1.0 + distance)
}
/// Sparsify graph keeping top-k important edges
pub fn sparsify(&self, graph: &DynamicGraph, k: usize) -> SparseGraph {
let mut edge_scores: Vec<_> = graph.edges()
.map(|e| (e, self.edge_importance(e.source, e.target)))
.collect();
// Sort by importance descending
edge_scores.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap());
// Keep top k edges
let kept_edges: Vec<_> = edge_scores.into_iter()
.take(k)
.map(|(e, _)| e)
.collect();
SparseGraph::from_edges(kept_edges)
}
/// Update embeddings based on cut preservation loss
pub fn train_step(&mut self, original_cuts: &[(VertexId, VertexId, f64)]) {
// Compute gradients based on cut preservation
let gradients = self.compute_cut_gradients(original_cuts);
// Update via WASM
self.neural.update_embeddings(
&gradients,
self.learning_rate,
self.embedding_dim as u32,
);
}
/// Compute gradients to preserve cut values
fn compute_cut_gradients(&self, cuts: &[(VertexId, VertexId, f64)]) -> Vec<f64> {
let mut gradients = vec![0.0; self.neural.vertex_count() * self.embedding_dim];
for &(s, t, true_cut) in cuts {
let predicted_cut = self.neural.semantic_distance(s as u32, t as u32);
let error = predicted_cut - true_cut;
// Gradient for embedding update
// (simplified - actual implementation would use autograd)
let s_offset = s as usize * self.embedding_dim;
let t_offset = t as usize * self.embedding_dim;
for d in 0..self.embedding_dim {
gradients[s_offset + d] += error * 0.5;
gradients[t_offset + d] += error * 0.5;
}
}
gradients
}
}
```
### 3. Full Integration with Predictive j-Tree
```rust
/// Predictive j-tree with BMSSP acceleration
pub struct BmsspPredictiveJTree {
/// J-tree levels backed by BMSSP
levels: Vec<BmsspJTreeLevel>,
/// Neural sparsifier
sparsifier: BmsspNeuralSparsifier,
/// SNN prediction engine (from SOTA addendum)
snn_predictor: PolicySNN,
/// Exact verifier (Tier 2)
exact: SubpolynomialMinCut,
}
impl BmsspPredictiveJTree {
/// Build hierarchy with BMSSP at each level
pub fn build(graph: &DynamicGraph, epsilon: f64) -> Self {
let alpha = compute_alpha(epsilon);
let num_levels = (graph.vertex_count() as f64).log(alpha).ceil() as usize;
// Build neural sparsifier first
let sparsifier = BmsspNeuralSparsifier::new(graph, 64);
let sparse = sparsifier.sparsify(graph, graph.vertex_count() * 10);
// Build BMSSP-backed levels
let mut levels = Vec::with_capacity(num_levels);
let mut current = sparse.clone();
for level in 0..num_levels {
let bmssp_level = BmsspJTreeLevel::from_contracted(&current, level);
levels.push(bmssp_level);
current = contract_graph(&current, alpha);
}
Self {
levels,
sparsifier,
snn_predictor: PolicySNN::new(),
exact: SubpolynomialMinCut::new(graph),
}
}
/// Query with BMSSP acceleration
pub fn min_cut(&mut self, s: VertexId, t: VertexId) -> CutResult {
// Use SNN to predict optimal level to query
let optimal_level = self.snn_predictor.predict_level(s, t);
// Query BMSSP at predicted level
let approx_cut = self.levels[optimal_level].min_cut(s, t);
// Decide if exact verification needed
if approx_cut < CRITICAL_THRESHOLD {
let exact_cut = self.exact.min_cut_between(s, t);
CutResult::exact(exact_cut)
} else {
CutResult::approximate(approx_cut, self.approximation_factor(optimal_level))
}
}
/// Batch queries with BMSSP multi-source
pub fn all_pairs_cuts(&mut self, vertices: &[VertexId]) -> AllPairsResult {
// BMSSP handles this efficiently via multi-source
let mut results = HashMap::new();
for level in &mut self.levels {
let level_cuts = level.multi_terminal_cut(vertices);
// Aggregate results across levels
}
AllPairsResult { cuts: results }
}
}
```
---
## Performance Analysis
### Complexity Comparison
| Operation | Without BMSSP | With BMSSP | Improvement |
|-----------|---------------|------------|-------------|
| Point-to-point cut | O(n log n) | O(m·log^(2/3) n) | ~log^(1/3) n |
| Multi-terminal (k) | O(k·n log n) | O(k·m·log^(2/3) n) | ~log^(1/3) n |
| All-pairs (n²) | O(n² log n) | O(n·m·log^(2/3) n) | ~n/m · log^(1/3) n |
| Neural sparsify | O(n² embeddings) | O(n·d) WASM | ~n/d |
### Benchmarks (from BMSSP)
| Graph Size | JS (ms) | BMSSP WASM (ms) | Speedup |
|------------|---------|-----------------|---------|
| 1K nodes | 12.5 | 1.0 | **12.5x** |
| 10K nodes | 145.3 | 12.0 | **12.1x** |
| 100K nodes | 1,523.7 | 45.0 | **33.9x** |
| 1M nodes | 15,234.2 | 180.0 | **84.6x** |
### Expected j-Tree Speedup
```
J-tree query (10K graph):
├── Without BMSSP: ~50ms (Rust native)
├── With BMSSP: ~12ms (WASM accelerated)
└── Improvement: ~4x for path-based queries
J-tree + Neural Sparsify (10K graph):
├── Without BMSSP: ~200ms (native + neural)
├── With BMSSP: ~25ms (WASM + embeddings)
└── Improvement: ~8x for full pipeline
```
---
## Deployment Scenarios
### 1. Browser/Edge (Primary Use Case)
```typescript
// Browser deployment with BMSSP
import init, { WasmGraph, WasmNeuralBMSSP } from '@ruvnet/bmssp';
async function initJTreeBrowser() {
await init(); // Load 27KB WASM
const graph = new WasmGraph(1000, false);
// Build j-tree hierarchy in browser
// 10-15x faster than pure JS implementation
}
```
### 2. Node.js with Native Fallback
```typescript
// Hybrid: BMSSP for queries, native Rust for exact
import { WasmGraph } from '@ruvnet/bmssp';
import { SubpolynomialMinCut } from 'ruvector-mincut-napi';
const bmsspLevel = new WasmGraph(n, false);
const exactVerifier = new SubpolynomialMinCut(graph);
// Use BMSSP for fast approximate
const approx = bmsspLevel.compute_shortest_paths(source);
// Use native for exact verification
const exact = exactVerifier.min_cut();
```
### 3. 256-Core Agentic Chip
```rust
// Each core gets its own BMSSP instance for a j-tree level
// 27KB WASM fits within 8KB constraint when compiled to native
impl CoreExecutor {
pub fn init_bmssp_level(&mut self, level: &ContractedGraph) {
// WASM compiles to native instructions
// Memory footprint: ~6KB for 256-vertex level
self.bmssp = WasmGraph::new(level.vertex_count(), false);
}
}
```
---
## Implementation Priority
| Phase | Task | Effort | Impact |
|-------|------|--------|--------|
| **P0** | Add `@ruvnet/bmssp` to package.json | 1 hour | Enable integration |
| **P0** | `BmsspJTreeLevel` wrapper | 1 week | Core functionality |
| **P1** | Neural sparsifier integration | 2 weeks | Learned edge selection |
| **P1** | Multi-source batch queries | 1 week | All-pairs acceleration |
| **P2** | SNN predictor + BMSSP fusion | 2 weeks | Optimal level selection |
| **P2** | Browser deployment bundle | 1 week | Edge deployment |
---
## References
1. **BMSSP**: "Breaking the Sorting Barrier for SSSP" (arXiv:2501.00660)
2. **Package**: https://www.npmjs.com/package/@ruvnet/bmssp
3. **Integration**: ADR-002, ADR-002-addendum-sota-optimizations
---
## Appendix: BMSSP API Quick Reference
```typescript
// Core Graph
class WasmGraph {
constructor(vertices: number, directed: boolean);
add_edge(from: number, to: number, weight: number): boolean;
compute_shortest_paths(source: number): Float64Array;
readonly vertex_count: number;
readonly edge_count: number;
free(): void;
}
// Neural Extension
class WasmNeuralBMSSP {
constructor(vertices: number, embedding_dim: number);
set_embedding(node: number, embedding: Float64Array): boolean;
add_semantic_edge(from: number, to: number, alpha: number): void;
compute_neural_paths(source: number): Float64Array;
semantic_distance(node1: number, node2: number): number;
update_embeddings(gradients: Float64Array, lr: number, dim: number): boolean;
free(): void;
}
```

View File

@@ -0,0 +1,650 @@
# ADR-002 Addendum: SOTA Optimizations for Dynamic Hierarchical j-Tree
**Status**: Proposed
**Date**: 2026-01-25
**Extends**: ADR-002 (Dynamic Hierarchical j-Tree Decomposition)
---
## Executive Summary
This addendum pushes ADR-002 to true state-of-the-art by integrating:
1. **Predictive Dynamics** - SNN predicts updates before they happen
2. **Neural Sparsification** - Learned edge selection via SpecNet
3. **Lazy Hierarchical Evaluation** - Demand-paged j-tree levels
4. **Warm-Start Cut-Matching** - Reuse computation across updates
5. **256-Core Parallel Hierarchy** - Each core owns j-tree levels
6. **Streaming Sketch Fallback** - O(n log n) space for massive graphs
**Target**: Sub-microsecond approximate queries, <100μs exact verification
---
## Architecture: Predictive Dynamic j-Tree
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ PREDICTIVE DYNAMIC J-TREE ARCHITECTURE │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────────────────┐│
│ │ LAYER 0: PREDICTION ENGINE ││
│ │ ││
│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ││
│ │ │ SNN Policy │───►│ TD Learner │───►│ Prefetcher │ ││
│ │ │ (R-STDP) │ │ (Value Net) │ │ (Speculate) │ ││
│ │ └──────────────┘ └──────────────┘ └──────────────┘ ││
│ │ │ │ │ ││
│ │ ▼ ▼ ▼ ││
│ │ Predict which Estimate cut Pre-compute ││
│ │ levels change value change likely queries ││
│ │ ││
│ └─────────────────────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────────────┐│
│ │ LAYER 1: NEURAL SPARSIFIER ││
│ │ ││
│ │ ┌────────────────────────────────────────────────────────────────────┐ ││
│ │ │ SpecNet Integration (arXiv:2510.27474) │ ││
│ │ │ │ ││
│ │ │ Loss = λ₁·Laplacian_Alignment + λ₂·Feature_Preserve + λ₃·Sparsity │ ││
│ │ │ │ ││
│ │ │ • Joint Graph Evolution layer │ ││
│ │ │ • Spectral Concordance preservation │ ││
│ │ │ • Degree-based fast presparse (DSpar: 5.9x speedup) │ ││
│ │ └────────────────────────────────────────────────────────────────────┘ ││
│ │ ││
│ └─────────────────────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────────────┐│
│ │ LAYER 2: LAZY HIERARCHICAL J-TREE ││
│ │ ││
│ │ Level L ──┐ ││
│ │ Level L-1 ├── Demand-paged: Only materialize when queried ││
│ │ Level L-2 ├── Dirty marking: Track which levels need recomputation ││
│ │ ... │ Warm-start: Reuse cut-matching state across updates ││
│ │ Level 0 ──┘ ││
│ │ ││
│ │ Memory: O(active_levels × n_level) instead of O(L × n) ││
│ │ ││
│ └─────────────────────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────────────┐│
│ │ LAYER 3: 256-CORE PARALLEL DISTRIBUTION ││
│ │ ││
│ │ ┌─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐ ││
│ │ │Core 0-31│Core32-63│Core64-95│Core96-127│Core128+ │Core 255│ ││
│ │ │ Level 0 │ Level 1 │ Level 2 │ Level 3 │ ... │ Level L│ ││
│ │ └─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘ ││
│ │ ││
│ │ Work Stealing: Imbalanced levels redistribute to idle cores ││
│ │ Atomic CAS: SharedCoordinator for global min-cut updates ││
│ │ 8KB/core: CompactCoreState fits entire j-tree level ││
│ │ ││
│ └─────────────────────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────────────┐│
│ │ LAYER 4: STREAMING SKETCH FALLBACK ││
│ │ ││
│ │ When n > 100K vertices: ││
│ │ ┌────────────────────────────────────────────────────────────────────┐ ││
│ │ │ Semi-Streaming Cut Sketch │ ││
│ │ │ • O(n log n) space (two edges per vertex) │ ││
│ │ │ • Reservoir sampling for edge selection │ ││
│ │ │ • (1+ε) approximation maintained incrementally │ ││
│ │ └────────────────────────────────────────────────────────────────────┘ ││
│ │ ││
│ └─────────────────────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────────────┐│
│ │ LAYER 5: EXACT VERIFICATION ││
│ │ ││
│ │ El-Hayek/Henzinger/Li (arXiv:2512.13105) ││
│ │ • Triggered only when approximate cut < threshold ││
│ │ • O(n^{o(1)}) exact verification ││
│ │ • Deterministic, no randomization ││
│ │ ││
│ └─────────────────────────────────────────────────────────────────────────────┘│
│ │
└─────────────────────────────────────────────────────────────────────────────────┘
```
---
## Component 1: SNN Prediction Engine
Exploits the triple isomorphism already in the codebase:
| Graph Theory | Dynamical Systems | Neuromorphic |
|--------------|-------------------|--------------|
| MinCut value | Lyapunov exponent | Spike synchrony |
| Edge contraction | Phase space flow | Synaptic plasticity |
| Hierarchy level | Attractor basin | Memory consolidation |
```rust
/// Predictive j-tree using SNN dynamics
pub struct PredictiveJTree {
/// Core j-tree hierarchy
hierarchy: JTreeHierarchy,
/// SNN policy network for update prediction
policy: PolicySNN,
/// Value network for cut estimation
value_net: ValueNetwork,
/// Prefetch cache for speculative computation
prefetch: PrefetchCache,
/// SONA hooks for continuous adaptation
sona_hooks: [usize; 4], // Layers 8, 16, 24, 28
}
impl PredictiveJTree {
/// Predict which levels will need updates after edge change
pub fn predict_affected_levels(&self, edge: (VertexId, VertexId)) -> Vec<usize> {
// SNN encodes edge as spike pattern
let spike_input = self.edge_to_spikes(edge);
// Policy network predicts affected regions
let activity = self.policy.forward(&spike_input);
// Low activity regions are stable, high activity needs update
activity.iter()
.enumerate()
.filter(|(_, &a)| a > ACTIVITY_THRESHOLD)
.map(|(level, _)| level)
.collect()
}
/// Speculative update: pre-compute before edge actually changes
pub fn speculative_update(&mut self, likely_edge: (VertexId, VertexId), prob: f64) {
if prob > SPECULATION_THRESHOLD {
let affected = self.predict_affected_levels(likely_edge);
// Pre-compute in background cores
for level in affected {
self.prefetch.schedule(level, likely_edge);
}
}
}
/// TD-learning update after observing actual cut change
pub fn learn_from_observation(&mut self, predicted_cut: f64, actual_cut: f64) {
let td_error = actual_cut - predicted_cut;
// R-STDP: Reward-modulated spike-timing-dependent plasticity
self.policy.apply_rstdp(td_error);
// Update value network
self.value_net.td_update(td_error);
}
}
```
**Performance Target**: Predict 80%+ of affected levels correctly → skip 80% of unnecessary recomputation
---
## Component 2: Neural Sparsifier (SpecNet Integration)
Based on arXiv:2510.27474, learn which edges to keep:
```rust
/// Neural graph sparsifier with spectral concordance
pub struct NeuralSparsifier {
/// Graph evolution layer (learned edge selection)
evolution_layer: GraphEvolutionLayer,
/// Spectral concordance loss weights
lambda_laplacian: f64, // λ₁ = 1.0
lambda_feature: f64, // λ₂ = 0.5
lambda_sparsity: f64, // λ₃ = 0.1
/// Degree-based presparse threshold (DSpar optimization)
degree_threshold: f64,
}
impl NeuralSparsifier {
/// Fast presparse using degree heuristic (DSpar: 5.9x speedup)
pub fn degree_presparse(&self, graph: &DynamicGraph) -> DynamicGraph {
let mut sparse = graph.clone();
// Effective resistance ≈ 1/(deg_u × deg_v)
// Keep edges with high effective resistance
for edge in graph.edges() {
let deg_u = graph.degree(edge.source) as f64;
let deg_v = graph.degree(edge.target) as f64;
let eff_resistance = 1.0 / (deg_u * deg_v);
// Sample with probability proportional to effective resistance
if eff_resistance < self.degree_threshold {
sparse.remove_edge(edge.source, edge.target);
}
}
sparse
}
/// Spectral concordance loss for training
pub fn spectral_concordance_loss(
&self,
original: &DynamicGraph,
sparsified: &DynamicGraph,
) -> f64 {
// L₁: Laplacian eigenvalue alignment
let laplacian_loss = self.laplacian_alignment(original, sparsified);
// L₂: Feature geometry preservation (cut values)
let feature_loss = self.cut_preservation_loss(original, sparsified);
// L₃: Sparsity inducing trace penalty
let sparsity_loss = sparsified.edge_count() as f64 / original.edge_count() as f64;
self.lambda_laplacian * laplacian_loss
+ self.lambda_feature * feature_loss
+ self.lambda_sparsity * sparsity_loss
}
/// End-to-end learnable sparsification
pub fn learn_sparsify(&mut self, graph: &DynamicGraph) -> SparseGraph {
// 1. Fast presparse (DSpar)
let presparse = self.degree_presparse(graph);
// 2. Neural refinement (SpecNet)
let edge_scores = self.evolution_layer.forward(&presparse);
// 3. Top-k selection preserving spectral properties
let k = (graph.vertex_count() as f64 * (graph.vertex_count() as f64).ln()) as usize;
let selected = edge_scores.top_k(k);
SparseGraph::from_edges(selected)
}
}
```
**Performance Target**: 90% edge reduction while maintaining 95%+ cut accuracy
---
## Component 3: Lazy Hierarchical Evaluation
Don't compute levels until needed:
```rust
/// Lazy j-tree with demand-paged levels
pub struct LazyJTreeHierarchy {
/// Level states
levels: Vec<LazyLevel>,
/// Which levels are materialized
materialized: BitSet,
/// Dirty flags for incremental update
dirty: BitSet,
/// Cut-matching state for warm-start
warm_state: Vec<CutMatchingState>,
}
#[derive(Clone)]
enum LazyLevel {
/// Not yet computed
Unmaterialized,
/// Computed and valid
Materialized(JTree),
/// Needs recomputation
Dirty(JTree),
}
impl LazyJTreeHierarchy {
/// Query with lazy materialization
pub fn approximate_min_cut(&mut self) -> ApproximateCut {
// Only materialize levels needed for query
let mut current_level = self.levels.len() - 1;
while current_level > 0 {
self.ensure_materialized(current_level);
let cut = self.levels[current_level].as_materialized().min_cut();
// Early termination if cut is good enough
if cut.approximation_factor < ACCEPTABLE_APPROX {
return cut;
}
current_level -= 1;
}
self.levels[0].as_materialized().min_cut()
}
/// Ensure level is materialized (demand-paging)
fn ensure_materialized(&mut self, level: usize) {
match &self.levels[level] {
LazyLevel::Unmaterialized => {
// First-time computation
let jtree = self.compute_level(level);
self.levels[level] = LazyLevel::Materialized(jtree);
self.materialized.insert(level);
}
LazyLevel::Dirty(old_jtree) => {
// Warm-start from previous state (arXiv:2511.02943)
let jtree = self.warm_start_recompute(level, old_jtree);
self.levels[level] = LazyLevel::Materialized(jtree);
self.dirty.remove(level);
}
LazyLevel::Materialized(_) => {
// Already valid, no-op
}
}
}
/// Warm-start recomputation avoiding full recursion cost
fn warm_start_recompute(&self, level: usize, old: &JTree) -> JTree {
// Reuse cut-matching game state from warm_state
let state = &self.warm_state[level];
// Only recompute affected regions
let mut new_jtree = old.clone();
for node in state.affected_nodes() {
new_jtree.recompute_node(node, state);
}
new_jtree
}
/// Mark levels dirty after edge update
pub fn mark_dirty(&mut self, affected_levels: &[usize]) {
for &level in affected_levels {
if self.materialized.contains(level) {
if let LazyLevel::Materialized(jtree) = &self.levels[level] {
self.levels[level] = LazyLevel::Dirty(jtree.clone());
self.dirty.insert(level);
}
}
}
}
}
```
**Performance Target**: 70% reduction in level computations for typical query patterns
---
## Component 4: 256-Core Parallel Distribution
Leverage the existing agentic chip architecture:
```rust
/// Parallel j-tree across 256 cores
pub struct ParallelJTree {
/// Core assignments: which cores handle which levels
level_assignments: Vec<CoreRange>,
/// Shared coordinator for atomic updates
coordinator: SharedCoordinator,
/// Per-core executors
executors: [CoreExecutor; 256],
}
struct CoreRange {
start_core: u8,
end_core: u8,
level: usize,
}
impl ParallelJTree {
/// Distribute L levels across 256 cores
pub fn distribute_levels(num_levels: usize) -> Vec<CoreRange> {
let cores_per_level = 256 / num_levels;
(0..num_levels)
.map(|level| {
let start = (level * cores_per_level) as u8;
let end = ((level + 1) * cores_per_level - 1) as u8;
CoreRange { start_core: start, end_core: end, level }
})
.collect()
}
/// Parallel update across all affected levels
pub fn parallel_update(&mut self, edge: (VertexId, VertexId)) {
// Phase 1: Distribute update to affected cores
self.coordinator.phase.store(SharedCoordinator::PHASE_DISTRIBUTE, Ordering::Release);
for assignment in &self.level_assignments {
for core_id in assignment.start_core..=assignment.end_core {
self.executors[core_id as usize].queue_update(edge);
}
}
// Phase 2: Parallel compute
self.coordinator.phase.store(SharedCoordinator::PHASE_COMPUTE, Ordering::Release);
// Each core processes independently
// Work stealing if some cores finish early
while !self.coordinator.all_completed() {
// Idle cores steal from busy cores
self.work_stealing_pass();
}
// Phase 3: Collect results
self.coordinator.phase.store(SharedCoordinator::PHASE_COLLECT, Ordering::Release);
let global_min = self.coordinator.global_min_cut.load(Ordering::Acquire);
}
/// Work stealing for load balancing
fn work_stealing_pass(&mut self) {
for core_id in 0..256u8 {
if self.executors[core_id as usize].is_idle() {
// Find busy core to steal from
if let Some(victim) = self.find_busy_core() {
let work = self.executors[victim].steal_work();
self.executors[core_id as usize].accept_work(work);
}
}
}
}
}
```
**Performance Target**: Near-linear speedup up to 256× for independent level updates
---
## Component 5: Streaming Sketch Fallback
For graphs with n > 100K vertices:
```rust
/// Semi-streaming cut sketch for massive graphs
pub struct StreamingCutSketch {
/// Two edges per vertex (reservoir sampling)
sampled_edges: HashMap<VertexId, [Option<Edge>; 2]>,
/// Total vertices seen
vertex_count: usize,
/// Reservoir sampling state
reservoir: ReservoirSampler,
}
impl StreamingCutSketch {
/// Process edge in streaming fashion: O(1) per edge
pub fn process_edge(&mut self, edge: Edge) {
// Update reservoir for source vertex
self.reservoir.sample(edge.source, edge);
// Update reservoir for target vertex
self.reservoir.sample(edge.target, edge);
}
/// Approximate min-cut from sketch: O(n) query
pub fn approximate_min_cut(&self) -> ApproximateCut {
// Build sparse graph from sampled edges
let sparse = self.build_sparse_graph();
// Run exact algorithm on sparse graph
// O(n log n) edges → tractable
let cut = exact_min_cut(&sparse);
ApproximateCut {
value: cut.value,
approximation_factor: 1.0 + self.epsilon(),
partition: cut.partition,
}
}
/// Memory usage: O(n log n)
pub fn memory_bytes(&self) -> usize {
self.vertex_count * 2 * std::mem::size_of::<Edge>()
}
}
/// Adaptive system that switches between full j-tree and streaming
pub struct AdaptiveJTree {
full_jtree: Option<LazyJTreeHierarchy>,
streaming_sketch: Option<StreamingCutSketch>,
threshold: usize, // Switch point (default: 100K vertices)
}
impl AdaptiveJTree {
pub fn new(graph: &DynamicGraph) -> Self {
if graph.vertex_count() > 100_000 {
Self {
full_jtree: None,
streaming_sketch: Some(StreamingCutSketch::from_graph(graph)),
threshold: 100_000,
}
} else {
Self {
full_jtree: Some(LazyJTreeHierarchy::build(graph)),
streaming_sketch: None,
threshold: 100_000,
}
}
}
}
```
**Performance Target**: Handle 1M+ vertex graphs in <1GB memory
---
## Performance Comparison
| Metric | ADR-002 Baseline | SOTA Optimized | Improvement |
|--------|------------------|----------------|-------------|
| **Update Time** | O(n^ε) | O(n^ε) / 256 cores | ~100× |
| **Query Time (approx)** | O(log n) | O(1) cached | ~10× |
| **Query Time (exact)** | O(n^{o(1)}) | O(n^{o(1)}) lazy | ~5× |
| **Memory** | O(n log n) | O(active × n) | ~3× |
| **Prediction Accuracy** | N/A | 80%+ | New |
| **Edge Reduction** | 1 - ε | 90% neural | ~9× |
| **Max Graph Size** | ~100K | 1M+ streaming | ~10× |
---
## Integration with Existing Codebase
### SNN Integration Points
```rust
// Use existing SNN components from src/snn/
use crate::snn::{
PolicySNN, // For prediction engine
ValueNetwork, // For TD learning
NeuralGraphOptimizer, // For neural sparsification
compute_synchrony, // For stability detection
compute_energy, // For attractor dynamics
};
// Connect j-tree to SNN energy landscape
impl PredictiveJTree {
pub fn snn_energy(&self) -> f64 {
let mincut = self.hierarchy.approximate_min_cut().value;
let synchrony = compute_synchrony(&self.policy.recent_spikes(), 10.0);
compute_energy(mincut, synchrony)
}
}
```
### Parallel Architecture Integration
```rust
// Use existing parallel components from src/parallel/
use crate::parallel::{
SharedCoordinator, // Atomic coordination
CoreExecutor, // Per-core execution
CoreDistributor, // Work distribution
ResultAggregator, // Result collection
NUM_CORES, // 256 cores
};
// Extend CoreExecutor for j-tree levels
impl CoreExecutor {
pub fn process_jtree_level(&mut self, level: &JTree) -> CoreResult {
// Process assigned level within 8KB memory budget
self.state.process_compact_jtree(level)
}
}
```
### SONA Integration
```rust
// Connect to SONA hooks for continuous adaptation
const SONA_HOOKS: [usize; 4] = [8, 16, 24, 28];
impl PredictiveJTree {
pub fn enable_sona(&mut self) {
for &hook in &SONA_HOOKS {
self.policy.enable_hook(hook);
}
// Adaptation latency: <0.05ms per hook
}
}
```
---
## Implementation Priority
| Phase | Component | Effort | Impact | Dependencies |
|-------|-----------|--------|--------|--------------|
| **P0** | Degree-based presparse | 1 week | High | None |
| **P0** | 256-core distribution | 2 weeks | High | parallel/mod.rs |
| **P1** | Lazy hierarchy | 2 weeks | High | ADR-002 base |
| **P1** | Warm-start cut-matching | 2 weeks | High | Lazy hierarchy |
| **P2** | SNN prediction | 3 weeks | Medium | snn/optimizer.rs |
| **P2** | Neural sparsifier | 3 weeks | Medium | SNN prediction |
| **P3** | Streaming fallback | 2 weeks | Medium | None |
| **P3** | SONA integration | 1 week | Medium | SNN prediction |
---
## References
### New Research (2024-2026)
1. **SpecNet**: "Spectral Neural Graph Sparsification" (arXiv:2510.27474)
2. **DSpar**: "Degree-based Sparsification" (OpenReview)
3. **Warm-Start**: "Faster Weak Expander Decomposition" (arXiv:2511.02943)
4. **Parallel Expander**: "Near-Optimal Parallel Expander Decomposition" (SODA 2025)
5. **Semi-Streaming**: "Semi-Streaming Min-Cut" (Dudeja et al.)
### Existing Codebase
- `src/snn/mod.rs` - SNN integration (triple isomorphism)
- `src/snn/optimizer.rs` - PolicySNN, ValueNetwork, R-STDP
- `src/parallel/mod.rs` - 256-core architecture
- `src/compact/mod.rs` - 8KB per-core state
---
## Appendix: Complexity Summary
| Operation | Baseline | + Prediction | + Neural | + Parallel | + Streaming |
|-----------|----------|--------------|----------|------------|-------------|
| Insert Edge | O(n^ε) | O(n^ε) × 0.2 | O(n^ε) × 0.1 | O(n^ε / 256) | O(1) |
| Delete Edge | O(n^ε) | O(n^ε) × 0.2 | O(n^ε) × 0.1 | O(n^ε / 256) | O(1) |
| Approx Query | O(log n) | O(1) cached | O(1) | O(1) | O(n) |
| Exact Query | O(n^{o(1)}) | O(n^{o(1)}) × 0.2 | - | - | - |
| Memory | O(n log n) | O(n log n) | O(n log n / 10) | O(n log n) | O(n log n) |
**Combined**: Average case approaches O(1) for queries, O(n^ε / 256) for updates, with graceful degradation to streaming for massive graphs.

View File

@@ -0,0 +1,683 @@
# ADR-002: Dynamic Hierarchical j-Tree Decomposition for Approximate Cut Structure
**Status**: Proposed
**Date**: 2026-01-25
**Authors**: ruv.io, RuVector Team
**Deciders**: Architecture Review Board
**SDK**: Claude-Flow
## Version History
| Version | Date | Author | Changes |
|---------|------|--------|---------|
| 0.1 | 2026-01-25 | ruv.io | Initial draft based on arXiv:2601.09139 research |
---
## Plain Language Summary
**What is it?**
A new algorithmic framework for maintaining an approximate view of a graph's cut structure that updates in near-constant time even as edges are added and removed. It complements our existing exact min-cut implementation by providing a fast "global radar" that can answer approximate cut queries instantly.
**Why does it matter?**
Our current implementation (arXiv:2512.13105, El-Hayek/Henzinger/Li) excels at **exact** min-cut for superpolylogarithmic cuts but is optimized for a specific cut-size regime. The new j-tree decomposition (arXiv:2601.09139, Goranci/Henzinger/Kiss/Momeni/Zöcklein, January 2026) provides:
- **Broader coverage**: Poly-logarithmic approximation for ALL cut-based problems (sparsest cut, multi-way cut, multi-cut, all-pairs min-cuts)
- **Faster updates**: O(n^ε) amortized for any arbitrarily small ε > 0
- **Low recourse**: The underlying cut-sparsifier tolerates vertex splits with poly-logarithmic recourse
**The Two-Tier Strategy**:
| Tier | Algorithm | Purpose | When to Use |
|------|-----------|---------|-------------|
| **Tier 1** | j-Tree Decomposition | Fast approximate hierarchy for global structure | Continuous monitoring, routing decisions |
| **Tier 2** | El-Hayek/Henzinger/Li | Exact deterministic min-cut | When Tier 1 detects critical cuts |
Think of it like sonar and radar: the j-tree is your wide-area radar that shows approximate threat positions instantly, while the exact algorithm is your precision sonar that confirms exact details when needed.
---
## Context
### Current State
RuVector MinCut implements the December 2025 breakthrough (arXiv:2512.13105) achieving:
| Property | Current Implementation |
|----------|----------------------|
| **Update Time** | O(n^{o(1)}) amortized |
| **Approximation** | Exact |
| **Deterministic** | Yes |
| **Cut Regime** | Superpolylogarithmic (λ > log^c n) |
| **Verified Scaling** | n^0.12 empirically |
This works excellently for the coherence gate (ADR-001) where we need exact cut values for safety decisions. However, several use cases require:
1. **Broader cut-based queries**: Sparsest cut, multi-way cut, multi-cut, all-pairs min-cuts
2. **Even faster updates**: When monitoring 10K+ updates/second
3. **Global structure awareness**: Understanding the overall cut landscape, not just the minimum
### The January 2026 Breakthrough
The paper "Dynamic Hierarchical j-Tree Decomposition and Its Applications" (arXiv:2601.09139, SODA 2026) by Goranci, Henzinger, Kiss, Momeni, and Zöcklein addresses the open question:
> "Is there a fully dynamic algorithm for cut-based optimization problems that achieves poly-logarithmic approximation with very small polynomial update time?"
**Key Results**:
| Result | Complexity | Significance |
|--------|------------|--------------|
| **Update Time** | O(n^ε) amortized for any ε ∈ (0,1) | Arbitrarily close to polylog |
| **Approximation** | Poly-logarithmic | Sufficient for structure detection |
| **Query Support** | All cut-based problems | Not just min-cut |
| **Recourse** | Poly-logarithmic total | Sparsifier doesn't explode |
### Technical Innovation: Vertex-Split-Tolerant Cut Sparsifier
The core innovation is a **dynamic cut-sparsifier** that handles vertex splits with low recourse:
```
Traditional approach: Vertex splits cause O(n) cascading updates
New approach: Forest packing with lazy repair → poly-log recourse
```
The sparsifier maintains (1±ε) approximation of all cuts while:
- Tolerating vertex splits (critical for dynamic hierarchies)
- Adjusting only poly-logarithmically many edges per update
- Serving as a backbone for the j-tree hierarchy
### The (L,j) Hierarchy
The j-tree hierarchy reflects increasingly coarse views of the graph's cut landscape:
```
Level 0: Original graph G
Level 1: Contracted graph with j-tree quality α
Level 2: Further contracted with quality α²
...
Level L: Root (O(1) vertices)
L = O(log n / log α)
```
Each level preserves cut structure within an α^ factor, enabling:
- **Fast approximate queries**: Traverse O(log n) levels
- **Local updates**: Changes propagate through O(log n) levels
- **Multi-scale view**: See both fine and coarse structure
---
## Decision
### Adopt Two-Tier Dynamic Cut Architecture
We will implement the j-tree decomposition as a complementary layer to our existing exact min-cut, creating a two-tier system:
```
┌─────────────────────────────────────────────────────────────────────────┐
│ TWO-TIER DYNAMIC CUT ARCHITECTURE │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌────────────────────────────────────────────────────────────────────┐ │
│ │ TIER 1: J-TREE HIERARCHY (NEW) │ │
│ │ │ │
│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │
│ │ │ Level L │ │ Level L-1 │ │ Level 0 │ │ │
│ │ │ (Root) │◄───│ (Coarse) │◄───│ (Original) │ │ │
│ │ │ O(1) vtx │ │ α^(L-1) cut │ │ Exact cuts │ │ │
│ │ └──────────────┘ └──────────────┘ └──────────────┘ │ │
│ │ │ │
│ │ Purpose: Fast approximate answers for global structure │ │
│ │ Update: O(n^ε) amortized for any ε > 0 │ │
│ │ Query: Poly-log approximation for all cut problems │ │
│ │ │ │
│ └────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ Trigger: Approximate cut below threshold │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────────────┐ │
│ │ TIER 2: EXACT MIN-CUT (EXISTING) │ │
│ │ │ │
│ │ ┌──────────────────────────────────────────────────────────────┐ │ │
│ │ │ SubpolynomialMinCut (arXiv:2512.13105) │ │ │
│ │ │ • O(n^{o(1)}) amortized exact updates │ │ │
│ │ │ • Verified n^0.12 scaling │ │ │
│ │ │ • Deterministic, no randomization │ │ │
│ │ │ • For superpolylogarithmic cuts (λ > log^c n) │ │ │
│ │ └──────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ Purpose: Exact verification when precision required │ │
│ │ Trigger: Tier 1 detects potential critical cut │ │
│ │ │ │
│ └────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
```
### Module Structure
```
ruvector-mincut/
├── src/
│ ├── jtree/ # NEW: j-Tree Decomposition
│ │ ├── mod.rs # Module exports
│ │ ├── hierarchy.rs # (L,j) hierarchical decomposition
│ │ ├── sparsifier.rs # Vertex-split-tolerant cut sparsifier
│ │ ├── forest_packing.rs # Forest packing for sparsification
│ │ ├── vertex_split.rs # Vertex split handling with low recourse
│ │ ├── contraction.rs # Graph contraction for hierarchy levels
│ │ └── queries/ # Cut-based query implementations
│ │ ├── mod.rs
│ │ ├── all_pairs_mincut.rs
│ │ ├── sparsest_cut.rs
│ │ ├── multiway_cut.rs
│ │ └── multicut.rs
│ ├── tiered/ # NEW: Two-tier coordination
│ │ ├── mod.rs
│ │ ├── coordinator.rs # Tier 1/Tier 2 routing logic
│ │ ├── trigger.rs # Escalation trigger policies
│ │ └── cache.rs # Cross-tier result caching
│ └── ...existing modules...
```
### Core Data Structures
#### j-Tree Hierarchy
```rust
/// Hierarchical j-tree decomposition for approximate cut structure
pub struct JTreeHierarchy {
/// Number of levels (L = O(log n / log α))
levels: usize,
/// Approximation quality per level
alpha: f64,
/// Contracted graphs at each level
contracted_graphs: Vec<ContractedGraph>,
/// Cut sparsifier backbone
sparsifier: DynamicCutSparsifier,
/// j-trees at each level
jtrees: Vec<JTree>,
}
/// Single level j-tree
pub struct JTree {
/// Tree structure
tree: DynamicTree,
/// Mapping from original vertices to tree nodes
vertex_map: HashMap<VertexId, TreeNodeId>,
/// Cached cut values between tree nodes
cut_cache: CutCache,
/// Level index
level: usize,
}
impl JTreeHierarchy {
/// Build hierarchy from graph
pub fn build(graph: &DynamicGraph, epsilon: f64) -> Self {
let alpha = compute_alpha(epsilon);
let levels = (graph.vertex_count() as f64).log(alpha as f64).ceil() as usize;
// Build sparsifier first
let sparsifier = DynamicCutSparsifier::build(graph, epsilon);
// Build contracted graphs level by level
let mut contracted_graphs = Vec::with_capacity(levels);
let mut current = sparsifier.sparse_graph();
for level in 0..levels {
contracted_graphs.push(current.clone());
current = contract_to_jtree(&current, alpha);
}
Self {
levels,
alpha,
contracted_graphs,
sparsifier,
jtrees: build_jtrees(&contracted_graphs),
}
}
/// Insert edge with O(n^ε) amortized update
pub fn insert_edge(&mut self, u: VertexId, v: VertexId, weight: f64) -> Result<(), Error> {
// Update sparsifier (handles vertex splits internally)
self.sparsifier.insert_edge(u, v, weight)?;
// Propagate through hierarchy levels
for level in 0..self.levels {
self.update_level(level, EdgeUpdate::Insert(u, v, weight))?;
}
Ok(())
}
/// Delete edge with O(n^ε) amortized update
pub fn delete_edge(&mut self, u: VertexId, v: VertexId) -> Result<(), Error> {
self.sparsifier.delete_edge(u, v)?;
for level in 0..self.levels {
self.update_level(level, EdgeUpdate::Delete(u, v))?;
}
Ok(())
}
/// Query approximate min-cut (poly-log approximation)
pub fn approximate_min_cut(&self) -> ApproximateCut {
// Start from root level and refine
let mut cut = self.jtrees[self.levels - 1].min_cut();
for level in (0..self.levels - 1).rev() {
cut = self.jtrees[level].refine_cut(&cut);
}
ApproximateCut {
value: cut.value,
approximation_factor: self.alpha.powi(self.levels as i32),
partition: cut.partition,
}
}
}
```
#### Vertex-Split-Tolerant Cut Sparsifier
```rust
/// Dynamic cut sparsifier with low recourse under vertex splits
pub struct DynamicCutSparsifier {
/// Forest packing for edge sampling
forest_packing: ForestPacking,
/// Sparse graph maintaining (1±ε) cut approximation
sparse_graph: DynamicGraph,
/// Epsilon parameter
epsilon: f64,
/// Recourse counter for complexity verification
recourse: RecourseTracker,
}
impl DynamicCutSparsifier {
/// Handle vertex split with poly-log recourse
pub fn split_vertex(&mut self, v: VertexId, v1: VertexId, v2: VertexId,
partition: &[EdgeId]) -> Result<RecourseStats, Error> {
let before_edges = self.sparse_graph.edge_count();
// Forest packing handles the split
let affected_forests = self.forest_packing.split_vertex(v, v1, v2, partition)?;
// Lazy repair: only fix forests that actually need it
for forest_id in affected_forests {
self.repair_forest(forest_id)?;
}
let recourse = (self.sparse_graph.edge_count() as i64 - before_edges as i64).abs();
self.recourse.record(recourse as usize);
Ok(self.recourse.stats())
}
/// The key insight: forest packing limits cascading updates
fn repair_forest(&mut self, forest_id: ForestId) -> Result<(), Error> {
// Only O(log n) edges need adjustment per forest
// Total forests = O(log n / ε²)
// Total recourse = O(log² n / ε²) per vertex split
self.forest_packing.repair(forest_id, &mut self.sparse_graph)
}
}
```
### Two-Tier Coordinator
```rust
/// Coordinates between j-tree approximation (Tier 1) and exact min-cut (Tier 2)
pub struct TwoTierCoordinator {
/// Tier 1: Fast approximate hierarchy
jtree: JTreeHierarchy,
/// Tier 2: Exact min-cut for verification
exact: SubpolynomialMinCut,
/// Trigger policy for escalation
trigger: EscalationTrigger,
/// Result cache to avoid redundant computation
cache: TierCache,
}
/// When to escalate from Tier 1 to Tier 2
pub struct EscalationTrigger {
/// Approximate cut threshold below which we verify exactly
critical_threshold: f64,
/// Maximum approximation factor before requiring exact
max_approx_factor: f64,
/// Whether the query requires exact answer
exact_required: bool,
}
impl TwoTierCoordinator {
/// Query min-cut with tiered strategy
pub fn min_cut(&mut self, exact_required: bool) -> CutResult {
// Check cache first
if let Some(cached) = self.cache.get() {
if !exact_required || cached.is_exact {
return cached.clone();
}
}
// Tier 1: Fast approximate query
let approx = self.jtree.approximate_min_cut();
// Decide whether to escalate
let should_escalate = exact_required
|| approx.value < self.trigger.critical_threshold
|| approx.approximation_factor > self.trigger.max_approx_factor;
if should_escalate {
// Tier 2: Exact verification
let exact_value = self.exact.min_cut_value();
let exact_partition = self.exact.partition();
let result = CutResult {
value: exact_value,
partition: exact_partition,
is_exact: true,
approximation_factor: 1.0,
tier_used: Tier::Exact,
};
self.cache.store(result.clone());
result
} else {
let result = CutResult {
value: approx.value,
partition: approx.partition,
is_exact: false,
approximation_factor: approx.approximation_factor,
tier_used: Tier::Approximate,
};
self.cache.store(result.clone());
result
}
}
/// Insert edge, updating both tiers
pub fn insert_edge(&mut self, u: VertexId, v: VertexId, weight: f64) -> Result<(), Error> {
self.cache.invalidate();
// Update Tier 1 (fast)
self.jtree.insert_edge(u, v, weight)?;
// Update Tier 2 (also fast, but only if we're tracking that edge regime)
self.exact.insert_edge(u, v, weight)?;
Ok(())
}
}
```
### Extended Query Support
The j-tree hierarchy enables queries beyond min-cut:
```rust
impl JTreeHierarchy {
/// All-pairs minimum cuts (approximate)
pub fn all_pairs_min_cuts(&self) -> AllPairsResult {
// Use hierarchy to avoid O(n²) explicit computation
// Query time: O(n log n) for all pairs
let mut results = HashMap::new();
for (u, v) in self.vertex_pairs() {
let cut = self.min_cut_between(u, v);
results.insert((u, v), cut);
}
AllPairsResult { cuts: results }
}
/// Sparsest cut (approximate)
pub fn sparsest_cut(&self) -> SparsestCutResult {
// Leverage hierarchy for O(n^ε) approximate sparsest cut
let mut best_sparsity = f64::INFINITY;
let mut best_cut = None;
for level in 0..self.levels {
let candidate = self.jtrees[level].sparsest_cut_candidate();
let sparsity = candidate.value / candidate.size.min() as f64;
if sparsity < best_sparsity {
best_sparsity = sparsity;
best_cut = Some(candidate);
}
}
SparsestCutResult {
cut: best_cut.unwrap(),
sparsity: best_sparsity,
approximation: self.alpha.powi(self.levels as i32),
}
}
/// Multi-way cut (approximate)
pub fn multiway_cut(&self, terminals: &[VertexId]) -> MultiwayCutResult {
// Use j-tree hierarchy to find approximate multiway cut
// Approximation: O(log k) where k = number of terminals
self.compute_multiway_cut(terminals)
}
/// Multi-cut (approximate)
pub fn multicut(&self, pairs: &[(VertexId, VertexId)]) -> MulticutResult {
// Approximate multicut using hierarchy
self.compute_multicut(pairs)
}
}
```
### Integration with Coherence Gate (ADR-001)
The j-tree hierarchy integrates with the Anytime-Valid Coherence Gate:
```rust
/// Enhanced coherence gate using two-tier cut architecture
pub struct TieredCoherenceGate {
/// Two-tier cut coordinator
cut_coordinator: TwoTierCoordinator,
/// Conformal prediction component
conformal: ShiftAdaptiveConformal,
/// E-process evidence accumulator
evidence: EProcessAccumulator,
/// Gate thresholds
thresholds: GateThresholds,
}
impl TieredCoherenceGate {
/// Fast structural check using Tier 1
pub fn fast_structural_check(&self, action: &Action) -> QuickDecision {
// Use j-tree for O(n^ε) approximate check
let approx_cut = self.cut_coordinator.jtree.approximate_min_cut();
if approx_cut.value > self.thresholds.definitely_safe {
QuickDecision::Permit
} else if approx_cut.value < self.thresholds.definitely_unsafe {
QuickDecision::Deny
} else {
QuickDecision::NeedsExactCheck
}
}
/// Full evaluation with exact verification if needed
pub fn evaluate(&mut self, action: &Action, context: &Context) -> GateDecision {
// Quick check first
let quick = self.fast_structural_check(action);
match quick {
QuickDecision::Permit => {
// Fast path: structure is definitely safe
self.issue_permit_fast(action)
}
QuickDecision::Deny => {
// Fast path: structure is definitely unsafe
self.issue_denial_fast(action)
}
QuickDecision::NeedsExactCheck => {
// Invoke Tier 2 for exact verification
let exact_cut = self.cut_coordinator.min_cut(true);
self.evaluate_with_exact_cut(action, context, exact_cut)
}
}
}
}
```
### Performance Characteristics
| Operation | Tier 1 (j-Tree) | Tier 2 (Exact) | Combined |
|-----------|-----------------|----------------|----------|
| **Insert Edge** | O(n^ε) | O(n^{o(1)}) | O(n^ε) |
| **Delete Edge** | O(n^ε) | O(n^{o(1)}) | O(n^ε) |
| **Min-Cut Query** | O(log n) approx | O(1) exact | O(1) - O(log n) |
| **All-Pairs Min-Cut** | O(n log n) | N/A | O(n log n) |
| **Sparsest Cut** | O(n^ε) | N/A | O(n^ε) |
| **Multi-Way Cut** | O(k log k · n^ε) | N/A | O(k log k · n^ε) |
### Recourse Guarantees
The vertex-split-tolerant sparsifier provides:
| Metric | Guarantee |
|--------|-----------|
| **Edges adjusted per update** | O(log² n / ε²) |
| **Total recourse over m updates** | O(m · log² n / ε²) |
| **Forest repairs per vertex split** | O(log n) |
This is critical for maintaining hierarchy stability under dynamic changes.
---
## Implementation Phases
### Phase 1: Core Sparsifier (Weeks 1-3)
- [ ] Implement `ForestPacking` with edge sampling
- [ ] Implement `DynamicCutSparsifier` with vertex split handling
- [ ] Add recourse tracking and verification
- [ ] Unit tests for sparsifier correctness
### Phase 2: j-Tree Hierarchy (Weeks 4-6)
- [ ] Implement `JTree` single-level structure
- [ ] Implement `JTreeHierarchy` multi-level decomposition
- [ ] Add contraction algorithms for level construction
- [ ] Integration tests for hierarchy maintenance
### Phase 3: Query Support (Weeks 7-9)
- [ ] Implement approximate min-cut queries
- [ ] Implement all-pairs min-cut
- [ ] Implement sparsest cut
- [ ] Implement multi-way cut and multi-cut
- [ ] Benchmark query performance
### Phase 4: Two-Tier Integration (Weeks 10-12)
- [ ] Implement `TwoTierCoordinator`
- [ ] Define escalation trigger policies
- [ ] Integrate with coherence gate
- [ ] End-to-end testing with coherence scenarios
---
## Feature Flags
```toml
[features]
# Existing features
default = ["exact", "approximate"]
exact = []
approximate = []
# New features
jtree = [] # j-Tree hierarchical decomposition
tiered = ["jtree", "exact"] # Two-tier coordinator
all-cut-queries = ["jtree"] # Sparsest cut, multiway, multicut
```
---
## Consequences
### Benefits
1. **Broader Query Support**: Sparsest cut, multi-way cut, multi-cut, all-pairs - not just minimum cut
2. **Faster Continuous Monitoring**: O(n^ε) updates enable 10K+ updates/second even on large graphs
3. **Global Structure Awareness**: Hierarchical view shows cut landscape at multiple scales
4. **Graceful Degradation**: Approximate answers when exact isn't needed, exact when it is
5. **Low Recourse**: Sparsifier stability prevents update cascades
6. **Coherence Gate Enhancement**: Fast structural checks with exact fallback
### Risks & Mitigations
| Risk | Probability | Impact | Mitigation |
|------|-------------|--------|------------|
| Implementation complexity | High | Medium | Phase incrementally, extensive testing |
| Approximation too loose | Medium | Medium | Tunable α parameter, exact fallback |
| Memory overhead from hierarchy | Medium | Low | Lazy level construction |
| Integration complexity with existing code | Medium | Medium | Clean interface boundaries |
### Complexity Analysis
| Component | Space | Time (Update) | Time (Query) |
|-----------|-------|---------------|--------------|
| Forest Packing | O(m log n / ε²) | O(log² n / ε²) | O(1) |
| j-Tree Level | O(n_) | O(n_^ε) | O(log n_) |
| Full Hierarchy | O(n log n) | O(n^ε) | O(log n) |
| Two-Tier Cache | O(n) | O(1) | O(1) |
---
## References
### Primary
1. Goranci, G., Henzinger, M., Kiss, P., Momeni, A., & Zöcklein, G. (January 2026). "Dynamic Hierarchical j-Tree Decomposition and Its Applications." *arXiv:2601.09139*. SODA 2026. **[Core paper for this ADR]**
### Complementary
2. El-Hayek, A., Henzinger, M., & Li, J. (December 2025). "Deterministic and Exact Fully-dynamic Minimum Cut of Superpolylogarithmic Size in Subpolynomial Time." *arXiv:2512.13105*. **[Existing Tier 2 implementation]**
3. Mądry, A. (2010). "Fast Approximation Algorithms for Cut-Based Problems in Undirected Graphs." *FOCS 2010*. **[Original j-tree decomposition]**
### Background
4. Benczúr, A. A., & Karger, D. R. (1996). "Approximating s-t Minimum Cuts in Õ(n²) Time." *STOC*. **[Cut sparsification foundations]**
5. Thorup, M. (2007). "Fully-Dynamic Min-Cut." *Combinatorica*. **[Dynamic min-cut foundations]**
---
## Related Decisions
- **ADR-001**: Anytime-Valid Coherence Gate (uses Tier 2 exact min-cut)
- **ADR-014**: Coherence Engine Architecture (coherence computation)
- **ADR-CE-001**: Sheaf Laplacian Coherence (structural coherence foundation)
---
## Appendix: Paper Comparison
### El-Hayek/Henzinger/Li (Dec 2025) vs Goranci et al. (Jan 2026)
| Aspect | arXiv:2512.13105 | arXiv:2601.09139 |
|--------|------------------|------------------|
| **Focus** | Exact min-cut | Approximate cut hierarchy |
| **Update Time** | O(n^{o(1)}) | O(n^ε) for any ε > 0 |
| **Approximation** | Exact | Poly-logarithmic |
| **Cut Regime** | Superpolylogarithmic | All sizes |
| **Query Types** | Min-cut only | All cut problems |
| **Deterministic** | Yes | Yes |
| **Key Technique** | Cluster hierarchy + LocalKCut | j-Tree + vertex-split sparsifier |
**Synergy**: The two approaches complement each other perfectly:
- Use Goranci et al. for fast global monitoring and diverse cut queries
- Use El-Hayek et al. for exact verification when critical cuts are detected
This two-tier strategy provides both breadth (approximate queries on all cut problems) and depth (exact min-cut when needed).

View File

@@ -0,0 +1,392 @@
# Appendix: Applications Spectrum for Anytime-Valid Coherence Gate
**Related**: ADR-001, DDC-001, ROADMAP
This appendix maps the Anytime-Valid Coherence Gate to concrete market applications across three horizons.
---
## Practical Applications (0-18 months)
These convert pilots into procurement. Target: Enterprise buyers who need auditable safety now.
### 1. Network Security Control Plane
**Use Case**: Detect and suppress lateral movement, credential abuse, and tool misuse in real time.
**How the Gate Helps**:
- When coherence drops (new relationships, weird graph cuts, novel access paths), actions get deferred or denied automatically
- Witness partitions identify the exact boundary crossing that triggered intervention
- E-process accumulates evidence of anomalous behavior over time
**Demo Scenario**:
```
1. Ingest NetFlow + auth logs into RuVector graph
2. Fire simulated attack (credential stuffing → lateral movement)
3. Show Permit/Deny decisions with witness cut visualization
4. Highlight "here's exactly why this action was blocked"
```
**Metric to Own**: Mean time to safe containment (MTTC)
**Integration Points**:
- SIEM integration via `GatePacket` events
- Witness receipts feed into incident response workflows
- E-process thresholds map to SOC escalation tiers
---
### 2. Cloud Operations Autopilot
**Use Case**: Auto-remediation of incidents without runaway automation.
**How the Gate Helps**:
- Only allow remediation steps that stay inside stable partitions of dependency graphs
- Coherence drop triggers "Defer to human" instead of cascading rollback
- Conformal prediction sets quantify uncertainty about remediation outcomes
**Demo Scenario**:
```
1. Service dependency graph + deploy pipeline in RuVector
2. Inject failure (service A crashes)
3. Autopilot proposes rollback
4. Gate checks: "Does rollback stay within stable partition?"
5. If boundary crossing detected → DEFER with witness
```
**Metric to Own**: Reduction in incident blast radius
**Integration Points**:
- Kubernetes operator for deployment gating
- Terraform plan validation via graph analysis
- PagerDuty integration for DEFER escalations
---
### 3. Data Governance and Exfiltration Prevention
**Use Case**: Prevent agents from leaking sensitive data across boundaries.
**How the Gate Helps**:
- Boundary witnesses become enforceable "do not cross" lines
- Memory shards and tool scopes mapped as graph partitions
- Any action crossing partition → immediate DENY + audit
**Metric to Own**: Unauthorized cross-domain action suppression rate
**Architecture**:
```
┌─────────────────┐ ┌─────────────────┐
│ PII Zone │ │ Public Zone │
│ (Partition A) │ │ (Partition B) │
│ │ │ │
│ • User records │ │ • Analytics │
│ • Credentials │ │ • Reports │
└────────┬────────┘ └────────┬────────┘
│ │
└──────┬───────────────┘
┌──────▼──────┐
│ COHERENCE │
│ GATE │
│ │
│ Witness: │
│ "Action │
│ crosses │
│ PII→Public" │
│ │
│ Decision: │
│ DENY │
└─────────────┘
```
---
### 4. Agent Routing and Budget Control
**Use Case**: Stop agents from spiraling, chattering, or tool thrashing.
**How the Gate Helps**:
- Coherence signal detects when agent is "lost" (exploration without progress)
- E-value evidence decides whether escalation/continuation is justified
- Conformal sets bound expected cost of next action
**Metric to Own**: Cost per resolved task with fixed safety constraints
**Decision Logic**:
```
IF action_count > threshold AND coherence < target:
→ Check e-process: "Is progress being made?"
→ IF e_value < τ_deny: DENY (stop the spiral)
→ IF e_value < τ_permit: DEFER (escalate to human)
→ ELSE: PERMIT (continue but monitor)
```
---
## Advanced Practical (18 months - 3 years)
These start to look like "new infrastructure."
### 5. Autonomous SOC and NOC
**Use Case**: Always-on detection, triage, and response with bounded actions.
**How the Gate Helps**:
- System stays calm until boundary crossings spike
- Then concentrates attention on anomalous regions
- Human analysts handle DEFER decisions only
**Metric to Own**: Analyst-hours saved per month without increased risk
**Architecture**:
```
┌─────────────────────────────────────────────────────────┐
│ AUTONOMOUS SOC │
├─────────────────────────────────────────────────────────┤
│ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Detect │──▶│ Triage │──▶│ Respond │──▶│ Learn │ │
│ └────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘ │
│ │ │ │ │ │
│ └─────────────┴─────────────┴─────────────┘ │
│ │ │
│ ┌──────▼──────┐ │
│ │ COHERENCE │ │
│ │ GATE │ │
│ └──────┬──────┘ │
│ │ │
│ ┌──────────────┼──────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ PERMIT DEFER DENY │
│ (automated) (to analyst) (blocked) │
│ │
└─────────────────────────────────────────────────────────┘
```
---
### 6. Supply Chain Integrity and Firmware Trust
**Use Case**: Devices that self-audit software changes and refuse unsafe upgrades.
**How the Gate Helps**:
- Signed event logs feed into coherence computation
- Deterministic replay verifies state transitions
- Boundary gating on what updates may alter
**Metric to Own**: Mean time to recover from compromised update attempt
**Witness Receipt Structure**:
```json
{
"update_id": "firmware-v2.3.1",
"source_hash": "abc123...",
"coherence_before": 0.95,
"coherence_after_sim": 0.72,
"boundary_violations": [
"bootloader partition",
"secure enclave boundary"
],
"decision": "DENY",
"e_value": 0.003,
"receipt_hash": "def456..."
}
```
---
### 7. Multi-Tenant AI Safety Partitioning
**Use Case**: Same hardware, many customers, no cross-tenant drift or bleed.
**How the Gate Helps**:
- RuVector partitions model tenant boundaries
- Cut-witness enforcement prevents cross-tenant actions
- Per-tenant e-processes track coherence independently
**Metric to Own**: Cross-tenant anomaly leakage probability (measured, not promised)
**Guarantee Structure**:
```
For each tenant T_i:
P(action from T_i affects T_j, j≠i) ≤ ε
Where ε is bounded by:
- Min-cut between T_i and T_j partitions
- Conformal prediction set overlap
- E-process independence verification
```
---
## Exotic Applications (3-10 years)
These are the ones that make people say "wait, that's a different kind of computer."
### 8. Machines that "Refuse to Hallucinate with Actions"
**Use Case**: A system that can still be uncertain, but cannot act uncertainly.
**Principle**:
- It can generate hypotheses all day
- But action requires coherence AND evidence
- Creativity without incident
**How It Works**:
```
WHILE generating:
hypotheses ← LLM.generate() # Unconstrained creativity
FOR action in proposed_actions:
IF NOT coherence_gate.permits(action):
CONTINUE # Skip uncertain actions
# Only reaches here if:
# 1. Action stays in stable partition
# 2. Conformal set is small (confident prediction)
# 3. E-process shows sufficient evidence
EXECUTE(action)
```
**Outcome**: You get creativity without incident. The system can explore freely in thought-space but must be grounded before acting.
---
### 9. Continuous Self-Healing Software and Infrastructure
**Use Case**: Systems that grow calmer over time, not more fragile.
**Principle**:
- Coherence becomes the homeostasis signal
- Learning pauses when unstable, resumes when stable
- Optimization is built-in, not bolt-on
**Homeostasis Loop**:
```
┌─────────────────────────────────────────┐
│ │
│ ┌─────────┐ │
│ │ Observe │◀──────────────────┐ │
│ └────┬────┘ │ │
│ │ │ │
│ ▼ │ │
│ ┌─────────┐ │ │
│ │ Compute │──▶ coherence │ │
│ │Coherence│ │ │
│ └────┬────┘ │ │
│ │ │ │
│ ▼ │ │
│ ┌─────────────────────┐ │ │
│ │ coherence > target? │ │ │
│ └──────────┬──────────┘ │ │
│ │ │ │
│ ┌──────┴──────┐ │ │
│ │ │ │ │
│ ▼ ▼ │ │
│ ┌───────┐ ┌────────┐ │ │
│ │ LEARN │ │ PAUSE │ │ │
│ └───┬───┘ └────────┘ │ │
│ │ │ │
│ └─────────────────────────┘ │
│ │
└─────────────────────────────────────────┘
```
**Outcome**: "Built-in optimization" instead of built-in obsolescence. Systems that maintain themselves.
---
### 10. Nervous-System Computing for Fleets
**Use Case**: Millions of devices that coordinate without central control.
**Principle**:
- Local coherence gates at each node
- Only boundary deltas shared upstream
- Scale without noise
**Architecture**:
```
┌─────────────────────────────────────┐
│ GLOBAL AGGREGATE │
│ (boundary deltas only) │
└──────────────────┬──────────────────┘
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
┌───────────┐ ┌───────────┐ ┌───────────┐
│ Region A │ │ Region B │ │ Region C │
│ Gate │ │ Gate │ │ Gate │
└─────┬─────┘ └─────┬─────┘ └─────┬─────┘
│ │ │
┌─────┴─────┐ ┌─────┴─────┐ ┌─────┴─────┐
│ • • • • • │ │ • • • • • │ │ • • • • • │
│ Devices │ │ Devices │ │ Devices │
│ (local │ │ (local │ │ (local │
│ gates) │ │ gates) │ │ gates) │
└───────────┘ └───────────┘ └───────────┘
```
**Key Insight**: Most decisions stay local. Only boundary crossings escalate. This is how biological nervous systems achieve scale—not by centralizing everything, but by making most decisions locally and only propagating what matters.
**Outcome**: Scale without noise. Decisions stay local, escalation stays rare.
---
### 11. Synthetic Institutions
**Use Case**: Autonomous org-like systems that maintain rules, budgets, and integrity over decades.
**Principle**:
- Deterministic governance receipts become the operating fabric
- Every decision has a witness
- Institutional memory is cryptographically anchored
**What This Looks Like**:
```
SYNTHETIC INSTITUTION
├── Constitution (immutable rules)
│ └── Encoded as min-cut constraints
├── Governance (decision procedures)
│ └── Gate policies with e-process thresholds
├── Memory (institutional history)
│ └── Merkle tree of witness receipts
├── Budget (resource allocation)
│ └── Conformal bounds on expenditure
└── Evolution (rule changes)
└── Requires super-majority e-process evidence
```
**Outcome**: A new class of durable, auditable autonomy. Organizations that can outlive their creators while remaining accountable.
---
## Summary: The Investment Thesis
| Horizon | Applications | Market Signal |
|---------|--------------|---------------|
| **0-18 months** | Network security, cloud ops, data governance, agent routing | "Buyers will pay for this next quarter" |
| **18 months - 3 years** | Autonomous SOC/NOC, supply chain, multi-tenant AI | "New infrastructure" |
| **3-10 years** | Action-grounded AI, self-healing systems, fleet nervous systems, synthetic institutions | "A different kind of computer" |
The coherence gate is the primitive that enables all of these. It converts the category thesis (bounded autonomy with receipts) into a product primitive that:
1. **Buyers understand**: "Permit / Defer / Deny with audit trail"
2. **Auditors accept**: "Every decision has a cryptographic witness"
3. **Engineers can build on**: "Clear API with formal guarantees"
---
## Next Steps
1. **Phase 1 Demo**: Network security control plane (shortest path to revenue)
2. **Phase 2 Platform**: Agent routing SDK (developer adoption)
3. **Phase 3 Infrastructure**: Multi-tenant AI safety (enterprise lock-in)
4. **Phase 4 Research**: Exotic applications (thought leadership)

View File

@@ -0,0 +1,370 @@
# DDC-001: Anytime-Valid Coherence Gate - Design Decision Criteria
**Version**: 1.0
**Date**: 2026-01-17
**Related ADR**: ADR-001-anytime-valid-coherence-gate
## Purpose
This document specifies the design decision criteria for implementing the Anytime-Valid Coherence Gate (AVCG). It provides concrete guidance for architectural choices, implementation trade-offs, and acceptance criteria.
---
## 1. Graph Model Design Decisions
### DDC-1.1: Action Graph Construction
**Decision Required**: How to construct the action graph G_t from agent state?
| Option | Description | Pros | Cons | Recommendation |
|--------|-------------|------|------|----------------|
| **A. State-Action Pairs** | Nodes = (state, action), Edges = transitions | Fine-grained control; precise cuts | Large graphs; O(|S|·|A|) nodes | Use for high-stakes domains |
| **B. Abstract State Clusters** | Nodes = state clusters, Edges = aggregate transitions | Smaller graphs; faster updates | May miss nuanced boundaries | **Recommended for v0** |
| **C. Learned Embeddings** | Nodes = learned state embeddings | Adaptive; captures latent structure | Requires training data; less interpretable | Future enhancement |
**Acceptance Criteria**:
- [ ] Graph construction completes in < 100μs for typical agent states
- [ ] Graph accurately represents reachability to unsafe states
- [ ] Witness partitions are human-interpretable
### DDC-1.2: Edge Weight Semantics
**Decision Required**: What do edge weights represent?
| Option | Interpretation | Use Case |
|--------|---------------|----------|
| **A. Risk Scores** | Higher weight = higher risk of unsafe outcome | Min-cut = minimum total risk to unsafe |
| **B. Inverse Probability** | Higher weight = less likely transition | Min-cut = least likely path to unsafe |
| **C. Unit Weights** | All edges weight 1.0 | Min-cut = fewest actions to unsafe |
| **D. Conformal Set Size** | Weight = |C_t| for that action | Natural integration with predictive uncertainty |
**Recommendation**: Option D creates natural integration between min-cut and conformal prediction.
**Acceptance Criteria**:
- [ ] Weight semantics are documented and consistent
- [ ] Min-cut value has interpretable meaning for operators
- [ ] Weights update correctly on new observations
---
## 2. Conformal Predictor Architecture
### DDC-2.1: Base Predictor Selection
**Decision Required**: Which base predictor to wrap with conformal prediction?
| Option | Characteristics | Computational Cost |
|--------|----------------|-------------------|
| **A. Neural Network** | High capacity; requires calibration | Medium-High |
| **B. Random Forest** | Built-in uncertainty; robust | Medium |
| **C. Gaussian Process** | Natural uncertainty; O(n³) training | High |
| **D. Ensemble with Dropout** | Approximate Bayesian; scalable | Medium |
**Recommendation**: Option D (Ensemble with Dropout) for balance of capacity and uncertainty.
**Acceptance Criteria**:
- [ ] Base predictor achieves acceptable accuracy on held-out data
- [ ] Prediction latency < 10ms for single action
- [ ] Uncertainty estimates correlate with actual error rates
### DDC-2.2: Non-Conformity Score Function
**Decision Required**: How to compute non-conformity scores?
| Option | Formula | Properties |
|--------|---------|------------|
| **A. Absolute Residual** | s(x,y) = |y - ŷ(x)| | Simple; symmetric |
| **B. Normalized Residual** | s(x,y) = |y - ŷ(x)| / σ̂(x) | Scale-invariant |
| **C. CQR** | s(x,y) = max(q̂_lo - y, y - q̂_hi) | Heteroscedastic coverage |
**Recommendation**: Option C (CQR) for heteroscedastic agent environments.
**Acceptance Criteria**:
- [ ] Marginal coverage ≥ 1 - α over calibration window
- [ ] Conditional coverage approximately uniform across feature space
- [ ] Prediction sets are not trivially large
### DDC-2.3: Shift Adaptation Method
**Decision Required**: How to adapt conformal predictor to distribution shift?
| Method | Adaptation Speed | Conservativeness |
|--------|-----------------|------------------|
| **A. ACI (Adaptive Conformal)** | Medium | High |
| **B. Retrospective Adjustment** | Fast | Medium |
| **C. COP (Conformal Optimistic)** | Fastest | Low (but valid) |
| **D. CORE (RL-based)** | Adaptive | Task-dependent |
**Recommendation**: Hybrid approach:
- Use COP for normal operation (fast, less conservative)
- Fall back to ACI under detected severe shift
- Use retrospective adjustment for post-hoc correction
**Acceptance Criteria**:
- [ ] Coverage maintained during gradual shift (δ < 0.1/step)
- [ ] Recovery to target coverage within 100 steps after abrupt shift
- [ ] No catastrophic coverage failures (coverage never < 0.5)
---
## 3. E-Process Construction
### DDC-3.1: E-Value Computation Method
**Decision Required**: How to compute per-action e-values?
| Method | Requirements | Robustness |
|--------|--------------|------------|
| **A. Likelihood Ratio** | Density models for H₀ and H₁ | Low (model-dependent) |
| **B. Universal Inference** | Split data; no density needed | Medium |
| **C. Mixture E-Values** | Multiple alternatives | High (hedged) |
| **D. Betting E-Values** | Online learning framework | High (adaptive) |
**Recommendation**: Option C (Mixture E-Values) for robustness:
```
e_t = (1/K) Σ_k e_t^{(k)}
```
Where each e_t^{(k)} tests a different alternative hypothesis.
**Acceptance Criteria**:
- [ ] E[e_t | H₀] ≤ 1 verified empirically
- [ ] Power against reasonable alternatives > 0.5
- [ ] Computation time < 1ms per e-value
### DDC-3.2: E-Process Update Rule
**Decision Required**: How to update the e-process over time?
| Rule | Formula | Properties |
|------|---------|------------|
| **A. Product** | E_t = Π_{i=1}^t e_i | Aggressive; exponential power |
| **B. Average** | E_t = (1/t) Σ_{i=1}^t e_i | Conservative; bounded |
| **C. Exponential Moving** | E_t = λ·e_t + (1-λ)·E_{t-1} | Balanced; forgetting |
| **D. Mixture Supermartingale** | E_t = Σ_j w_j · E_t^{(j)} | Robust; hedged |
**Recommendation**:
- Option A (Product) for high-stakes single decisions
- Option D (Mixture) for continuous monitoring
**Acceptance Criteria**:
- [ ] E_t remains nonnegative supermartingale
- [ ] Stopping time τ has valid Type I error: P(E_τ ≥ 1/α) ≤ α
- [ ] Power grows with evidence accumulation
### DDC-3.3: Null Hypothesis Specification
**Decision Required**: What constitutes the "coherence" null hypothesis?
| Formulation | Meaning |
|-------------|---------|
| **A. Action Safety** | H₀: P(action leads to unsafe state) ≤ p₀ |
| **B. State Stability** | H₀: P(state deviates from normal) ≤ p₀ |
| **C. Policy Consistency** | H₀: Current policy ≈ reference policy |
| **D. Composite** | H₀: (A) ∧ (B) ∧ (C) |
**Recommendation**: Start with Option A, extend to Option D for production.
**Acceptance Criteria**:
- [ ] H₀ is well-specified and testable
- [ ] False alarm rate matches target α
- [ ] Null violations are meaningfully dangerous
---
## 4. Integration Architecture
### DDC-4.1: Signal Combination Strategy
**Decision Required**: How to combine the three signals into a gate decision?
| Strategy | Logic | Properties |
|----------|-------|------------|
| **A. Sequential Short-Circuit** | Cut → Conformal → E-process | Fast rejection; ordered |
| **B. Parallel with Voting** | All evaluate; majority rules | Robust; slower |
| **C. Weighted Integration** | score = w₁·cut + w₂·conf + w₃·e | Flexible; needs tuning |
| **D. Hierarchical** | E-process gates conformal gates cut | Layered authority |
**Recommendation**: Option A (Sequential Short-Circuit):
1. Min-cut DENY is immediate (structural safety)
2. Conformal uncertainty gates e-process (no point accumulating evidence if outcome unpredictable)
3. E-process makes final permit/defer decision
**Acceptance Criteria**:
- [ ] Gate latency < 50ms for typical decisions
- [ ] No single-point-of-failure (graceful degradation)
- [ ] Decision audit trail is complete
### DDC-4.2: Graceful Degradation
**Decision Required**: How should the gate behave when components fail?
| Component Failure | Fallback Behavior |
|-------------------|-------------------|
| Min-cut unavailable | Defer all actions; alert operator |
| Conformal predictor fails | Use widened prediction sets (conservative) |
| E-process computation fails | Use last valid e-value; decay confidence |
| All components fail | Full DENY; require human approval |
**Acceptance Criteria**:
- [ ] Failure detection within 100ms
- [ ] Fallback never less safe than full DENY
- [ ] Recovery is automatic when component restores
### DDC-4.3: Latency Budget Allocation
**Decision Required**: How to allocate total latency budget across components?
Given total budget T_total (e.g., 50ms):
| Component | Allocation | Rationale |
|-----------|------------|-----------|
| Min-cut update | 0.2 · T | Amortized; subpolynomial |
| Conformal prediction | 0.4 · T | Main computation |
| E-process update | 0.2 · T | Arithmetic; fast |
| Decision logic | 0.1 · T | Simple rules |
| Receipt generation | 0.1 · T | Hashing; logging |
**Acceptance Criteria**:
- [ ] p99 latency < T_total
- [ ] No component exceeds 2× its budget
- [ ] Latency monitoring in place
---
## 5. Operational Parameters
### DDC-5.1: Threshold Configuration
| Parameter | Symbol | Default | Range | Tuning Guidance |
|-----------|--------|---------|-------|-----------------|
| E-process deny threshold | τ_deny | 0.01 | [0.001, 0.1] | Lower = more conservative |
| E-process permit threshold | τ_permit | 100 | [10, 1000] | Higher = more evidence required |
| Uncertainty threshold | θ_uncertainty | 0.5 | [0.1, 1.0] | Fraction of outcome space |
| Confidence threshold | θ_confidence | 0.1 | [0.01, 0.3] | Fraction of outcome space |
| Conformal coverage target | 1-α | 0.9 | [0.8, 0.99] | Higher = larger sets |
### DDC-5.2: Audit Requirements
| Requirement | Specification |
|-------------|---------------|
| Receipt retention | 90 days minimum |
| Receipt format | JSON + protobuf |
| Receipt signing | Ed25519 signature |
| Receipt searchability | Indexed by action_id, timestamp, decision |
| Receipt integrity | Merkle tree for batch verification |
---
## 6. Testing & Validation Criteria
### DDC-6.1: Unit Test Coverage
| Module | Coverage Target | Critical Paths |
|--------|-----------------|----------------|
| conformal/ | ≥ 90% | Prediction set generation; shift adaptation |
| eprocess/ | ≥ 95% | E-value validity; supermartingale property |
| anytime_gate/ | ≥ 90% | Decision logic; receipt generation |
### DDC-6.2: Integration Test Scenarios
| Scenario | Expected Behavior |
|----------|-------------------|
| Normal operation | Permit rate > 90% |
| Gradual shift | Coverage maintained; permit rate may decrease |
| Abrupt shift | Temporary DEFER; recovery within 100 steps |
| Adversarial probe | DENY rate increases; alerts generated |
| Component failure | Graceful degradation; no unsafe permits |
### DDC-6.3: Benchmark Requirements
| Metric | Target | Measurement Method |
|--------|--------|-------------------|
| Gate latency p50 | < 10ms | Continuous profiling |
| Gate latency p99 | < 50ms | Continuous profiling |
| False deny rate | < 5% | Simulation with known-safe actions |
| Missed unsafe rate | < 0.1% | Simulation with known-unsafe actions |
| Coverage maintenance | ≥ 85% | Real distribution shift scenarios |
---
## 7. Implementation Phases
### Phase 1: Foundation (v0.1)
- [ ] E-value and e-process core implementation
- [ ] Basic conformal prediction with ACI
- [ ] Integration with existing `GateController`
- [ ] Simple witness receipts
### Phase 2: Adaptation (v0.2)
- [ ] COP and retrospective adjustment
- [ ] Mixture e-values for robustness
- [ ] Graph model with conformal-based weights
- [ ] Enhanced audit trail
### Phase 3: Production (v1.0)
- [ ] CORE RL-based adaptation
- [ ] Learned graph construction
- [ ] Cryptographic receipt signing
- [ ] Full monitoring and alerting
---
## 8. Open Questions for Review
1. **Graph Model Scope**: Should the action graph include only immediate actions or multi-step lookahead?
2. **E-Process Null**: Is "action safety" the right null hypothesis, or should we test "policy consistency"?
3. **Threshold Learning**: Should thresholds be fixed or learned via meta-optimization?
4. **Human-in-Loop**: How should DEFER decisions be presented to human operators?
5. **Adversarial Robustness**: How does AVCG perform against adaptive adversaries who observe gate decisions?
---
## 9. Sign-Off
| Role | Name | Date | Signature |
|------|------|------|-----------|
| Architecture Lead | | | |
| Security Lead | | | |
| ML Lead | | | |
| Engineering Lead | | | |
---
## Appendix A: Glossary
| Term | Definition |
|------|------------|
| **E-value** | Nonnegative test statistic with E[e] ≤ 1 under null |
| **E-process** | Sequence of e-values forming a nonnegative supermartingale |
| **Conformal Prediction** | Distribution-free method for calibrated uncertainty |
| **Witness Partition** | Explicit (S, V\S) showing which vertices are separated |
| **Anytime-Valid** | Guarantee holds at any stopping time |
| **COP** | Conformal Optimistic Prediction |
| **CORE** | Conformal Regression via Reinforcement Learning |
| **ACI** | Adaptive Conformal Inference |
## Appendix B: Key Equations
### E-Value Validity
```
E_H₀[e] ≤ 1
```
### Anytime-Valid Type I Error
```
P_H₀(∃t: E_t ≥ 1/α) ≤ α
```
### Conformal Coverage
```
P(Y_{t+1} ∈ C_t(X_{t+1})) ≥ 1 - α
```
### E-Value Composition
```
e₁ · e₂ is valid if e₁, e₂ independent
```

View File

@@ -0,0 +1,559 @@
# Implementation Roadmap: Anytime-Valid Coherence Gate
**Version**: 1.0
**Date**: 2026-01-17
**Related**: ADR-001, DDC-001
## Executive Summary
This document provides a phased implementation roadmap for the Anytime-Valid Coherence Gate (AVCG), integrating:
1. **Dynamic Min-Cut** (existing, enhanced)
2. **Online Conformal Prediction** (new)
3. **E-Values/E-Processes** (new)
The implementation is designed for incremental delivery with each phase providing standalone value.
---
## Phase 0: Preparation (Current State Analysis)
### Existing Infrastructure ✅
| Component | Location | Status |
|-----------|----------|--------|
| `SubpolynomialMinCut` | `src/subpolynomial/mod.rs` | Production-ready |
| `WitnessTree` | `src/witness/mod.rs` | Production-ready |
| `CutCertificate` | `src/certificate/mod.rs` | Production-ready |
| `DeterministicLocalKCut` | `src/localkcut/` | Production-ready |
| `GateController` | `mincut-gated-transformer/src/gate.rs` | Production-ready |
| `GatePacket` | `mincut-gated-transformer/src/packets.rs` | Production-ready |
### Dependencies to Add
```toml
# Cargo.toml additions for ruvector-mincut
[dependencies]
# Statistics
statrs = "0.17" # Statistical distributions
rand = "0.8" # Random number generation
rand_distr = "0.4" # Probability distributions
# Serialization for receipts
serde_json = "1.0"
bincode = "1.3"
blake3 = "1.5" # Fast cryptographic hashing
# Optional: async support
tokio = { version = "1", features = ["sync"], optional = true }
```
---
## Phase 1: E-Process Foundation
**Goal**: Implement core e-value and e-process infrastructure.
### Task 1.1: E-Value Module
Create `src/eprocess/evalue.rs`:
```rust
/// Core e-value type with validity guarantees
pub struct EValue {
value: f64,
/// Null hypothesis under which E[e] ≤ 1
null: NullHypothesis,
/// Computation timestamp
timestamp: u64,
}
/// Supported null hypotheses
pub enum NullHypothesis {
/// P(unsafe outcome) ≤ p0
ActionSafety { p0: f64 },
/// Current state ~ reference distribution
StateStability { reference: DistributionId },
/// Policy matches reference
PolicyConsistency { reference: PolicyId },
}
impl EValue {
/// Create from likelihood ratio
pub fn from_likelihood_ratio(
likelihood_h1: f64,
likelihood_h0: f64,
) -> Self;
/// Create mixture e-value for robustness
pub fn from_mixture(
components: &[EValue],
weights: &[f64],
) -> Self;
/// Verify E[e] ≤ 1 property empirically
pub fn verify_validity(&self, samples: &[f64]) -> bool;
}
```
### Task 1.2: E-Process Module
Create `src/eprocess/process.rs`:
```rust
/// E-process for continuous monitoring
pub struct EProcess {
/// Current accumulated value
current: f64,
/// History for audit
history: Vec<EValue>,
/// Update rule
update_rule: UpdateRule,
}
pub enum UpdateRule {
/// E_t = Π e_i (aggressive)
Product,
/// E_t = (1/t) Σ e_i (conservative)
Average,
/// E_t = λe_t + (1-λ)E_{t-1}
ExponentialMoving { lambda: f64 },
/// E_t = Σ w_j E_t^{(j)}
Mixture { weights: Vec<f64> },
}
impl EProcess {
pub fn new(rule: UpdateRule) -> Self;
pub fn update(&mut self, e: EValue);
pub fn current_value(&self) -> f64;
/// Check stopping condition
pub fn should_stop(&self, threshold: f64) -> bool;
/// Export for audit
pub fn to_evidence_receipt(&self) -> EvidenceReceipt;
}
```
### Task 1.3: Stopping Rules
Create `src/eprocess/stopping.rs`:
```rust
/// Anytime-valid stopping rule
pub struct StoppingRule {
/// Threshold for rejection
reject_threshold: f64, // typically 1/α
/// Threshold for acceptance (optional)
accept_threshold: Option<f64>,
}
impl StoppingRule {
/// Check if we can stop now
pub fn can_stop(&self, e_process: &EProcess) -> StoppingDecision;
/// Get confidence at current stopping time
pub fn confidence_at_stop(&self, e_process: &EProcess) -> f64;
}
pub enum StoppingDecision {
/// Continue accumulating evidence
Continue,
/// Reject null (evidence of incoherence)
Reject { confidence: f64 },
/// Accept null (evidence of coherence)
Accept { confidence: f64 },
}
```
### Deliverables Phase 1
- [ ] `src/eprocess/mod.rs` - module organization
- [ ] `src/eprocess/evalue.rs` - e-value implementation
- [ ] `src/eprocess/process.rs` - e-process implementation
- [ ] `src/eprocess/stopping.rs` - stopping rules
- [ ] `src/eprocess/mixture.rs` - mixture e-values
- [ ] Unit tests with ≥95% coverage
- [ ] Integration with `CutCertificate`
### Acceptance Criteria Phase 1
- [ ] E[e] ≤ 1 verified for all implemented e-value types
- [ ] E-process maintains supermartingale property
- [ ] Stopping rule provides valid Type I error control
- [ ] Computation time < 1ms for single e-value
---
## Phase 2: Conformal Prediction
**Goal**: Implement online conformal prediction with shift adaptation.
### Task 2.1: Prediction Set Core
Create `src/conformal/prediction_set.rs`:
```rust
/// Conformal prediction set
pub struct PredictionSet<T> {
/// Elements in the set
elements: Vec<T>,
/// Coverage target
coverage: f64,
/// Non-conformity scores
scores: Vec<f64>,
}
impl<T> PredictionSet<T> {
/// Check if outcome is in set
pub fn contains(&self, outcome: &T) -> bool;
/// Get set size (measure of uncertainty)
pub fn size(&self) -> usize;
/// Get normalized uncertainty measure
pub fn uncertainty(&self) -> f64;
}
```
### Task 2.2: Non-Conformity Scores
Create `src/conformal/scores.rs`:
```rust
/// Non-conformity score function
pub trait NonConformityScore {
type Input;
type Output;
fn score(&self, input: &Self::Input, output: &Self::Output) -> f64;
}
/// Absolute residual score
pub struct AbsoluteResidual<P: Predictor> {
predictor: P,
}
/// Normalized residual score
pub struct NormalizedResidual<P: Predictor + UncertaintyEstimator> {
predictor: P,
}
/// Conformalized Quantile Regression (CQR)
pub struct CQRScore<Q: QuantilePredictor> {
quantile_predictor: Q,
}
```
### Task 2.3: Online Conformal with Adaptation
Create `src/conformal/online.rs`:
```rust
/// Online conformal predictor with shift adaptation
pub struct OnlineConformal<S: NonConformityScore> {
score_fn: S,
/// Calibration buffer
calibration: RingBuffer<f64>,
/// Current quantile
quantile: f64,
/// Adaptation method
adaptation: AdaptationMethod,
}
pub enum AdaptationMethod {
/// Adaptive Conformal Inference
ACI { learning_rate: f64 },
/// Retrospective adjustment
Retrospective { window: usize },
/// Conformal Optimistic Prediction
COP { cdf_estimator: Box<dyn CDFEstimator> },
}
impl<S: NonConformityScore> OnlineConformal<S> {
/// Generate prediction set
pub fn predict(&self, input: &S::Input) -> PredictionSet<S::Output>;
/// Update with observed outcome
pub fn update(&mut self, input: &S::Input, outcome: &S::Output);
/// Get current coverage estimate
pub fn coverage_estimate(&self) -> f64;
}
```
### Task 2.4: CORE RL-Based Adaptation
Create `src/conformal/core.rs`:
```rust
/// CORE: RL-based conformal adaptation
pub struct COREConformal<S: NonConformityScore> {
base: OnlineConformal<S>,
/// RL agent for quantile adjustment
agent: QuantileAgent,
/// Coverage as reward signal
coverage_target: f64,
}
/// Simple TD-learning agent for quantile adjustment
struct QuantileAgent {
q_value: f64,
learning_rate: f64,
discount: f64,
}
impl<S: NonConformityScore> COREConformal<S> {
/// Predict with RL-adjusted quantile
pub fn predict(&self, input: &S::Input) -> PredictionSet<S::Output>;
/// Update agent and base conformal
pub fn update(&mut self, input: &S::Input, outcome: &S::Output, covered: bool);
}
```
### Deliverables Phase 2
- [ ] `src/conformal/mod.rs` - module organization
- [ ] `src/conformal/prediction_set.rs` - prediction set types
- [ ] `src/conformal/scores.rs` - non-conformity scores
- [ ] `src/conformal/online.rs` - online conformal with ACI
- [ ] `src/conformal/retrospective.rs` - retrospective adjustment
- [ ] `src/conformal/cop.rs` - Conformal Optimistic Prediction
- [ ] `src/conformal/core.rs` - RL-based adaptation
- [ ] Unit tests with ≥90% coverage
### Acceptance Criteria Phase 2
- [ ] Marginal coverage ≥ 1 - α on exchangeable data
- [ ] Coverage maintained under gradual shift (δ < 0.1/step)
- [ ] Recovery within 100 steps after abrupt shift
- [ ] Prediction latency < 10ms
---
## Phase 3: Gate Integration
**Goal**: Integrate all components into unified gate controller.
### Task 3.1: Anytime Gate Policy
Create `src/anytime_gate/policy.rs`:
```rust
/// Policy for anytime-valid gate
pub struct AnytimeGatePolicy {
/// E-process thresholds
pub e_deny_threshold: f64, // τ_deny
pub e_permit_threshold: f64, // τ_permit
/// Conformal thresholds
pub uncertainty_threshold: f64, // θ_uncertainty
pub confidence_threshold: f64, // θ_confidence
/// Min-cut thresholds (from existing GatePolicy)
pub lambda_min: u32,
pub boundary_max: u16,
/// Adaptation settings
pub adaptive_thresholds: bool,
pub threshold_learning_rate: f64,
}
```
### Task 3.2: Unified Gate Controller
Create `src/anytime_gate/controller.rs`:
```rust
/// Unified anytime-valid coherence gate
pub struct AnytimeGateController<S: NonConformityScore> {
/// Existing min-cut infrastructure
mincut: SubpolynomialMinCut,
/// Conformal predictor
conformal: OnlineConformal<S>,
/// E-process for evidence
e_process: EProcess,
/// Policy
policy: AnytimeGatePolicy,
}
impl<S: NonConformityScore> AnytimeGateController<S> {
/// Evaluate gate for action
pub fn evaluate(&mut self, action: &Action, context: &Context) -> GateResult;
/// Update after observing outcome
pub fn update(&mut self, action: &Action, outcome: &Outcome);
/// Generate witness receipt
pub fn receipt(&self, decision: &GateDecision) -> WitnessReceipt;
}
pub struct GateResult {
pub decision: GateDecision,
// From min-cut
pub cut_value: f64,
pub witness_partition: Option<WitnessPartition>,
// From conformal
pub prediction_set_size: f64,
pub uncertainty: f64,
// From e-process
pub e_value: f64,
pub evidence_sufficient: bool,
}
```
### Task 3.3: Witness Receipt
Create `src/anytime_gate/receipt.rs`:
```rust
/// Cryptographically sealed witness receipt
#[derive(Serialize, Deserialize)]
pub struct WitnessReceipt {
/// Receipt metadata
pub id: Uuid,
pub timestamp: u64,
pub action_id: ActionId,
pub decision: GateDecision,
/// Structural witness (from min-cut)
pub structural: StructuralWitness,
/// Predictive witness (from conformal)
pub predictive: PredictiveWitness,
/// Evidential witness (from e-process)
pub evidential: EvidentialWitness,
/// Cryptographic seal
pub hash: [u8; 32],
pub signature: Option<[u8; 64]>,
}
#[derive(Serialize, Deserialize)]
pub struct StructuralWitness {
pub cut_value: f64,
pub partition_hash: [u8; 32],
pub critical_edge_count: usize,
}
#[derive(Serialize, Deserialize)]
pub struct PredictiveWitness {
pub prediction_set_size: usize,
pub coverage_target: f64,
pub adaptation_rate: f64,
}
#[derive(Serialize, Deserialize)]
pub struct EvidentialWitness {
pub e_value: f64,
pub e_process_cumulative: f64,
pub null_hypothesis: String,
pub stopping_valid: bool,
}
impl WitnessReceipt {
pub fn seal(&mut self) {
self.hash = blake3::hash(&self.to_bytes()).into();
}
pub fn verify(&self) -> bool {
self.hash == blake3::hash(&self.to_bytes_without_hash()).into()
}
}
```
### Deliverables Phase 3
- [ ] `src/anytime_gate/mod.rs` - module organization
- [ ] `src/anytime_gate/policy.rs` - gate policy
- [ ] `src/anytime_gate/controller.rs` - unified controller
- [ ] `src/anytime_gate/decision.rs` - decision types
- [ ] `src/anytime_gate/receipt.rs` - witness receipts
- [ ] Integration tests with full pipeline
- [ ] Benchmarks for latency validation
### Acceptance Criteria Phase 3
- [ ] Gate latency p99 < 50ms
- [ ] All three signals integrated correctly
- [ ] Witness receipts pass verification
- [ ] Graceful degradation on component failure
---
## Phase 4: Production Hardening
**Goal**: Production-ready implementation with monitoring and optimization.
### Task 4.1: Performance Optimization
- [ ] SIMD-optimized e-value computation
- [ ] Lazy evaluation for conformal sets
- [ ] Batched graph updates for min-cut
- [ ] Memory-mapped receipt storage
### Task 4.2: Monitoring & Alerting
- [ ] Prometheus metrics for gate decisions
- [ ] Coverage drift detection
- [ ] E-process anomaly alerts
- [ ] Latency histogram tracking
### Task 4.3: Operational Tooling
- [ ] Receipt query API
- [ ] Threshold tuning dashboard
- [ ] A/B testing framework for policy comparison
- [ ] Incident replay from receipts
### Task 4.4: Documentation
- [ ] API documentation
- [ ] Operator runbook
- [ ] Threshold tuning guide
- [ ] Troubleshooting guide
---
## Timeline Summary
| Phase | Duration | Dependencies | Deliverable |
|-------|----------|--------------|-------------|
| Phase 0 | Complete | - | Requirements analysis |
| Phase 1 | 2 weeks | None | E-process module |
| Phase 2 | 3 weeks | Phase 1 | Conformal module |
| Phase 3 | 2 weeks | Phase 1, 2 | Unified gate |
| Phase 4 | 2 weeks | Phase 3 | Production hardening |
**Total estimated effort**: 9 weeks
---
## Risk Register
| Risk | Probability | Impact | Mitigation |
|------|------------|--------|------------|
| E-value power too low | Medium | High | Mixture e-values; tuned alternatives |
| Conformal sets too large | Medium | Medium | COP for tighter sets; better base predictor |
| Latency exceeds budget | Low | High | Early profiling; lazy evaluation |
| Integration complexity | Medium | Medium | Phased delivery; isolated modules |
| Threshold tuning difficulty | High | Medium | Adaptive thresholds; meta-learning |
---
## Success Metrics
| Metric | Target | Measurement |
|--------|--------|-------------|
| False deny rate | < 5% | Simulation |
| Missed unsafe rate | < 0.1% | Simulation |
| Gate latency p99 | < 50ms | Production |
| Coverage maintenance | ≥ 85% | Production |
| Receipt verification pass | 100% | Audit |
---
## References
1. El-Hayek, Henzinger, Li. arXiv:2512.13105 (Dec 2025)
2. Online Conformal with Retrospective. arXiv:2511.04275 (Nov 2025)
3. Ramdas, Wang. "Hypothesis Testing with E-values" (2025)
4. ICML 2025 Tutorial on SAVI
5. Distribution-informed Conformal (COP). arXiv:2512.07770 (Dec 2025)