Squashed 'vendor/ruvector/' content from commit b64c2172

git-subtree-dir: vendor/ruvector git-subtree-split: b64c21726f2bb37286d9ee36a7869fef60cc6900
2026-02-28 14:39:40 -05:00
commit d803bfe2b1
7854 changed files with 3522914 additions and 0 deletions
--- a/crates/ruvector-mincut/docs/adr/ROADMAP-coherence-gate-implementation.md
+++ b/crates/ruvector-mincut/docs/adr/ROADMAP-coherence-gate-implementation.md
@@ -0,0 +1,559 @@
+# Implementation Roadmap: Anytime-Valid Coherence Gate
+
+**Version**: 1.0
+**Date**: 2026-01-17
+**Related**: ADR-001, DDC-001
+
+## Executive Summary
+
+This document provides a phased implementation roadmap for the Anytime-Valid Coherence Gate (AVCG), integrating:
+1. **Dynamic Min-Cut** (existing, enhanced)
+2. **Online Conformal Prediction** (new)
+3. **E-Values/E-Processes** (new)
+
+The implementation is designed for incremental delivery with each phase providing standalone value.
+
+---
+
+## Phase 0: Preparation (Current State Analysis)
+
+### Existing Infrastructure ✅
+
+| Component | Location | Status |
+|-----------|----------|--------|
+| `SubpolynomialMinCut` | `src/subpolynomial/mod.rs` | Production-ready |
+| `WitnessTree` | `src/witness/mod.rs` | Production-ready |
+| `CutCertificate` | `src/certificate/mod.rs` | Production-ready |
+| `DeterministicLocalKCut` | `src/localkcut/` | Production-ready |
+| `GateController` | `mincut-gated-transformer/src/gate.rs` | Production-ready |
+| `GatePacket` | `mincut-gated-transformer/src/packets.rs` | Production-ready |
+
+### Dependencies to Add
+
+```toml
+# Cargo.toml additions for ruvector-mincut
+[dependencies]
+# Statistics
+statrs = "0.17"           # Statistical distributions
+rand = "0.8"              # Random number generation
+rand_distr = "0.4"        # Probability distributions
+
+# Serialization for receipts
+serde_json = "1.0"
+bincode = "1.3"
+blake3 = "1.5"            # Fast cryptographic hashing
+
+# Optional: async support
+tokio = { version = "1", features = ["sync"], optional = true }
+```
+
+---
+
+## Phase 1: E-Process Foundation
+
+**Goal**: Implement core e-value and e-process infrastructure.
+
+### Task 1.1: E-Value Module
+
+Create `src/eprocess/evalue.rs`:
+
+```rust
+/// Core e-value type with validity guarantees
+pub struct EValue {
+    value: f64,
+    /// Null hypothesis under which E[e] ≤ 1
+    null: NullHypothesis,
+    /// Computation timestamp
+    timestamp: u64,
+}
+
+/// Supported null hypotheses
+pub enum NullHypothesis {
+    /// P(unsafe outcome) ≤ p0
+    ActionSafety { p0: f64 },
+    /// Current state ~ reference distribution
+    StateStability { reference: DistributionId },
+    /// Policy matches reference
+    PolicyConsistency { reference: PolicyId },
+}
+
+impl EValue {
+    /// Create from likelihood ratio
+    pub fn from_likelihood_ratio(
+        likelihood_h1: f64,
+        likelihood_h0: f64,
+    ) -> Self;
+
+    /// Create mixture e-value for robustness
+    pub fn from_mixture(
+        components: &[EValue],
+        weights: &[f64],
+    ) -> Self;
+
+    /// Verify E[e] ≤ 1 property empirically
+    pub fn verify_validity(&self, samples: &[f64]) -> bool;
+}
+```
+
+### Task 1.2: E-Process Module
+
+Create `src/eprocess/process.rs`:
+
+```rust
+/// E-process for continuous monitoring
+pub struct EProcess {
+    /// Current accumulated value
+    current: f64,
+    /// History for audit
+    history: Vec<EValue>,
+    /// Update rule
+    update_rule: UpdateRule,
+}
+
+pub enum UpdateRule {
+    /// E_t = Π e_i (aggressive)
+    Product,
+    /// E_t = (1/t) Σ e_i (conservative)
+    Average,
+    /// E_t = λe_t + (1-λ)E_{t-1}
+    ExponentialMoving { lambda: f64 },
+    /// E_t = Σ w_j E_t^{(j)}
+    Mixture { weights: Vec<f64> },
+}
+
+impl EProcess {
+    pub fn new(rule: UpdateRule) -> Self;
+    pub fn update(&mut self, e: EValue);
+    pub fn current_value(&self) -> f64;
+
+    /// Check stopping condition
+    pub fn should_stop(&self, threshold: f64) -> bool;
+
+    /// Export for audit
+    pub fn to_evidence_receipt(&self) -> EvidenceReceipt;
+}
+```
+
+### Task 1.3: Stopping Rules
+
+Create `src/eprocess/stopping.rs`:
+
+```rust
+/// Anytime-valid stopping rule
+pub struct StoppingRule {
+    /// Threshold for rejection
+    reject_threshold: f64,  // typically 1/α
+    /// Threshold for acceptance (optional)
+    accept_threshold: Option<f64>,
+}
+
+impl StoppingRule {
+    /// Check if we can stop now
+    pub fn can_stop(&self, e_process: &EProcess) -> StoppingDecision;
+
+    /// Get confidence at current stopping time
+    pub fn confidence_at_stop(&self, e_process: &EProcess) -> f64;
+}
+
+pub enum StoppingDecision {
+    /// Continue accumulating evidence
+    Continue,
+    /// Reject null (evidence of incoherence)
+    Reject { confidence: f64 },
+    /// Accept null (evidence of coherence)
+    Accept { confidence: f64 },
+}
+```
+
+### Deliverables Phase 1
+- [ ] `src/eprocess/mod.rs` - module organization
+- [ ] `src/eprocess/evalue.rs` - e-value implementation
+- [ ] `src/eprocess/process.rs` - e-process implementation
+- [ ] `src/eprocess/stopping.rs` - stopping rules
+- [ ] `src/eprocess/mixture.rs` - mixture e-values
+- [ ] Unit tests with ≥95% coverage
+- [ ] Integration with `CutCertificate`
+
+### Acceptance Criteria Phase 1
+- [ ] E[e] ≤ 1 verified for all implemented e-value types
+- [ ] E-process maintains supermartingale property
+- [ ] Stopping rule provides valid Type I error control
+- [ ] Computation time < 1ms for single e-value
+
+---
+
+## Phase 2: Conformal Prediction
+
+**Goal**: Implement online conformal prediction with shift adaptation.
+
+### Task 2.1: Prediction Set Core
+
+Create `src/conformal/prediction_set.rs`:
+
+```rust
+/// Conformal prediction set
+pub struct PredictionSet<T> {
+    /// Elements in the set
+    elements: Vec<T>,
+    /// Coverage target
+    coverage: f64,
+    /// Non-conformity scores
+    scores: Vec<f64>,
+}
+
+impl<T> PredictionSet<T> {
+    /// Check if outcome is in set
+    pub fn contains(&self, outcome: &T) -> bool;
+
+    /// Get set size (measure of uncertainty)
+    pub fn size(&self) -> usize;
+
+    /// Get normalized uncertainty measure
+    pub fn uncertainty(&self) -> f64;
+}
+```
+
+### Task 2.2: Non-Conformity Scores
+
+Create `src/conformal/scores.rs`:
+
+```rust
+/// Non-conformity score function
+pub trait NonConformityScore {
+    type Input;
+    type Output;
+
+    fn score(&self, input: &Self::Input, output: &Self::Output) -> f64;
+}
+
+/// Absolute residual score
+pub struct AbsoluteResidual<P: Predictor> {
+    predictor: P,
+}
+
+/// Normalized residual score
+pub struct NormalizedResidual<P: Predictor + UncertaintyEstimator> {
+    predictor: P,
+}
+
+/// Conformalized Quantile Regression (CQR)
+pub struct CQRScore<Q: QuantilePredictor> {
+    quantile_predictor: Q,
+}
+```
+
+### Task 2.3: Online Conformal with Adaptation
+
+Create `src/conformal/online.rs`:
+
+```rust
+/// Online conformal predictor with shift adaptation
+pub struct OnlineConformal<S: NonConformityScore> {
+    score_fn: S,
+    /// Calibration buffer
+    calibration: RingBuffer<f64>,
+    /// Current quantile
+    quantile: f64,
+    /// Adaptation method
+    adaptation: AdaptationMethod,
+}
+
+pub enum AdaptationMethod {
+    /// Adaptive Conformal Inference
+    ACI { learning_rate: f64 },
+    /// Retrospective adjustment
+    Retrospective { window: usize },
+    /// Conformal Optimistic Prediction
+    COP { cdf_estimator: Box<dyn CDFEstimator> },
+}
+
+impl<S: NonConformityScore> OnlineConformal<S> {
+    /// Generate prediction set
+    pub fn predict(&self, input: &S::Input) -> PredictionSet<S::Output>;
+
+    /// Update with observed outcome
+    pub fn update(&mut self, input: &S::Input, outcome: &S::Output);
+
+    /// Get current coverage estimate
+    pub fn coverage_estimate(&self) -> f64;
+}
+```
+
+### Task 2.4: CORE RL-Based Adaptation
+
+Create `src/conformal/core.rs`:
+
+```rust
+/// CORE: RL-based conformal adaptation
+pub struct COREConformal<S: NonConformityScore> {
+    base: OnlineConformal<S>,
+    /// RL agent for quantile adjustment
+    agent: QuantileAgent,
+    /// Coverage as reward signal
+    coverage_target: f64,
+}
+
+/// Simple TD-learning agent for quantile adjustment
+struct QuantileAgent {
+    q_value: f64,
+    learning_rate: f64,
+    discount: f64,
+}
+
+impl<S: NonConformityScore> COREConformal<S> {
+    /// Predict with RL-adjusted quantile
+    pub fn predict(&self, input: &S::Input) -> PredictionSet<S::Output>;
+
+    /// Update agent and base conformal
+    pub fn update(&mut self, input: &S::Input, outcome: &S::Output, covered: bool);
+}
+```
+
+### Deliverables Phase 2
+- [ ] `src/conformal/mod.rs` - module organization
+- [ ] `src/conformal/prediction_set.rs` - prediction set types
+- [ ] `src/conformal/scores.rs` - non-conformity scores
+- [ ] `src/conformal/online.rs` - online conformal with ACI
+- [ ] `src/conformal/retrospective.rs` - retrospective adjustment
+- [ ] `src/conformal/cop.rs` - Conformal Optimistic Prediction
+- [ ] `src/conformal/core.rs` - RL-based adaptation
+- [ ] Unit tests with ≥90% coverage
+
+### Acceptance Criteria Phase 2
+- [ ] Marginal coverage ≥ 1 - α on exchangeable data
+- [ ] Coverage maintained under gradual shift (δ < 0.1/step)
+- [ ] Recovery within 100 steps after abrupt shift
+- [ ] Prediction latency < 10ms
+
+---
+
+## Phase 3: Gate Integration
+
+**Goal**: Integrate all components into unified gate controller.
+
+### Task 3.1: Anytime Gate Policy
+
+Create `src/anytime_gate/policy.rs`:
+
+```rust
+/// Policy for anytime-valid gate
+pub struct AnytimeGatePolicy {
+    /// E-process thresholds
+    pub e_deny_threshold: f64,      // τ_deny
+    pub e_permit_threshold: f64,    // τ_permit
+
+    /// Conformal thresholds
+    pub uncertainty_threshold: f64,  // θ_uncertainty
+    pub confidence_threshold: f64,   // θ_confidence
+
+    /// Min-cut thresholds (from existing GatePolicy)
+    pub lambda_min: u32,
+    pub boundary_max: u16,
+
+    /// Adaptation settings
+    pub adaptive_thresholds: bool,
+    pub threshold_learning_rate: f64,
+}
+```
+
+### Task 3.2: Unified Gate Controller
+
+Create `src/anytime_gate/controller.rs`:
+
+```rust
+/// Unified anytime-valid coherence gate
+pub struct AnytimeGateController<S: NonConformityScore> {
+    /// Existing min-cut infrastructure
+    mincut: SubpolynomialMinCut,
+
+    /// Conformal predictor
+    conformal: OnlineConformal<S>,
+
+    /// E-process for evidence
+    e_process: EProcess,
+
+    /// Policy
+    policy: AnytimeGatePolicy,
+}
+
+impl<S: NonConformityScore> AnytimeGateController<S> {
+    /// Evaluate gate for action
+    pub fn evaluate(&mut self, action: &Action, context: &Context) -> GateResult;
+
+    /// Update after observing outcome
+    pub fn update(&mut self, action: &Action, outcome: &Outcome);
+
+    /// Generate witness receipt
+    pub fn receipt(&self, decision: &GateDecision) -> WitnessReceipt;
+}
+
+pub struct GateResult {
+    pub decision: GateDecision,
+
+    // From min-cut
+    pub cut_value: f64,
+    pub witness_partition: Option<WitnessPartition>,
+
+    // From conformal
+    pub prediction_set_size: f64,
+    pub uncertainty: f64,
+
+    // From e-process
+    pub e_value: f64,
+    pub evidence_sufficient: bool,
+}
+```
+
+### Task 3.3: Witness Receipt
+
+Create `src/anytime_gate/receipt.rs`:
+
+```rust
+/// Cryptographically sealed witness receipt
+#[derive(Serialize, Deserialize)]
+pub struct WitnessReceipt {
+    /// Receipt metadata
+    pub id: Uuid,
+    pub timestamp: u64,
+    pub action_id: ActionId,
+    pub decision: GateDecision,
+
+    /// Structural witness (from min-cut)
+    pub structural: StructuralWitness,
+
+    /// Predictive witness (from conformal)
+    pub predictive: PredictiveWitness,
+
+    /// Evidential witness (from e-process)
+    pub evidential: EvidentialWitness,
+
+    /// Cryptographic seal
+    pub hash: [u8; 32],
+    pub signature: Option<[u8; 64]>,
+}
+
+#[derive(Serialize, Deserialize)]
+pub struct StructuralWitness {
+    pub cut_value: f64,
+    pub partition_hash: [u8; 32],
+    pub critical_edge_count: usize,
+}
+
+#[derive(Serialize, Deserialize)]
+pub struct PredictiveWitness {
+    pub prediction_set_size: usize,
+    pub coverage_target: f64,
+    pub adaptation_rate: f64,
+}
+
+#[derive(Serialize, Deserialize)]
+pub struct EvidentialWitness {
+    pub e_value: f64,
+    pub e_process_cumulative: f64,
+    pub null_hypothesis: String,
+    pub stopping_valid: bool,
+}
+
+impl WitnessReceipt {
+    pub fn seal(&mut self) {
+        self.hash = blake3::hash(&self.to_bytes()).into();
+    }
+
+    pub fn verify(&self) -> bool {
+        self.hash == blake3::hash(&self.to_bytes_without_hash()).into()
+    }
+}
+```
+
+### Deliverables Phase 3
+- [ ] `src/anytime_gate/mod.rs` - module organization
+- [ ] `src/anytime_gate/policy.rs` - gate policy
+- [ ] `src/anytime_gate/controller.rs` - unified controller
+- [ ] `src/anytime_gate/decision.rs` - decision types
+- [ ] `src/anytime_gate/receipt.rs` - witness receipts
+- [ ] Integration tests with full pipeline
+- [ ] Benchmarks for latency validation
+
+### Acceptance Criteria Phase 3
+- [ ] Gate latency p99 < 50ms
+- [ ] All three signals integrated correctly
+- [ ] Witness receipts pass verification
+- [ ] Graceful degradation on component failure
+
+---
+
+## Phase 4: Production Hardening
+
+**Goal**: Production-ready implementation with monitoring and optimization.
+
+### Task 4.1: Performance Optimization
+- [ ] SIMD-optimized e-value computation
+- [ ] Lazy evaluation for conformal sets
+- [ ] Batched graph updates for min-cut
+- [ ] Memory-mapped receipt storage
+
+### Task 4.2: Monitoring & Alerting
+- [ ] Prometheus metrics for gate decisions
+- [ ] Coverage drift detection
+- [ ] E-process anomaly alerts
+- [ ] Latency histogram tracking
+
+### Task 4.3: Operational Tooling
+- [ ] Receipt query API
+- [ ] Threshold tuning dashboard
+- [ ] A/B testing framework for policy comparison
+- [ ] Incident replay from receipts
+
+### Task 4.4: Documentation
+- [ ] API documentation
+- [ ] Operator runbook
+- [ ] Threshold tuning guide
+- [ ] Troubleshooting guide
+
+---
+
+## Timeline Summary
+
+| Phase | Duration | Dependencies | Deliverable |
+|-------|----------|--------------|-------------|
+| Phase 0 | Complete | - | Requirements analysis |
+| Phase 1 | 2 weeks | None | E-process module |
+| Phase 2 | 3 weeks | Phase 1 | Conformal module |
+| Phase 3 | 2 weeks | Phase 1, 2 | Unified gate |
+| Phase 4 | 2 weeks | Phase 3 | Production hardening |
+
+**Total estimated effort**: 9 weeks
+
+---
+
+## Risk Register
+
+| Risk | Probability | Impact | Mitigation |
+|------|------------|--------|------------|
+| E-value power too low | Medium | High | Mixture e-values; tuned alternatives |
+| Conformal sets too large | Medium | Medium | COP for tighter sets; better base predictor |
+| Latency exceeds budget | Low | High | Early profiling; lazy evaluation |
+| Integration complexity | Medium | Medium | Phased delivery; isolated modules |
+| Threshold tuning difficulty | High | Medium | Adaptive thresholds; meta-learning |
+
+---
+
+## Success Metrics
+
+| Metric | Target | Measurement |
+|--------|--------|-------------|
+| False deny rate | < 5% | Simulation |
+| Missed unsafe rate | < 0.1% | Simulation |
+| Gate latency p99 | < 50ms | Production |
+| Coverage maintenance | ≥ 85% | Production |
+| Receipt verification pass | 100% | Audit |
+
+---
+
+## References
+
+1. El-Hayek, Henzinger, Li. arXiv:2512.13105 (Dec 2025)
+2. Online Conformal with Retrospective. arXiv:2511.04275 (Nov 2025)
+3. Ramdas, Wang. "Hypothesis Testing with E-values" (2025)
+4. ICML 2025 Tutorial on SAVI
+5. Distribution-informed Conformal (COP). arXiv:2512.07770 (Dec 2025)