d803bfe2b1
git-subtree-dir: vendor/ruvector git-subtree-split: b64c21726f2bb37286d9ee36a7869fef60cc6900
493 lines
18 KiB
Markdown
493 lines
18 KiB
Markdown
# RuVector DAG - Neural Self-Learning DAG
|
|
|
|
**Make your queries faster automatically.** RuVector DAG learns from every query execution and continuously optimizes performance—no manual tuning required.
|
|
|
|
## What is This?
|
|
|
|
RuVector DAG is a **self-learning query optimization system**. Think of it as a "nervous system" for your database queries that:
|
|
|
|
1. **Watches** how queries execute and identifies bottlenecks
|
|
2. **Learns** which optimization strategies work best for different query patterns
|
|
3. **Adapts** in real-time, switching strategies when conditions change
|
|
4. **Heals** itself by detecting anomalies and fixing problems before they impact users
|
|
|
|
Unlike traditional query optimizers that use static rules, RuVector DAG learns from actual execution patterns and gets smarter over time.
|
|
|
|
## Who Should Use This?
|
|
|
|
| Use Case | Why RuVector DAG Helps |
|
|
|----------|------------------------|
|
|
| **Vector Search Applications** | Optimize similarity searches that traditional databases struggle with |
|
|
| **High-Traffic APIs** | Automatically adapt to changing query patterns throughout the day |
|
|
| **Real-Time Analytics** | Learn which aggregation paths are fastest for your specific data |
|
|
| **Edge/Embedded Systems** | 58KB WASM build runs in browsers and IoT devices |
|
|
| **Multi-Tenant Platforms** | Learn per-tenant query patterns without manual per-tenant tuning |
|
|
|
|
## Key Benefits
|
|
|
|
### Automatic Performance Improvement
|
|
Queries get faster over time without any code changes. In benchmarks, repeated queries show **50-80% latency reduction** after the system learns optimal execution paths.
|
|
|
|
### Zero-Downtime Adaptation
|
|
When query patterns change (new features, traffic spikes, data growth), the system adapts automatically. No need to rebuild indexes or rewrite queries.
|
|
|
|
### Predictive Problem Prevention
|
|
The system detects rising "tension" (early warning signs of bottlenecks) and intervenes *before* users experience slowdowns.
|
|
|
|
### Works Everywhere
|
|
- **PostgreSQL** via the ruvector-postgres extension
|
|
- **Browsers** via 58KB WASM module
|
|
- **Embedded systems** with minimal memory footprint
|
|
- **Distributed systems** with quantum-resistant sync between nodes
|
|
|
|
## How It Works (Simple Version)
|
|
|
|
```
|
|
Query comes in → DAG analyzes execution plan → Best attention mechanism selected
|
|
↓
|
|
Query executes → Results returned → Learning system records what worked
|
|
↓
|
|
Next similar query benefits from learned optimizations
|
|
```
|
|
|
|
The system maintains a "MinCut tension" score that acts as a health indicator. When tension rises, the system automatically switches to more aggressive optimization strategies and triggers predictive healing.
|
|
|
|
## Features
|
|
|
|
- **7 DAG Attention Mechanisms**: Topological, Causal Cone, Critical Path, MinCut Gated, Hierarchical Lorentz, Parallel Branch, Temporal BTSP
|
|
- **SONA Learning**: Self-Optimizing Neural Architecture with MicroLoRA adaptation (<100μs)
|
|
- **Subpolynomial MinCut**: O(n^0.12) bottleneck detection—the coherence boundary everything listens to
|
|
- **Self-Healing**: Autonomous anomaly detection, reactive repair, and predictive intervention
|
|
- **QuDAG Integration**: Quantum-resistant distributed pattern learning with bounded sync
|
|
- **WASM Target**: 58KB gzipped for browser and embedded systems
|
|
|
|
## Design Philosophy
|
|
|
|
MinCut is not an optimization trick here. It is the coherence boundary that everything else listens to. Attention mechanisms, SONA learning, and self-healing all respond to MinCut stress signals—creating a unified nervous system for query optimization.
|
|
|
|
## Quick Start
|
|
|
|
```rust
|
|
use ruvector_dag::{QueryDag, OperatorNode, OperatorType};
|
|
use ruvector_dag::attention::{TopologicalAttention, DagAttention};
|
|
|
|
// Build a query DAG
|
|
let mut dag = QueryDag::new();
|
|
let scan = dag.add_node(OperatorNode::hnsw_scan(0, "vectors_idx", 64));
|
|
let filter = dag.add_node(OperatorNode::filter(1, "score > 0.5"));
|
|
let result = dag.add_node(OperatorNode::new(2, OperatorType::Result));
|
|
|
|
dag.add_edge(scan, filter).unwrap();
|
|
dag.add_edge(filter, result).unwrap();
|
|
|
|
// Compute attention scores
|
|
let attention = TopologicalAttention::new(Default::default());
|
|
let scores = attention.forward(&dag).unwrap();
|
|
```
|
|
|
|
## Modules
|
|
|
|
- `dag` - Core DAG data structures and algorithms
|
|
- `attention` - 7 attention mechanisms + policy-driven selection
|
|
- `sona` - Self-Optimizing Neural Architecture with adaptive learning
|
|
- `mincut` - Subpolynomial bottleneck detection (the central control signal)
|
|
- `healing` - Reactive + predictive self-healing
|
|
- `qudag` - QuDAG network integration with bounded sync frequency
|
|
|
|
## Core Components
|
|
|
|
### DAG (Directed Acyclic Graph)
|
|
|
|
The `QueryDag` structure represents query execution plans as directed acyclic graphs. Each node represents an operator (scan, filter, join, etc.) and edges represent data flow.
|
|
|
|
```rust
|
|
use ruvector_dag::{QueryDag, OperatorNode, OperatorType};
|
|
|
|
let mut dag = QueryDag::new();
|
|
let scan = dag.add_node(OperatorNode::seq_scan(0, "users"));
|
|
let filter = dag.add_node(OperatorNode::filter(1, "age > 18"));
|
|
dag.add_edge(scan, filter).unwrap();
|
|
```
|
|
|
|
### Attention Mechanisms + Policy Layer
|
|
|
|
Seven attention mechanisms with dynamic policy-driven selection:
|
|
|
|
| Mechanism | When to Use | Trigger |
|
|
|-----------|-------------|---------|
|
|
| Topological | Default baseline | Low variance |
|
|
| Causal Cone | Downstream impact analysis | Write-heavy patterns |
|
|
| Critical Path | Latency-bound queries | p99 > 2x p50 |
|
|
| MinCut Gated | Bottleneck-aware weighting | Cut tension rising |
|
|
| Hierarchical Lorentz | Deep hierarchical queries | Depth > 10 |
|
|
| Parallel Branch | Wide parallel execution | Branch count > 3 |
|
|
| Temporal BTSP | Time-series workloads | Temporal patterns |
|
|
|
|
```rust
|
|
use ruvector_dag::attention::{AttentionSelector, SelectionPolicy};
|
|
use ruvector_dag::mincut::DagMinCutEngine;
|
|
|
|
// Policy-driven attention selection based on MinCut stress
|
|
let mut selector = AttentionSelector::new();
|
|
let mut mincut = DagMinCutEngine::new(Default::default());
|
|
|
|
// Dynamic switching based on cut tension
|
|
let analysis = mincut.analyze_bottlenecks(&dag)?;
|
|
let policy = if analysis.max_tension > 0.7 {
|
|
SelectionPolicy::MinCutGated // High stress: gate by flow
|
|
} else if analysis.latency_variance > 2.0 {
|
|
SelectionPolicy::CriticalPath // Variance: focus on bottlenecks
|
|
} else {
|
|
SelectionPolicy::Topological // Stable: use position-based
|
|
};
|
|
|
|
let scores = selector.select_and_apply(policy, &dag)?;
|
|
```
|
|
|
|
### SONA (Self-Optimizing Neural Architecture)
|
|
|
|
Adaptive learning with explicit data structures. SONA runs post-query in background, never blocking execution.
|
|
|
|
**State Vector Structure:**
|
|
```rust
|
|
/// SONA maintains per-DAG-pattern state vectors
|
|
pub struct SonaState {
|
|
/// Base embedding: pattern signature (256-dim)
|
|
pub embedding: [f32; 256],
|
|
|
|
/// MicroLoRA weights: scoped per operator type
|
|
/// Shape: [num_operator_types, rank, rank] where rank=2
|
|
pub lora_weights: HashMap<OperatorType, [[f32; 2]; 2]>,
|
|
|
|
/// Trajectory statistics for this pattern
|
|
pub trajectory_stats: TrajectoryStats,
|
|
}
|
|
|
|
pub struct TrajectoryStats {
|
|
pub count: u64,
|
|
pub mean_improvement: f32, // vs baseline
|
|
pub variance: f32,
|
|
pub best_mechanism: AttentionType,
|
|
}
|
|
```
|
|
|
|
```rust
|
|
use ruvector_dag::sona::{DagSonaEngine, SonaConfig};
|
|
|
|
let config = SonaConfig {
|
|
embedding_dim: 256,
|
|
lora_rank: 2, // Rank-2 for <100μs updates
|
|
ewc_lambda: 5000.0, // Catastrophic forgetting prevention
|
|
trajectory_capacity: 10_000,
|
|
};
|
|
let mut sona = DagSonaEngine::new(config);
|
|
|
|
// Pre-query: Get enhanced embedding (fast path)
|
|
let enhanced = sona.pre_query(&dag);
|
|
|
|
// Execute query... (SONA doesn't block here)
|
|
let execution_time = execute_query(&dag);
|
|
|
|
// Post-query: Record trajectory (async, background)
|
|
sona.post_query(&dag, execution_time, baseline_time, "topological");
|
|
|
|
// Background learning (runs in separate thread)
|
|
sona.background_learn(); // Updates LoRA weights, EWC consolidation
|
|
```
|
|
|
|
### MinCut Optimization (Central Control Signal)
|
|
|
|
The MinCut engine is the coherence boundary. Rising cut tension triggers attention switching, SONA re-weighting, and predictive healing.
|
|
|
|
```rust
|
|
use ruvector_dag::mincut::{DagMinCutEngine, MinCutConfig};
|
|
|
|
let mut engine = DagMinCutEngine::new(MinCutConfig {
|
|
update_complexity: 0.12, // O(n^0.12) amortized
|
|
tension_threshold: 0.7,
|
|
emit_signals: true, // Broadcast to other subsystems
|
|
});
|
|
|
|
let analysis = engine.analyze_bottlenecks(&dag)?;
|
|
|
|
// Tension signal drives the whole system
|
|
if analysis.max_tension > 0.7 {
|
|
// High tension: trigger predictive healing
|
|
healing.predict_and_prepare(&analysis);
|
|
|
|
// Switch attention to MinCut-aware mechanism
|
|
selector.force_mechanism(AttentionType::MinCutGated);
|
|
|
|
// Accelerate SONA learning for this pattern
|
|
sona.boost_learning_rate(2.0);
|
|
}
|
|
|
|
for bottleneck in &analysis.bottlenecks {
|
|
println!("Bottleneck at nodes {:?}: capacity {}, tension {}",
|
|
bottleneck.cut_nodes, bottleneck.capacity, bottleneck.tension);
|
|
}
|
|
```
|
|
|
|
### Self-Healing (Reactive + Predictive)
|
|
|
|
Self-healing responds to anomalies (reactive) and rising MinCut tension (predictive).
|
|
|
|
```rust
|
|
use ruvector_dag::healing::{HealingOrchestrator, AnomalyConfig, PredictiveConfig};
|
|
|
|
let mut orchestrator = HealingOrchestrator::new();
|
|
|
|
// Reactive: Z-score anomaly detection
|
|
orchestrator.add_detector("query_latency", AnomalyConfig {
|
|
z_threshold: 3.0,
|
|
window_size: 100,
|
|
min_samples: 10,
|
|
});
|
|
|
|
// Predictive: Rising cut tension triggers early intervention
|
|
orchestrator.enable_predictive(PredictiveConfig {
|
|
tension_threshold: 0.6, // Intervene before 0.7 crisis
|
|
variance_threshold: 1.5, // Rising variance = trouble coming
|
|
lookahead_window: 50, // Predict 50 queries ahead
|
|
});
|
|
|
|
// Observe metrics
|
|
orchestrator.observe("query_latency", latency);
|
|
orchestrator.observe_mincut(&mincut_analysis);
|
|
|
|
// Healing cycle: reactive + predictive
|
|
let result = orchestrator.run_cycle();
|
|
println!("Reactive repairs: {}, Predictive interventions: {}",
|
|
result.reactive_repairs, result.predictive_interventions);
|
|
```
|
|
|
|
### External Cost Model Trait
|
|
|
|
Plug in cost models for PostgreSQL, embedded, or chip-level schedulers without forking logic.
|
|
|
|
```rust
|
|
/// Trait for external cost estimation
|
|
pub trait CostModel: Send + Sync {
|
|
/// Estimate execution cost for an operator
|
|
fn estimate_cost(&self, op: &OperatorNode, context: &CostContext) -> f64;
|
|
|
|
/// Estimate cardinality (row count) for an operator
|
|
fn estimate_cardinality(&self, op: &OperatorNode, context: &CostContext) -> u64;
|
|
|
|
/// Platform-specific overhead factor
|
|
fn platform_overhead(&self) -> f64 { 1.0 }
|
|
}
|
|
|
|
/// PostgreSQL cost model (uses pg_catalog statistics)
|
|
pub struct PostgresCostModel { /* ... */ }
|
|
|
|
/// Embedded systems cost model (memory-bound)
|
|
pub struct EmbeddedCostModel {
|
|
pub ram_kb: u32,
|
|
pub flash_latency_ns: u32,
|
|
}
|
|
|
|
/// Chip-level cost model (cycle-accurate)
|
|
pub struct ChipCostModel {
|
|
pub clock_mhz: u32,
|
|
pub pipeline_depth: u8,
|
|
pub cache_line_bytes: u8,
|
|
}
|
|
|
|
// Plug into DAG analysis
|
|
let mut dag = QueryDag::with_cost_model(Box::new(EmbeddedCostModel {
|
|
ram_kb: 512,
|
|
flash_latency_ns: 100,
|
|
}));
|
|
```
|
|
|
|
### QuDAG Integration (Bounded Sync)
|
|
|
|
Quantum-resistant distributed learning with explicit sync frequency bounds.
|
|
|
|
```rust
|
|
use ruvector_dag::qudag::{QuDagClient, SyncConfig};
|
|
|
|
let client = QuDagClient::new(SyncConfig {
|
|
// Sync frequency bounds (critical for distributed scale)
|
|
min_sync_interval: Duration::from_secs(60), // At least 1 min apart
|
|
max_sync_interval: Duration::from_secs(3600), // At most 1 hour
|
|
adaptive_backoff: true, // Backoff under network pressure
|
|
|
|
// Batch settings
|
|
max_patterns_per_sync: 100,
|
|
pattern_age_threshold: Duration::from_secs(300), // 5 min maturity
|
|
|
|
// Privacy
|
|
differential_privacy_epsilon: 0.1,
|
|
noise_mechanism: NoiseMechanism::Laplace,
|
|
});
|
|
|
|
// Sync only mature, validated patterns
|
|
client.sync_patterns(
|
|
sona.get_mature_patterns(),
|
|
&crypto_identity,
|
|
).await?;
|
|
|
|
// Receive network-learned patterns (also bounded)
|
|
let network_patterns = client.receive_patterns().await?;
|
|
sona.merge_network_patterns(network_patterns);
|
|
```
|
|
|
|
## End-to-End Example: Query Convergence
|
|
|
|
A slow query converges over several runs. One file, no prose, just logs.
|
|
|
|
```text
|
|
$ cargo run --example convergence_demo
|
|
|
|
[run 1] query: SELECT * FROM vectors WHERE embedding <-> $1 < 0.5
|
|
dag: 4 nodes, 3 edges
|
|
attention: topological (default)
|
|
mincut_tension: 0.23
|
|
latency: 847ms (baseline: 850ms, improvement: 0.4%)
|
|
sona: recorded trajectory, pattern_id=0x7a3f
|
|
|
|
[run 2] same query, different params
|
|
attention: topological
|
|
mincut_tension: 0.31 (rising)
|
|
latency: 812ms (improvement: 4.5%)
|
|
sona: pattern match, applying lora_weights
|
|
|
|
[run 3]
|
|
attention: topological
|
|
mincut_tension: 0.58 (approaching threshold)
|
|
latency: 623ms (improvement: 26.7%)
|
|
sona: lora adaptation complete, ewc consolidating
|
|
|
|
[run 4]
|
|
mincut_tension: 0.71 > 0.7 (THRESHOLD)
|
|
--> switching attention: topological -> mincut_gated
|
|
--> healing: predictive intervention queued
|
|
attention: mincut_gated
|
|
latency: 412ms (improvement: 51.5%)
|
|
sona: boosting learning rate 2x for this pattern
|
|
|
|
[run 5]
|
|
attention: mincut_gated (sticky after tension spike)
|
|
mincut_tension: 0.45 (stabilizing)
|
|
latency: 398ms (improvement: 53.2%)
|
|
healing: predictive reindex completed in background
|
|
|
|
[run 10]
|
|
attention: mincut_gated
|
|
mincut_tension: 0.22 (stable)
|
|
latency: 156ms (improvement: 81.6%)
|
|
sona: pattern mature, queued for qudag sync
|
|
|
|
[qudag sync] pattern 0x7a3f synced to network
|
|
peers learning from our optimization
|
|
```
|
|
|
|
## Examples
|
|
|
|
The `examples/` directory contains:
|
|
|
|
- `basic_usage.rs` - DAG creation and basic operations
|
|
- `attention_selection.rs` - Policy-driven attention switching
|
|
- `learning_workflow.rs` - SONA learning with explicit state vectors
|
|
- `self_healing.rs` - Reactive and predictive healing
|
|
- `convergence_demo.rs` - End-to-end query convergence logs
|
|
|
|
```bash
|
|
cargo run --example basic_usage
|
|
cargo run --example attention_selection
|
|
cargo run --example learning_workflow
|
|
cargo run --example self_healing
|
|
```
|
|
|
|
## WASM Target
|
|
|
|
Minimal WASM build for browser and embedded systems.
|
|
|
|
| Metric | Value |
|
|
|--------|-------|
|
|
| Raw size | 130 KB |
|
|
| Gzipped | 58 KB |
|
|
| API surface | 13 methods |
|
|
|
|
```bash
|
|
# Build WASM
|
|
wasm-pack build crates/ruvector-dag-wasm --target web --release
|
|
|
|
# With wee_alloc for even smaller size
|
|
wasm-pack build crates/ruvector-dag-wasm --target web --release -- --features wee_alloc
|
|
```
|
|
|
|
## Performance Targets
|
|
|
|
| Component | Target | Notes |
|
|
|-----------|--------|-------|
|
|
| Attention (100 nodes) | <100μs | All 7 mechanisms |
|
|
| MicroLoRA adaptation | <100μs | Rank-2, per-operator |
|
|
| Pattern search (10K) | <2ms | K-means++ indexing |
|
|
| MinCut update | O(n^0.12) | Subpolynomial amortized |
|
|
| Anomaly detection | <50μs | Z-score, streaming |
|
|
| Predictive healing | <1ms | Tension-based lookahead |
|
|
| QuDAG sync | Bounded | 1min-1hr adaptive |
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Query DAG Layer │
|
|
│ (Operators, Edges, Topological Sort) │
|
|
│ + External Cost Model Trait │
|
|
└───────────────────────────┬─────────────────────────────────┘
|
|
│
|
|
┌─────────────┴─────────────┐
|
|
│ │
|
|
┌──────────▼──────────┐ ┌─────────▼─────────┐
|
|
│ Attention Layer │ │ MinCut Engine │
|
|
│ (7 mechanisms) │◄────│ (Control Signal) │
|
|
│ + Policy Selector │ │ O(n^0.12) │
|
|
└──────────┬──────────┘ └─────────┬─────────┘
|
|
│ │
|
|
│ ┌─────────────────────┤
|
|
│ │ │
|
|
┌──────────▼────▼─────┐ ┌─────────▼─────────┐
|
|
│ SONA Engine │ │ Self-Healing │
|
|
│ (Post-Query Learn) │ │ (Reactive + Pred) │
|
|
│ MicroLoRA + EWC │ │ Tension-Driven │
|
|
└──────────┬──────────┘ └─────────┬─────────┘
|
|
│ │
|
|
└────────────┬────────────┘
|
|
│
|
|
┌────────────▼────────────┐
|
|
│ QuDAG Sync Layer │
|
|
│ (Bounded Frequency) │
|
|
│ ML-KEM + Differential │
|
|
└─────────────────────────┘
|
|
```
|
|
|
|
## Development
|
|
|
|
```bash
|
|
# Run tests
|
|
cargo test -p ruvector-dag
|
|
|
|
# Run benchmarks
|
|
cargo bench -p ruvector-dag
|
|
|
|
# Check documentation
|
|
cargo doc -p ruvector-dag --open
|
|
```
|
|
|
|
## Integration with RuVector
|
|
|
|
This crate is part of the RuVector ecosystem:
|
|
|
|
- `ruvector-core` - Core vector operations
|
|
- `ruvector-dag-wasm` - Browser/embedded WASM target (58KB gzipped)
|
|
- `ruvector-postgres` - PostgreSQL extension with 50+ SQL functions
|
|
- `ruvector-qudag` - Full QuDAG consensus client
|
|
|
|
## License
|
|
|
|
Apache-2.0 OR MIT
|