Squashed 'vendor/ruvector/' content from commit b64c2172

git-subtree-dir: vendor/ruvector git-subtree-split: b64c21726f2bb37286d9ee36a7869fef60cc6900
2026-02-28 14:39:40 -05:00
commit d803bfe2b1
7854 changed files with 3522914 additions and 0 deletions
--- a/examples/data/framework/docs/cut_aware_hnsw.md
+++ b/examples/data/framework/docs/cut_aware_hnsw.md
@@ -0,0 +1,370 @@
+# Cut-Aware HNSW: Dynamic Min-Cut Integration with Vector Search
+
+## Overview
+
+`cut_aware_hnsw.rs` implements a coherence-aware extension to HNSW (Hierarchical Navigable Small World) graphs that respects semantic boundaries in vector spaces. Traditional HNSW blindly follows similarity edges during search. Cut-aware HNSW adds "coherence gates" that halt expansion at weak cuts, keeping searches within semantically coherent regions.
+
+## Architecture
+
+### Core Components
+
+1. **DynamicCutWatcher** - Tracks minimum cuts and graph coherence
+   - Implements Stoer-Wagner algorithm for global min-cut
+   - Incremental updates with caching for efficiency
+   - Identifies boundary edges crossing partitions
+
+2. **CutAwareHNSW** - Extended HNSW with coherence gating
+   - Wraps standard HNSW index
+   - Maintains cut watcher for edge weights
+   - Supports both gated and ungated search modes
+
+3. **CoherenceZone** - Regions of strong internal connectivity
+   - Computed from min-cut partitions
+   - Tracked with coherence ratios
+   - Used for zone-aware queries
+
+## Key Features
+
+### 1. Coherence-Gated Search
+
+```rust
+let config = CutAwareConfig {
+    coherence_gate_threshold: 0.3,  // Cuts below this are "weak"
+    max_cross_cut_hops: 2,           // Max boundary crossings
+    ..Default::default()
+};
+
+let mut index = CutAwareHNSW::new(config);
+
+// Insert vectors
+index.insert(node_id, &vector)?;
+
+// Gated search (respects boundaries)
+let gated_results = index.search_gated(&query, k);
+
+// Ungated search (baseline)
+let ungated_results = index.search_ungated(&query, k);
+```
+
+**Gated Search** will:
+- Track cut crossings for each result
+- Gate expansion at weak cuts (below threshold)
+- Return coherence scores (1.0 = no cuts crossed)
+- Prune expansions exceeding max_cross_cut_hops
+
+### 2. Coherent Neighborhoods
+
+Find all nodes reachable without crossing weak cuts:
+
+```rust
+let neighbors = index.coherent_neighborhood(node_id, radius);
+// Returns nodes within `radius` hops that don't cross weak cuts
+```
+
+### 3. Zone-Based Queries
+
+Partition the graph into coherence zones and query specific regions:
+
+```rust
+// Compute zones
+let zones = index.compute_zones();
+
+// Search within specific zones
+let results = index.cross_zone_search(&query, k, &[zone_0, zone_1]);
+```
+
+### 4. Dynamic Updates
+
+Efficiently handle graph changes with incremental cut recomputation:
+
+```rust
+// Single edge update
+index.add_edge(u, v, weight);
+index.remove_edge(u, v);
+
+// Batch updates
+let updates = vec![
+    EdgeUpdate { kind: UpdateKind::Insert, u: 0, v: 1, weight: Some(0.8) },
+    EdgeUpdate { kind: UpdateKind::Delete, u: 2, v: 3, weight: None },
+];
+let stats = index.batch_update(updates);
+```
+
+### 5. Cut Pruning
+
+Remove weak edges to improve coherence:
+
+```rust
+let pruned_count = index.prune_weak_edges(threshold);
+```
+
+## Performance Characteristics
+
+### Time Complexity
+
+| Operation | Complexity | Notes |
+|-----------|-----------|-------|
+| Insert | O(log n × M) | Same as HNSW |
+| Search (ungated) | O(log n) | Same as HNSW |
+| Search (gated) | O(log n) | Plus gate checks |
+| Min-cut | O(n³) | Stoer-Wagner, cached |
+| Zone computation | O(n²) | Periodic recomputation |
+
+### Space Complexity
+
+- **Base HNSW**: O(n × M × L) where L is layer count
+- **Cut tracking**: O(n²) for adjacency (sparse in practice)
+- **Total**: O(n × M × L + e) where e is edge count
+
+### Optimizations
+
+1. **Cached Min-Cut**: Recomputes only when graph changes
+2. **Incremental Updates**: Version-tracked cache invalidation
+3. **Sparse Adjacency**: HashMap-based for efficiency
+4. **Periodic Recomputation**: Configurable via `cut_recompute_interval`
+
+## Use Cases
+
+### 1. Multi-Domain Discovery
+
+Search within specific research domains without crossing into others:
+
+```rust
+// Climate papers in one cluster, finance in another
+// Query climate without getting finance results
+let climate_results = index.search_gated(&climate_query, 10);
+```
+
+### 2. Anomaly Detection
+
+Identify nodes that bridge disparate clusters:
+
+```rust
+let zones = index.compute_zones();
+for zone in zones {
+    if zone.coherence_ratio < threshold {
+        // Low coherence = potential boundary/anomaly
+    }
+}
+```
+
+### 3. Hierarchical Exploration
+
+Navigate from abstract to specific within a coherent region:
+
+```rust
+let l1_neighbors = index.coherent_neighborhood(root, 1);
+let l2_neighbors = index.coherent_neighborhood(root, 2);
+// Expand without crossing semantic boundaries
+```
+
+### 4. Cross-Domain Linking
+
+Explicitly find connections between domains:
+
+```rust
+// Find papers that bridge climate and finance
+let bridging_papers = index.cross_zone_search(
+    &interdisciplinary_query,
+    10,
+    &[climate_zone, finance_zone]
+);
+```
+
+## Metrics and Monitoring
+
+Track performance and behavior:
+
+```rust
+let metrics = index.metrics();
+println!("Searches: {}", metrics.searches_performed.load(Ordering::Relaxed));
+println!("Gates triggered: {}", metrics.cut_gates_triggered.load(Ordering::Relaxed));
+println!("Expansions pruned: {}", metrics.expansions_pruned.load(Ordering::Relaxed));
+
+// Export as JSON
+let json = index.export_metrics();
+
+// Get cut distribution
+let dist = index.cut_distribution();
+for layer_stats in dist {
+    println!("Layer {}: avg_cut={:.3}", layer_stats.layer, layer_stats.avg_cut);
+}
+```
+
+## Configuration Guide
+
+### CutAwareConfig Parameters
+
+```rust
+pub struct CutAwareConfig {
+    // Standard HNSW
+    pub m: usize,                    // Max connections per node (default: 16)
+    pub ef_construction: usize,      // Construction quality (default: 200)
+    pub ef_search: usize,            // Search quality (default: 50)
+
+    // Cut-aware
+    pub coherence_gate_threshold: f64,    // Weak cut threshold (default: 0.3)
+    pub max_cross_cut_hops: usize,        // Max boundary crossings (default: 2)
+    pub enable_cut_pruning: bool,         // Auto-prune weak edges (default: false)
+    pub cut_recompute_interval: usize,    // Recompute frequency (default: 100)
+    pub min_zone_size: usize,             // Min nodes per zone (default: 5)
+}
+```
+
+### Tuning Guidelines
+
+| Workload | `coherence_gate_threshold` | `max_cross_cut_hops` | Notes |
+|----------|---------------------------|---------------------|-------|
+| Strict coherence | 0.5-0.8 | 0-1 | Stay within zones |
+| Moderate | 0.3-0.5 | 2-3 | Some flexibility |
+| Exploratory | 0.1-0.3 | 3-5 | Cross boundaries |
+| No gating | 0.0 | ∞ | Ungated search |
+
+## Examples
+
+### Basic Usage
+
+```rust
+use ruvector_data_framework::cut_aware_hnsw::{CutAwareHNSW, CutAwareConfig};
+
+let config = CutAwareConfig::default();
+let mut index = CutAwareHNSW::new(config);
+
+// Build index
+for i in 0..100 {
+    let vector = generate_vector(i);
+    index.insert(i as u32, &vector)?;
+}
+
+// Query
+let results = index.search_gated(&query, 10);
+for result in results {
+    println!("Node {}: distance={:.4}, coherence={:.3}",
+        result.node_id, result.distance, result.coherence_score);
+}
+```
+
+### Advanced: Multi-Cluster Discovery
+
+See `examples/cut_aware_demo.rs` for a complete example demonstrating:
+- Three distinct semantic clusters
+- Gated vs ungated search comparison
+- Coherent neighborhood exploration
+- Cross-zone queries
+- Metrics tracking
+
+## Testing
+
+The implementation includes 16 comprehensive tests:
+
+```bash
+cargo test --lib cut_aware_hnsw
+```
+
+**Test Coverage:**
+- ✅ Dynamic cut watcher (basic, partition, triangle)
+- ✅ Cut-aware insert and search
+- ✅ Gated vs ungated comparison
+- ✅ Coherent neighborhoods
+- ✅ Zone computation
+- ✅ Cross-zone search
+- ✅ Edge updates (single and batch)
+- ✅ Weak edge pruning
+- ✅ Metrics tracking and export
+- ✅ Boundary edge identification
+
+## Benchmarks
+
+Compare gated vs ungated search performance:
+
+```bash
+cargo bench --bench cut_aware_hnsw_bench
+```
+
+**Benchmarks:**
+- Gated vs ungated search (100, 500, 1000 nodes)
+- Coherent neighborhood (radius 2, 5)
+- Zone computation
+- Batch updates (10, 50, 100 edges)
+- Cross-zone search
+
+**Expected Results:**
+- Ungated search: ~10-50 μs for 1000 nodes
+- Gated search: ~15-70 μs (overhead from gate checks)
+- Zone computation: ~1-5 ms for 1000 nodes
+
+## Integration with RuVector
+
+### With ruvector-core
+
+```rust
+// Use ruvector-core for production HNSW
+use ruvector_core::hnsw::HnswIndex as RuvectorHNSW;
+
+// Wrap with cut-awareness
+let base_index = RuvectorHNSW::new(dimension);
+let cut_aware = CutAwareHNSW::with_base(base_index, config);
+```
+
+### With ruvector-mincut
+
+```rust
+// Use ruvector-mincut for production min-cut
+use ruvector_mincut::StoerWagner;
+
+// Replace DynamicCutWatcher backend
+let mincut = StoerWagner::new();
+let watcher = DynamicCutWatcher::with_backend(mincut);
+```
+
+## Limitations
+
+1. **Min-Cut Complexity**: O(n³) Stoer-Wagner limits scalability to ~10k nodes
+2. **Memory**: Stores full adjacency (sparse) for cut computation
+3. **Static Partitions**: Zones recomputed periodically, not incrementally
+4. **Threshold Sensitivity**: Results depend on `coherence_gate_threshold`
+
+## Future Enhancements
+
+### Planned Features
+
+1. **Euler Tour Trees** - O(log n) dynamic connectivity for faster updates
+2. **Hierarchical Cuts** - Multi-level zone hierarchy
+3. **Approximate Min-Cut** - Karger's algorithm for large graphs
+4. **Persistent Zones** - Incremental zone maintenance
+5. **SIMD Distance** - Accelerated vector comparisons
+
+### Research Directions
+
+1. **Learned Gates** - ML-based coherence threshold prediction
+2. **Temporal Coherence** - Track coherence evolution over time
+3. **Multi-Metric Cuts** - Combine similarity, citation, correlation
+4. **Distributed Cuts** - Partition across machines
+
+## References
+
+1. **Stoer-Wagner Algorithm**
+   - Stoer & Wagner (1997). "A simple min-cut algorithm"
+
+2. **HNSW**
+   - Malkov & Yashunin (2018). "Efficient and robust approximate nearest neighbor search"
+
+3. **Dynamic Connectivity**
+   - Holm et al. (2001). "Poly-logarithmic deterministic fully-dynamic algorithms"
+
+4. **Applications**
+   - Cross-domain research discovery
+   - Hierarchical document clustering
+   - Anomaly detection in graphs
+
+## License
+
+Same as RuVector (MIT/Apache-2.0)
+
+## Contributing
+
+See `CONTRIBUTING.md` for guidelines on:
+- Adding new distance metrics
+- Optimizing cut algorithms
+- Improving zone computation
+- Adding tests and benchmarks