dearsky/wifi-densepose
1
0
Fork 0
Code Issues 20 Pull Requests 5 Actions Packages Projects Releases 1 Wiki Activity

Merge commit 'd803bfe2b1fe7f5e219e50ac20d6801a0a58ac75' as 'vendor/ruvector'

Browse Source
This commit is contained in:
ruv
2026-02-28 14:39:40 -05:00
parent 7885bf6278 d803bfe2b1
commit cd5943df23
7854 changed files with 3522914 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

684
vendor/ruvector/crates/ruvector-bench/README.md vendored Normal file
View File

@@ -0,0 +1,684 @@
# Ruvector-Bench
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![Rust](https://img.shields.io/badge/rust-1.77%2B-orange.svg)](https://www.rust-lang.org)
**Comprehensive benchmarking suite for measuring Ruvector performance across different operations and configurations.**
> Professional-grade performance testing tools for validating sub-millisecond vector search, HNSW optimization, quantization efficiency, and cross-system comparisons. Built for developers who demand data-driven insights.
## 🎯 Overview
The `ruvector-bench` crate provides a complete benchmarking infrastructure to measure and analyze Ruvector's performance characteristics. It includes standardized test suites compatible with [ann-benchmarks.com](http://ann-benchmarks.com), comprehensive latency profiling, memory usage analysis, and cross-system performance comparison tools.
### Key Features
- ⚡ **ANN-Benchmarks Compatible**: Standard datasets (SIFT1M, GIST1M, Deep1M) and metrics
- 📊 **Latency Profiling**: High-precision measurement of p50, p95, p99, p99.9 percentiles
- 💾 **Memory Analysis**: Track memory usage with quantization and optimization techniques
- 🔬 **AgenticDB Workloads**: Simulate real-world AI agent memory patterns
- 🏆 **Cross-System Comparison**: Compare against Python baselines and other vector databases
- 📈 **Comprehensive Reporting**: JSON, CSV, and Markdown output formats
- 🔥 **Performance Profiling**: CPU flamegraphs and memory profiling support
## 📦 Installation
Add to your `Cargo.toml`:
```toml
[dev-dependencies]
ruvector-bench = { path = "../ruvector-bench" }
# Optional: Enable profiling features
ruvector-bench = { path = "../ruvector-bench", features = ["profiling"] }
# Optional: Enable HDF5 dataset loading
ruvector-bench = { path = "../ruvector-bench", features = ["hdf5-datasets"] }
```
## 🚀 Available Benchmarks
The suite includes 6 specialized benchmark binaries:
| Benchmark | Purpose | Metrics |
|-----------|---------|---------|
| **ann-benchmark** | ANN-Benchmarks compatibility | QPS, latency, recall@k, memory |
| **agenticdb-benchmark** | AI agent memory workloads | Insert/search/update latency, memory |
| **latency-benchmark** | Detailed latency profiling | p50/p95/p99/p99.9 latencies |
| **memory-benchmark** | Memory usage analysis | Memory per vector, quantization savings |
| **comparison-benchmark** | Cross-system performance | Ruvector vs baselines (10-100x faster) |
| **profiling-benchmark** | CPU/memory profiling | Flamegraphs, allocation tracking |
## ⚡ Quick Start
### Running Basic Benchmarks
```bash
# Run ANN-Benchmarks suite with default settings
cargo run --bin ann-benchmark --release
# Run with custom parameters
cargo run --bin ann-benchmark --release -- \
--num-vectors 100000 \
--dimensions 384 \
--ef-search-values 50,100,200 \
--output bench_results
# Run latency profiling
cargo run --bin latency-benchmark --release
# Run AgenticDB workload simulation
cargo run --bin agenticdb-benchmark --release
# Run cross-system comparison
cargo run --bin comparison-benchmark --release
```
### Running with Profiling
```bash
# Build with profiling enabled
cargo build --bin profiling-benchmark --release --features profiling
# Run and generate flamegraph
cargo run --bin profiling-benchmark --release --features profiling -- \
--enable-flamegraph \
--output profiling_results
```
## 📊 Benchmark Categories
### 1. ANN-Benchmarks Suite (`ann-benchmark`)
Standard benchmarking compatible with [ann-benchmarks.com](http://ann-benchmarks.com) methodology.
**Supported Datasets:**
- **SIFT1M**: 1M vectors, 128 dimensions (image descriptors)
- **GIST1M**: 1M vectors, 960 dimensions (scene recognition)
- **Deep1M**: 1M vectors, 96 dimensions (deep learning embeddings)
- **Synthetic**: Configurable size and distribution
**Usage:**
```bash
# Test with synthetic data (default)
cargo run --bin ann-benchmark --release -- \
--dataset synthetic \
--num-vectors 100000 \
--dimensions 384 \
--k 10
# Test with SIFT1M (requires dataset download)
cargo run --bin ann-benchmark --release -- \
--dataset sift1m \
--ef-search-values 50,100,200,400
```
**Measured Metrics:**
- Queries per second (QPS)
- Latency percentiles (p50, p95, p99, p99.9)
- Recall@1, Recall@10, Recall@100
- Memory usage (MB)
- Build/index time
**Example Output:**
```
╔════════════════════════════════════════╗
║ Ruvector ANN-Benchmarks Suite ║
╚════════════════════════════════════════╝
✓ Dataset loaded: 100000 vectors, 1000 queries
============================================================
Testing with ef_search = 100
============================================================
┌───────────┬──────┬──────────┬──────────┬───────────┬─────────────┐
│ ef_search │ QPS │ p50 (ms) │ p99 (ms) │ Recall@10 │ Memory (MB) │
├───────────┼──────┼──────────┼──────────┼───────────┼─────────────┤
│ 100 │ 5243 │ 0.19 │ 0.45 │ 95.23% │ 246.8 │
└───────────┴──────┴──────────┴──────────┴───────────┴─────────────┘
```
### 2. AgenticDB Workload Simulation (`agenticdb-benchmark`)
Simulates real-world AI agent memory patterns with mixed read/write workloads.
**Workload Types:**
- **Conversational AI**: High read ratio (70/30 read/write)
- **Learning Agents**: Balanced read/write (50/50)
- **Batch Processing**: Write-heavy (30/70 read/write)
**Usage:**
```bash
cargo run --bin agenticdb-benchmark --release -- \
--workload conversational \
--num-vectors 50000 \
--num-operations 10000
```
**Measured Operations:**
- Insert latency
- Search latency
- Update latency
- Batch operation throughput
- Memory efficiency
### 3. Latency Profiling (`latency-benchmark`)
Detailed latency analysis across different configurations and concurrency levels.
**Test Scenarios:**
- Single-threaded vs multi-threaded search
- Effect of `ef_search` parameter on latency
- Effect of quantization on latency/recall tradeoff
- Concurrent query handling
**Usage:**
```bash
# Test with different thread counts
cargo run --bin latency-benchmark --release -- \
--threads 1,4,8,16 \
--num-vectors 50000 \
--queries 1000
```
**Example Output:**
```
Test 1: Single-threaded Latency
- p50: 0.42ms
- p95: 1.23ms
- p99: 2.15ms
- p99.9: 4.87ms
Test 2: Multi-threaded Latency (8 threads)
- p50: 0.38ms
- p95: 1.05ms
- p99: 1.89ms
- p99.9: 3.92ms
```
### 4. Memory Benchmarks (`memory-benchmark`)
Analyzes memory usage with different quantization strategies.
**Quantization Tests:**
- **None**: Full precision (baseline)
- **Scalar**: 4x compression
- **Binary**: 32x compression
**Usage:**
```bash
cargo run --bin memory-benchmark --release -- \
--num-vectors 100000 \
--dimensions 384
```
**Measured Metrics:**
- Memory per vector (bytes)
- Compression ratio
- Memory overhead
- Quantization impact on recall
**Example Results:**
```
┌──────────────┬─────────────┬───────────────┬────────────┐
│ Quantization │ Memory (MB) │ Bytes/Vector │ Recall@10 │
├──────────────┼─────────────┼───────────────┼────────────┤
│ None │ 147.5 │ 1536 │ 100.00% │
│ Scalar │ 38.2 │ 398 │ 95.80% │
│ Binary │ 4.7 │ 49 │ 87.20% │
└──────────────┴─────────────┴───────────────┴────────────┘
✓ Scalar quantization: 4.0x memory reduction, 4.2% recall loss
✓ Binary quantization: 31.4x memory reduction, 12.8% recall loss
```
### 5. Cross-System Comparison (`comparison-benchmark`)
Compare Ruvector against other implementations and baselines.
**Comparison Targets:**
- Ruvector (optimized: SIMD + Quantization + HNSW)
- Ruvector (no quantization)
- Simulated Python baseline (numpy)
- Simulated brute-force search
**Usage:**
```bash
cargo run --bin comparison-benchmark --release -- \
--num-vectors 50000 \
--dimensions 384
```
**Example Results:**
```
┌──────────────────────────┬──────┬──────────┬─────────────┬────────────┐
│ System │ QPS │ p50 (ms) │ Memory (MB) │ Speedup │
├──────────────────────────┼──────┼──────────┼─────────────┼────────────┤
│ Ruvector (optimized) │ 5243 │ 0.19 │ 38.2 │ 1.0x │
│ Ruvector (no quant) │ 4891 │ 0.20 │ 147.5 │ 0.93x │
│ Python baseline │ 89 │ 11.2 │ 153.6 │ 58.9x │
│ Brute-force │ 12 │ 83.3 │ 147.5 │ 437x │
└──────────────────────────┴──────┴──────────┴─────────────┴────────────┘
✓ Ruvector is 58.9x faster than Python baseline
✓ Ruvector uses 74.1% less memory with quantization
```
### 6. Performance Profiling (`profiling-benchmark`)
CPU and memory profiling with flamegraph generation (requires `profiling` feature).
**Usage:**
```bash
# Build with profiling support
cargo build --bin profiling-benchmark --release --features profiling
# Run with flamegraph generation
cargo run --bin profiling-benchmark --release --features profiling -- \
--enable-flamegraph \
--num-vectors 50000 \
--output profiling_results
# View flamegraph
open profiling_results/flamegraph.svg
```
**Generated Artifacts:**
- CPU flamegraph (SVG)
- Memory allocation profile
- Hotspot analysis
- Function-level timing breakdown
## 📈 Interpreting Results
### Latency Metrics
| Percentile | Meaning | Target |
|------------|---------|--------|
| **p50** | Median latency - typical query performance | <0.5ms |
| **p95** | 95% of queries complete within this time | <1.5ms |
| **p99** | 99% of queries complete within this time | <3.0ms |
| **p99.9** | 99.9% of queries (tail latency) | <5.0ms |
### Recall Metrics
- **Recall@k**: Fraction of true nearest neighbors found in top-k results
- **Target Recall@10**: ≥95% for most applications
- **Trade-off**: Higher `ef_search` → better recall, higher latency
### Memory Efficiency
```
Memory per vector = Total Memory / Number of Vectors
Typical values:
- No quantization: ~1536 bytes (384D float32)
- Scalar quantization: ~400 bytes (4x compression)
- Binary quantization: ~50 bytes (32x compression)
```
## 🔧 Benchmark Configuration Options
### Common Options (All Benchmarks)
```bash
--num-vectors <N> # Number of vectors to index (default: 50000)
--dimensions <D> # Vector dimensions (default: 384)
--output <PATH> # Output directory for results (default: bench_results)
```
### ANN-Benchmark Specific
```bash
--dataset <NAME> # Dataset: sift1m, gist1m, deep1m, synthetic
--num-queries <N> # Number of search queries (default: 1000)
--k <K> # Number of nearest neighbors to retrieve (default: 10)
--m <M> # HNSW M parameter (default: 32)
--ef-construction <EF> # HNSW build parameter (default: 200)
--ef-search-values <EF> # Comma-separated ef_search values to test (default: 50,100,200,400)
--metric <METRIC> # Distance metric: cosine, euclidean, dot (default: cosine)
--quantization <TYPE> # Quantization: none, scalar, binary (default: scalar)
```
### Latency-Benchmark Specific
```bash
--threads <THREADS> # Comma-separated thread counts (default: 1,4,8,16)
```
### AgenticDB-Benchmark Specific
```bash
--workload <TYPE> # Workload type: conversational, learning, batch
--num-operations <N> # Number of operations to perform (default: 10000)
```
### Profiling-Benchmark Specific
```bash
--enable-flamegraph # Generate CPU flamegraph (requires profiling feature)
--enable-memory-profile # Enable detailed memory profiling
```
## 🎨 Custom Benchmark Creation
Create your own benchmarks using the `ruvector-bench` library:
```rust
use ruvector_bench::{
BenchmarkResult, DatasetGenerator, LatencyStats,
MemoryProfiler, ResultWriter, VectorDistribution,
};
use ruvector_core::{VectorDB, DbOptions, SearchQuery, VectorEntry};
use std::time::Instant;
fn my_custom_benchmark() -> anyhow::Result<()> {
// Generate test data
let gen = DatasetGenerator::new(384, VectorDistribution::Normal {
mean: 0.0,
std_dev: 1.0,
});
let vectors = gen.generate(10000);
let queries = gen.generate(100);
// Create database
let db = VectorDB::new(DbOptions::default())?;
// Measure indexing
let mem_profiler = MemoryProfiler::new();
let build_start = Instant::now();
for (idx, vector) in vectors.iter().enumerate() {
db.insert(VectorEntry {
id: Some(idx.to_string()),
vector: vector.clone(),
metadata: None,
})?;
}
let build_time = build_start.elapsed();
// Measure search performance
let mut latency_stats = LatencyStats::new()?;
for query in &queries {
let start = Instant::now();
db.search(SearchQuery {
vector: query.clone(),
k: 10,
filter: None,
ef_search: None,
})?;
latency_stats.record(start.elapsed())?;
}
// Print results
println!("Build time: {:.2}s", build_time.as_secs_f64());
println!("p50 latency: {:.2}ms", latency_stats.percentile(0.50).as_secs_f64() * 1000.0);
println!("p99 latency: {:.2}ms", latency_stats.percentile(0.99).as_secs_f64() * 1000.0);
println!("Memory usage: {:.2}MB", mem_profiler.current_usage_mb());
Ok(())
}
```
## 🔄 CI/CD Integration
### GitHub Actions Example
```yaml
name: Benchmarks
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
benchmark:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Install Rust
uses: actions-rs/toolchain@v1
with:
toolchain: stable
profile: minimal
- name: Run benchmarks
run: |
cd crates/ruvector-bench
cargo run --bin ann-benchmark --release -- --output ci_results
cargo run --bin latency-benchmark --release -- --output ci_results
- name: Upload results
uses: actions/upload-artifact@v3
with:
name: benchmark-results
path: crates/ruvector-bench/ci_results/
- name: Check performance regression
run: |
python scripts/check_regression.py ci_results/ann_benchmark.json
```
## 📉 Performance Regression Testing
Track performance over time using historical benchmark data:
```bash
# Run baseline benchmarks (on main branch)
git checkout main
cargo run --bin ann-benchmark --release -- --output baseline_results
# Run comparison benchmarks (on feature branch)
git checkout feature-branch
cargo run --bin ann-benchmark --release -- --output feature_results
# Compare results
python scripts/compare_benchmarks.py \
baseline_results/ann_benchmark.json \
feature_results/ann_benchmark.json
```
**Regression Thresholds:**
- ✅ **Pass**: <5% latency regression, <10% memory regression
- ⚠️ **Warning**: 5-10% latency regression, 10-20% memory regression
- ❌ **Fail**: >10% latency regression, >20% memory regression
## 📊 Results Visualization
Benchmark results are automatically saved in multiple formats:
### JSON Format
```json
{
"name": "ruvector-ef100",
"dataset": "synthetic",
"dimensions": 384,
"num_vectors": 100000,
"qps": 5243.2,
"latency_p50": 0.19,
"latency_p99": 2.15,
"recall_at_10": 0.9523,
"memory_mb": 38.2
}
```
### CSV Format
```csv
name,dataset,dimensions,num_vectors,qps,p50,p99,recall@10,memory_mb
ruvector-ef100,synthetic,384,100000,5243.2,0.19,2.15,0.9523,38.2
```
### Markdown Report
Results include automatically generated markdown reports with detailed performance analysis.
### Custom Visualization
Generate performance charts using the provided data:
```python
import pandas as pd
import matplotlib.pyplot as plt
# Load benchmark results
df = pd.read_csv('bench_results/ann_benchmark.csv')
# Plot QPS vs Recall tradeoff
plt.figure(figsize=(10, 6))
plt.scatter(df['recall@10'] * 100, df['qps'])
plt.xlabel('Recall@10 (%)')
plt.ylabel('Queries per Second')
plt.title('Ruvector Performance: QPS vs Recall')
plt.grid(True)
plt.savefig('qps_vs_recall.png')
```
## 🔗 Links to Benchmark Reports
- [Latest Benchmark Results](../../benchmarks/LOAD_TEST_SCENARIOS.md)
- [Performance Optimization Guide](../../docs/cloud-architecture/PERFORMANCE_OPTIMIZATION_GUIDE.md)
- [Implementation Summary](../../docs/IMPLEMENTATION_SUMMARY.md)
- [ANN-Benchmarks.com](http://ann-benchmarks.com) - Standard vector search benchmarks
## 🎯 Optimization Based on Benchmarks
### Use Benchmark Results to Tune Performance
1. **Optimize for Latency** (sub-millisecond queries):
```rust
HnswConfig {
m: 16, // Lower M = faster search, less recall
ef_construction: 100,
ef_search: 50, // Lower ef_search = faster, less recall
max_elements: 100000,
}
```
2. **Optimize for Recall** (95%+ accuracy):
```rust
HnswConfig {
m: 64, // Higher M = better recall
ef_construction: 400,
ef_search: 200, // Higher ef_search = better recall
max_elements: 100000,
}
```
3. **Optimize for Memory** (minimal footprint):
```rust
DbOptions {
quantization: Some(QuantizationConfig::Binary), // 32x compression
..Default::default()
}
```
### Recommended Configurations by Use Case
| Use Case | M | ef_construction | ef_search | Quantization | Expected Performance |
|----------|---|----------------|-----------|--------------|----------------------|
| **Low-Latency Search** | 16 | 100 | 50 | Scalar | <0.5ms p50, 90%+ recall |
| **Balanced** | 32 | 200 | 100 | Scalar | <1ms p50, 95%+ recall |
| **High Accuracy** | 64 | 400 | 200 | None | <2ms p50, 98%+ recall |
| **Memory Constrained** | 16 | 100 | 50 | Binary | <1ms p50, 85%+ recall, 32x compression |
## 🛠️ Development
### Running Tests
```bash
# Run unit tests
cargo test -p ruvector-bench
# Run specific benchmark
cargo test -p ruvector-bench --test latency_stats_test
```
### Building Documentation
```bash
# Generate API documentation
cargo doc -p ruvector-bench --open
```
### Adding New Benchmarks
1. Create a new binary in `src/bin/`:
```bash
touch src/bin/my_benchmark.rs
```
2. Add to `Cargo.toml`:
```toml
[[bin]]
name = "my-benchmark"
path = "src/bin/my_benchmark.rs"
```
3. Implement using `ruvector-bench` utilities:
```rust
use ruvector_bench::{LatencyStats, ResultWriter};
```
## 📚 API Reference
### Core Types
- **`BenchmarkResult`**: Comprehensive benchmark result structure
- **`LatencyStats`**: HDR histogram-based latency measurement
- **`DatasetGenerator`**: Synthetic vector data generation
- **`MemoryProfiler`**: Memory usage tracking
- **`ResultWriter`**: Multi-format result output (JSON, CSV, Markdown)
### Utilities
- **`calculate_recall()`**: Compute recall@k metric
- **`create_progress_bar()`**: Terminal progress indication
- **`VectorDistribution`**: Uniform, Normal, or Clustered vector generation
See [full API documentation](https://docs.rs/ruvector-bench) for details.
## 🤝 Contributing
We welcome contributions to improve the benchmarking suite!
### Areas for Contribution
- 📊 Additional benchmark scenarios (concurrent writes, updates, deletes)
- 🔌 Integration with other vector databases (Pinecone, Qdrant, Milvus)
- 📈 Enhanced visualization and reporting
- 🎯 Real-world dataset support (SIFT, GIST, Deep1M loaders)
- 🚀 Performance optimization insights
See [Contributing Guidelines](../../docs/development/CONTRIBUTING.md) for details.
## 📜 License
This crate is part of the Ruvector project and is licensed under the MIT License.
---
<div align="center">
**Part of [Ruvector](../../README.md) - Next-generation vector database built in Rust**
Built by [rUv](https://ruv.io) • [GitHub](https://github.com/ruvnet/ruvector) • [Documentation](../../docs/README.md)
</div>