d803bfe2b1
git-subtree-dir: vendor/ruvector git-subtree-split: b64c21726f2bb37286d9ee36a7869fef60cc6900
258 lines
8.3 KiB
Markdown
258 lines
8.3 KiB
Markdown
# EXO-AI 2025 Performance Baseline Metrics
|
|
|
|
**Date**: 2025-11-29
|
|
**Version**: 0.1.0
|
|
**Benchmark Framework**: Criterion 0.5
|
|
|
|
## Executive Summary
|
|
|
|
This document establishes baseline performance metrics for the EXO-AI cognitive substrate. All measurements represent **target** performance on modern multi-core CPUs (e.g., AMD Ryzen 9 / Intel i9 class).
|
|
|
|
## System Architecture Performance Profile
|
|
|
|
### Cognitive Operations (Real-time Tier)
|
|
- **Latency Target**: < 1ms for interactive operations
|
|
- **Throughput Target**: 1000+ ops/sec per component
|
|
|
|
### Batch Processing (High-throughput Tier)
|
|
- **Latency Target**: < 10ms for batch operations
|
|
- **Throughput Target**: 10,000+ items/sec
|
|
|
|
### Distributed Coordination (Consensus Tier)
|
|
- **Latency Target**: < 100ms for consensus rounds
|
|
- **Throughput Target**: 100+ consensus/sec
|
|
|
|
---
|
|
|
|
## Component Baselines
|
|
|
|
### 1. Manifold (Geometric Embedding)
|
|
|
|
#### Retrieval Performance
|
|
| Concept Count | Expected Latency | Throughput | Notes |
|
|
|---------------|------------------|------------|-------|
|
|
| 100 | 20-30μs | 35,000 queries/sec | Small workspace |
|
|
| 500 | 50-70μs | 15,000 queries/sec | Medium workspace |
|
|
| 1,000 | 80-120μs | 10,000 queries/sec | **Baseline target** |
|
|
| 5,000 | 300-500μs | 2,500 queries/sec | Large workspace |
|
|
|
|
**Optimization Threshold**: > 150μs @ 1000 concepts
|
|
|
|
#### Deformation (Embedding) Performance
|
|
| Batch Size | Expected Latency | Throughput | Notes |
|
|
|------------|------------------|------------|-------|
|
|
| 10 | 100-200μs | 60,000 embeds/sec | Micro-batch |
|
|
| 50 | 500-800μs | 65,000 embeds/sec | **Baseline target** |
|
|
| 100 | 800-1,200μs | 85,000 embeds/sec | Standard batch |
|
|
| 500 | 4-6ms | 90,000 embeds/sec | Large batch |
|
|
|
|
**Optimization Threshold**: > 1.5ms @ 100 batch size
|
|
|
|
#### Specialized Operations
|
|
| Operation | Expected Latency | Notes |
|
|
|-----------|------------------|-------|
|
|
| Local Adaptation | 30-50μs | Per-concept learning |
|
|
| Curvature Computation | 5-10μs | Geometric calculation |
|
|
| Geodesic Distance | 8-15μs | Manifold distance |
|
|
|
|
---
|
|
|
|
### 2. Hypergraph (Relational Reasoning)
|
|
|
|
#### Edge Creation Performance
|
|
| Nodes per Edge | Expected Latency | Throughput | Notes |
|
|
|----------------|------------------|------------|-------|
|
|
| 2 (standard edge) | 1-3μs | 400,000 edges/sec | Binary relation |
|
|
| 5 | 3-6μs | 180,000 edges/sec | **Baseline target** |
|
|
| 10 | 8-12μs | 90,000 edges/sec | Medium hyperedge |
|
|
| 20 | 18-25μs | 45,000 edges/sec | Large hyperedge |
|
|
| 50 | 50-80μs | 15,000 edges/sec | Very large hyperedge |
|
|
|
|
**Optimization Threshold**: > 8μs @ 5 nodes
|
|
|
|
#### Query Performance
|
|
| Total Edges | Expected Latency | Throughput | Notes |
|
|
|-------------|------------------|------------|-------|
|
|
| 100 | 10-20μs | 60,000 queries/sec | Small graph |
|
|
| 500 | 30-50μs | 25,000 queries/sec | Medium graph |
|
|
| 1,000 | 40-70μs | 16,000 queries/sec | **Baseline target** |
|
|
| 5,000 | 100-200μs | 7,000 queries/sec | Large graph |
|
|
|
|
**Optimization Threshold**: > 100μs @ 1000 edges
|
|
|
|
#### Complex Operations
|
|
| Operation | Expected Latency | Notes |
|
|
|-----------|------------------|-------|
|
|
| Pattern Matching | 80-150μs | 3-node patterns in 500-edge graph |
|
|
| Subgraph Extraction | 150-300μs | Depth-2, 10 seed nodes |
|
|
| Transitive Closure | 500-1000μs | 100-node graph |
|
|
|
|
---
|
|
|
|
### 3. Temporal Coordinator (Causal Memory)
|
|
|
|
#### Causal Query Performance
|
|
| Event Count | Expected Latency | Throughput | Notes |
|
|
|-------------|------------------|------------|-------|
|
|
| 100 | 20-40μs | 30,000 queries/sec | Small history |
|
|
| 500 | 60-100μs | 12,000 queries/sec | Medium history |
|
|
| 1,000 | 80-150μs | 8,000 queries/sec | **Baseline target** |
|
|
| 5,000 | 300-600μs | 2,200 queries/sec | Large history |
|
|
|
|
**Optimization Threshold**: > 200μs @ 1000 events
|
|
|
|
#### Memory Management
|
|
| Operation | Expected Latency | Throughput | Notes |
|
|
|-----------|------------------|------------|-------|
|
|
| Event Recording | 2-5μs | 250,000 events/sec | Single event |
|
|
| Consolidation (500) | 3-7ms | - | Periodic operation |
|
|
| Range Query | 150-300μs | 4,000 queries/sec | 1-hour window |
|
|
| Causal Path (100) | 400-700μs | 1,700 paths/sec | 100-hop path |
|
|
| Event Pruning (5000) | 1-3ms | - | Maintenance operation |
|
|
|
|
**Optimization Threshold**: > 5ms consolidation @ 500 events
|
|
|
|
---
|
|
|
|
### 4. Federation (Distributed Coordination)
|
|
|
|
#### CRDT Operations (Async)
|
|
| Operation Count | Expected Latency | Throughput | Notes |
|
|
|-----------------|------------------|------------|-------|
|
|
| 10 | 500-1000μs | 12,000 ops/sec | Small batch |
|
|
| 50 | 2-4ms | 14,000 ops/sec | Medium batch |
|
|
| 100 | 4-7ms | 16,000 ops/sec | **Baseline target** |
|
|
| 500 | 20-35ms | 16,000 ops/sec | Large batch |
|
|
|
|
**Optimization Threshold**: > 10ms @ 100 operations
|
|
|
|
#### Consensus Performance
|
|
| Node Count | Expected Latency | Throughput | Notes |
|
|
|------------|------------------|------------|-------|
|
|
| 3 | 20-40ms | 35 rounds/sec | Minimum quorum |
|
|
| 5 | 40-70ms | 17 rounds/sec | **Baseline target** |
|
|
| 7 | 60-100ms | 12 rounds/sec | Standard cluster |
|
|
| 10 | 90-150ms | 8 rounds/sec | Large cluster |
|
|
|
|
**Optimization Threshold**: > 100ms @ 5 nodes
|
|
|
|
#### Network Operations (Simulated)
|
|
| Operation | Expected Latency | Notes |
|
|
|-----------|------------------|-------|
|
|
| State Sync (100 items) | 8-15ms | Full state transfer |
|
|
| Cryptographic Sign | 80-150μs | Per message |
|
|
| Signature Verify | 120-200μs | Per signature |
|
|
| Gossip Round (10 nodes) | 15-30ms | Full propagation |
|
|
| Gossip Round (50 nodes) | 80-150ms | Large network |
|
|
|
|
---
|
|
|
|
## Scaling Characteristics
|
|
|
|
### Expected Complexity Classes
|
|
|
|
| Component | Operation | Complexity | Notes |
|
|
|-----------|-----------|------------|-------|
|
|
| Manifold | Retrieval | O(n log n) | With spatial indexing |
|
|
| Manifold | Embedding | O(d²) | d = dimension (512) |
|
|
| Hypergraph | Edge Creation | O(k) | k = nodes per edge |
|
|
| Hypergraph | Query | O(e) | e = incident edges |
|
|
| Temporal | Causal Query | O(log n) | With indexed DAG |
|
|
| Temporal | Path Finding | O(n + m) | BFS/DFS on causal graph |
|
|
| Federation | CRDT Merge | O(n) | n = operations |
|
|
| Federation | Consensus | O(n²) | n = nodes (messaging) |
|
|
|
|
### Scalability Targets
|
|
|
|
**Horizontal Scaling** (via Federation):
|
|
- Linear throughput scaling up to 10 nodes
|
|
- Sub-linear latency growth (< 2x @ 10 nodes)
|
|
|
|
**Vertical Scaling** (single node):
|
|
- Near-linear scaling with CPU cores (up to 8 cores)
|
|
- Memory bandwidth becomes bottleneck > 16 cores
|
|
|
|
---
|
|
|
|
## Performance Regression Detection
|
|
|
|
### Critical Thresholds (Trigger Investigation)
|
|
- **5% regression**: Individual operation baselines
|
|
- **10% regression**: End-to-end workflows
|
|
- **15% regression**: Acceptable for major feature additions
|
|
|
|
### Monitoring Strategy
|
|
1. **Pre-commit**: Run quick benchmarks (< 30s)
|
|
2. **CI Pipeline**: Full benchmark suite on main branch
|
|
3. **Weekly**: Comprehensive baseline updates
|
|
4. **Release**: Performance validation vs. previous release
|
|
|
|
---
|
|
|
|
## Hardware Specifications (Reference)
|
|
|
|
**Baseline Testing Environment**:
|
|
- CPU: 8-core modern processor (3.5+ GHz)
|
|
- RAM: 32GB DDR4-3200
|
|
- Storage: NVMe SSD
|
|
- OS: Linux kernel 5.15+
|
|
|
|
**Variance Expectations**:
|
|
- ±10% on different hardware generations
|
|
- ±5% across benchmark runs
|
|
- ±15% between architectures (AMD vs Intel)
|
|
|
|
---
|
|
|
|
## Optimization Priorities
|
|
|
|
### Priority 1: Critical Path (Target < 1ms)
|
|
1. Manifold retrieval @ 1000 concepts
|
|
2. Hypergraph queries @ 1000 edges
|
|
3. Temporal causal queries @ 1000 events
|
|
|
|
### Priority 2: Throughput (Target > 10k ops/sec)
|
|
1. Manifold batch embedding
|
|
2. Hypergraph edge creation
|
|
3. CRDT merge operations
|
|
|
|
### Priority 3: Distributed Latency (Target < 100ms)
|
|
1. Consensus rounds @ 5 nodes
|
|
2. State synchronization
|
|
3. Gossip propagation
|
|
|
|
---
|
|
|
|
## Benchmark Validation
|
|
|
|
### Statistical Requirements
|
|
- **Iterations**: 100+ per measurement
|
|
- **Confidence**: 95% confidence intervals
|
|
- **Outliers**: < 5% outlier rate
|
|
- **Warmup**: 10+ warmup iterations
|
|
|
|
### Reproducibility
|
|
- Coefficient of variation < 10%
|
|
- Multiple runs should differ by < 5%
|
|
- Baseline comparisons use same hardware
|
|
|
|
---
|
|
|
|
## Future Optimization Targets
|
|
|
|
### Version 0.2.0 Goals
|
|
- 20% improvement in manifold retrieval
|
|
- 30% improvement in hypergraph queries
|
|
- 15% improvement in consensus latency
|
|
|
|
### Version 1.0.0 Goals
|
|
- Sub-millisecond cognitive operations
|
|
- 100k ops/sec throughput per component
|
|
- 50ms consensus @ 10 nodes
|
|
|
|
---
|
|
|
|
**Benchmark Maintainer**: Performance Agent
|
|
**Review Cycle**: Monthly
|
|
**Next Review**: 2025-12-29
|