8.3 KiB
8.3 KiB
EXO-AI 2025 Performance Baseline Metrics
Date: 2025-11-29 Version: 0.1.0 Benchmark Framework: Criterion 0.5
Executive Summary
This document establishes baseline performance metrics for the EXO-AI cognitive substrate. All measurements represent target performance on modern multi-core CPUs (e.g., AMD Ryzen 9 / Intel i9 class).
System Architecture Performance Profile
Cognitive Operations (Real-time Tier)
- Latency Target: < 1ms for interactive operations
- Throughput Target: 1000+ ops/sec per component
Batch Processing (High-throughput Tier)
- Latency Target: < 10ms for batch operations
- Throughput Target: 10,000+ items/sec
Distributed Coordination (Consensus Tier)
- Latency Target: < 100ms for consensus rounds
- Throughput Target: 100+ consensus/sec
Component Baselines
1. Manifold (Geometric Embedding)
Retrieval Performance
| Concept Count | Expected Latency | Throughput | Notes |
|---|---|---|---|
| 100 | 20-30μs | 35,000 queries/sec | Small workspace |
| 500 | 50-70μs | 15,000 queries/sec | Medium workspace |
| 1,000 | 80-120μs | 10,000 queries/sec | Baseline target |
| 5,000 | 300-500μs | 2,500 queries/sec | Large workspace |
Optimization Threshold: > 150μs @ 1000 concepts
Deformation (Embedding) Performance
| Batch Size | Expected Latency | Throughput | Notes |
|---|---|---|---|
| 10 | 100-200μs | 60,000 embeds/sec | Micro-batch |
| 50 | 500-800μs | 65,000 embeds/sec | Baseline target |
| 100 | 800-1,200μs | 85,000 embeds/sec | Standard batch |
| 500 | 4-6ms | 90,000 embeds/sec | Large batch |
Optimization Threshold: > 1.5ms @ 100 batch size
Specialized Operations
| Operation | Expected Latency | Notes |
|---|---|---|
| Local Adaptation | 30-50μs | Per-concept learning |
| Curvature Computation | 5-10μs | Geometric calculation |
| Geodesic Distance | 8-15μs | Manifold distance |
2. Hypergraph (Relational Reasoning)
Edge Creation Performance
| Nodes per Edge | Expected Latency | Throughput | Notes |
|---|---|---|---|
| 2 (standard edge) | 1-3μs | 400,000 edges/sec | Binary relation |
| 5 | 3-6μs | 180,000 edges/sec | Baseline target |
| 10 | 8-12μs | 90,000 edges/sec | Medium hyperedge |
| 20 | 18-25μs | 45,000 edges/sec | Large hyperedge |
| 50 | 50-80μs | 15,000 edges/sec | Very large hyperedge |
Optimization Threshold: > 8μs @ 5 nodes
Query Performance
| Total Edges | Expected Latency | Throughput | Notes |
|---|---|---|---|
| 100 | 10-20μs | 60,000 queries/sec | Small graph |
| 500 | 30-50μs | 25,000 queries/sec | Medium graph |
| 1,000 | 40-70μs | 16,000 queries/sec | Baseline target |
| 5,000 | 100-200μs | 7,000 queries/sec | Large graph |
Optimization Threshold: > 100μs @ 1000 edges
Complex Operations
| Operation | Expected Latency | Notes |
|---|---|---|
| Pattern Matching | 80-150μs | 3-node patterns in 500-edge graph |
| Subgraph Extraction | 150-300μs | Depth-2, 10 seed nodes |
| Transitive Closure | 500-1000μs | 100-node graph |
3. Temporal Coordinator (Causal Memory)
Causal Query Performance
| Event Count | Expected Latency | Throughput | Notes |
|---|---|---|---|
| 100 | 20-40μs | 30,000 queries/sec | Small history |
| 500 | 60-100μs | 12,000 queries/sec | Medium history |
| 1,000 | 80-150μs | 8,000 queries/sec | Baseline target |
| 5,000 | 300-600μs | 2,200 queries/sec | Large history |
Optimization Threshold: > 200μs @ 1000 events
Memory Management
| Operation | Expected Latency | Throughput | Notes |
|---|---|---|---|
| Event Recording | 2-5μs | 250,000 events/sec | Single event |
| Consolidation (500) | 3-7ms | - | Periodic operation |
| Range Query | 150-300μs | 4,000 queries/sec | 1-hour window |
| Causal Path (100) | 400-700μs | 1,700 paths/sec | 100-hop path |
| Event Pruning (5000) | 1-3ms | - | Maintenance operation |
Optimization Threshold: > 5ms consolidation @ 500 events
4. Federation (Distributed Coordination)
CRDT Operations (Async)
| Operation Count | Expected Latency | Throughput | Notes |
|---|---|---|---|
| 10 | 500-1000μs | 12,000 ops/sec | Small batch |
| 50 | 2-4ms | 14,000 ops/sec | Medium batch |
| 100 | 4-7ms | 16,000 ops/sec | Baseline target |
| 500 | 20-35ms | 16,000 ops/sec | Large batch |
Optimization Threshold: > 10ms @ 100 operations
Consensus Performance
| Node Count | Expected Latency | Throughput | Notes |
|---|---|---|---|
| 3 | 20-40ms | 35 rounds/sec | Minimum quorum |
| 5 | 40-70ms | 17 rounds/sec | Baseline target |
| 7 | 60-100ms | 12 rounds/sec | Standard cluster |
| 10 | 90-150ms | 8 rounds/sec | Large cluster |
Optimization Threshold: > 100ms @ 5 nodes
Network Operations (Simulated)
| Operation | Expected Latency | Notes |
|---|---|---|
| State Sync (100 items) | 8-15ms | Full state transfer |
| Cryptographic Sign | 80-150μs | Per message |
| Signature Verify | 120-200μs | Per signature |
| Gossip Round (10 nodes) | 15-30ms | Full propagation |
| Gossip Round (50 nodes) | 80-150ms | Large network |
Scaling Characteristics
Expected Complexity Classes
| Component | Operation | Complexity | Notes |
|---|---|---|---|
| Manifold | Retrieval | O(n log n) | With spatial indexing |
| Manifold | Embedding | O(d²) | d = dimension (512) |
| Hypergraph | Edge Creation | O(k) | k = nodes per edge |
| Hypergraph | Query | O(e) | e = incident edges |
| Temporal | Causal Query | O(log n) | With indexed DAG |
| Temporal | Path Finding | O(n + m) | BFS/DFS on causal graph |
| Federation | CRDT Merge | O(n) | n = operations |
| Federation | Consensus | O(n²) | n = nodes (messaging) |
Scalability Targets
Horizontal Scaling (via Federation):
- Linear throughput scaling up to 10 nodes
- Sub-linear latency growth (< 2x @ 10 nodes)
Vertical Scaling (single node):
- Near-linear scaling with CPU cores (up to 8 cores)
- Memory bandwidth becomes bottleneck > 16 cores
Performance Regression Detection
Critical Thresholds (Trigger Investigation)
- 5% regression: Individual operation baselines
- 10% regression: End-to-end workflows
- 15% regression: Acceptable for major feature additions
Monitoring Strategy
- Pre-commit: Run quick benchmarks (< 30s)
- CI Pipeline: Full benchmark suite on main branch
- Weekly: Comprehensive baseline updates
- Release: Performance validation vs. previous release
Hardware Specifications (Reference)
Baseline Testing Environment:
- CPU: 8-core modern processor (3.5+ GHz)
- RAM: 32GB DDR4-3200
- Storage: NVMe SSD
- OS: Linux kernel 5.15+
Variance Expectations:
- ±10% on different hardware generations
- ±5% across benchmark runs
- ±15% between architectures (AMD vs Intel)
Optimization Priorities
Priority 1: Critical Path (Target < 1ms)
- Manifold retrieval @ 1000 concepts
- Hypergraph queries @ 1000 edges
- Temporal causal queries @ 1000 events
Priority 2: Throughput (Target > 10k ops/sec)
- Manifold batch embedding
- Hypergraph edge creation
- CRDT merge operations
Priority 3: Distributed Latency (Target < 100ms)
- Consensus rounds @ 5 nodes
- State synchronization
- Gossip propagation
Benchmark Validation
Statistical Requirements
- Iterations: 100+ per measurement
- Confidence: 95% confidence intervals
- Outliers: < 5% outlier rate
- Warmup: 10+ warmup iterations
Reproducibility
- Coefficient of variation < 10%
- Multiple runs should differ by < 5%
- Baseline comparisons use same hardware
Future Optimization Targets
Version 0.2.0 Goals
- 20% improvement in manifold retrieval
- 30% improvement in hypergraph queries
- 15% improvement in consensus latency
Version 1.0.0 Goals
- Sub-millisecond cognitive operations
- 100k ops/sec throughput per component
- 50ms consensus @ 10 nodes
Benchmark Maintainer: Performance Agent Review Cycle: Monthly Next Review: 2025-12-29