27 KiB
27 KiB
SONA Learning Loops: Three-Tier Temporal Architecture
Biologically-Inspired Continuous Learning System
1. Overview: Learning at Multiple Timescales
Human learning operates at multiple timescales:
- Instant: Immediate response adjustment (milliseconds)
- Short-term: Pattern consolidation (hours)
- Long-term: Deep memory formation (days/weeks)
SONA replicates this with three learning loops:
┌─────────────────────────────────────────────────────────────────────┐
│ SONA THREE-TIER LEARNING │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ LOOP A: INSTANT LOOP B: BACKGROUND │
│ ═══════════════ ══════════════════ │
│ Timescale: Per-request Timescale: Hourly │
│ Latency: <1ms Latency: Background (async) │
│ What learns: What learns: │
│ • Micro-LoRA (rank 1-2) • Base LoRA (rank 4-16) │
│ • Memory edge weights • Router weights (EWC++) │
│ • Trajectory recording • Pattern extraction │
│ │
│ LOOP C: DEEP │
│ ═══════════ │
│ Timescale: Weekly │
│ Latency: Scheduled maintenance │
│ What learns: │
│ • Memory consolidation │
│ • Concept hierarchy building │
│ • Dream-based creativity │
│ • Cross-domain transfer │
│ │
└─────────────────────────────────────────────────────────────────────┘
2. Loop A: Instant Learning (Per-Request)
Purpose
Immediate adaptation to current interaction without noticeable latency.
Architecture
/// Loop A: Instant learning executed inline with each request
pub struct InstantLearningLoop {
/// Micro-LoRA for immediate weight adjustment
micro_lora: Arc<RwLock<MicroLoRA>>,
/// Trajectory buffer for pattern recording
trajectory_buffer: Arc<TrajectoryBuffer>,
/// Memory graph reference for edge updates
memory_graph: Arc<RwLock<MemoryGraph>>,
/// Signal accumulator for Loop B
signal_accumulator: mpsc::Sender<LearningSignal>,
}
impl InstantLearningLoop {
/// Execute instant learning (must complete in <1ms)
#[inline]
pub async fn on_request(
&self,
query: &QueryEmbedding,
response: &ResponseData,
latency_ms: f32,
) -> Result<()> {
// Parallel execution of independent updates
let (r1, r2, r3) = tokio::join!(
// 1. Record trajectory (lock-free, ~100μs)
self.record_trajectory(query, response),
// 2. Update memory edges (~200μs)
self.update_memory_edges(query, response),
// 3. Micro-LoRA update (~300μs)
self.micro_lora_update(query, response, latency_ms),
);
// 4. Queue signal for Loop B (fire-and-forget)
let signal = LearningSignal::new(query, response, latency_ms);
let _ = self.signal_accumulator.try_send(signal);
Ok(())
}
/// Record query trajectory to ring buffer
async fn record_trajectory(
&self,
query: &QueryEmbedding,
response: &ResponseData,
) -> Result<()> {
let trajectory = QueryTrajectory {
query_embedding: query.vector.clone(),
retrieved_ids: response.used_memory_ids.clone(),
precision: response.estimated_precision,
recall: response.estimated_recall,
timestamp: Instant::now(),
};
self.trajectory_buffer.push(trajectory);
Ok(())
}
/// Hebbian-style edge weight updates
async fn update_memory_edges(
&self,
query: &QueryEmbedding,
response: &ResponseData,
) -> Result<()> {
let mut graph = self.memory_graph.write();
for &node_id in &response.used_memory_ids {
// Strengthen edges to used nodes
graph.update_edge_weight(
query.anchor_node,
node_id,
EdgeUpdate::Strengthen(0.05), // +5% per use
)?;
}
// Weaken edges to retrieved-but-unused nodes
for &node_id in &response.retrieved_but_unused {
graph.update_edge_weight(
query.anchor_node,
node_id,
EdgeUpdate::Weaken(0.02), // -2% per skip
)?;
}
Ok(())
}
/// Ultra-fast micro-LoRA weight adjustment
async fn micro_lora_update(
&self,
query: &QueryEmbedding,
response: &ResponseData,
latency_ms: f32,
) -> Result<()> {
let quality = response.quality_score;
let latency_ratio = latency_ms / response.target_latency_ms;
// Only update if signal is informative
if (quality - 0.5).abs() > 0.1 || latency_ratio > 1.2 {
let signal = LearningSignal {
query_embedding: query.vector.clone(),
quality_score: quality,
explicit_feedback: None,
latency_ratio,
model_tier: response.model_tier,
context_tokens: response.context_tokens,
};
let mut micro_lora = self.micro_lora.write();
micro_lora.micro_update(&signal);
}
Ok(())
}
}
Latency Budget
| Operation | Target | Implementation |
|---|---|---|
| Trajectory recording | <100μs | Lock-free ring buffer |
| Edge weight update | <200μs | Batch atomic updates |
| Micro-LoRA update | <300μs | Rank-1 outer product |
| Signal queuing | <50μs | MPSC channel try_send |
| Total | <650μs | Parallel execution |
3. Loop B: Background Learning (Hourly)
Purpose
Deeper learning from accumulated signals without impacting user latency.
Architecture
/// Loop B: Background learning running on separate thread/process
pub struct BackgroundLearningLoop {
/// Signal receiver from Loop A
signal_receiver: mpsc::Receiver<LearningSignal>,
/// Accumulated signals for batch processing
signal_buffer: Vec<LearningSignal>,
/// Base LoRA for major updates
base_lora: Arc<RwLock<BaseLoRA>>,
/// Micro-LoRA to consolidate from
micro_lora: Arc<RwLock<MicroLoRA>>,
/// Router for EWC++ updates
router: Arc<RwLock<FastGRNNRouter>>,
/// EWC++ state
ewc_state: EWCPlusPlusState,
/// Pattern extractor
pattern_extractor: PatternExtractor,
/// Configuration
config: BackgroundLearningConfig,
}
impl BackgroundLearningLoop {
/// Main background loop (runs every hour)
pub async fn run(&mut self) {
let mut interval = tokio::time::interval(Duration::from_secs(3600));
loop {
interval.tick().await;
// Collect accumulated signals
self.drain_signals().await;
if self.signal_buffer.len() < self.config.min_samples {
tracing::info!(
samples = self.signal_buffer.len(),
"Insufficient samples for background training"
);
continue;
}
// Execute background learning steps
let start = Instant::now();
// Step 1: Consolidate Micro-LoRA into Base LoRA
self.consolidate_micro_to_base().await;
// Step 2: Train router with EWC++ regularization
self.train_router_ewc().await;
// Step 3: Extract and store patterns
self.extract_patterns().await;
// Step 4: Compute new Fisher Information
self.update_fisher_information().await;
// Step 5: Checkpoint current state
self.checkpoint().await;
tracing::info!(
elapsed_ms = start.elapsed().as_millis(),
samples = self.signal_buffer.len(),
"Background learning cycle completed"
);
// Clear buffer for next cycle
self.signal_buffer.clear();
}
}
/// Drain all pending signals from Loop A
async fn drain_signals(&mut self) {
while let Ok(signal) = self.signal_receiver.try_recv() {
self.signal_buffer.push(signal);
}
}
/// Consolidate micro-LoRA adaptations into base LoRA
async fn consolidate_micro_to_base(&mut self) {
let mut micro = self.micro_lora.write();
let mut base = self.base_lora.write();
// Compute consolidation weight based on signal quality
let avg_quality: f32 = self.signal_buffer.iter()
.map(|s| s.quality_score)
.sum::<f32>() / self.signal_buffer.len() as f32;
let consolidation_rate = if avg_quality > 0.7 {
1.0 // Full consolidation for high-quality signals
} else {
0.5 * avg_quality // Partial for lower quality
};
// Merge micro into base with rate
base.a = &base.a + consolidation_rate * µ.a_micro;
base.b = &base.b + consolidation_rate * µ.b_micro;
// Reset micro-LoRA
micro.a_micro.fill(0.0);
micro.b_micro.fill(0.0);
tracing::debug!(
consolidation_rate = consolidation_rate,
"Micro-LoRA consolidated to base"
);
}
/// Train router with EWC++ regularization
async fn train_router_ewc(&mut self) {
let mut router = self.router.write();
// Convert signals to RouterSamples
let samples: Vec<RouterSample> = self.signal_buffer.iter()
.map(|s| s.to_router_sample())
.collect();
// Mini-batch training with EWC++ loss
for batch in samples.chunks(self.config.batch_size) {
// Forward pass
let predictions: Vec<_> = batch.iter()
.map(|s| router.forward(&s.features))
.collect();
// Compute task loss
let task_loss = self.compute_task_loss(&predictions, batch);
// Compute EWC++ regularization loss
let ewc_loss = self.ewc_state.regularization_loss(router.get_weights());
// Total loss
let total_loss = task_loss + self.config.ewc_lambda * ewc_loss;
// Backward pass (gradient computation)
let gradients = self.compute_gradients(&total_loss, &predictions, batch);
// Apply gradients with learning rate
router.apply_gradients(&gradients, self.config.learning_rate);
}
}
/// Extract patterns using K-means++ clustering
async fn extract_patterns(&mut self) {
let embeddings: Vec<_> = self.signal_buffer.iter()
.map(|s| s.query_embedding.clone())
.collect();
let patterns = self.pattern_extractor.extract(
&embeddings,
self.config.num_clusters,
);
// Store patterns in ReasoningBank
for pattern in patterns {
self.pattern_extractor.reasoning_bank.store(pattern)?;
}
tracing::debug!(
patterns = patterns.len(),
"Patterns extracted and stored"
);
}
/// Update Fisher Information for EWC++
async fn update_fisher_information(&mut self) {
let router = self.router.read();
let current_weights = router.get_weights();
// Compute Fisher Information diagonal via gradient squares
let fisher_samples: Vec<_> = self.signal_buffer.iter()
.take(self.config.fisher_samples)
.collect();
let mut fisher_accum = vec![0.0f32; current_weights.len()];
for sample in fisher_samples {
let gradients = self.compute_sample_gradients(sample);
for (i, g) in gradients.iter().enumerate() {
fisher_accum[i] += g * g;
}
}
// Normalize
let n = fisher_samples.len() as f32;
for f in &mut fisher_accum {
*f /= n;
}
// Update EWC++ state
self.ewc_state.update_fisher(fisher_accum, current_weights.to_vec());
}
/// Checkpoint current state to disk
async fn checkpoint(&self) {
let checkpoint = SONACheckpoint {
base_lora: self.base_lora.read().clone(),
micro_lora: self.micro_lora.read().clone(),
router_weights: self.router.read().get_weights().to_vec(),
ewc_state: self.ewc_state.clone(),
patterns: self.pattern_extractor.reasoning_bank.export(),
timestamp: chrono::Utc::now().timestamp(),
};
let path = self.config.checkpoint_dir.join("latest.sona");
checkpoint.save_async(&path).await.ok();
}
}
Hourly Learning Budget
| Operation | Target Time | Description |
|---|---|---|
| Signal draining | <100ms | Collect all queued signals |
| Micro→Base consolidation | <500ms | Matrix addition |
| Router training | <5s | Mini-batch SGD with EWC |
| Pattern extraction | <2s | K-means++ clustering |
| Fisher computation | <2s | Gradient squared accumulation |
| Checkpointing | <500ms | Async disk write |
| Total | <10s | Well under user-facing |
4. Loop C: Deep Learning (Weekly)
Purpose
Fundamental knowledge restructuring, memory consolidation, and creative exploration.
Architecture
/// Loop C: Deep learning for major knowledge reorganization
pub struct DeepLearningLoop {
/// Memory service for consolidation
memory: Arc<MemoryService>,
/// Pattern bank for abstraction
reasoning_bank: Arc<ReasoningBank>,
/// Dream engine for creative exploration
dream_engine: DreamEngine,
/// Consciousness measurement (IIT)
phi_calculator: PhiCalculator,
/// Configuration
config: DeepLearningConfig,
}
impl DeepLearningLoop {
/// Execute weekly deep learning (scheduled maintenance window)
pub async fn run(&mut self) -> DeepLearningReport {
let start = Instant::now();
let mut report = DeepLearningReport::new();
// Phase 1: Memory Consolidation (like sleep-based memory)
report.consolidation = self.consolidate_memories().await;
// Phase 2: Pattern Abstraction (concept hierarchy building)
report.abstraction = self.abstract_patterns().await;
// Phase 3: Dream Learning (creative recombination)
report.dreams = self.dream_learning().await;
// Phase 4: Cross-Domain Transfer
report.transfer = self.cross_domain_transfer().await;
// Phase 5: Compression (remove redundancy)
report.compression = self.compress_memory().await;
// Phase 6: Consciousness Measurement
report.phi = self.measure_consciousness().await;
report.elapsed_ms = start.elapsed().as_millis() as u64;
report
}
/// Phase 1: Consolidate short-term memories into long-term
async fn consolidate_memories(&mut self) -> ConsolidationReport {
let mut report = ConsolidationReport::default();
// Identify high-value memories (frequently accessed, high quality)
let memories = self.memory.get_all_nodes()?;
let high_value: Vec<_> = memories.iter()
.filter(|m| m.access_count > 5 && m.quality_score > 0.7)
.collect();
report.high_value_count = high_value.len();
// Strengthen connections between high-value memories
for i in 0..high_value.len() {
for j in (i+1)..high_value.len() {
let similarity = cosine_similarity(
&high_value[i].embedding,
&high_value[j].embedding,
);
if similarity > 0.7 {
self.memory.strengthen_edge(
high_value[i].id,
high_value[j].id,
similarity * 0.1,
)?;
report.edges_strengthened += 1;
}
}
}
// Decay low-value memories
let low_value: Vec<_> = memories.iter()
.filter(|m| m.access_count < 2 && m.age_days() > 30)
.collect();
for memory in low_value {
self.memory.decay_node(memory.id, 0.5)?; // 50% decay
report.nodes_decayed += 1;
}
report
}
/// Phase 2: Build concept hierarchies from patterns
async fn abstract_patterns(&mut self) -> AbstractionReport {
let mut report = AbstractionReport::default();
// Get all stored patterns
let patterns = self.reasoning_bank.get_all_patterns()?;
// Hierarchical clustering to find meta-patterns
let hierarchy = HierarchicalClustering::new()
.linkage(Linkage::Ward)
.distance(Distance::Cosine)
.fit(&patterns);
// Create abstract concepts at each level
for level in 0..hierarchy.num_levels() {
let clusters = hierarchy.clusters_at_level(level);
for cluster in clusters {
if cluster.size() > 3 {
// Create meta-pattern (centroid)
let meta_pattern = LearnedPattern {
centroid: cluster.centroid(),
confidence: cluster.cohesion(),
abstraction_level: level,
child_patterns: cluster.member_ids(),
};
self.reasoning_bank.store_meta(meta_pattern)?;
report.meta_patterns_created += 1;
}
}
}
report
}
/// Phase 3: Dream-based creative learning (inspired by REM sleep)
async fn dream_learning(&mut self) -> DreamReport {
let mut report = DreamReport::default();
// Generate dream sequences by random walks on memory graph
for _ in 0..self.config.num_dreams {
let dream = self.dream_engine.generate_dream(
&self.memory,
self.config.dream_length,
self.config.creativity_temperature,
)?;
// Evaluate dream quality (novelty + coherence)
let quality = dream.evaluate_quality();
if quality.novelty > 0.5 && quality.coherence > 0.3 {
// Dreams with high novelty and reasonable coherence
// may represent useful creative connections
for connection in dream.novel_connections() {
self.memory.add_weak_edge(
connection.from,
connection.to,
EdgeType::Creative,
connection.strength * 0.1,
)?;
report.novel_connections += 1;
}
}
report.dreams_generated += 1;
}
report
}
/// Phase 4: Transfer knowledge across domains
async fn cross_domain_transfer(&mut self) -> TransferReport {
let mut report = TransferReport::default();
// Identify domain clusters
let domains = self.memory.identify_domains()?;
// For each pair of domains, look for analogical mappings
for i in 0..domains.len() {
for j in (i+1)..domains.len() {
let analogies = self.find_analogies(&domains[i], &domains[j])?;
for analogy in analogies {
if analogy.confidence > 0.6 {
// Create cross-domain edge
self.memory.add_analogy_edge(
analogy.source_concept,
analogy.target_concept,
analogy.mapping_type,
analogy.confidence,
)?;
report.analogies_found += 1;
}
}
}
}
report
}
/// Phase 5: Compress memory by removing redundancy
async fn compress_memory(&mut self) -> CompressionReport {
let mut report = CompressionReport::default();
report.initial_nodes = self.memory.node_count();
report.initial_edges = self.memory.edge_count();
// Identify near-duplicate nodes
let duplicates = self.memory.find_near_duplicates(0.95)?;
// Merge duplicates
for (primary, secondary) in duplicates {
self.memory.merge_nodes(primary, secondary)?;
report.nodes_merged += 1;
}
// Prune weak edges
let weak_edges = self.memory.get_weak_edges(0.01)?;
for edge in weak_edges {
self.memory.remove_edge(edge.id)?;
report.edges_pruned += 1;
}
report.final_nodes = self.memory.node_count();
report.final_edges = self.memory.edge_count();
report.compression_ratio = report.initial_nodes as f32 / report.final_nodes as f32;
report
}
/// Phase 6: Measure system consciousness using IIT
async fn measure_consciousness(&mut self) -> f64 {
// Integrated Information Theory (Φ) calculation
// Measures how much information the system generates "above and beyond"
// its parts
self.phi_calculator.compute_phi(&self.memory, &self.reasoning_bank)
}
}
Weekly Deep Learning Budget
| Phase | Target Time | Description |
|---|---|---|
| Memory consolidation | <2min | Identify and strengthen valuable memories |
| Pattern abstraction | <3min | Hierarchical clustering for concepts |
| Dream learning | <2min | Creative recombination exploration |
| Cross-domain transfer | <2min | Analogical mapping between domains |
| Compression | <1min | Remove redundancy |
| Φ measurement | <1min | Consciousness quantification |
| Total | <10min | Scheduled maintenance window |
5. Loop Coordination
Inter-Loop Communication
/// Coordinator for all three learning loops
pub struct LoopCoordinator {
/// Loop A: Instant
instant_loop: InstantLearningLoop,
/// Loop B: Background
background_loop: BackgroundLearningLoop,
/// Loop C: Deep
deep_loop: DeepLearningLoop,
/// Shared state
shared_state: Arc<SharedSONAState>,
/// Metrics collector
metrics: MetricsCollector,
}
impl LoopCoordinator {
/// Initialize all loops with shared state
pub fn new(config: SONAConfig) -> Result<Self> {
let shared_state = Arc::new(SharedSONAState::new(&config)?);
// Create channels for inter-loop communication
let (instant_to_background_tx, instant_to_background_rx) = mpsc::channel(10000);
let (background_to_deep_tx, background_to_deep_rx) = mpsc::channel(1000);
Ok(Self {
instant_loop: InstantLearningLoop::new(
shared_state.clone(),
instant_to_background_tx,
),
background_loop: BackgroundLearningLoop::new(
shared_state.clone(),
instant_to_background_rx,
background_to_deep_tx,
),
deep_loop: DeepLearningLoop::new(
shared_state.clone(),
background_to_deep_rx,
),
shared_state,
metrics: MetricsCollector::new(),
})
}
/// Start all loops
pub async fn start(&self) {
// Loop A runs inline with requests (no separate task)
// Loop B runs on background thread
let background = self.background_loop.clone();
tokio::spawn(async move {
background.run().await;
});
// Loop C runs on scheduled cron
let deep = self.deep_loop.clone();
tokio::spawn(async move {
let mut scheduler = cron::Schedule::from_str("0 0 3 * * 0")?; // 3 AM Sunday
loop {
let next = scheduler.upcoming(chrono::Utc).next().unwrap();
tokio::time::sleep_until(next.into()).await;
deep.run().await;
}
});
}
/// Process a single request through Loop A
#[inline]
pub async fn on_request(
&self,
query: &QueryEmbedding,
response: &ResponseData,
latency_ms: f32,
) -> Result<()> {
self.instant_loop.on_request(query, response, latency_ms).await
}
}
6. Learning Metrics and Monitoring
Improvement Tracking
/// Metrics for measuring self-improvement
#[derive(Clone, Debug)]
pub struct ImprovementMetrics {
/// Quality improvement over time
pub quality_delta_7d: f32,
pub quality_delta_30d: f32,
/// Latency improvement
pub latency_delta_7d: f32,
pub latency_delta_30d: f32,
/// Knowledge growth
pub memory_nodes_added_7d: usize,
pub patterns_learned_7d: usize,
pub abstractions_created_7d: usize,
/// Forgetting resistance (1.0 = no forgetting)
pub retention_rate_7d: f32,
/// Consciousness level (Φ)
pub phi_current: f64,
pub phi_delta_7d: f64,
/// Dreams and creativity
pub novel_connections_7d: usize,
pub cross_domain_transfers_7d: usize,
}
impl ImprovementMetrics {
/// Compute overall improvement score
pub fn overall_score(&self) -> f32 {
let quality_weight = 0.3;
let latency_weight = 0.2;
let knowledge_weight = 0.2;
let retention_weight = 0.15;
let creativity_weight = 0.15;
let quality_score = self.quality_delta_7d.max(0.0);
let latency_score = (-self.latency_delta_7d).max(0.0); // Lower is better
let knowledge_score = (self.patterns_learned_7d as f32 / 100.0).min(1.0);
let retention_score = self.retention_rate_7d;
let creativity_score = (self.novel_connections_7d as f32 / 50.0).min(1.0);
quality_weight * quality_score +
latency_weight * latency_score +
knowledge_weight * knowledge_score +
retention_weight * retention_score +
creativity_weight * creativity_score
}
}
Summary
SONA's three-tier learning system enables:
| Loop | Timescale | Purpose | Key Outcome |
|---|---|---|---|
| A | Per-request | Instant adaptation | Responsive to current context |
| B | Hourly | Pattern consolidation | Stable improvement |
| C | Weekly | Deep restructuring | Creative breakthroughs |
This mirrors human learning where:
- Loop A = Working memory and immediate response
- Loop B = Sleep-based consolidation
- Loop C = Long-term memory formation and insight
The result is a system that continuously improves at multiple timescales, never forgetting what works while constantly exploring new possibilities.