dearsky/wifi-densepose
1
0
Fork 0
Code Issues 20 Pull Requests 5 Actions Packages Projects Releases 1 Wiki Activity
Files
main
wifi-densepose/vendor/ruvector/docs/adr/ADR-057-federated-rvf-transfer-learning.md

16 KiB
Raw Permalink Blame History

ADR-057: Federated RVF Format for Real-Time Transfer Learning

Status: Proposed Date: 2026-02-26 Authors: ruv.io, RuVector Architecture Team Deciders: Architecture Review Board SDK: Claude-Flow Supersedes: None Related: ADR-029 (RVF Canonical Format), ADR-030 (Cognitive Containers), ADR-056 (Knowledge Export)

Context

The Federation Problem

RuVector users independently develop modules and crates, each accumulating valuable learning patterns: SONA weight trajectories, policy kernel configurations, domain expansion priors, HNSW tuning parameters, and convergence data. Today, this learning is siloed. User A discovers that a specific LoRA rank and EWC lambda combination works well for code review tasks, but User B must rediscover this independently.

The existing infrastructure already supports local federated learning within a single deployment:

  1. SONA FederatedCoordinator (crates/sona/src/training/federated.rs) aggregates AgentExport from EphemeralAgent instances, replaying trajectories above a quality threshold into a master engine. Supports Star, Hierarchical, and PeerToPeer topologies.

  2. Domain Expansion Engine (crates/ruvector-domain-expansion/) implements cross-domain transfer via MetaThompsonEngine with TransferPrior (compact Beta posteriors), PolicyKernel (population-based policy search), and CostCurve (acceleration scoreboard). The rvf_bridge module already serializes these into RVF segments 0x30, 0x31, 0x32.

  3. RVF Format (crates/rvf/) provides 25 segment types with 64-byte headers, SHAKE-256 hashing, Ed25519 signing, WITNESS_SEG audit trails, and forward-compatible unknown-segment passthrough. Segments TransferPrior (0x30), PolicyKernel (0x31), and CostCurve (0x32) already exist.

  4. Google Cloud example (examples/google-cloud/) demonstrates Cloud Run deployment with axum HTTP server, GPU benchmarking, and self-learning models.

What is missing is the inter-user federation layer: the ability to strip PII, package transferable learning as RVF segments, publish them to a shared registry, and merge incoming learning with differential privacy guarantees.

Why Now

  • The RVF segment model is stable with 25 types and a clear allocation map
  • The rvf_bridge proves that TransferPrior/PolicyKernel/CostCurve round-trip cleanly through RVF segments
  • SONA's FederatedCoordinator demonstrates that trajectory aggregation with quality gating works
  • The Google Cloud example provides the deployment foundation
  • Users are building domain-specific crates and would benefit from shared learning

Design Principles

  1. Optional: Core RuVector works without federation. All new crates are feature-gated.
  2. Privacy-First: PII stripping happens before any data leaves the local system. Differential privacy noise is injected at the export boundary.
  3. RVF-Native: Learning is exchanged as RVF segments, not custom wire formats. Unknown segments pass through unchanged.
  4. Cryptographically Verifiable: Every export carries a WITNESS_SEG chain and Ed25519/ML-DSA-65 signatures.
  5. Incremental: Users can share only what they choose. No all-or-nothing.

Decision

1. New Segment Types

Add four new segment types to the 0x33-0x36 range in rvf-types:

Code Name Purpose
0x33 FederatedManifest Describes a federated learning export: contributor pseudonym, export timestamp, included segment IDs, privacy budget spent, format version
0x34 DiffPrivacyProof Differential privacy attestation: epsilon/delta values, noise mechanism used, sensitivity bounds, clipping parameters
0x35 RedactionLog PII stripping attestation: which fields were redacted, which rules fired, hash of pre-redaction content (for audit without revealing content)
0x36 AggregateWeights Federated-averaged SONA weights: aggregated LoRA deltas, participation count, round number, convergence metrics

The existing TransferPrior (0x30), PolicyKernel (0x31), CostCurve (0x32), Witness (0x0A), Crypto (0x0C), and Meta (0x07) segments are reused as-is.

2. New Crates

Nine new crates (seven within crates/rvf/, two interface crates):

Crate Path no_std Purpose
rvf-federation crates/rvf/rvf-federation no (std-only) Core federation protocol: export builder, import merger, version-aware conflict resolution, selective sharing
rvf-pii-strip crates/rvf/rvf-pii-strip core: yes, full: no PII detection and stripping pipeline: regex patterns, path normalization, credential detection, configurable rules, REDACTION_LOG segment generation
rvf-diff-privacy crates/rvf/rvf-diff-privacy core: yes, full: no Differential privacy primitives: Gaussian/Laplace mechanisms, privacy accountant (RDP), gradient clipping, per-parameter noise calibration
rvf-gcloud crates/rvf/rvf-gcloud no (std-only) Google Cloud integration: Pub/Sub publisher/subscriber, GCS object store, Firestore metadata registry, Cloud IAM auth
rvf-fed-aggregate crates/rvf/rvf-fed-aggregate no (std-only) Federated aggregation server: FedAvg, FedProx, weighted averaging, Byzantine-tolerant aggregation, round management
rvf-fed-wasm crates/rvf/rvf-fed-wasm no (wasm32) WASM-compatible export path: browser-side PII stripping and export packaging
mcp-federation crates/mcp-federation no (std-only) MCP server for AI agent access: 6 tools + 4 resources over JSON-RPC 2.0 stdio
rvf-fed-server crates/rvf/rvf-fed-server no (std-only) REST API server (axum): export/import/aggregate endpoints, SSE events, Prometheus metrics
rvf-adapters/federation crates/rvf/rvf-adapters/federation no (std-only) Adapter connecting SONA's FederatedCoordinator and domain expansion's MetaThompsonEngine to the federation protocol

3. PII Stripping Pipeline

The rvf-pii-strip crate implements a three-stage pipeline:

Stage 1: Detection -- Scan all string fields in RVF segment payloads for PII patterns:

  • File paths (/home/user/..., C:\Users\...)
  • IP addresses (IPv4, IPv6, loopback)
  • Email addresses
  • API keys (common patterns: sk-..., AKIA..., ghp_..., Bearer tokens)
  • Usernames and hostnames
  • Environment variable references ($HOME, %USERPROFILE%)
  • Custom regex rules from configuration

Stage 2: Redaction -- Replace detected PII with deterministic pseudonyms:

  • Paths become <PATH_N> where N is a per-export incrementing counter
  • IPs become <IP_N>
  • Keys become <REDACTED_KEY>
  • Usernames become <USER_N>
  • Preserves structural relationships (same path always maps to same pseudonym within one export)

Stage 3: Attestation -- Generate a RedactionLog (0x35) segment containing:

  • Count of each redaction type
  • SHAKE-256 hash of the pre-redaction content (proves content was scanned without revealing it)
  • Rules that fired
  • Timestamp

4. Differential Privacy

The rvf-diff-privacy crate provides mathematical privacy guarantees:

  • Gradient Clipping: Before aggregation, clip per-user gradient norms to bound sensitivity
  • Noise Injection: Add calibrated Gaussian noise (for (epsilon, delta)-DP) to aggregated weights
  • Privacy Accountant: Track cumulative privacy loss using Renyi Differential Privacy (RDP) composition
  • Per-Export Budget: Each federated export consumes a portion of the user's privacy budget. The DiffPrivacyProof (0x34) segment records the spent budget.
  • Configurable Epsilon: Users set their comfort level. Default: epsilon=1.0, delta=1e-5 (strong privacy)

5. Google Cloud Architecture

The rvf-gcloud crate integrates with Google Cloud Platform:

Pub/Sub: Real-time learning event propagation

  • Topic: ruvector-federation-events
  • Messages: serialized FederatedManifest headers (small, <1KB)
  • Subscribers filter by domain, version, and contributor reputation

Cloud Storage (GCS): RVF file exchange

  • Bucket: ruvector-federation-{region}
  • Object naming: {domain}/{version}/{contributor_pseudonym}/{timestamp}.rvf
  • Lifecycle: auto-archive after 90 days, delete after 365 days
  • Server-side encryption with CMEK

Firestore: Metadata registry

  • Collection: federation_manifests
  • Documents: manifest metadata, contributor reputation scores, merge history
  • Real-time listeners for new contribution notifications

Cloud Run: Aggregation service

  • Extends the existing examples/google-cloud/ server
  • New endpoints: POST /federation/submit, GET /federation/pull, POST /federation/aggregate
  • Rate limiting: 100 submissions/hour per contributor
  • IAM-based access control

6. Transfer Learning Protocol

Export Flow:

  1. User triggers export (CLI: rvf federation export --domain <id> --epsilon 1.0)
  2. rvf-adapters/federation extracts TransferPrior, PolicyKernel, CostCurve, and SONA weights from local engines
  3. rvf-pii-strip scans and redacts all payloads, generating RedactionLog segment
  4. rvf-diff-privacy adds calibrated noise to numerical parameters, generating DiffPrivacyProof segment
  5. rvf-federation assembles the export: FederatedManifest + learning segments + RedactionLog + DiffPrivacyProof + Witness chain + Crypto signature
  6. rvf-gcloud uploads to GCS and publishes notification to Pub/Sub

Import Flow:

  1. User subscribes to federation updates (CLI: rvf federation subscribe --domains <ids>)
  2. rvf-gcloud receives Pub/Sub notification, downloads RVF file from GCS
  3. rvf-federation validates: signature check, witness chain verification, privacy proof verification, version compatibility check
  4. rvf-federation merges: version-aware prior dampening (same sqrt-scaling as MetaThompsonEngine::init_domain_with_transfer), conflict resolution for competing patterns
  5. rvf-adapters/federation imports merged learning into local SONA and domain expansion engines

Federated Averaging:

  1. Aggregation server collects N exports for a given domain/version
  2. rvf-fed-aggregate computes weighted average (weight = contributor reputation * trajectory count * quality score)
  3. Byzantine tolerance: exclude outliers beyond 2 standard deviations from the mean
  4. Generate aggregate AggregateWeights (0x36) segment
  5. Publish aggregate back to GCS for all subscribers

7. Version-Aware Merging

Learning from different RVF versions must be handled:

  • Same version: Direct merge using federated averaging
  • Newer to older: Newer learning carries a version tag; older clients skip segments they cannot parse (RVF forward compatibility)
  • Older to newer: Accepted with dampened confidence (lower weight in averaging)
  • Conflict resolution: When two priors disagree on a bucket/arm, merge using BetaParams::merge() (sum parameters minus uniform prior)

8. MCP Server Interface

A dedicated mcp-federation crate provides AI agent access to federation through MCP (JSON-RPC 2.0 over stdio), following the same pattern as the existing mcp-gate crate:

Tool Purpose
federation_export Extract learning, strip PII, apply DP noise, sign, and upload
federation_import Pull, validate, and merge federated learning into local engines
federation_status Read privacy budget, recent activity, contributor reputation
federation_search Query the registry for available learning by domain/quality
federation_budget Check remaining privacy budget and export history
federation_aggregate Trigger server-side aggregation round

Resources (read-only): federation://domains, federation://contributors, federation://rounds/{id}, federation://budget

Registration: claude mcp add mcp-federation -- cargo run -p mcp-federation

9. REST API Interface

The rvf-fed-server crate provides a REST API (axum-based, deployed on Cloud Run) for programmatic access:

  • Export/Import: POST /v1/exports, GET /v1/exports/{id}, DELETE /v1/exports/{id}
  • Aggregation: POST /v1/aggregates, GET /v1/aggregates/{round_id}, GET /v1/aggregates/latest
  • Registry: GET /v1/domains, GET /v1/contributors/{pseudonym}, GET /v1/contributors/{pseudonym}/budget
  • Events: GET /v1/events?domain=X (Server-Sent Events for real-time notifications)
  • Health: GET /v1/health, GET /v1/metrics (Prometheus)

Authentication: API key (Bearer token) or Ed25519 signed requests. Rate-limited per contributor.

SDKs: Rust (rvf_federation::client::FederationClient) and TypeScript (@ruvector/rvf-federation).

10. Selective Sharing

Users control what they share via a FederationPolicy:

  • Allowlist/Denylist: Specific segment types or domains to include/exclude
  • Quality Gate: Only export learning from trajectories above a quality threshold (reuses SONA's quality_threshold)
  • Minimum Evidence: Only export priors with sufficient observations (reuses TransferPrior::extract_summary()'s >12 observation filter)
  • Rate Limit: Maximum exports per time period
  • Privacy Budget: Cumulative epsilon limit before exports are blocked

Consequences

Benefits

  1. Knowledge acceleration: New users bootstrap from community learning instead of starting cold
  2. Privacy-preserving: PII stripping + differential privacy ensure no sensitive data leaks
  3. RVF-native: No new wire formats; everything is standard RVF segments
  4. Cryptographically auditable: Witness chains prove provenance without revealing content
  5. Incremental adoption: Feature-gated, optional, selective sharing
  6. Cloud-native: Google Cloud Pub/Sub + GCS + Firestore provide scalable infrastructure
  7. WASM-compatible: Browser-based exports via rvf-fed-wasm
  8. MCP-integrated: AI agents access federation through standard MCP tools (JSON-RPC 2.0)
  9. API-first: REST API with SSE events for programmatic access, Rust and TypeScript SDKs

Risks

Risk Severity Mitigation
Poisoning attacks (malicious learning) High Byzantine-tolerant aggregation, reputation system, signature verification
Privacy budget exhaustion Medium Configurable epsilon, budget tracking per-export, admin alerts at 80% budget
Version skew causing merge failures Medium RVF forward compatibility, version-tagged manifests, graceful skip of unknown segments
GCS cost escalation Low Lifecycle policies, per-contributor quotas, compression (ZSTD segment compression)
Latency of federated averaging Low Async aggregation, Pub/Sub decoupling, local-first operation
Regulatory compliance (GDPR, CCPA) High PII stripping attestation, data retention policies, right-to-deletion via contributor pseudonym revocation

Segment Allocation Map (Updated)

0x00       Invalid
0x01-0x0F  Core segments (Vec, Index, Overlay, Journal, Manifest, Quant, Meta, Hot, Sketch, Witness, Profile, Crypto, MetaIdx, Kernel, Ebpf)
0x10-0x11  Extension segments (Wasm, Dashboard)
0x12-0x1F  RESERVED (12 slots available)
0x20-0x23  Storage segments (CowMap, Refcount, Membership, Delta)
0x24-0x2F  RESERVED (12 slots available)
0x30-0x32  Domain expansion (TransferPrior, PolicyKernel, CostCurve)
0x33-0x36  Federation (FederatedManifest, DiffPrivacyProof, RedactionLog, AggregateWeights)  <-- NEW
0x37-0xEF  RESERVED (future use)
0xF0-0xFF  RESERVED (system)

Compliance

  • GDPR Article 25: Privacy by design -- PII stripping is mandatory before export, not optional
  • GDPR Article 17: Right to erasure -- contributor pseudonym revocation removes all associated exports from GCS
  • CCPA Section 1798.105: Deletion requests honored via pseudonym revocation
  • NIST SP 800-188: De-identification via differential privacy with formal epsilon guarantees

References

  • McMahan et al., "Communication-Efficient Learning of Deep Networks from Decentralized Data" (FedAvg)
  • Abadi et al., "Deep Learning with Differential Privacy" (DP-SGD)
  • Mironov, "Renyi Differential Privacy" (RDP composition)
  • Blanchard et al., "Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent" (Byzantine tolerance)
  • RVF Format Specification (ADR-029)
  • SONA Architecture (crates/sona)
  • Domain Expansion Engine (crates/ruvector-domain-expansion)
Reference in New Issue View Git Blame Copy Permalink