# Graph Condensation (SFGC) - Implementation Plan

## Overview

### Problem Statement

HNSW graphs in production vector databases face critical deployment challenges:

1. **Memory Footprint**: Full HNSW graphs require 40-120 bytes per vector for connectivity metadata
2. **Edge Deployment**: Mobile/IoT devices cannot store million-node graphs (400MB-4.8GB overhead)
3. **Federated Learning**: Transferring full graphs between nodes is bandwidth-prohibitive
4. **Cold Start**: Initial graph construction is expensive for dynamic applications

### Proposed Solution

Implement Structure-Preserving Graph Condensation (SFGC) that creates synthetic "super-nodes" representing clusters of original nodes. The condensed graph:

- Reduces graph size by 10-100x (configurable compression ratio)
- Preserves topological properties (small-world, scale-free characteristics)
- Maintains search accuracy within 2-5% of full graph
- Enables progressive graph expansion from condensed to full representation

**Core Innovation**: Unlike naive graph coarsening, SFGC learns synthetic node embeddings that maximize structural fidelity using a differentiable graph neural network.

### Expected Benefits (Quantified)

| Metric | Current (Full HNSW) | With SFGC (50x) | Improvement |
|--------|---------------------|-----------------|-------------|
| Memory footprint | 4.8GB (1M vectors) | 96MB | 50x reduction |
| Transfer bandwidth | 4.8GB | 96MB | 50x reduction |
| Edge device compatibility | Limited to 100K vectors | 5M vectors | 50x capacity |
| Cold start time | 120s | 8s + progressive | 15x faster |
| Search accuracy (recall@10) | 0.95 | 0.92-0.94 | 2-3% degradation |
| Search latency | 1.2ms | 1.5ms (initial), 1.2ms (expanded) | 25% slower → same |

**ROI Calculation**:
- Edge deployment: enables $500 devices vs $2000 workstations
- Federated learning: 50x faster synchronization (2.4s vs 120s)
- Multi-tenant SaaS: 50x more graphs per server

## Technical Design

### Architecture Diagram (ASCII)

```
┌─────────────────────────────────────────────────────────────────┐
│                     Graph Condensation Pipeline                  │
└─────────────────────────────────────────────────────────────────┘
                                  │
                    ┌─────────────┴─────────────┐
                    ▼                           ▼
        ┌───────────────────────┐   ┌───────────────────────┐
        │  Offline Condensation │   │  Online Expansion     │
        │      (Training)       │   │     (Runtime)         │
        └───────────────────────┘   └───────────────────────┘
                    │                           │
        ┌───────────┼───────────┐              │
        ▼           ▼           ▼              ▼
    ┌────────┐ ┌────────┐ ┌─────────┐  ┌─────────────┐
    │Cluster │ │Synth   │ │Edge     │  │Progressive  │
    │ ing    │ │Node    │ │Preserv  │  │Decompression│
    │        │ │Learn   │ │ation    │  │             │
    └────────┘ └────────┘ └─────────┘  └─────────────┘
        │           │           │              │
        └───────────┼───────────┘              │
                    ▼                          ▼
            ┌──────────────┐          ┌──────────────┐
            │  Condensed   │─────────▶│  Hybrid      │
            │  Graph File  │  Load    │  Graph Store │
            │  (.cgraph)   │          │              │
            └──────────────┘          └──────────────┘
                                              │
                                              ▼
                                      ┌──────────────┐
                                      │  Search API  │
                                      │  (adaptive)  │
                                      └──────────────┘
```

**Component Flow**:

1. **Offline Condensation** (Training Phase):
   - Hierarchical clustering of original graph
   - GNN-based synthetic node embedding learning
   - Edge weight optimization via structure preservation loss
   - Export to `.cgraph` format

2. **Online Expansion** (Runtime):
   - Load condensed graph for fast cold start
   - Progressive decompression on cache misses
   - Adaptive switching between condensed/full graph

### Core Data Structures (Rust)

```rust
/// Condensed graph representation with synthetic nodes
#[derive(Clone, Debug)]
pub struct CondensedGraph {
    /// Synthetic node embeddings (learned via GNN)
    pub synthetic_nodes: Vec<SyntheticNode>,

    /// Condensed HNSW layers (smaller topology)
    pub condensed_layers: Vec<HnswLayer>,

    /// Compression ratio (e.g., 50.0 for 50x)
    pub compression_ratio: f32,

    /// Mapping from synthetic node to original node IDs
    pub expansion_map: HashMap<NodeId, Vec<NodeId>>,

    /// Graph statistics for adaptive expansion
    pub stats: GraphStatistics,
}

/// Synthetic node representing a cluster of original nodes
#[derive(Clone, Debug)]
pub struct SyntheticNode {
    /// Learned embedding (centroid of cluster, refined via GNN)
    pub embedding: Vec<f32>,

    /// Original node IDs in this cluster
    pub cluster_members: Vec<NodeId>,

    /// Cluster radius (for expansion threshold)
    pub radius: f32,

    /// Connectivity in condensed graph
    pub neighbors: Vec<(NodeId, f32)>, // (neighbor_id, edge_weight)

    /// Access frequency (for adaptive expansion)
    pub access_count: AtomicU64,
}

/// Configuration for graph condensation process
#[derive(Clone, Debug)]
pub struct CondensationConfig {
    /// Target compression ratio (10-100)
    pub compression_ratio: f32,

    /// Clustering method
    pub clustering_method: ClusteringMethod,

    /// GNN training epochs for synthetic nodes
    pub gnn_epochs: usize,

    /// Structure preservation weight (vs embedding quality)
    pub structure_weight: f32,

    /// Edge preservation strategy
    pub edge_strategy: EdgePreservationStrategy,
}

#[derive(Clone, Debug)]
pub enum ClusteringMethod {
    /// Hierarchical agglomerative clustering
    Hierarchical { linkage: LinkageType },

    /// Louvain modularity-based clustering
    Louvain { resolution: f32 },

    /// Spectral clustering via graph Laplacian
    Spectral { n_components: usize },

    /// Custom clustering function
    Custom(Box<dyn Fn(&HnswIndex) -> Vec<Vec<NodeId>>>),
}

#[derive(Clone, Debug)]
pub enum EdgePreservationStrategy {
    /// Keep edges if both endpoints map to different synthetic nodes
    InterCluster,

    /// Weighted by cluster similarity
    WeightedSimilarity,

    /// Learn edge weights via GNN
    Learned,
}

/// Hybrid graph store supporting both condensed and full graphs
pub struct HybridGraphStore {
    /// Condensed graph (always loaded)
    condensed: CondensedGraph,

    /// Full graph (lazily loaded regions)
    full_graph: Option<Arc<RwLock<HnswIndex>>>,

    /// Expanded regions cache
    expanded_cache: LruCache<NodeId, ExpandedRegion>,

    /// Expansion policy
    policy: ExpansionPolicy,
}

/// Policy for when to expand condensed nodes to full graph
#[derive(Clone, Debug)]
pub enum ExpansionPolicy {
    /// Never expand (use condensed graph only)
    Never,

    /// Expand on cache miss
    OnDemand { cache_size: usize },

    /// Expand regions with high query frequency
    Adaptive { threshold: f64 },

    /// Always use full graph
    Always,
}

/// Expanded region of the full graph
struct ExpandedRegion {
    /// Full node data for this region
    nodes: Vec<FullNode>,

    /// Last access timestamp
    last_access: Instant,

    /// Access count
    access_count: u64,
}

/// Statistics for monitoring condensation quality
#[derive(Clone, Debug, Default)]
pub struct GraphStatistics {
    /// Average cluster size
    pub avg_cluster_size: f32,

    /// Cluster size variance
    pub cluster_variance: f32,

    /// Edge preservation ratio (condensed edges / original edges)
    pub edge_preservation: f32,

    /// Average path length increase
    pub path_length_delta: f32,

    /// Clustering coefficient preservation
    pub clustering_coef_ratio: f32,
}
```

### Key Algorithms (Pseudocode)

#### Algorithm 1: Graph Condensation (Offline Training)

```
function condense_graph(hnsw_index, config):
    # Step 1: Hierarchical clustering
    clusters = hierarchical_cluster(
        hnsw_index.nodes,
        target_clusters = hnsw_index.size / config.compression_ratio
    )

    # Step 2: Initialize synthetic node embeddings
    synthetic_nodes = []
    for cluster in clusters:
        centroid = compute_centroid(cluster.members)
        synthetic_nodes.append(SyntheticNode {
            embedding: centroid,
            cluster_members: cluster.members,
            radius: compute_cluster_radius(cluster),
            neighbors: [],
            access_count: 0
        })

    # Step 3: Build condensed edges
    condensed_edges = build_condensed_edges(
        hnsw_index,
        clusters,
        config.edge_strategy
    )

    # Step 4: GNN-based refinement
    gnn_model = GraphNeuralNetwork(
        input_dim = embedding_dim,
        hidden_dims = [128, 64],
        output_dim = embedding_dim
    )

    optimizer = Adam(gnn_model.parameters(), lr=0.001)

    for epoch in 1..config.gnn_epochs:
        # Forward pass: refine synthetic embeddings
        refined_embeddings = gnn_model.forward(
            synthetic_nodes.embeddings,
            condensed_edges
        )

        # Compute structure preservation loss
        loss = compute_structure_loss(
            refined_embeddings,
            condensed_edges,
            original_graph = hnsw_index,
            expansion_map = clusters,
            structure_weight = config.structure_weight
        )

        # Backward pass
        loss.backward()
        optimizer.step()

        # Update synthetic embeddings
        for i, node in enumerate(synthetic_nodes):
            node.embedding = refined_embeddings[i]

    # Step 5: Build condensed HNSW layers
    condensed_layers = build_hnsw_layers(
        synthetic_nodes,
        condensed_edges,
        max_layer = hnsw_index.max_layer
    )

    return CondensedGraph {
        synthetic_nodes,
        condensed_layers,
        compression_ratio: config.compression_ratio,
        expansion_map: clusters,
        stats: compute_statistics(...)
    }

function compute_structure_loss(embeddings, edges, original_graph, expansion_map, structure_weight):
    # Part 1: Embedding quality (centroid fidelity)
    embedding_loss = 0
    for i, synthetic_node in enumerate(embeddings):
        cluster_members = expansion_map[i]
        original_embeddings = [original_graph.get_embedding(id) for id in cluster_members]
        true_centroid = mean(original_embeddings)
        embedding_loss += mse(synthetic_node, true_centroid)

    # Part 2: Structure preservation (edge connectivity)
    structure_loss = 0
    for (u, v, weight) in edges:
        # Check if original graph had path between clusters u and v
        cluster_u = expansion_map[u]
        cluster_v = expansion_map[v]
        original_connectivity = compute_inter_cluster_connectivity(
            original_graph, cluster_u, cluster_v
        )
        predicted_connectivity = cosine_similarity(embeddings[u], embeddings[v])
        structure_loss += mse(predicted_connectivity, original_connectivity)

    # Part 3: Topological invariants
    topo_loss = 0
    condensed_clustering_coef = compute_clustering_coefficient(embeddings, edges)
    original_clustering_coef = original_graph.clustering_coefficient
    topo_loss += abs(condensed_clustering_coef - original_clustering_coef)

    return (1 - structure_weight) * embedding_loss +
           structure_weight * (structure_loss + 0.1 * topo_loss)
```

#### Algorithm 2: Progressive Expansion (Online Runtime)

```
function search_hybrid_graph(query, k, hybrid_store):
    # Step 1: Search in condensed graph
    condensed_results = search_condensed(
        query,
        hybrid_store.condensed,
        k_initial = k * 2  # oversample
    )

    # Step 2: Decide whether to expand
    if hybrid_store.policy == ExpansionPolicy::Never:
        return refine_condensed_results(condensed_results, k)

    # Step 3: Identify expansion candidates
    expansion_candidates = []
    for result in condensed_results:
        synthetic_node = result.node

        # Expand if: high uncertainty OR cache miss OR high query frequency
        should_expand = (
            result.distance < synthetic_node.radius * 1.5 OR  # uncertainty
            not hybrid_store.expanded_cache.contains(synthetic_node.id) OR  # cache miss
            synthetic_node.access_count.load() > adaptive_threshold  # hot region
        )

        if should_expand:
            expansion_candidates.append(synthetic_node.id)

    # Step 4: Expand regions (lazily load from full graph)
    if len(expansion_candidates) > 0:
        expanded_regions = hybrid_store.expand_regions(expansion_candidates)

        # Step 5: Refine search in expanded regions
        refined_results = []
        for region in expanded_regions:
            local_results = search_full_graph(
                query,
                region.nodes,
                k_local = k
            )
            refined_results.extend(local_results)

        # Merge condensed and expanded results
        all_results = merge_results(condensed_results, refined_results)
        return top_k(all_results, k)
    else:
        # No expansion needed
        return refine_condensed_results(condensed_results, k)

function expand_regions(hybrid_store, synthetic_node_ids):
    expanded = []
    for node_id in synthetic_node_ids:
        # Check cache first
        if hybrid_store.expanded_cache.contains(node_id):
            expanded.append(hybrid_store.expanded_cache.get(node_id))
            continue

        # Load from full graph (disk or memory)
        synthetic_node = hybrid_store.condensed.synthetic_nodes[node_id]
        cluster_member_ids = synthetic_node.cluster_members

        full_nodes = []
        if hybrid_store.full_graph.is_some():
            # Full graph in memory
            full_graph = hybrid_store.full_graph.unwrap()
            for member_id in cluster_member_ids:
                full_nodes.append(full_graph.get_node(member_id))
        else:
            # Load from disk (mmap)
            full_nodes = load_nodes_from_disk(cluster_member_ids)

        region = ExpandedRegion {
            nodes: full_nodes,
            last_access: now(),
            access_count: 1
        }

        # Add to cache (evict LRU if full)
        hybrid_store.expanded_cache.put(node_id, region)
        expanded.append(region)

    return expanded
```

### API Design (Function Signatures)

```rust
// ============================================================
// Public API for Graph Condensation
// ============================================================

pub trait GraphCondensation {
    /// Condense an HNSW index into a smaller graph
    fn condense(
        &self,
        config: CondensationConfig,
    ) -> Result<CondensedGraph, CondensationError>;

    /// Save condensed graph to disk
    fn save_condensed(&self, path: &Path) -> Result<(), io::Error>;

    /// Load condensed graph from disk
    fn load_condensed(path: &Path) -> Result<CondensedGraph, io::Error>;

    /// Validate condensation quality
    fn validate_condensation(
        &self,
        condensed: &CondensedGraph,
        test_queries: &[Vec<f32>],
    ) -> ValidationMetrics;
}

pub trait HybridGraphSearch {
    /// Search using hybrid condensed/full graph
    fn search_hybrid(
        &self,
        query: &[f32],
        k: usize,
        policy: ExpansionPolicy,
    ) -> Result<Vec<SearchResult>, SearchError>;

    /// Adaptive search with automatic expansion
    fn search_adaptive(
        &self,
        query: &[f32],
        k: usize,
        recall_target: f32,  // e.g., 0.95
    ) -> Result<Vec<SearchResult>, SearchError>;

    /// Get current cache statistics
    fn cache_stats(&self) -> CacheStatistics;

    /// Preload hot regions into cache
    fn warmup_cache(&mut self, query_log: &[Vec<f32>]) -> Result<(), CacheError>;
}

// ============================================================
// Configuration API
// ============================================================

impl CondensationConfig {
    /// Default configuration for 50x compression
    pub fn default_50x() -> Self {
        Self {
            compression_ratio: 50.0,
            clustering_method: ClusteringMethod::Hierarchical {
                linkage: LinkageType::Ward,
            },
            gnn_epochs: 100,
            structure_weight: 0.7,
            edge_strategy: EdgePreservationStrategy::Learned,
        }
    }

    /// Aggressive compression for edge devices (100x)
    pub fn edge_device() -> Self {
        Self {
            compression_ratio: 100.0,
            clustering_method: ClusteringMethod::Louvain {
                resolution: 1.2,
            },
            gnn_epochs: 50,
            structure_weight: 0.5,
            edge_strategy: EdgePreservationStrategy::InterCluster,
        }
    }

    /// Conservative compression for high accuracy (10x)
    pub fn high_accuracy() -> Self {
        Self {
            compression_ratio: 10.0,
            clustering_method: ClusteringMethod::Spectral {
                n_components: 128,
            },
            gnn_epochs: 200,
            structure_weight: 0.9,
            edge_strategy: EdgePreservationStrategy::Learned,
        }
    }
}

// ============================================================
// Monitoring and Metrics
// ============================================================

#[derive(Clone, Debug)]
pub struct ValidationMetrics {
    /// Recall at different k values
    pub recall_at_k: HashMap<usize, f32>,

    /// Average path length increase
    pub avg_path_length_ratio: f32,

    /// Search latency comparison
    pub latency_ratio: f32,

    /// Memory reduction achieved
    pub memory_reduction: f32,

    /// Graph property preservation
    pub property_preservation: PropertyPreservation,
}

#[derive(Clone, Debug)]
pub struct PropertyPreservation {
    pub clustering_coefficient: f32,
    pub average_degree: f32,
    pub diameter_ratio: f32,
}

#[derive(Clone, Debug)]
pub struct CacheStatistics {
    pub hit_rate: f32,
    pub eviction_count: u64,
    pub avg_expansion_time: Duration,
    pub total_expansions: u64,
}
```

## Integration Points

### Affected Crates/Modules

1. **`ruvector-gnn` (Core GNN crate)**:
   - Add `condensation/` module for graph compression
   - Extend `HnswIndex` with `condense()` method
   - Add GNN training loop for synthetic node refinement

2. **`ruvector-core`**:
   - Add `CondensedGraph` serialization format (`.cgraph`)
   - Extend search API with hybrid search modes
   - Add `HybridGraphStore` as alternative index backend

3. **`ruvector-gnn-node` (Node.js bindings)**:
   - Expose `condense()` API to JavaScript/TypeScript
   - Add configuration builder for condensation parameters
   - Provide progress callbacks for offline condensation

4. **`ruvector-cli`**:
   - Add `ruvector condense` command for offline condensation
   - Add `ruvector validate-condensed` for quality testing
   - Add visualization for condensed graph statistics

5. **`ruvector-distributed`**:
   - Use condensed graphs for federated learning synchronization
   - Implement condensed graph transfer protocol
   - Add merge logic for condensed graphs from multiple nodes

### New Modules to Create

```
crates/ruvector-gnn/src/condensation/
├── mod.rs                    # Public API
├── clustering.rs             # Hierarchical/Louvain/Spectral clustering
├── synthetic_node.rs         # Synthetic node learning via GNN
├── edge_preservation.rs      # Edge weight computation
├── gnn_trainer.rs            # GNN training loop
├── structure_loss.rs         # Loss functions for structure preservation
├── serialization.rs          # .cgraph format I/O
└── validation.rs             # Quality metrics

crates/ruvector-core/src/hybrid/
├── mod.rs                    # HybridGraphStore
├── expansion_policy.rs       # Adaptive expansion logic
├── cache.rs                  # LRU cache for expanded regions
└── search.rs                 # Hybrid search algorithms

crates/ruvector-gnn-node/condensation/
├── bindings.rs               # NAPI bindings
└── typescript/
    └── condensation.d.ts     # TypeScript definitions
```

### Dependencies on Other Features

1. **Prerequisite: Attention Mechanisms (Tier 1)**:
   - SFGC uses attention-weighted clustering
   - Synthetic node embeddings benefit from attention-based aggregation
   - **Action**: Ensure attention module is stable before SFGC integration

2. **Synergy: Adaptive HNSW (Tier 2, Feature #5)**:
   - Adaptive HNSW can use condensed graph for cold start
   - Layer-wise compression ratios (compress higher layers more aggressively)
   - **Integration**: Shared `ExpansionPolicy` trait

3. **Optional: Neuromorphic Spiking (Tier 2, Feature #6)**:
   - Spiking networks can accelerate GNN training for synthetic nodes
   - **Integration**: Conditional compilation flag for spiking backend

4. **Complementary: Sparse Attention (Tier 3, Feature #8)**:
   - Sparse attention patterns can guide clustering
   - **Integration**: Use learned attention masks as clustering hints

## Regression Prevention

### Existing Functionality at Risk

1. **HNSW Search Accuracy**:
   - **Risk**: Condensed graph returns lower-quality results
   - **Mitigation**:
     - Validate recall@10 >= 0.92 on standard benchmarks (SIFT1M, GIST1M)
     - Add A/B testing framework for condensed vs full graph
     - Default to conservative 10x compression

2. **Memory Safety (Rust)**:
   - **Risk**: Expansion cache causes use-after-free or data races
   - **Mitigation**:
     - Use `Arc<RwLock<...>>` for shared ownership
     - Fuzz testing with ThreadSanitizer
     - Property-based testing with `proptest`

3. **Serialization Format Compatibility**:
   - **Risk**: `.cgraph` format breaks existing index loading
   - **Mitigation**:
     - Separate file extension (`.cgraph` vs `.hnsw`)
     - Version magic number in header
     - Fallback to full graph if condensation fails

4. **Node.js Bindings Performance**:
   - **Risk**: Condensation adds latency to JavaScript API
   - **Mitigation**:
     - Make condensation opt-in (separate method)
     - Async/non-blocking condensation API
     - Progress callbacks to avoid blocking event loop

### Test Cases to Prevent Regressions

```rust
// Test 1: Search quality preservation
#[test]
fn test_condensed_search_recall() {
    let full_index = build_test_index(10000);
    let condensed = full_index.condense(CondensationConfig::default_50x()).unwrap();

    let test_queries = generate_test_queries(100);

    for query in test_queries {
        let full_results = full_index.search(&query, 10);
        let condensed_results = condensed.search(&query, 10);

        let recall = compute_recall(&full_results, &condensed_results);
        assert!(recall >= 0.92, "Recall dropped below 92%: {}", recall);
    }
}

// Test 2: Memory reduction
#[test]
fn test_memory_footprint() {
    let full_index = build_test_index(100000);
    let condensed = full_index.condense(CondensationConfig::default_50x()).unwrap();

    let full_size = full_index.memory_usage();
    let condensed_size = condensed.memory_usage();

    let reduction = full_size as f32 / condensed_size as f32;
    assert!(reduction >= 40.0, "Memory reduction below 40x: {}", reduction);
}

// Test 3: Serialization round-trip
#[test]
fn test_condensed_serialization() {
    let original = build_test_index(1000).condense(CondensationConfig::default_50x()).unwrap();

    let path = "/tmp/test.cgraph";
    original.save_condensed(Path::new(path)).unwrap();
    let loaded = CondensedGraph::load_condensed(Path::new(path)).unwrap();

    assert_eq!(original.synthetic_nodes.len(), loaded.synthetic_nodes.len());
    assert_eq!(original.compression_ratio, loaded.compression_ratio);
}

// Test 4: Hybrid search correctness
#[test]
fn test_hybrid_search_equivalence() {
    let full_index = build_test_index(5000);
    let condensed = full_index.condense(CondensationConfig::default_50x()).unwrap();

    let hybrid_store = HybridGraphStore::new(condensed, Some(Arc::new(RwLock::new(full_index))));

    let query = generate_random_query();

    // With ExpansionPolicy::Always, hybrid should match full graph
    let hybrid_results = hybrid_store.search_hybrid(&query, 10, ExpansionPolicy::Always).unwrap();
    let full_results = full_index.search(&query, 10);

    assert_eq!(hybrid_results, full_results);
}

// Test 5: Concurrent expansion safety
#[test]
fn test_concurrent_expansion() {
    let hybrid_store = Arc::new(RwLock::new(build_hybrid_store()));

    let handles: Vec<_> = (0..10).map(|_| {
        let store = Arc::clone(&hybrid_store);
        thread::spawn(move || {
            let query = generate_random_query();
            let results = store.write().unwrap().search_hybrid(
                &query, 10, ExpansionPolicy::OnDemand { cache_size: 100 }
            );
            assert!(results.is_ok());
        })
    }).collect();

    for handle in handles {
        handle.join().unwrap();
    }
}
```

### Backward Compatibility Strategy

1. **API Level**:
   - Keep existing `HnswIndex::search()` unchanged
   - Add new `HnswIndex::condense()` method (opt-in)
   - Condensed search via separate `HybridGraphStore` type

2. **File Format**:
   - Condensed graphs use `.cgraph` extension
   - Original `.hnsw` format unchanged
   - Metadata includes version + compression ratio

3. **Node.js Bindings**:
   - Add `index.condense(config)` method (returns new `CondensedIndex` instance)
   - Keep `index.search()` behavior identical
   - Add `condensedIndex.searchHybrid()` for hybrid mode

4. **CLI**:
   - `ruvector build` unchanged (builds full graph)
   - New `ruvector condense` command (separate step)
   - Auto-detect `.cgraph` vs `.hnsw` on load

## Implementation Phases

### Phase 1: Core Implementation (Weeks 1-3)

**Goals**:
- Implement clustering algorithms (hierarchical, Louvain)
- Build basic synthetic node creation (centroid-based, no GNN)
- Implement condensed HNSW layer construction
- Basic serialization (`.cgraph` format)

**Deliverables**:
```rust
// Week 1: Clustering
crates/ruvector-gnn/src/condensation/clustering.rs
  ✓ hierarchical_cluster()
  ✓ louvain_cluster()
  ✓ spectral_cluster()

// Week 2: Synthetic nodes + edges
crates/ruvector-gnn/src/condensation/synthetic_node.rs
  ✓ create_synthetic_nodes() // centroid-based
  ✓ build_condensed_edges()

// Week 3: Condensed graph + serialization
crates/ruvector-gnn/src/condensation/mod.rs
  ✓ CondensedGraph::from_hnsw()
  ✓ save_condensed() / load_condensed()
```

**Success Criteria**:
- Can condense 100K vector index to 2K synthetic nodes
- Serialization round-trip preserves graph structure
- Unit tests pass for clustering algorithms

### Phase 2: Integration (Weeks 4-6)

**Goals**:
- Integrate with `HnswIndex` API
- Add GNN-based synthetic node refinement
- Implement hybrid search with basic expansion policy
- Node.js bindings

**Deliverables**:
```rust
// Week 4: HNSW integration
crates/ruvector-gnn/src/hnsw/index.rs
  ✓ impl GraphCondensation for HnswIndex

// Week 5: GNN training
crates/ruvector-gnn/src/condensation/gnn_trainer.rs
  ✓ train_synthetic_embeddings()
  ✓ structure_preservation_loss()

// Week 6: Hybrid search
crates/ruvector-core/src/hybrid/
  ✓ HybridGraphStore::search_hybrid()
  ✓ ExpansionPolicy::OnDemand
```

**Success Criteria**:
- Recall@10 >= 0.90 on SIFT1M benchmark
- GNN training converges in <100 epochs
- Hybrid search passes correctness tests

### Phase 3: Optimization (Weeks 7-9)

**Goals**:
- Performance tuning (SIMD, caching)
- Adaptive expansion policy (query frequency tracking)
- Distributed condensation for federated learning
- CLI tool for offline condensation

**Deliverables**:
```rust
// Week 7: Performance optimization
crates/ruvector-gnn/src/condensation/
  ✓ SIMD-optimized centroid computation
  ✓ Parallel clustering (rayon)

// Week 8: Adaptive expansion
crates/ruvector-core/src/hybrid/
  ✓ ExpansionPolicy::Adaptive
  ✓ Query frequency tracking
  ✓ LRU cache tuning

// Week 9: CLI + distributed
crates/ruvector-cli/src/commands/condense.rs
  ✓ ruvector condense --ratio 50
crates/ruvector-distributed/src/sync.rs
  ✓ Condensed graph synchronization
```

**Success Criteria**:
- Condensation time <10s for 1M vectors
- Adaptive expansion improves latency by 20%+
- CLI can condense production-scale graphs

### Phase 4: Production Hardening (Weeks 10-12)

**Goals**:
- Comprehensive testing (property-based, fuzz, benchmarks)
- Documentation + examples
- Performance regression suite
- Multi-platform validation

**Deliverables**:
```rust
// Week 10: Testing
tests/condensation/
  ✓ Property-based tests (proptest)
  ✓ Fuzz testing (cargo-fuzz)
  ✓ Regression test suite

// Week 11: Documentation
docs/
  ✓ Graph Condensation Guide (user-facing)
  ✓ API documentation (rustdoc)
  ✓ Examples (edge device deployment)

// Week 12: Benchmarks + validation
benches/condensation.rs
  ✓ Condensation time benchmarks
  ✓ Search quality benchmarks
  ✓ Memory footprint benchmarks
```

**Success Criteria**:
- 100% code coverage for condensation module
- Passes all regression tests
- Documentation complete with 3+ examples
- Validated on ARM64, x86-64, WASM targets

## Success Metrics

### Performance Benchmarks

| Benchmark | Metric | Target | Measurement Method |
|-----------|--------|--------|-------------------|
| Condensation Time | Time to condense 1M vectors | <10s | `cargo bench condense_1m` |
| Memory Reduction | Footprint ratio (full/condensed) | 50x | `malloc_count` |
| Search Latency (condensed only) | p99 latency | <2ms | `criterion` benchmark |
| Search Latency (hybrid, cold) | p99 latency on first query | <3ms | Cache miss scenario |
| Search Latency (hybrid, warm) | p99 latency after warmup | <1.5ms | Cache hit scenario |
| Expansion Time | Time to expand 1 cluster | <0.5ms | `expand_regions()` profiling |

### Accuracy Metrics

| Dataset | Metric | Target | Baseline (Full Graph) |
|---------|--------|--------|-----------------------|
| SIFT1M | Recall@10 (50x compression) | >=0.92 | 0.95 |
| SIFT1M | Recall@100 (50x compression) | >=0.90 | 0.94 |
| GIST1M | Recall@10 (50x compression) | >=0.90 | 0.93 |
| GloVe-200 | Recall@10 (100x compression) | >=0.85 | 0.92 |
| Custom high-dim (1536d) | Recall@10 (50x compression) | >=0.88 | 0.94 |

### Memory/Latency Targets

| Configuration | Memory Footprint | Search Latency (p99) | Use Case |
|---------------|------------------|----------------------|----------|
| Full HNSW (1M vectors) | 4.8GB | 1.2ms | Server deployment |
| Condensed 50x (baseline) | 96MB | 1.5ms (cold), 1.2ms (warm) | Edge device |
| Condensed 100x (aggressive) | 48MB | 2.0ms (cold), 1.5ms (warm) | IoT device |
| Condensed 10x (conservative) | 480MB | 1.3ms (cold), 1.2ms (warm) | Embedded system |
| Hybrid (50x + on-demand) | 96MB + cache | 1.3ms (adaptive) | Mobile app |

**Measurement Tools**:
- Memory: `massif` (Valgrind), `heaptrack`, custom `malloc_count`
- Latency: `criterion` (Rust), `perf` (Linux profiling)
- Accuracy: Custom recall calculator against ground truth

### Quality Gates

All gates must pass before production release:

1. **Functional**:
   - ✓ All unit tests pass (100% coverage for core logic)
   - ✓ Integration tests pass on 3+ datasets
   - ✓ Serialization round-trip is lossless

2. **Performance**:
   - ✓ Memory reduction >= 40x (for 50x target config)
   - ✓ Condensation time <= 15s for 1M vectors
   - ✓ Search latency penalty <= 30% (cold start)

3. **Accuracy**:
   - ✓ Recall@10 >= 0.92 on SIFT1M (50x compression)
   - ✓ Recall@10 >= 0.85 on GIST1M (100x compression)
   - ✓ No catastrophic failures (recall < 0.5)

4. **Compatibility**:
   - ✓ Works on Linux x86-64, ARM64, macOS
   - ✓ Node.js bindings pass all tests
   - ✓ Backward compatible with existing indexes

## Risks and Mitigations

### Technical Risks

#### Risk 1: GNN Training Instability

**Description**:
Synthetic node embeddings may not converge during GNN training, leading to poor structure preservation.

**Probability**: Medium (30%)

**Impact**: High (blocks Phase 2)

**Mitigation**:
1. **Fallback**: Start with centroid-only embeddings (no GNN) in Phase 1
2. **Hyperparameter Tuning**: Grid search over learning rates (1e-4 to 1e-2)
3. **Loss Function Design**: Add regularization term to prevent mode collapse
4. **Early Stopping**: Monitor validation recall and stop if plateauing
5. **Alternative**: Use pre-trained graph embeddings (Node2Vec, GraphSAGE) if GNN fails

**Contingency Plan**:
If GNN training is unstable after 2 weeks of tuning, fall back to attention-weighted centroids (use existing attention mechanisms from Tier 1).

#### Risk 2: Cold Start Latency Regression

**Description**:
Condensed graph search may be slower than expected due to poor synthetic node placement.

**Probability**: Medium (40%)

**Impact**: Medium (user-facing latency)

**Mitigation**:
1. **Profiling**: Use `perf` to identify bottlenecks (likely distance computations)
2. **SIMD Optimization**: Vectorize distance calculations for synthetic nodes
3. **Caching**: Precompute pairwise distances between synthetic nodes
4. **Pruning**: Reduce condensed graph connectivity (fewer edges per node)
5. **Hybrid Strategy**: Always expand top-3 synthetic nodes to reduce uncertainty

**Contingency Plan**:
If cold start latency exceeds 2x full graph, add "warm cache" mode that preloads frequently accessed clusters based on query distribution.

#### Risk 3: Memory Overhead from Expansion Cache

**Description**:
LRU cache for expanded regions may consume more memory than expected, negating compression benefits.

**Probability**: Low (20%)

**Impact**: Medium (defeats purpose on edge devices)

**Mitigation**:
1. **Adaptive Cache Size**: Dynamically adjust cache size based on available memory
2. **Partial Expansion**: Only expand k-nearest neighbors within cluster (not full cluster)
3. **Compression**: Store expanded regions in quantized format (int8 instead of float32)
4. **Eviction Policy**: Evict based on access frequency + recency (LFU + LRU hybrid)

**Contingency Plan**:
If cache overhead exceeds 20% of condensed graph size, make expansion fully on-demand (no caching) and optimize expansion from disk (mmap).

#### Risk 4: Clustering Quality for High-Dimensional Data

**Description**:
Hierarchical clustering may produce imbalanced clusters in high-dimensional spaces (curse of dimensionality).

**Probability**: High (60%)

**Impact**: Medium (poor compression or accuracy)

**Mitigation**:
1. **Dimensionality Reduction**: Apply PCA or UMAP before clustering
2. **Alternative Algorithms**: Try spectral clustering or Louvain (graph-based, not distance-based)
3. **Cluster Validation**: Measure silhouette score and reject poor clusterings
4. **Adaptive Compression**: Use variable compression ratios per region (dense regions = higher compression)

**Contingency Plan**:
If clustering quality is poor (silhouette score < 0.3), switch to graph-based Louvain clustering using HNSW edges as adjacency matrix.

#### Risk 5: Serialization Format Bloat

**Description**:
`.cgraph` format may be larger than expected due to storing expansion maps and GNN weights.

**Probability**: Medium (35%)

**Impact**: Low (reduces compression benefits)

**Mitigation**:
1. **Sparse Storage**: Use sparse matrix formats (CSR) for expansion maps
2. **Quantization**: Store GNN embeddings in int8 (8x smaller)
3. **Compression**: Apply zstd compression to `.cgraph` file
4. **Lazy Loading**: Only load expansion map on-demand (not upfront)

**Contingency Plan**:
If `.cgraph` file exceeds 50% of condensed graph target size, remove GNN weights from serialization and recompute on load (trade disk space for CPU time).

### Operational Risks

#### Risk 6: User Confusion with Hybrid API

**Description**:
Users may not understand when to use condensed vs full vs hybrid graphs.

**Probability**: High (70%)

**Impact**: Low (documentation issue)

**Mitigation**:
1. **Clear Documentation**: Add decision tree (edge device → condensed, server → full, mobile → hybrid)
2. **Smart Defaults**: Auto-detect environment (check available memory) and choose policy
3. **Examples**: Provide 3 reference implementations (edge, mobile, server)
4. **Validation**: Add `validate_condensed()` method that warns if recall is too low

#### Risk 7: Debugging Difficulty

**Description**:
When condensed search returns wrong results, debugging is harder (no direct mapping to original nodes).

**Probability**: Medium (50%)

**Impact**: Medium (developer experience)

**Mitigation**:
1. **Logging**: Add verbose logging for expansion decisions
2. **Visualization**: Provide tool to visualize condensed graph + clusters
3. **Explain API**: Add `explain_search()` method that shows which clusters were searched
4. **Metrics**: Expose per-cluster recall metrics

---

## Appendix: Related Research

This design is based on:

1. **Graph Condensation for GNNs** (Jin et al., 2021): Core SFGC algorithm
2. **Structure-Preserving Graph Coarsening** (Loukas, 2019): Topological invariants
3. **Hierarchical Navigable Small Worlds** (Malkov & Yashunin, 2018): HNSW baseline
4. **Federated Graph Learning** (Wu et al., 2022): Distributed graph synchronization

Key differences from prior work:
- **Novel**: GNN-based synthetic node learning (prior work used simple centroids)
- **Novel**: Hybrid search with adaptive expansion (prior work only used condensed graph)
- **Engineering**: Production-ready Rust implementation with SIMD optimization