wifi-densepose/docs/research/latent-space/attention-mechanisms-research.md

# Alternative Attention Mechanisms for GNN Latent Space

## Executive Summary

This document explores alternative attention mechanisms beyond the current scaled dot-product multi-head attention used in RuVector. We analyze mechanisms that could better bridge the gap between high-dimensional latent spaces and graph topology, with emphasis on efficiency, expressiveness, and geometric awareness.

**Current**: Multi-head scaled dot-product attention (O(n²) complexity)
**Goal**: Enhance attention to capture graph structure, reduce complexity, and improve latent-graph interplay

---

## 1. Current Attention Mechanism Analysis

### 1.1 Scaled Dot-Product Attention (Current Implementation)

**File**: `crates/ruvector-gnn/src/layer.rs:84-205`

```
Attention(Q, K, V) = softmax(QK^T / √d_k) V
```

**Strengths**:
- ✓ Permutation invariant
- ✓ Differentiable
- ✓ Well-understood training dynamics
- ✓ Parallel computation

**Weaknesses**:
- ✗ No explicit edge features
- ✗ No positional/structural encoding
- ✗ Uniform geometric assumptions (Euclidean)
- ✗ O(d·h²) computational cost
- ✗ Attention scores independent of graph topology

### 1.2 Multi-Head Decomposition (Current)

```
MultiHead(Q, K, V) = Concat(head_1, ..., head_h) W_o
```

**Strengths**:
- ✓ Multiple representation subspaces
- ✓ Different aspects of neighborhood

**Weaknesses**:
- ✗ Fixed number of heads
- ✗ Heads learn similar patterns (redundancy)
- ✗ No explicit head specialization

---

## 2. Graph Attention Networks (GAT) Extensions

### 2.1 Edge-Featured Attention

**Key Innovation**: Incorporate edge attributes into attention computation

```
e_{ij} = LeakyReLU(a^T [W h_i || W h_j || W_e edge_{ij}])
α_{ij} = softmax_j(e_{ij})
h'_i = σ(Σ_{j∈N(i)} α_{ij} W h_j)
```

**Implementation Proposal**:

```rust
pub struct EdgeFeaturedAttention {
    w_node: Linear,      // Node transformation
    w_edge: Linear,      // Edge transformation
    a: Vec<f32>,         // Attention coefficients
    activation: LeakyReLU,
}

impl EdgeFeaturedAttention {
    fn forward(
        &self,
        query_node: &[f32],
        neighbor_nodes: &[Vec<f32>],
        edge_features: &[Vec<f32>],  // NEW
    ) -> Vec<f32> {
        // 1. Transform nodes and edges
        let q_trans = self.w_node.forward(query_node);
        let n_trans: Vec<_> = neighbor_nodes.iter()
            .map(|n| self.w_node.forward(n))
            .collect();
        let e_trans: Vec<_> = edge_features.iter()
            .map(|e| self.w_edge.forward(e))
            .collect();

        // 2. Compute attention with edge features
        let mut scores = Vec::new();
        for (n, e) in n_trans.iter().zip(e_trans.iter()) {
            // Concatenate [query || neighbor || edge]
            let concat = [&q_trans[..], &n[..], &e[..]].concat();
            let score = dot_product(&self.a, &concat);
            scores.push(self.activation.forward(score));
        }

        // 3. Softmax and aggregate
        let weights = softmax(&scores);
        weighted_sum(&n_trans, &weights)
    }
}
```

**Benefits for RuVector**:
- Edge weights (distances) become learnable features
- HNSW layer information can be encoded in edges
- Better captures graph topology in latent space

**Complexity**: O(d·(h_node + h_edge + h_attn))

---

## 3. Hyperbolic Attention

### 3.1 Motivation

**Problem**: HNSW has hierarchical structure, but Euclidean space poorly represents trees/hierarchies

**Solution**: Operate in hyperbolic space (Poincaré ball or hyperboloid model)

### 3.2 Poincaré Ball Attention

**Poincaré Ball Model**:
```
B^d = {x ∈ R^d : ||x|| < 1}
Distance: d(x, y) = arcosh(1 + 2||x - y||² / ((1-||x||²)(1-||y||²)))
```

**Hyperbolic Attention Mechanism**:

```
# Key differences from Euclidean:
1. Use hyperbolic distance for similarity
2. Exponential map for transformations
3. Logarithmic map for aggregation

HyperbolicAttention(q, k, v):
    # Compute hyperbolic similarity
    sim_ij = -d_poincare(q, k_j)  # Negative distance

    # Softmax in tangent space
    α_ij = softmax(sim_ij / τ)

    # Aggregate in hyperbolic space
    result = ⊕_{j} (α_ij ⊗ v_j)  # Möbius addition

    return result
```

**Implementation Sketch**:

```rust
pub struct HyperbolicAttention {
    curvature: f32,  // Negative curvature (e.g., -1.0)
}

impl HyperbolicAttention {
    // Poincaré distance
    fn poincare_distance(&self, x: &[f32], y: &[f32]) -> f32 {
        let diff_norm_sq = l2_norm_squared(&subtract(x, y));
        let x_norm_sq = l2_norm_squared(x);
        let y_norm_sq = l2_norm_squared(y);

        let numerator = 2.0 * diff_norm_sq;
        let denominator = (1.0 - x_norm_sq) * (1.0 - y_norm_sq);

        self.curvature.abs().sqrt() * (1.0 + numerator / denominator).acosh()
    }

    // Möbius addition (hyperbolic vector addition)
    fn mobius_add(&self, x: &[f32], y: &[f32]) -> Vec<f32> {
        let x_norm_sq = l2_norm_squared(x);
        let y_norm_sq = l2_norm_squared(y);
        let xy_dot = dot_product(x, y);

        let numerator_coef = (1.0 + 2.0*xy_dot + y_norm_sq) / (1.0 - x_norm_sq);
        let denominator_coef = (1.0 + 2.0*xy_dot + x_norm_sq*y_norm_sq) / (1.0 - x_norm_sq);

        // (1+2⟨x,y⟩+||y||²)x + (1-||x||²)y / (1+2⟨x,y⟩+||x||²||y||²)
        let numerator = add(
            &scale(x, numerator_coef),
            &scale(y, 1.0 - x_norm_sq)
        );
        scale(&numerator, 1.0 / denominator_coef)
    }

    fn forward(
        &self,
        query: &[f32],
        keys: &[Vec<f32>],
        values: &[Vec<f32>],
    ) -> Vec<f32> {
        // 1. Compute hyperbolic similarities (negative distances)
        let scores: Vec<f32> = keys.iter()
            .map(|k| -self.poincare_distance(query, k))
            .collect();

        // 2. Softmax
        let weights = softmax(&scores);

        // 3. Hyperbolic aggregation
        let mut result = vec![0.0; values[0].len()];
        for (v, &w) in values.iter().zip(weights.iter()) {
            let scaled = self.mobius_scalar_mult(w, v);
            result = self.mobius_add(&result, &scaled);
        }

        result
    }
}
```

**Benefits for HNSW**:
- Natural representation of hierarchical layers
- Exponential capacity (tree-like structures)
- Distance preserves hierarchy

**Challenges**:
- Numerical instability near ball boundary (||x|| → 1)
- More complex backpropagation
- Requires hyperbolic embeddings throughout pipeline

---

## 4. Sparse Attention Patterns

### 4.1 Local + Global Attention (Longformer-style)

**Motivation**: Full attention is O(n²), wasteful for graphs with local structure

**Pattern**:
```
Attention Matrix Structure:
  [L L L G 0 0 0 0]
  [L L L L G 0 0 0]
  [L L L L L G 0 0]
  [G L L L L L G 0]
  [0 G L L L L L G]
  [0 0 G L L L L L]
  [0 0 0 G L L L L]
  [0 0 0 0 G L L L]

L = Local attention (1-hop neighbors)
G = Global attention (HNSW higher layers)
0 = No attention
```

**Implementation**:

```rust
pub struct SparseGraphAttention {
    local_attn: MultiHeadAttention,
    global_attn: MultiHeadAttention,
    local_window: usize,  // K-hop neighborhood
}

impl SparseGraphAttention {
    fn forward(
        &self,
        query: &[f32],
        neighbor_embeddings: &[Vec<f32>],
        neighbor_layers: &[usize],  // HNSW layer for each neighbor
    ) -> Vec<f32> {
        // Split neighbors by locality
        let (local_neighbors, local_indices): (Vec<_>, Vec<_>) =
            neighbor_embeddings.iter().enumerate()
                .filter(|(i, _)| neighbor_layers[*i] == 0)  // Layer 0 = local
                .unzip();

        let (global_neighbors, global_indices): (Vec<_>, Vec<_>) =
            neighbor_embeddings.iter().enumerate()
                .filter(|(i, _)| neighbor_layers[*i] > 0)  // Higher layers = global
                .unzip();

        // Compute local attention
        let local_output = if !local_neighbors.is_empty() {
            self.local_attn.forward(query, &local_neighbors, &local_neighbors)
        } else {
            vec![0.0; query.len()]
        };

        // Compute global attention
        let global_output = if !global_neighbors.is_empty() {
            self.global_attn.forward(query, &global_neighbors, &global_neighbors)
        } else {
            vec![0.0; query.len()]
        };

        // Combine (learned gating)
        combine_local_global(&local_output, &global_output)
    }
}
```

**Complexity**: O(k_local + k_global) instead of O(n²)

---

## 5. Linear Attention (O(n) complexity)

### 5.1 Kernel-Based Linear Attention

**Key Idea**: Replace softmax with kernel feature map

```
Standard: Attention(Q, K, V) = softmax(QK^T) V
Linear:   Attention(Q, K, V) = φ(Q) (φ(K)^T V) / (φ(Q) (φ(K)^T 1))

where φ: R^d → R^D is a feature map
```

**Random Feature Approximation** (Performer):

```rust
pub struct LinearAttention {
    num_features: usize,  // D (typically 256-512)
    random_features: Array2<f32>,  // Random projection matrix
}

impl LinearAttention {
    fn feature_map(&self, x: &[f32]) -> Vec<f32> {
        // Random Fourier Features
        let proj = self.random_features.dot(&Array1::from_vec(x.to_vec()));
        let scale = 1.0 / (self.num_features as f32).sqrt();

        proj.mapv(|z| {
            scale * (z.cos() + z.sin())  // Simplified RFF
        }).to_vec()
    }

    fn forward(
        &self,
        query: &[f32],
        keys: &[Vec<f32>],
        values: &[Vec<f32>],
    ) -> Vec<f32> {
        // 1. Apply feature map
        let q_feat = self.feature_map(query);
        let k_feats: Vec<_> = keys.iter().map(|k| self.feature_map(k)).collect();

        // 2. Compute K^T V (sum over neighbors)
        let mut kv = vec![0.0; values[0].len()];
        for (k_feat, v) in k_feats.iter().zip(values.iter()) {
            for (i, &v_i) in v.iter().enumerate() {
                kv[i] += k_feat.iter().sum::<f32>() * v_i;
            }
        }

        // 3. Compute Q (K^T V)
        let numerator: Vec<f32> = kv.iter()
            .map(|&kv_i| q_feat.iter().sum::<f32>() * kv_i)
            .collect();

        // 4. Normalize by Q (K^T 1)
        let denominator: f32 = q_feat.iter().sum::<f32>()
            * k_feats.iter().map(|k| k.iter().sum::<f32>()).sum::<f32>();

        numerator.iter().map(|&n| n / denominator).collect()
    }
}
```

**Benefits**:
- **O(n) complexity**: Scales linearly with graph size
- **Theoretically grounded**: Approximates softmax attention
- **Parallel friendly**: Matrix operations

**Tradeoffs**:
- Approximation error vs. exact softmax
- Requires more random features for accuracy
- Less interpretable attention weights

---

## 6. Rotary Position Embeddings (RoPE) for Graphs

### 6.1 Motivation

**Problem**: Graph attention has no notion of "position" or "distance" beyond explicit edge features

**Solution**: Encode relative distances/positions via rotation

### 6.2 RoPE Mathematics

**Standard RoPE** (for sequences):
```
RoPE(x, m) = [
    x₀ cos(mθ₀) - x₁ sin(mθ₀),
    x₀ sin(mθ₀) + x₁ cos(mθ₀),
    x₂ cos(mθ₁) - x₃ sin(mθ₁),
    ...
]

where m = position index, θᵢ = 10000^(-2i/d)
```

**Graph RoPE Adaptation**:
```
Instead of sequential position m, use:
- Graph distance (shortest path length)
- HNSW layer index
- Normalized edge weight
```

**Implementation**:

```rust
pub struct GraphRoPE {
    dim: usize,
    base: f32,  // Base frequency (default 10000)
}

impl GraphRoPE {
    fn apply_rotation(&self, embedding: &[f32], distance: f32) -> Vec<f32> {
        let mut rotated = vec![0.0; embedding.len()];

        for i in (0..self.dim).step_by(2) {
            let theta = distance / self.base.powf(2.0 * i as f32 / self.dim as f32);
            let cos_theta = theta.cos();
            let sin_theta = theta.sin();

            rotated[i] = embedding[i] * cos_theta - embedding[i+1] * sin_theta;
            rotated[i+1] = embedding[i] * sin_theta + embedding[i+1] * cos_theta;
        }

        rotated
    }

    fn forward_attention(
        &self,
        query: &[f32],
        keys: &[Vec<f32>],
        values: &[Vec<f32>],
        distances: &[f32],  // NEW: graph distances
    ) -> Vec<f32> {
        // Apply RoPE to query and keys based on relative distance
        let q_rotated = self.apply_rotation(query, 0.0);  // Query at "origin"

        let mut scores = Vec::new();
        for (k, &dist) in keys.iter().zip(distances.iter()) {
            let k_rotated = self.apply_rotation(k, dist);
            let score = dot_product(&q_rotated, &k_rotated);
            scores.push(score);
        }

        let weights = softmax(&scores);
        weighted_sum(values, &weights)
    }
}
```

**Benefits**:
- Encodes distance without explicit features
- Relative position encoding (rotation-invariant)
- Efficient (just rotations, no extra parameters)

**Graph-Specific Applications**:
1. **HNSW Layer Distance**: Encode which layer neighbors come from
2. **Shortest Path Distance**: Penalize far nodes in latent space
3. **Edge Weight Encoding**: Continuous rotation based on edge weight

---

## 7. Flash Attention (Memory-Efficient)

### 7.1 Problem

Standard attention materializes the full attention matrix in memory:
```
Memory: O(n²)  for n neighbors
```

For dense graphs or large neighborhoods, this is prohibitive.

### 7.2 Flash Attention Algorithm

**Key Ideas**:
1. Tile the attention computation
2. Recompute attention on-the-fly during backward pass
3. Never materialize full attention matrix

**Pseudocode**:

```
FlashAttention(Q, K, V):
    # Divide Q, K, V into blocks
    Q_blocks = split(Q, block_size)
    K_blocks = split(K, block_size)
    V_blocks = split(V, block_size)

    O = zeros_like(Q)

    # Outer loop: iterate over query blocks
    for Q_i in Q_blocks:
        row_max = -inf
        row_sum = 0

        # Inner loop: iterate over key blocks
        for K_j, V_j in zip(K_blocks, V_blocks):
            # Compute attention block
            S_ij = Q_i @ K_j^T / sqrt(d)

            # Online softmax (numerically stable)
            new_max = max(row_max, max(S_ij))
            exp_S = exp(S_ij - new_max)

            # Update running statistics
            correction = exp(row_max - new_max)
            row_sum = row_sum * correction + sum(exp_S)
            row_max = new_max

            # Accumulate output
            O_i += exp_S @ V_j

        # Final normalization
        O_i /= row_sum

    return O
```

**Implementation Note**:

Flash Attention requires careful low-level optimization (CUDA kernels, tiling, SRAM management). For RuVector:

```rust
// Simplified tiled version for CPU
pub struct TiledAttention {
    block_size: usize,
}

impl TiledAttention {
    fn forward_tiled(
        &self,
        query: &[f32],
        keys: &[Vec<f32>],
        values: &[Vec<f32>],
    ) -> Vec<f32> {
        let n = keys.len();
        let mut output = vec![0.0; query.len()];
        let mut row_sum = 0.0;
        let mut row_max = f32::NEG_INFINITY;

        // Process keys in blocks
        for chunk_start in (0..n).step_by(self.block_size) {
            let chunk_end = (chunk_start + self.block_size).min(n);

            // Compute attention for this block
            let chunk_keys = &keys[chunk_start..chunk_end];
            let chunk_values = &values[chunk_start..chunk_end];

            let scores: Vec<f32> = chunk_keys.iter()
                .map(|k| dot_product(query, k))
                .collect();

            // Online softmax update
            let new_max = scores.iter().copied().fold(row_max, f32::max);
            let exp_scores: Vec<f32> = scores.iter()
                .map(|&s| (s - new_max).exp())
                .collect();

            let correction = (row_max - new_max).exp();
            row_sum = row_sum * correction + exp_scores.iter().sum::<f32>();
            row_max = new_max;

            // Accumulate weighted values
            for (v, &weight) in chunk_values.iter().zip(exp_scores.iter()) {
                for (o, &v_i) in output.iter_mut().zip(v.iter()) {
                    *o = *o * correction + weight * v_i;
                }
            }
        }

        // Final normalization
        output.iter().map(|&o| o / row_sum).collect()
    }
}
```

**Benefits**:
- **Memory**: O(n) instead of O(n²)
- **Speed**: Can be faster due to better cache locality
- **Scalability**: Handle larger neighborhoods

---

## 8. Mixture of Experts (MoE) Attention

### 8.1 Concept

Different attention mechanisms for different graph patterns:

```
MoE-Attention(query, keys, values):
    # Router decides which expert(s) to use
    router_scores = Router(query)
    expert_indices = topk(router_scores, k=2)

    # Apply selected experts
    outputs = []
    for expert_idx in expert_indices:
        expert_output = Experts[expert_idx](query, keys, values)
        outputs.append(expert_output * router_scores[expert_idx])

    return sum(outputs)
```

**Graph-Specific Experts**:
1. **Local Expert**: For 1-hop neighbors (standard attention)
2. **Hierarchical Expert**: For HNSW higher layers (hyperbolic attention)
3. **Global Expert**: For distant nodes (linear attention)
4. **Structural Expert**: Edge-featured attention

### 8.2 Implementation

```rust
pub enum AttentionExpert {
    Standard(MultiHeadAttention),
    Hyperbolic(HyperbolicAttention),
    Linear(LinearAttention),
    EdgeFeatured(EdgeFeaturedAttention),
}

pub struct MoEAttention {
    router: Linear,  // Maps query to expert scores
    experts: Vec<AttentionExpert>,
    top_k: usize,
}

impl MoEAttention {
    fn forward(
        &self,
        query: &[f32],
        keys: &[Vec<f32>],
        values: &[Vec<f32>],
        edge_features: Option<&[Vec<f32>]>,
    ) -> Vec<f32> {
        // 1. Route to experts
        let router_scores = self.router.forward(query);
        let expert_weights = softmax(&router_scores);
        let top_experts = topk_indices(&expert_weights, self.top_k);

        // 2. Compute weighted expert outputs
        let mut output = vec![0.0; query.len()];
        for &expert_idx in &top_experts {
            let expert_output = match &self.experts[expert_idx] {
                AttentionExpert::Standard(attn) =>
                    attn.forward(query, keys, values),
                AttentionExpert::Hyperbolic(attn) =>
                    attn.forward(query, keys, values),
                AttentionExpert::Linear(attn) =>
                    attn.forward(query, keys, values),
                AttentionExpert::EdgeFeatured(attn) =>
                    attn.forward(query, keys, values, edge_features.unwrap()),
            };

            let weight = expert_weights[expert_idx];
            for (o, &e) in output.iter_mut().zip(expert_output.iter()) {
                *o += weight * e;
            }
        }

        output
    }
}
```

**Benefits**:
- Adaptive to different graph neighborhoods
- Specialization reduces computation
- Router learns which mechanism suits which context

---

## 9. Cross-Attention Between Graph and Latent

### 9.1 Motivation

**Problem**: Current attention only looks at graph neighbors. What about latent space neighbors?

**Solution**: Cross-attention between topological neighbors (graph) and semantic neighbors (latent)

### 9.2 Dual-Space Attention

```
Given node v:
- Graph neighbors: N_G(v) = {u : (u,v) ∈ E}
- Latent neighbors: N_L(v) = TopK({u : sim(h_u, h_v) > threshold})

CrossAttention(v):
    # Graph attention
    graph_out = Attention(h_v, {h_u}_{u∈N_G}, {h_u}_{u∈N_G})

    # Latent attention
    latent_out = Attention(h_v, {h_u}_{u∈N_L}, {h_u}_{u∈N_L})

    # Cross-attention: graph queries latent
    cross_out = Attention(graph_out, {h_u}_{u∈N_L}, {h_u}_{u∈N_L})

    # Fusion
    return Combine(graph_out, latent_out, cross_out)
```

**Implementation**:

```rust
pub struct DualSpaceAttention {
    graph_attn: MultiHeadAttention,
    latent_attn: MultiHeadAttention,
    cross_attn: MultiHeadAttention,
    fusion: Linear,
}

impl DualSpaceAttention {
    fn forward(
        &self,
        query: &[f32],
        graph_neighbors: &[Vec<f32>],
        all_embeddings: &[Vec<f32>],  // For latent neighbor search
        k_latent: usize,
    ) -> Vec<f32> {
        // 1. Graph attention (topology-based)
        let graph_output = self.graph_attn.forward(
            query,
            graph_neighbors,
            graph_neighbors
        );

        // 2. Find latent neighbors (similarity-based)
        let latent_neighbors = self.find_latent_neighbors(
            query,
            all_embeddings,
            k_latent
        );

        // 3. Latent attention (embedding-based)
        let latent_output = self.latent_attn.forward(
            query,
            &latent_neighbors,
            &latent_neighbors
        );

        // 4. Cross-attention (graph context attends to latent space)
        let cross_output = self.cross_attn.forward(
            &graph_output,
            &latent_neighbors,
            &latent_neighbors
        );

        // 5. Fusion
        let concatenated = [
            &graph_output[..],
            &latent_output[..],
            &cross_output[..],
        ].concat();

        self.fusion.forward(&concatenated)
    }

    fn find_latent_neighbors(
        &self,
        query: &[f32],
        all_embeddings: &[Vec<f32>],
        k: usize,
    ) -> Vec<Vec<f32>> {
        // Compute similarities
        let mut similarities: Vec<(usize, f32)> = all_embeddings
            .iter()
            .enumerate()
            .map(|(i, emb)| (i, cosine_similarity(query, emb)))
            .collect();

        // Sort by similarity
        similarities.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap());

        // Return top-k
        similarities.iter()
            .take(k)
            .map(|(i, _)| all_embeddings[*i].clone())
            .collect()
    }
}
```

**Benefits**:
- Bridges topology and semantics
- Captures "similar but not connected" nodes
- Enriches latent space with graph structure

---

## 10. Comparison Matrix

| Mechanism | Complexity | Edge Features | Geometry | Memory | Use Case |
|-----------|------------|---------------|----------|--------|----------|
| **Current (MHA)** | O(d·h²) | ✗ | Euclidean | O(d·h) | General purpose |
| **GAT + Edges** | O(d·h²) | ✓ | Euclidean | O(d·h) | Rich edge info |
| **Hyperbolic** | O(d·h²) | ✗ | Hyperbolic | O(d·h) | Hierarchical graphs |
| **Sparse (Local+Global)** | O(k_l + k_g) | ✗ | Euclidean | O((k_l+k_g)·h) | Large graphs |
| **Linear (Performer)** | O(d·D) | ✗ | Euclidean | O(D·h) | Scalability |
| **RoPE** | O(d·h²) | Implicit | Euclidean | O(d·h) | Distance encoding |
| **Flash Attention** | O(d·h²) | ✗ | Euclidean | O(h) | Memory efficiency |
| **MoE** | Variable | ✓ | Mixed | Variable | Heterogeneous graphs |
| **Cross (Dual-Space)** | O(d·h² + k²·h) | ✗ | Dual | O((d+k)·h) | Latent-graph bridge |

---

## 11. Recommendations for RuVector

### 11.1 Short-Term (Immediate Implementation)

**1. Edge-Featured Attention**
- **Priority**: HIGH
- **Effort**: LOW-MEDIUM
- **Reason**: HNSW edge weights are currently underutilized
- **Implementation**: Extend current `MultiHeadAttention` to include edge features

**2. Sparse Attention (Local + Global)**
- **Priority**: HIGH
- **Effort**: MEDIUM
- **Reason**: Natural fit for HNSW's layered structure
- **Implementation**: Separate attention for layer 0 (local) vs. higher layers (global)

**3. RoPE for Distance Encoding**
- **Priority**: MEDIUM
- **Effort**: LOW
- **Reason**: Encode HNSW layer or edge distance without extra parameters
- **Implementation**: Apply rotation based on layer index or edge weight

### 11.2 Medium-Term (Next Quarter)

**4. Linear Attention (Performer)**
- **Priority**: MEDIUM
- **Effort**: MEDIUM-HIGH
- **Reason**: Scalability for large graphs
- **Implementation**: Replace softmax with random feature approximation

**5. Flash Attention**
- **Priority**: LOW-MEDIUM
- **Effort**: HIGH
- **Reason**: Memory efficiency for dense neighborhoods
- **Implementation**: Tiled computation, may need GPU optimization

### 11.3 Long-Term (Research Exploration)

**6. Hyperbolic Attention**
- **Priority**: MEDIUM
- **Effort**: HIGH
- **Reason**: Hierarchical HNSW structure naturally hyperbolic
- **Implementation**: Full pipeline change to hyperbolic embeddings

**7. Mixture of Experts**
- **Priority**: LOW
- **Effort**: HIGH
- **Reason**: Heterogeneous graph patterns
- **Implementation**: Multiple attention types with learned routing

**8. Cross-Attention (Dual-Space)**
- **Priority**: HIGH (Research)
- **Effort**: HIGH
- **Reason**: Core to latent-graph interplay
- **Implementation**: Requires efficient latent neighbor search (ANN)

---

## 12. Implementation Roadmap

### Phase 1: Extend Current Attention (1-2 weeks)
```rust
// Add edge features to existing MultiHeadAttention
impl MultiHeadAttention {
    pub fn forward_with_edges(
        &self,
        query: &[f32],
        keys: &[Vec<f32>],
        values: &[Vec<f32>],
        edge_features: &[Vec<f32>],  // NEW
    ) -> Vec<f32> {
        // Modify attention score computation to include edges
    }
}
```

### Phase 2: Sparse Attention Variant (2-3 weeks)
```rust
// Separate local and global attention based on HNSW layer
pub struct HNSWAwareAttention {
    local: MultiHeadAttention,
    global: MultiHeadAttention,
}
```

### Phase 3: Alternative Mechanisms (1-2 months)
- Implement RoPE for distance encoding
- Prototype Linear Attention
- Benchmark all variants

### Phase 4: Research Exploration (Ongoing)
- Hyperbolic embeddings (full pipeline change)
- MoE attention routing
- Cross-attention with latent neighbors

---

## References

### Papers
1. **GAT**: Veličković et al. (2018) - Graph Attention Networks
2. **Hyperbolic**: Chami et al. (2019) - Hyperbolic Graph Convolutional Neural Networks
3. **Longformer**: Beltagy et al. (2020) - Longformer: The Long-Document Transformer
4. **Performer**: Choromanski et al. (2020) - Rethinking Attention with Performers
5. **RoPE**: Su et al. (2021) - RoFormer: Enhanced Transformer with Rotary Position Embedding
6. **Flash Attention**: Dao et al. (2022) - FlashAttention: Fast and Memory-Efficient Exact Attention
7. **MoE**: Shazeer et al. (2017) - Outrageously Large Neural Networks: The Sparsely-Gated MoE

### RuVector Code References
- `crates/ruvector-gnn/src/layer.rs:84-205` - Current MultiHeadAttention
- `crates/ruvector-gnn/src/search.rs:38-86` - Differentiable search with softmax

---

**Document Version**: 1.0
**Last Updated**: 2025-11-30
**Author**: RuVector Research Team