dearsky/wifi-densepose
1
0
Fork 0
Code Issues 20 Pull Requests 5 Actions Packages Projects Releases 1 Wiki Activity

Merge commit 'd803bfe2b1fe7f5e219e50ac20d6801a0a58ac75' as 'vendor/ruvector'

Browse Source
This commit is contained in:
ruv
2026-02-28 14:39:40 -05:00
parent 7885bf6278 d803bfe2b1
commit cd5943df23
7854 changed files with 3522914 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

813
vendor/ruvector/crates/ruvector-postgres/docs/API.md vendored Normal file
View File

@@ -0,0 +1,813 @@
# RuVector-Postgres API Reference
## Overview
Complete API reference for RuVector-Postgres extension, including SQL functions, operators, types, and GUC variables.
## Table of Contents
- [Data Types](#data-types)
- [SQL Functions](#sql-functions)
- [Operators](#operators)
- [Index Methods](#index-methods)
- [GUC Variables](#guc-variables)
- [Operator Classes](#operator-classes)
- [Usage Examples](#usage-examples)
## Data Types
### `ruvector(n)`
Primary vector type for dense floating-point vectors.
**Syntax:**
```sql
ruvector(dimensions)
```
**Parameters:**
- `dimensions`: Integer, 1 to 16,000
**Storage:**
- Header: 8 bytes
- Data: 4 bytes per dimension (f32)
- Total: 8 + (4 × dimensions) bytes
**Example:**
```sql
CREATE TABLE items (
id SERIAL PRIMARY KEY,
embedding ruvector(1536) -- OpenAI ada-002 dimensions
);
INSERT INTO items (embedding) VALUES ('[1.0, 2.0, 3.0]');
INSERT INTO items (embedding) VALUES (ARRAY[1.0, 2.0, 3.0]::ruvector);
```
### `halfvec(n)`
Half-precision (16-bit float) vector type.
**Syntax:**
```sql
halfvec(dimensions)
```
**Parameters:**
- `dimensions`: Integer, 1 to 16,000
**Storage:**
- Header: 8 bytes
- Data: 2 bytes per dimension (f16)
- Total: 8 + (2 × dimensions) bytes
**Benefits:**
- 50% memory reduction vs `ruvector`
- <0.01% accuracy loss for most embeddings
- SIMD f16 support on modern CPUs
**Example:**
```sql
CREATE TABLE items (
id SERIAL PRIMARY KEY,
embedding halfvec(1536) -- 3,080 bytes vs 6,152 for ruvector
);
-- Automatic conversion from ruvector
INSERT INTO items (embedding)
SELECT embedding::halfvec FROM ruvector_table;
```
### `sparsevec(n)`
Sparse vector type for high-dimensional sparse data.
**Syntax:**
```sql
sparsevec(dimensions)
```
**Parameters:**
- `dimensions`: Integer, 1 to 1,000,000
**Storage:**
- Header: 12 bytes
- Data: 8 bytes per non-zero element (u32 index + f32 value)
- Total: 12 + (8 × nnz) bytes
**Use Cases:**
- BM25 text embeddings
- TF-IDF vectors
- High-dimensional sparse features
**Example:**
```sql
CREATE TABLE documents (
id SERIAL PRIMARY KEY,
sparse_embedding sparsevec(50000) -- Only stores non-zero values
);
-- Sparse vector with 3 non-zero values
INSERT INTO documents (sparse_embedding)
VALUES ('{1:0.5, 100:0.8, 5000:0.3}/50000');
```
## SQL Functions
### Information Functions
#### `ruvector_version()`
Returns the extension version.
**Syntax:**
```sql
ruvector_version() → text
```
**Example:**
```sql
SELECT ruvector_version();
-- Output: '0.1.19'
```
#### `ruvector_simd_info()`
Returns detected SIMD capabilities.
**Syntax:**
```sql
ruvector_simd_info() → text
```
**Returns:**
- `'AVX512'`: AVX-512 support detected
- `'AVX2'`: AVX2 support detected
- `'NEON'`: ARM NEON support detected
- `'Scalar'`: No SIMD support
**Example:**
```sql
SELECT ruvector_simd_info();
-- Output: 'AVX2'
```
### Distance Functions
#### `ruvector_l2_distance(a, b)`
Compute L2 (Euclidean) distance.
**Syntax:**
```sql
ruvector_l2_distance(a ruvector, b ruvector) → float4
```
**Formula:**
```
L2(a, b) = sqrt(Σ(a[i] - b[i])²)
```
**Properties:**
- SIMD optimized
- Parallel safe
- Immutable
**Example:**
```sql
SELECT ruvector_l2_distance(
'[1.0, 2.0, 3.0]'::ruvector,
'[4.0, 5.0, 6.0]'::ruvector
);
-- Output: 5.196...
```
#### `ruvector_cosine_distance(a, b)`
Compute cosine distance.
**Syntax:**
```sql
ruvector_cosine_distance(a ruvector, b ruvector) → float4
```
**Formula:**
```
Cosine(a, b) = 1 - (a·b) / (||a|| ||b||)
```
**Range:** [0, 2]
- 0: Vectors point in same direction
- 1: Vectors are orthogonal
- 2: Vectors point in opposite directions
**Example:**
```sql
SELECT ruvector_cosine_distance(
'[1.0, 0.0]'::ruvector,
'[0.0, 1.0]'::ruvector
);
-- Output: 1.0 (orthogonal)
```
#### `ruvector_ip_distance(a, b)`
Compute inner product (negative dot product) distance.
**Syntax:**
```sql
ruvector_ip_distance(a ruvector, b ruvector) → float4
```
**Formula:**
```
IP(a, b) = -Σ(a[i] * b[i])
```
**Note:** Negative to work with `ORDER BY ASC`.
**Example:**
```sql
SELECT ruvector_ip_distance(
'[1.0, 2.0, 3.0]'::ruvector,
'[4.0, 5.0, 6.0]'::ruvector
);
-- Output: -32.0 (negative of 1*4 + 2*5 + 3*6)
```
#### `ruvector_l1_distance(a, b)`
Compute L1 (Manhattan) distance.
**Syntax:**
```sql
ruvector_l1_distance(a ruvector, b ruvector) → float4
```
**Formula:**
```
L1(a, b) = Σ|a[i] - b[i]|
```
**Example:**
```sql
SELECT ruvector_l1_distance(
'[1.0, 2.0, 3.0]'::ruvector,
'[4.0, 5.0, 6.0]'::ruvector
);
-- Output: 9.0
```
### Utility Functions
#### `ruvector_norm(v)`
Compute L2 norm (magnitude) of a vector.
**Syntax:**
```sql
ruvector_norm(v ruvector) → float4
```
**Formula:**
```
||v|| = sqrt(Σv[i]²)
```
**Example:**
```sql
SELECT ruvector_norm('[3.0, 4.0]'::ruvector);
-- Output: 5.0
```
#### `ruvector_normalize(v)`
Normalize vector to unit length.
**Syntax:**
```sql
ruvector_normalize(v ruvector) → ruvector
```
**Formula:**
```
normalize(v) = v / ||v||
```
**Example:**
```sql
SELECT ruvector_normalize('[3.0, 4.0]'::ruvector);
-- Output: [0.6, 0.8]
```
### Index Maintenance Functions
#### `ruvector_index_stats(index_name)`
Get statistics for a vector index.
**Syntax:**
```sql
ruvector_index_stats(index_name text) → TABLE(
index_name text,
index_size_mb numeric,
vector_count bigint,
dimensions int,
build_time_seconds numeric,
fragmentation_pct numeric
)
```
**Example:**
```sql
SELECT * FROM ruvector_index_stats('items_embedding_idx');
-- Output:
-- index_name | items_embedding_idx
-- index_size_mb | 512
-- vector_count | 1000000
-- dimensions | 1536
-- build_time_seconds | 45.2
-- fragmentation_pct | 2.3
```
#### `ruvector_index_maintenance(index_name)`
Perform maintenance on a vector index.
**Syntax:**
```sql
ruvector_index_maintenance(index_name text) → void
```
**Operations:**
- Removes deleted nodes
- Rebuilds fragmented layers
- Updates statistics
**Example:**
```sql
SELECT ruvector_index_maintenance('items_embedding_idx');
```
## Operators
### Distance Operators
| Operator | Name | Distance Metric | Order |
|----------|------|----------------|-------|
| `<->` | L2 | Euclidean | ASC |
| `<#>` | IP | Inner Product (negative) | ASC |
| `<=>` | Cosine | Cosine Distance | ASC |
| `<+>` | L1 | Manhattan | ASC |
**Properties:**
- All operators are IMMUTABLE
- All operators are PARALLEL SAFE
- All operators support index scans
### L2 Distance Operator (`<->`)
**Syntax:**
```sql
vector1 <-> vector2
```
**Example:**
```sql
SELECT * FROM items
ORDER BY embedding <-> '[1.0, 2.0, 3.0]'::ruvector
LIMIT 10;
```
### Cosine Distance Operator (`<=>`)
**Syntax:**
```sql
vector1 <=> vector2
```
**Example:**
```sql
SELECT * FROM items
ORDER BY embedding <=> '[1.0, 2.0, 3.0]'::ruvector
LIMIT 10;
```
### Inner Product Operator (`<#>`)
**Syntax:**
```sql
vector1 <#> vector2
```
**Note:** Returns negative dot product for ascending order.
**Example:**
```sql
SELECT * FROM items
ORDER BY embedding <#> '[1.0, 2.0, 3.0]'::ruvector
LIMIT 10;
```
### Manhattan Distance Operator (`<+>`)
**Syntax:**
```sql
vector1 <+> vector2
```
**Example:**
```sql
SELECT * FROM items
ORDER BY embedding <+> '[1.0, 2.0, 3.0]'::ruvector
LIMIT 10;
```
## Index Methods
### HNSW Index (`ruhnsw`)
Hierarchical Navigable Small World graph index.
**Syntax:**
```sql
CREATE INDEX index_name ON table_name
USING ruhnsw (column operator_class)
WITH (options);
```
**Options:**
| Option | Type | Default | Range | Description |
|--------|------|---------|-------|-------------|
| `m` | integer | 16 | 2-100 | Max connections per layer |
| `ef_construction` | integer | 64 | 4-1000 | Build-time search breadth |
| `quantization` | text | NULL | sq8, pq16, binary | Quantization method |
**Operator Classes:**
- `ruvector_l2_ops`: For `<->` operator
- `ruvector_ip_ops`: For `<#>` operator
- `ruvector_cosine_ops`: For `<=>` operator
**Example:**
```sql
-- Basic HNSW index
CREATE INDEX items_embedding_idx ON items
USING ruhnsw (embedding ruvector_l2_ops);
-- High recall HNSW index
CREATE INDEX items_embedding_idx ON items
USING ruhnsw (embedding ruvector_l2_ops)
WITH (m = 32, ef_construction = 200);
-- HNSW with quantization
CREATE INDEX items_embedding_idx ON items
USING ruhnsw (embedding ruvector_l2_ops)
WITH (m = 16, ef_construction = 100, quantization = 'sq8');
```
**Performance:**
- Search: O(log n)
- Insert: O(log n)
- Memory: ~1.5x vector data size
- Recall: 95-99%+ with tuned parameters
### IVFFlat Index (`ruivfflat`)
Inverted file with flat (uncompressed) vectors.
**Syntax:**
```sql
CREATE INDEX index_name ON table_name
USING ruivfflat (column operator_class)
WITH (lists = n);
```
**Options:**
| Option | Type | Default | Range | Description |
|--------|------|---------|-------|-------------|
| `lists` | integer | sqrt(rows) | 1-100000 | Number of clusters |
**Operator Classes:**
- `ruvector_l2_ops`: For `<->` operator
- `ruvector_ip_ops`: For `<#>` operator
- `ruvector_cosine_ops`: For `<=>` operator
**Example:**
```sql
-- Basic IVFFlat index
CREATE INDEX items_embedding_idx ON items
USING ruivfflat (embedding ruvector_l2_ops)
WITH (lists = 100);
-- IVFFlat for large dataset
CREATE INDEX items_embedding_idx ON items
USING ruivfflat (embedding ruvector_l2_ops)
WITH (lists = 1000);
```
**Performance:**
- Search: O(√n)
- Insert: O(1) after training
- Memory: Minimal overhead
- Recall: 90-95% with appropriate probes
**Training:**
IVFFlat requires training to find cluster centroids:
```sql
-- Index is automatically trained during creation
-- Training uses k-means on a sample of vectors
```
## GUC Variables
### `ruvector.ef_search`
Controls HNSW search quality (higher = better recall, slower).
**Syntax:**
```sql
SET ruvector.ef_search = value;
```
**Default:** 40
**Range:** 1-1000
**Scope:** Session, transaction, or global
**Example:**
```sql
-- Session-level
SET ruvector.ef_search = 200;
-- Transaction-level
BEGIN;
SET LOCAL ruvector.ef_search = 100;
SELECT ... ORDER BY embedding <-> query;
COMMIT;
-- Global
ALTER SYSTEM SET ruvector.ef_search = 100;
SELECT pg_reload_conf();
```
### `ruvector.probes`
Controls IVFFlat search quality (higher = better recall, slower).
**Syntax:**
```sql
SET ruvector.probes = value;
```
**Default:** 1
**Range:** 1-10000
**Recommended:** sqrt(lists) for 90%+ recall
**Example:**
```sql
-- For lists = 100, use probes = 10
SET ruvector.probes = 10;
```
## Operator Classes
### `ruvector_l2_ops`
For L2 (Euclidean) distance queries.
**Usage:**
```sql
CREATE INDEX ... USING ruhnsw (embedding ruvector_l2_ops);
SELECT ... ORDER BY embedding <-> query;
```
### `ruvector_ip_ops`
For inner product distance queries.
**Usage:**
```sql
CREATE INDEX ... USING ruhnsw (embedding ruvector_ip_ops);
SELECT ... ORDER BY embedding <#> query;
```
### `ruvector_cosine_ops`
For cosine distance queries.
**Usage:**
```sql
CREATE INDEX ... USING ruhnsw (embedding ruvector_cosine_ops);
SELECT ... ORDER BY embedding <=> query;
```
## Usage Examples
### Basic Vector Search
```sql
-- Create table
CREATE TABLE documents (
id SERIAL PRIMARY KEY,
content TEXT,
embedding ruvector(1536)
);
-- Insert vectors
INSERT INTO documents (content, embedding) VALUES
('Document 1', '[0.1, 0.2, ...]'::ruvector),
('Document 2', '[0.3, 0.4, ...]'::ruvector);
-- Create index
CREATE INDEX documents_embedding_idx ON documents
USING ruhnsw (embedding ruvector_l2_ops);
-- Search
SELECT content, embedding <-> '[0.5, 0.6, ...]'::ruvector AS distance
FROM documents
ORDER BY distance
LIMIT 10;
```
### Filtered Vector Search
```sql
-- Search with WHERE clause
SELECT content, embedding <-> query AS distance
FROM documents
WHERE category = 'technology'
ORDER BY distance
LIMIT 10;
```
### Batch Distance Calculation
```sql
-- Compute distances to multiple vectors
WITH queries AS (
SELECT id, embedding AS query FROM queries_table
)
SELECT
q.id AS query_id,
d.id AS doc_id,
d.embedding <-> q.query AS distance
FROM documents d
CROSS JOIN queries q
ORDER BY q.id, distance
LIMIT 100;
```
### Vector Arithmetic
```sql
-- Add vectors
SELECT (embedding1 + embedding2) AS sum FROM ...;
-- Subtract vectors
SELECT (embedding1 - embedding2) AS diff FROM ...;
-- Scalar multiplication
SELECT (embedding * 2.0) AS scaled FROM ...;
```
### Hybrid Search (Vector + Text)
```sql
-- Combine vector similarity with text search
SELECT
content,
embedding <-> query_vector AS vector_score,
ts_rank(to_tsvector(content), to_tsquery('search terms')) AS text_score,
(0.7 * (1 / (1 + embedding <-> query_vector)) +
0.3 * ts_rank(to_tsvector(content), to_tsquery('search terms'))) AS combined_score
FROM documents
WHERE to_tsvector(content) @@ to_tsquery('search terms')
ORDER BY combined_score DESC
LIMIT 10;
```
### Index Parameter Tuning
```sql
-- Test different ef_search values
DO $$
DECLARE
ef_val INTEGER;
BEGIN
FOR ef_val IN 10, 20, 40, 80, 160 LOOP
EXECUTE format('SET LOCAL ruvector.ef_search = %s', ef_val);
RAISE NOTICE 'ef_search = %', ef_val;
PERFORM * FROM items
ORDER BY embedding <-> '[...]'::ruvector
LIMIT 10;
END LOOP;
END $$;
```
## Performance Tips
1. **Choose the right index:**
- HNSW: Best for high recall, fast queries
- IVFFlat: Best for memory-constrained environments
2. **Tune index parameters:**
- Higher `m` and `ef_construction`: Better recall, larger index
- Higher `ef_search`: Better recall, slower queries
3. **Use appropriate vector type:**
- `ruvector`: Full precision
- `halfvec`: 50% memory savings, minimal accuracy loss
- `sparsevec`: Massive savings for sparse data
4. **Enable parallelism:**
```sql
SET max_parallel_workers_per_gather = 4;
```
5. **Use quantization for large datasets:**
```sql
WITH (quantization = 'sq8') -- 4x memory reduction
```
## See Also
- [ARCHITECTURE.md](./ARCHITECTURE.md) - System architecture
- [SIMD_OPTIMIZATION.md](./SIMD_OPTIMIZATION.md) - Performance details
- [MIGRATION.md](./MIGRATION.md) - Migrating from pgvector