Phoenix-Vector is a highly optimized vector database engine and AI-native ingestion platform built entirely from scratch in Rust. Designed with strict systems architecture principles, it focuses on memory efficiency, zero-cost abstractions, multi-threaded CPU saturation, and blazingly fast approximate nearest neighbor (ANN) search capabilities.
Unlike standard vector databases that only accept floating-point arrays, Phoenix-Vector is an end-to-end platform. It can ingest raw files (.pdf, .docx, .xlsx, .txt), automatically chunk them, embed them using neural networks (like OpenAI or local ONNX models), and build a highly concurrent HNSW graph index in seconds.
-
Blazing Fast HNSW Architecture: Navigates multi-layered undirected graphs for highly optimized sub-linear
$O(\log N)$ search complexity. -
Rayon-Powered Parallel Construction: Batch ingestion operations are heavily parallelized across all CPU cores using
rayonand granularparking_lot::RwLockinterior mutability. - End-to-End Ingestion Pipeline: Native support for parsing, chunking, and embedding PDF, DOCX, XLSX, and TXT files directly into the graph.
-
Cache-Optimized Memory Layout: Converts String IDs into sequential
usizeindices internally, allowing for$O(1)$ memory access during graph jumps with zero heap-allocations in the hot loop. -
High-Concurrency API: The engine is wrapped in an
Arc<RwLock>and served via a lightning-fastaxumandtokionetwork layer. -
Durable Persistence: Native
bincodebinary serialization viaserdeensures the entire engine state is safely saved and loaded from disk in milliseconds.
%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#4B0082', 'primaryBorderColor': '#008080', 'lineColor': '#008080', 'tertiaryColor': '#1A1A1A'}}}%%
graph TD
A[Client Request] -->|Files / Text / Vectors| B[Axum API Gateway]
B --> C(Phoenix Engine Arc RwLock)
subgraph Ingestion Layer
C --> D[File Parser]
D -->|Text| E[Chunker]
E -->|Chunks| F[Embedding Model]
F -->|OpenAI / ONNX| G[f32 Vectors]
end
subgraph Core Storage
G --> H{Collection}
H -->|ID Mapper| I[HashMap String -> usize]
H -->|Contiguous Memory| J[Vec Record]
end
subgraph HNSW Graph Index
K((HnswIndex)) -.->|Rayon Parallel Linking| H
K -.->|Zero Cost Math| L[DistanceMetrics]
end
Our custom-built, Rayon-parallelized HNSW algorithm achieves production-grade performance in pure Rust without external C++ bindings.
Testing Hardware Specifications:
- CPU: Intel Core i5 (10th Gen)
- RAM: 12 GB
- GPU: None (Pure CPU SIMD Execution)
Tested with ef_construction=100, M=16, and ef_search=50 (using target-cpu=native for SIMD auto-vectorization):
- Massive Batch Insertion: 100,000 vectors inserted and fully graph-linked in ~2.6 minutes (avg 1.58ms per vector).
- Sub-Millisecond Search: 100 queries executed sequentially in ~448ms (avg 4.48ms per query) with high recall.
- Extreme Scale Insertion: 1,000,000 vectors generated in-memory and graph-linked in ~18.8 minutes (avg 1.13ms per vector) across CPU cores.
-
Logarithmic Search Magic: 100 queries executed in ~327ms (avg 3.27ms per query). Despite the dataset being 10x larger, the search time stayed identical due to the
$O(\log N)$ HNSW graph traversal!
