Introducing AI-Lake Format 🏔️
We're open-sourcing ailake — a self-contained file format for AI Lakehouses, written in Rust.
The problem: Running BI and GenAI on the same data means juggling two stacks — a Data Lake for analytics and a separate vector DB (Pinecone, Milvus) for RAG. Two systems, two sources
of truth, double the ops cost.
Our answer: One file. Tabular data + embeddings + HNSW index — all inside a single extended Parquet file on S3.
Key properties:
- 100% Iceberg-compatible — Spark, Trino, DuckDB, PyIceberg read it natively with zero plugins
- Vector search built-in — HNSW index lives in the file footer; loaded via mmap, never pulled into RAM whole
- Geometric pruning — query vector compared to per-file centroids at manifest scan time; eliminates 95–99% of files before any S3 I/O
- Hybrid search — BM25 + HNSW + Tantivy FTS in the same query pipeline
- ACID via Iceberg — snapshots, time-travel, compaction, deletes — all standard
- Python-first — import ailake, zero native deps from user side (PyO3 wheel)
No new infra. Your existing Iceberg catalog, your existing S3 bucket. Add vector search without moving your data.
GitHub:
github.com/ailake-io/ai-lakehouse
Would love feedback — especially from anyone running RAG at scale on a Lakehouse.