Introducing AI-Lake Format :snow_capped_mountain: ...
# good-reads
t
Introducing AI-Lake Format 🏔️ We're open-sourcing ailake — a self-contained file format for AI Lakehouses, written in Rust. The problem: Running BI and GenAI on the same data means juggling two stacks — a Data Lake for analytics and a separate vector DB (Pinecone, Milvus) for RAG. Two systems, two sources of truth, double the ops cost. Our answer: One file. Tabular data + embeddings + HNSW index — all inside a single extended Parquet file on S3. Key properties: - 100% Iceberg-compatible — Spark, Trino, DuckDB, PyIceberg read it natively with zero plugins - Vector search built-in — HNSW index lives in the file footer; loaded via mmap, never pulled into RAM whole - Geometric pruning — query vector compared to per-file centroids at manifest scan time; eliminates 95–99% of files before any S3 I/O - Hybrid search — BM25 + HNSW + Tantivy FTS in the same query pipeline - ACID via Iceberg — snapshots, time-travel, compaction, deletes — all standard - Python-first — import ailake, zero native deps from user side (PyO3 wheel) No new infra. Your existing Iceberg catalog, your existing S3 bucket. Add vector search without moving your data. GitHub: github.com/ailake-io/ai-lakehouse Would love feedback — especially from anyone running RAG at scale on a Lakehouse.