LanceDB is the open-source vector database built on top of the Lance columnar file format. Two properties separate it from the rest of the OSS vector-DB landscape: it is append-only at the storage layer (new data lands as new fragments, no in-place mutation), and it stores vectors and metadata columnarly on disk (Lance is a Parquet-adjacent format, not a row-oriented store). Both choices fall out of the same engineering bet — that the bottleneck for production retrieval at scale is incremental indexing, not query latency, and the format should optimize for that.
It is also a young project with the operational maturity to match. The docs do not match the code. Behavior changes between minor versions. The fastest way to ship on it is to clone the Rust core and read the source. We use it anyway, because nothing else gives us the append-only property at our scale.
LanceDB is the only OSS vector DB that handles continuous ingestion without rebuilds. That single property is why we picked it. Everything else — the columnar storage, the Rust core, the cheap analytics on vectors — is a bonus. The cost is operational pain you eat in exchange.
What LanceDB Actually Is
Three layers, each independently meaningful:
- Lance — a columnar on-disk format, conceptually similar to Parquet but designed for ML workloads. Vectors, scalar metadata, and arbitrary nested types live in the same file. Random access is cheap (unlike Parquet, which is optimized for sequential scans). Schema evolution is first-class.
- The Lance index layer — IVF, IVF-PQ, and HNSW indexes built on top of Lance fragments. Each fragment can carry its own index; queries union across fragments and merge results.
- LanceDB — the database surface: tables, queries, filters, schema management. A thin wrapper over the format and index layer, with a Python and Rust API.
The split matters because most of the interesting properties live at the format layer, not the database layer. You can use Lance directly without LanceDB and lose almost nothing.
Lance files are columnar like Parquet, but with two key differences. First, random row access is fast — Lance keeps a row-offset index in the footer, so fetching row 47,213 of a 100M-row file is a single seek, not a scan. Parquet requires reading the row group containing that row. Second, vectors are first-class — Lance has native fixed-size-list types with vectorized layout, where Parquet treats them as nested lists with all the encoding overhead that implies.
The append-only property is the format’s most important guarantee. Writes produce new fragments; nothing in existing fragments is rewritten. A table is a set of fragments plus a manifest. Adding rows = write a fragment, update the manifest. Deleting rows = mark them as deleted in a deletion vector, never rewrite the fragment. Updating a row = delete + insert.
Why append-only matters at our scale
- Continuous embedding ingestion. New documents arrive every minute. Each batch becomes a new fragment; no rebuild, no rolling restart, no freshness lag.
- Multi-version snapshots for free. The manifest is the only thing that points at fragments; older manifests still see older data. Time travel is essentially free.
- Cheap point-in-time evals. Run yesterday’s reranker against yesterday’s index, today’s against today’s, no separate snapshotting infrastructure.
- Crash safety. A partial write leaves a fragment that’s never referenced by any manifest. No corruption, no half-rebuilt indexes.
Why Append-Only Matters at Our Scale
At ZeroEntropy we ingest hundreds of millions of embeddings across many tenants, with constant writes (new docs, re-embedded chunks, evaluation runs). The standard HNSW story breaks in two places at this scale:
- Rebuilds dominate operational cost. A pure hnswlib or FAISS-HNSW index has to be rebuilt periodically as deletes accumulate; at our churn rate that’s a multi-hour batch job per tenant per day.
- Online inserts are slow and degrade recall. HNSW supports per-vector insertion, but each insert is a graph traversal that gets more expensive as the index grows, and recall drifts as the graph diverges from a from-scratch build.
LanceDB’s IVF and HNSW indexes are scoped to fragments. Ingestion becomes: write a fragment, build a small index over its contents, append. Queries fan out across fragments and merge top-k. Periodic compaction merges small fragments into larger ones, but that runs in the background — it is not on the critical path of any write or query.
The win is a single architectural primitive: inserts and queries do not block each other and do not require rebuilds. At our scale, that is worth a significant amount of operational pain to obtain.
Operational Reality
This is the warning section. Treat it as the part you tell new engineers before they ship anything on Lance.
The specific failure modes we have hit:
- Index-build failures with no useful error message. A vector with a NaN, a fragment with a corrupted footer, a schema mismatch between fragments — these surface as opaque “failed to build index” errors. Recovery is reading the Rust panic backtrace and matching it against the source.
- Version churn. The Python wrapper, the Rust core, and the underlying Lance format all version independently and do not always agree. Pinning all three to known-good versions is non-optional.
- Compaction is not free. Background compaction merges small fragments, but it competes with foreground writes for I/O. Tuning compaction throughput vs ingest throughput is an operational decision the docs barely mention.
- Query planning is opaque. The query planner picks which fragments to read and which indexes to consult; when it picks wrong, query latency spikes and there is no built-in way to inspect the plan. We’ve patched in our own logging.
- Filter pushdown is partial. Some filter shapes push down into the fragment scan; some don’t. The docs do not enumerate which is which. The right answer is “test every filter shape you actually use.”
The team that operates a production Lance deployment has a specific shape. At minimum: one engineer fluent enough in Rust to read and patch the core; a fork of lance and lancedb pinned to specific commits; a CI pipeline that re-runs your full retrieval eval against any version bump; a runbook for the three or four failure modes you have personally hit; and a willingness to file (and sometimes write) upstream patches.
This is not the operational profile of most database deployments. It is closer to the profile of a team that ships on a young distributed system — Cassandra in 2012, Spark in 2014, ClickHouse in 2018. The maturity gap closes over time, but only if you contribute to closing it. We do, because the alternative is rebuilding append-only vector indexing ourselves, and that is a strictly larger engineering project.
When To Pick Something Else
LanceDB is the right answer for a narrow set of workloads. It is the wrong answer for most.
Pick something else when…
- You don’t need append-only. If your corpus is mostly static and re-indexed nightly, every property of LanceDB you’d benefit from is also delivered by Qdrant, Weaviate, or Milvus, with vastly better operational maturity. Pick one of them.
- You need rich filtering ergonomics. Qdrant’s payload filtering is years ahead of LanceDB’s. If your queries look like
WHERE tenant_id = X AND policy_version > Y AND published_at > Z, write Qdrant.
- You want hybrid search out of the box. Weaviate ships hybrid retrieval (BM25 + vector) and a polished GraphQL layer. LanceDB has hybrid in preview; the maturity gap is real.
- You want SQL alongside. pgvector inside Postgres is the obvious move when your vectors live next to relational data and you don’t need >50M vectors per index. The operational story is “your existing Postgres,” which is hard to beat.
- You’re at extreme scale with a dedicated infra team. Milvus is the heaviest of the OSS options, with the most enterprise features (replication, sharding, multi-tenancy primitives). At billion-scale with an infra team that wants tunables, it’s the right pick.
- You’re doing research / one-off jobs. Use FAISS directly. No database overhead, no ops, just numpy in and numpy out.
LanceDB is appropriate when (a) you have a continuous ingestion stream, (b) you cannot tolerate rebuild-driven freshness lag, (c) you want columnar analytics on top of your vectors (e.g., compute corpus-wide statistics without reshaping data), and (d) you have engineering capacity to absorb the maturity gap. If you don’t tick all four boxes, pick a more boring tool.
Why We Still Use It
We picked LanceDB for one specific reason — append-only indexing at scale — and we accept the cost. The Rust core is fast, the format is right, the bet is sound. The operational reality is that the project is still finding its surface area, and shipping on it requires a team that treats reading the source as part of normal development. We’d recommend it to others with the same caveat: pick it for the property only it has, build the operational muscle to absorb the rough edges, contribute fixes back upstream, and don’t expect the Postgres-grade polish you get from a 30-year-old database.
