What is the Lance File Format?
Lance is an open source (Apache 2.0) columnar data format built for machine learning and multimodal AI data: embeddings, images, video, text, and the tabular columns that describe them. It was created by the team behind LanceDB and is now developed in the lance-format GitHub organization.
“Lance” covers three related pieces:
- The Lance file format, a columnar file layout designed for fast random access to individual rows as well as scans.
- The Lance table format, which groups files into versioned datasets with ACID commits, time travel, tags, and branches.
- Lance Namespace, a catalog specification for connecting Lance tables to metadata services.
What Makes Lance Different
Parquet was designed for analytical scans: read a few columns across many rows. Machine learning workloads also need the opposite: fetch a few thousand specific rows, often with large binary or vector columns, in random order for training or retrieval. Lance targets that access pattern without giving up columnar scans. The project claims random access up to 100x faster than Parquet; treat that as a vendor benchmark and test with your own data.
Other notable features:
- Vector and full-text indexes stored with the dataset (for example IVF_PQ vector indexes), enabling similarity search directly on the files.
- Data evolution: adding a new column, such as a fresh set of embeddings, without rewriting existing data.
- Zero-copy versioning: each commit creates a new version that readers can time travel to.
Lance vs Parquet vs Apache Iceberg
| Lance | Apache Parquet | Apache Iceberg | |
|---|---|---|---|
| What it is | File format plus table format plus catalog spec | File format | Table format (uses Parquet, ORC, or Avro files) |
| Optimized for | Random row access, vectors, multimodal blobs | Analytical column scans | Reliable multi-engine tables over object storage |
| Built-in indexes | Vector, full-text, scalar | Statistics, optional bloom filters and page indexes | File and partition statistics in metadata |
| Versioning | Yes, at the dataset level | No (single file) | Yes, snapshots, branches, tags |
| Engine support | LanceDB, Python, Rust, DuckDB and Spark integrations, others | Nearly every analytics engine | Spark, Flink, Trino, Dremio, DuckDB, Snowflake, and many more |
Lance and Iceberg are usually complements, not rivals. A common split keeps business and analytical tables in Iceberg, queried by SQL engines, and keeps embeddings and training samples in Lance, where random access and vector search matter. Catalog support is converging: in January 2026 the Apache Polaris project described an integration that manages Lance tables through Polaris’s Generic Table API, so one catalog can govern both Iceberg and Lance tables.
For a deeper treatment, see Lance and Iceberg for Multimodal AI Data, The File Format Renaissance: Parquet, Lance, Vortex, Nimble, BtrBlocks, and Vector Search Directly Over Iceberg Tables.
Frequently Asked Questions
Is Lance a replacement for Parquet?
For ML training data and vector search, it can be. For general BI and SQL analytics, Parquet inside Iceberg tables still has far broader engine support.
Is Lance a vector database?
No. Lance is a format. LanceDB is a vector database built on it, and other tools can read and write Lance datasets directly. See vector database for the broader category.
How is Lance governed?
The format is open source under Apache 2.0. Governance features such as access control and credential vending come from the catalog layer, for example through Lance Namespace integrations with catalogs like Apache Polaris.
Learn More
To learn more about the Data Lakehouse, read The Lakehouse Built for Everyone by Alex Merced. You can find this and other books by Alex Merced at books.alexmerced.com.