The storage layer is the part of a data architecture that durably holds the data itself: the files, objects, or blocks that persist after every query engine and job has shut down. In a data lakehouse, the storage layer is usually cloud object storage such as Amazon S3, Azure Data Lake Storage, or Google Cloud Storage, or an S3-compatible system on premises, holding open file formats like Apache Parquet. Keeping this layer separate from compute is the defining move of the lakehouse design.
How the Storage Layer Works
A storage layer has three jobs: keep data durable, make it addressable, and serve reads and writes at the throughput engines need. Different technologies make different trade-offs:
| Storage type | Addressed by | Strengths | Typical role |
|---|---|---|---|
| Object storage | Key in a flat bucket namespace | Low cost per GB, high durability, scales with little administration | Lakehouse tables, raw landing zones, backups |
| Block storage | Block on an attached volume | Low latency, supports in-place updates | Database volumes, local engine caches |
| File storage | Path in a hierarchical file system | Familiar POSIX semantics, shared access | Legacy HDFS clusters, shared workspaces |
| Warehouse-managed storage | Proprietary internal format | Tight integration with one engine | Traditional cloud data warehouses |
Object storage is built around writing whole objects rather than editing bytes in place, so changing data generally means writing a new object. That property shapes how lakehouse formats work: they write immutable data files and track changes with metadata rather than rewriting files on every update.
Storage Layer Versus Table Format and Catalog
In a lakehouse, “storage layer” usually refers to the physical bytes, and several logical layers sit on top of it:
- File format. Parquet, ORC, or Avro files define how rows and columns are encoded on disk. Parquet’s columnar layout lets engines read only the columns a query needs.
- Table format. An open table format such as Apache Iceberg groups those files into a table, recording schema, partitioning, and snapshots in metadata files that live in the same storage. This is what adds ACID commits, schema evolution, and time travel on top of plain objects.
- Catalog. An Iceberg catalog such as Apache Polaris tracks the current metadata pointer for each table and can vend scoped credentials so engines only read the storage paths they are allowed to.
- Compute. Engines such as Dremio, Trino, Spark, Flink, and DuckDB read and write those files through the catalog. Because the data sits in open formats in storage the organization controls, several engines can work on the same tables without copying them.
Why the Storage Layer Matters
Separating storage from the compute layer changes how a platform is scaled and paid for. Storage grows with data volume, while compute grows with query demand, and each can be sized on its own. Compute clusters can shut down when idle without losing data, and a new engine can be added later without a migration.
The storage layer also sets limits that engines must work around. Object storage requests carry more latency than local disk, so engines rely on techniques such as reading only needed columns, skipping files using metadata statistics, caching hot data on local disks, and keeping file sizes reasonable through data compaction. Many small files mean many requests and slower planning, which is why table maintenance is part of running a lakehouse.
Governance starts here as well. Bucket policies, encryption at rest, and access logs operate at the storage level, while catalogs and engines enforce table, column, and row-level rules above it.
Related Reading
- Block vs. Object Storage: A Deep Dive Into the Foundation of Modern Data
- Decoupling Storage and Compute in Apache Iceberg
- Storage abstraction
Learn More
To learn more about the Data Lakehouse, read The Lakehouse Built for Everyone by Alex Merced. You can find this and other books by Alex Merced at books.alexmerced.com.