Designing Private, Air-Gapped Data Lakehouses: Scaling Iceberg in Highly Secure, On-Premises Clouds

Cross-posted. This article’s canonical home is Alex Merced’s Lakehouse Blog.

Some of the most important lakehouse work happens in environments that will never look like a simple public-cloud reference architecture. Defense, public sector, healthcare, financial services, manufacturing, energy, and research organizations often need private infrastructure, strict network boundaries, controlled software supply chains, and auditability that goes far beyond ordinary dashboard access.

That reality matters for AI. Agentic analytics does not remove security constraints. It makes them sharper. Agents can issue many queries, chain tools, inspect metadata, and request actions. If those systems operate over sensitive data, the platform has to be designed around least privilege, observability, and controlled execution from the start.

Apache Iceberg is useful in this context because it gives open table semantics over files that can live inside controlled object storage. But Iceberg alone does not make a deployment secure, compliant, or air-gapped. A secure lakehouse is an architecture. It includes identity, networking, encryption, catalogs, query engines, semantic layers, audit logs, upgrade paths, backup, disaster recovery, and operational discipline.

The Dremio-positive conclusion is that private environments still need modern lakehouse capabilities: open tables, fast SQL, query federation, semantic governance, and agent-safe access. The architecture should bring those capabilities inside the boundary rather than forcing sensitive data out of it.

Air-Gapped Is a Spectrum

People use “air-gapped” to describe several levels of isolation. Some environments have no network path to the public internet at all. Some run a data diode that only permits transfer in one direction. Some allow outbound access through a reviewed proxy with an allowlist. Some are private clouds with strict egress controls but a working path to vendor services. The word gets used for all of these, and the engineering consequences differ sharply between them.

The design should start by naming the actual boundary, in writing, before anyone picks a catalog or an engine. What systems can connect to what? How are patches imported? How do logs leave, and do they leave at all? How are users authenticated when the identity provider is internal? How are container images, language packages, and model weights approved? How is data transferred in or out, and who signs for it? An architecture that assumes normal SaaS connectivity will fail quietly in these environments, usually at the worst moment, which is the first production incident.

That answer cascades through every layer. Storage must be local or explicitly approved. Catalog services must run inside the boundary, because a catalog that phones home is not a catalog, it is a dependency. Query engines must authenticate against internal identity. Semantic metadata must be served internally. AI agents must use internal tools. Audit logs must land in systems that are already approved for the data classification in play. Any component that assumes a cloud control plane needs a substitute, and the substitute needs an owner.

What Iceberg contributes inside the boundary

Iceberg contributes an open table layer. It gives teams a way to organize data in object storage while maintaining table metadata, snapshots, schema evolution, partition evolution, and transactional behavior. In private environments that openness is valuable because it reduces dependence on a single closed engine. A team that writes Parquet to object storage with no table format gets files. A team that writes Iceberg gets a table with a history and a contract.

A private lakehouse usually needs several compute patterns at once: batch jobs, SQL engines for analysts, notebooks for data scientists, agentic workflows over semantic assets, and compliance review of lineage. Iceberg gives all of these a shared table foundation, provided that engine support for the specific features in use has been validated in your own environment and not just in a vendor compatibility matrix.

Snapshots are especially useful here. They let teams reason about table state at a point in time, which matters for audit, reproducibility, and incident investigation. Schema evolution matters too, because secure environments move slowly. Changing every consumer at once is often impossible when each change needs a review board, a maintenance window, and a rollback rehearsal. Open table semantics let tables evolve with the consumers following at their own approved pace. Partition evolution helps in the same way, since an early partitioning choice made under time pressure can be corrected without rewriting the whole dataset into a new table.

But Iceberg is not the whole security model. It does not replace network isolation, identity management, access control, encryption, monitoring, or compliance review. It is a strong table foundation inside a larger secure architecture, and the rest of this post is about the rest of that architecture.

Storage Inside the Boundary

Private lakehouses still need scalable storage. The usual answer is on-premises object storage speaking the S3 API, whether that is Ceph RGW, Swift with an S3 gateway, a vendor appliance, or a storage cluster that fronts cheaper hardware. Some organizations land on private cloud object storage inside a sovereign region. What matters is that the storage tier supports durability targets, encryption at rest with keys the organization controls, access logging, lifecycle controls, versioning, and a backup path that does not assume the same failure domain as production.

The storage tier sets the ceiling on everything above it. If the object store cannot do reliable multipart uploads at the sizes Iceberg writes, commits will time out under load. If it has no request-level audit log, no amount of catalog logging will reconstruct who read a data file. I went through the tradeoffs in Choosing Self-Hosted Object Storage for the Lakehouse After MinIO, and the short version is that request auditing, S3 API completeness, and operational maturity matter more than raw throughput benchmarks.

Direct storage access should be limited. Users and agents should not browse buckets casually, and engineers should not hold long-lived keys to the data lake because it was convenient during the build. Engines and maintenance services should reach storage through scoped credentials over controlled network paths, with the catalog and query layer mediating most interaction. Network policy should reflect the same idea: only the pods or hosts that need object storage endpoints get routes to them.

In highly secure environments, storage credentials should be short-lived, scoped to a prefix, rotated on a schedule, and audited on issue as well as on use. That is straightforward when a cloud STS exists and harder when it does not, which is the subject of the next section.

Performance pressure inside the boundary

Performance deserves its own attention because secure users still have real deadlines. If governed systems are slow, people ask for extracts, local copies, or special access paths. Those exceptions start as favors and end as security findings, and they are very hard to unwind once a team has built a workflow on a shadow copy.

A private lakehouse should treat performance as part of control. Iceberg table maintenance, file sizing, partition evolution, metadata cleanup, query acceleration, and workload management all reduce the temptation to bypass the platform. Fast governed access is safer than slow governed access plus informal workarounds, and the second option is what organizations actually get when the first one is neglected.

Two mechanics do most of the work. Compaction keeps file counts and file sizes in a range that scan planning can handle, and metadata cleanup keeps the manifest list from growing into a planning bottleneck. Both are cheap to run on a schedule and expensive to skip, because the cost arrives as a slow degradation that nobody attributes to table health until queries are ten times slower than they were a year earlier. On constrained hardware it is worth running compaction during off-peak windows and monitoring the commit rate rather than only the query latency.

Agentic workflows increase this pressure. An agent may run several queries to answer one question. It may compare time windows, inspect lineage, validate a threshold, and summarize results. If each step takes twenty seconds, the workflow becomes impractical and the team responds by giving the agent broader direct access, which weakens the security model in a way that is invisible in the moment. The better answer is to make the secure path performant. That is why acceleration and autonomous optimization belong in the private lakehouse conversation, not only in cloud analytics marketing.

Storage layout should also support recoverability. Iceberg metadata, data files, manifests, and snapshots need coordinated backup strategies. A table is only useful if the metadata and the data can be recovered together, and a data lake that backs up Parquet files but not the metadata directory has backed up nothing that a query engine can read.

Catalogs and Metadata Services

The catalog is the control plane. It tells engines what tables exist, where metadata lives, and how table operations are coordinated. In a private lakehouse the catalog must run inside the secure boundary, and it must be treated as a stateful production service with the same care as the database tier, because that is effectively what it is.

A practical reference deployment looks like this. Apache Polaris runs on Kubernetes inside the enclave, in its own namespace, with its management API and catalog API exposed only on cluster-internal service names. Polaris keeps its state in a relational database, so that database is deployed for high availability in the same boundary: PostgreSQL with a primary, a synchronous standby, and a connection pooler in front, or the equivalent managed service if the environment has one. Backups of that database ship to the same offline repository as everything else, and a restore is tested rather than assumed. Without the catalog database, an Iceberg warehouse is a pile of metadata files with no authoritative pointer, and rebuilding catalog state by scanning object storage is a slow, manual, error-prone exercise.

Policy lives in Polaris as catalog roles and grants scoped to namespaces, tables, and views. Catalog access should integrate with internal identity, whether that is LDAP, Kerberos, or an internal OIDC provider that the enclave runs itself. It should emit audit logs for every authorization decision, because the interesting question after an incident is not only who had access but who exercised it. It should restrict table discovery where necessary: in some environments, even knowing that a dataset exists is sensitive, so listing is a permission and not a default.

Credential vending without a cloud STS

The part teams underestimate is credential vending. In a public cloud, the catalog asks STS for a short-lived scoped token and hands it to the engine. Inside an air-gapped boundary there is no STS to ask, so the pattern has to be replaced with something that provides the same property: an engine gets limited, expiring access to exactly the files it needs, without ever holding a durable secret.

There are two workable patterns. The first is static scoped keys issued by the on-prem object store. An administrator creates a small number of keys per tenant or per workload class, each restricted by bucket policy to a prefix and a set of operations. Polaris vends these instead of cloud tokens. This works and it is simple, but the keys are static, so rotation has to be scheduled and monitored, and the blast radius of a leaked key is whatever its prefix covers. Keep the prefixes narrow, rotate on a fixed cadence, and log every use at the object store.

The second is remote signing. Polaris holds the storage secret itself and signs requests on behalf of the engine, so the engine never possesses a key at all. The catalog becomes the enforcement point and the log of record for storage access, which is a meaningful audit advantage in a regulated environment. The cost is that the signing endpoint is now on the query path for every read and write, so it needs to be replicated, low latency from the engine nodes, and sized for the peak scan rate rather than the average. It also becomes a single point of failure worth designing around, with more than one replica and a health check that actually fails over.

Either way, the credential TTL should be short, on the order of minutes rather than hours, and the engine should request creds per table load rather than holding them for the life of the process. And whichever pattern you choose, write it down. The next engineer to debug a permissions failure needs to know whether the key came from a bucket policy or a signing service, because the diagnostic path is completely different.

If you are running Polaris behind internal load balancers rather than a public endpoint, the networking details are worth reading before the first deployment: Keeping Lakehouse Traffic Off the Public Internet With Apache Polaris covers the service exposure and ingress choices.

Metadata itself needs protection. Schemas, table names, column names, tags, lineage, and statistics can reveal sensitive information without ever exposing a row. A column called referral_source_oncology is a finding on its own. Treat metadata as part of the security model, not as harmless decoration, and apply the same classification review to semantic definitions and metric names that you apply to tables.

Engines and the Semantic Layer

In a private or air-gapped environment, query engines should run close to the data. Pulling sensitive data into an external SaaS query service defeats the purpose of the boundary, and an “approved” exception to that rule tends to expand once one team has proved it works. The engine should operate inside the approved infrastructure, with local authentication, local logs, and local policy enforcement, and it should be deployed with enough capacity that analysts do not need to route around it.

Query federation still matters, but federation has to respect the network boundary. A federated engine may reach internal PostgreSQL, Oracle, MongoDB, Kafka, and approved application databases. It may not reach public endpoints, and it should not be able to attempt those connections silently. Egress policy should be default-deny at the namespace level so that a misconfigured source fails loudly at connection time rather than quietly exfiltrating a result set. Federation is the reason a private lakehouse can replace several copies of data with one governed access path, and that argument only holds if the federation layer itself is inside the perimeter.

This is where Dremio’s lakehouse direction is relevant. The need for fast SQL, federation, semantic access, and acceleration does not disappear in secure environments. If anything it becomes more important, because copying data out to easier tools is not an option and every consumer has to be served from the same controlled platform.

Semantic layer for secure analytics

The semantic layer is not optional in secure AI environments. It translates physical tables into approved business or mission concepts, and it is the difference between an agent that answers from a certified definition and an agent that guesses from a column name. It can define metrics, relationships, allowed dimensions, sensitivity levels, and usage notes, all of which are reviewable artifacts rather than prompt engineering.

For human users this reduces confusion and duplicated logic across dashboards. For agents it reduces risk. An agent should not infer the meaning of a sensitive table from its schema. It should query approved semantic assets with clear definitions and access rules, and the platform should refuse when the asset does not exist rather than reaching for the nearest physical table.

Semantic assets should carry classification metadata. A metric may be broadly visible while the underlying raw table is restricted. A view may mask or aggregate sensitive fields. A dimension may be allowed only for certain roles, or only in aggregate, or only above a threshold count. Agents need to know these boundaries because they will otherwise compose a query that is legal per object but illegal in combination, which is exactly the failure mode that row-level controls on single tables do not catch.

This is also how secure lakehouses support AI without exposing raw sensitive data. Agents work through curated metrics, governed views, and narrow tools, while raw data inspection stays limited to approved workflows with named humans in them. The semantic layer becomes the contract between the data platform team and every downstream consumer, human or machine, and it is the artifact auditors can actually read.

The hybrid version of this problem, where metadata is public but data stays private, is covered in Hybrid Lakehouse Design for Regulated Markets, which is a common pattern when a central team needs visibility without custody of the rows.

Agents and Models Inside the Boundary

If AI agents operate inside a private lakehouse, they should be treated as workload identities with specific permissions. Not a borrowed human admin account, not a service account with broad storage credentials, and not a default that can discover every dataset. An agent that can list the catalog can map your data estate from a prompt, and that map is itself sensitive.

Agent tools should be scoped and few. A tool might retrieve approved metric definitions, run a governed query, inspect lineage for a permitted asset, or request a maintenance action. Higher-risk actions should require human approval, and the approval should show the actual SQL or the actual parameters rather than a summary the agent wrote about itself. Every tool call should be logged with the agent identity, the tool name, the dataset, the semantic asset, the query text, the table snapshot where available, the policy decisions, and the response status.

Inside an enclave, the practical shape of this is an MCP gateway running in the same boundary as the data services. Agents connect to the gateway, the gateway holds the credentials and the allowlist, and the gateway enforces which tools exist for which agent identity. No agent process gets a direct route to object storage, and no tool call leaves the enclave. The gateway is also the natural place to enforce rate limits and to keep a single, complete log of agent activity for review, which is much easier to defend than reconstructing behavior from several systems afterward.

Agents should also have refusal behavior. If data is restricted, stale, undefined, or outside scope, the correct answer is a clear refusal or an escalation to a named person, not a plausible estimate. Testing for that behavior belongs in the release process, with a set of prompts that should be declined.

Models inside the boundary

Agentic lakehouse discussions often focus on data access, but model deployment matters just as much. In air-gapped or private environments, teams may not be allowed to call external model APIs at all. They need internally hosted models, approved weights, controlled inference services, and local evaluation workflows, usually on GPU capacity that is shared with other workloads and therefore contested.

The model supply chain should be treated like the software supply chain. Model artifacts need provenance, approval, scanning, versioning, and rollback. Prompt templates and tool definitions should be stored in version control and reviewed like code, because they encode policy. Evaluation datasets should stay inside the boundary if they contain sensitive examples, which they almost always do when they are drawn from real questions.

Agents should run close to the governed data services they use. If an agent calls a query tool, a semantic tool, or a lineage tool, that call stays on the approved network path. The agent should not be able to smuggle context to an external system, and a local model makes that guarantee easier to keep because there is no external endpoint to reach. Larger models are not automatically the right choice here; a smaller model with good tools and a tight semantic layer often outperforms a large model guessing at schema, and it fits the GPU budget.

This is where open lakehouse architecture and local AI deployment reinforce each other. The data stays in controlled storage. The table metadata stays in a local catalog. The query engine runs inside the boundary. The agent calls local tools through a gateway. The audit trail stays available for review. For more on how regulated production agents are being built around MCP, see MCP in Regulated Environments.

The Software Supply Chain

Air-gapped lakehouses require a plan for software supply chain, and this is usually more work than the architecture diagram suggests. Engines, libraries, connectors, container images, model artifacts, JDBC drivers, and security patches all have to be imported through an approved process, and each one is a chance to introduce something nobody reviewed.

The platform team should run an internal artifact repository and an offline container registry. Harbor is a common choice for images because it supports replication from an upstream registry when a connection window is open, vulnerability scanning on push, and content trust. For language packages, a Nexus or Artifactory instance inside the boundary mirrors PyPI, Maven Central, npm, and the OS package repositories, so build pipelines resolve dependencies locally and reproducibly. Builds that reach the internet are builds that cannot be repeated next quarter.

Signing is what makes the mirror trustworthy. Images should be signed, with cosign or the vendor’s equivalent, and the cluster should reject unsigned or unverified images at admission through a policy controller rather than relying on a human to check a digest. Keep the public keys inside the boundary so verification works when the network does not.

Every import should come with a software bill of materials and a review. The reviewer is asking a specific question: what new transitive dependencies arrived, do any of them need network access at runtime, and does anything in this list have a known vulnerability that matters for how we run it. SBOM diffing between the current and proposed version is the fastest way to answer that, and it is far more useful than reading release notes.

Upgrade cadence is the other half. In an environment with no internet access, “we will take the latest patch” is not a plan. Pick a cadence tied to the transfer window, for example one quarterly bundle with an out-of-band path for critical security fixes, and pin exact versions in the deployment manifests. Iceberg table compatibility should be part of every upgrade test: if a new engine or library version changes support for delete files, row-level deletes, schema evolution, catalog behavior, or table properties, the team needs to know before production workloads are affected. Format version upgrades deserve their own window, since a table upgraded to a newer spec version may not be readable by older engines that are still in service for a downstream team.

Rehearse in a staging environment that mirrors production constraints: the same restricted network, registry, and catalog topology. A staging environment with internet access tests nothing that matters here. Document the rollback and test it, because an upgrade that cannot be reversed inside the change window is a much larger decision than one that can.

Agent tooling adds another layer. Prompt templates, tool definitions, model versions, evaluation datasets, and policy rules belong in version control, reviewed alongside application code and deployed through the same pipeline. The agent stack is software, not configuration someone edits on a laptop.

Moving Data In and Out

Highly secure environments need an explicit data movement policy, written down and enforced by tooling rather than by convention. What data can enter? What data can leave? Who approves each transfer? How are transfers scanned, logged, and reconciled? Every one of those questions has a wrong answer that looks fine until an auditor asks.

Inbound data may come from operational systems, partner feeds, sensor networks, batch exports, or physical media. Each path needs validation. The common pattern is a one-way transfer into a landing zone using a data diode or a write-only share, so the sending side cannot read anything back and the receiving side cannot be reached from outside. From the landing zone, an import process checks integrity by checksum, scans for malware, verifies that the schema matches the registered contract, reads the classification label, and records provenance before the data becomes a lakehouse table.

The Iceberg-specific piece is branching. Land imports on a staging branch rather than writing directly to main, run validation queries against the branch, and merge to main only after the checks pass and the reviewer signs off. Branching gives the import a real rollback: if a feed arrives with a broken schema or a bad batch, the branch is discarded and main never moved. That is a materially better position than discovering the problem after a commit on main and rebuilding the table from scratch.

Outbound is even more sensitive. Aggregated reports, model outputs, audit extracts, and derived datasets may all carry information that policy treats as controlled, and small aggregates leak in ways that large ones do not. Export workflows should be explicit, reviewed, and logged: a query against approved semantic assets, an approved destination, a named approver, and a record of exactly what left. Users should not be able to bypass the governed layer by copying raw files, which means storage credentials should not exist in the hands of analysts in the first place.

For Iceberg tables, movement has to account for metadata as well as data files, because a partial transfer can create a table that looks fine until someone queries an older snapshot. Copy with a tool that understands the format rather than an object copy recursed over two prefixes, and verify the snapshot log on the far side before declaring the move complete.

This is another reason semantic assets are useful. In many cases users do not need raw data to leave the boundary. They need approved aggregates, metrics, or reports, and a governed metric endpoint is a far smaller export surface than a table. If the semantic layer covers the recurring asks, the number of approved raw exports drops, and every remaining one gets more scrutiny because it is genuinely exceptional.

Audit, Provenance, and Compliance

A private lakehouse needs deep observability, and it needs to be local. It should capture query latency, catalog latency, table maintenance health, file counts, metadata growth, failed authorization events, agent tool calls, and data quality signals. In air-gapped environments external telemetry is usually unavailable or disallowed, so the platform has to provide enough local observability to operate independently, including dashboards and alerting that do not depend on a vendor SaaS.

Audit logs should be protected, retained according to policy, and searchable by user, workload, dataset, table snapshot, semantic asset, and time window. Ship them to an append-only internal store, restrict deletion to a break-glass role, and keep enough history that an investigation spanning a year is possible. A log with thirty day retention in an environment with a two hundred day incident detection window is decoration.

Iceberg history as forensic provenance

Iceberg gives you something stronger than a query log, and it is worth using deliberately. The table’s own snapshot history is a forensic record. Every commit writes a snapshot with a sequence number, a timestamp, a summary of what the operation did, and the manifest list it points to. The snapshots, history, and files metadata tables let an investigator ask precise questions: which commit added these files, what was the row count delta, what was the operation type, when did the table change, and which snapshot was current when a given query ran. That is provenance the storage layer cannot reconstruct on its own.

Two mechanics make this work. First, record the snapshot ID in query audit records wherever the engine exposes it, so a logged query ties to an exact table state months later. Second, protect the snapshots audits depend on. Expiration is normally healthy, but a snapshot underpinning a regulatory filing, a model training run, or an open investigation must survive it.

The mechanism is tags with explicit retention. An Iceberg tag is a named reference to a snapshot, and you can set max-ref-age-ms on it so that snapshot expiration leaves it alone. Tag the table state you need to preserve, for example a month-end close or a dataset version used to train a model, and give it a retention longer than the audit requirement rather than longer than the next cleanup job. Then verify it: run expiration, and check that the tagged snapshot is still resolvable. A tag that was silently dropped by a expire_snapshots call configured with aggressive defaults is a very uncomfortable discovery during a regulator’s visit.

Compliance readiness without overclaiming

No table format or query engine makes an organization compliant by itself. Compliance depends on laws, policies, controls, evidence, and operating discipline. The lakehouse can help produce that evidence, but it cannot replace governance work, and any vendor claiming otherwise is selling something other than compliance.

What a private lakehouse should be able to show is concrete: who accessed data, which policies applied, which table version was queried, which semantic definition was used, and which agent tool produced a result. It should also show how data entered the system, how it changed, and how long it was retained. Those are answerable questions when the design captures the identifiers, and unanswerable ones when it does not.

That supports compliance readiness. It helps teams answer auditors and security reviewers without a six week archaeology project, and it supports internal accountability when an AI system produces a questionable answer, because you can see which snapshot it read and which definition it used. Design for evidence before the incident. Retrofitting auditability onto agentic workflows already in production is painful and usually incomplete.

Backup, Recovery, and Continuity

Secure lakehouses need recovery planning that understands the table format. Backing up data files without metadata is not enough. Backing up metadata without data files is not enough. Iceberg tables depend on the relationship between metadata files, manifests, snapshots, and data files, so the backup unit is the warehouse plus the catalog, not a directory.

In practice that means three coordinated things. Object storage content is replicated or snapshotted to a separate failure domain. The catalog database is backed up with point-in-time recovery, because it holds the pointer that makes the warehouse a set of tables. And the configuration that ties them together, including service definitions, bucket policies, and catalog role assignments, lives in version control so it can be rebuilt rather than remembered.

Recovery procedures should be tested on a schedule, with results recorded. Can the team restore a table to a previous snapshot? Can it recover after accidental metadata deletion within the retention window? Can it rebuild catalog state, and how long does that take? Can it prove that restored data matches audit expectations, using the snapshot history as the reference? An untested restore procedure is a hypothesis.

Continuity planning should also account for identity systems, query engines, semantic metadata, and audit logs. A lakehouse is a system, not a bucket. If the identity provider is down, the platform is down, and that dependency belongs in the recovery plan with its own RTO. If semantic definitions live only in a tool’s database, back that up too: a warehouse restored without its metric definitions is one nobody can use on day one.

A Day of Operations

It helps to see what all of this looks like when it is working, because the goal is ordinary operations rather than permanent incident posture.

Overnight, ingestion jobs land approved files in a landing prefix. Validation services check schemas, row counts, and checksums against the registered contract, then write to a staging branch. A reviewer approves the merge to main. Iceberg metadata commits through the internal Polaris catalog, with the catalog database replicating to its standby. A maintenance job compacts a high-use table and expires snapshots that are not tagged. The semantic layer refreshes certified metrics. Audit logs record each step with the actor, the snapshot, and the policy decisions.

Mid-morning, an analyst asks a question through a governed SQL interface. The engine authenticates against internal identity, resolves the semantic definition, reads the relevant Iceberg snapshot, applies column masking, and returns an answer. The audit record captures the snapshot ID, the definition version, and the policy decisions.

Later, an internal agent investigates an anomaly through the MCP gateway, which exposes four tools to that identity and nothing else. It inspects approved metrics, retrieves lineage for permitted assets, and runs scoped queries. It cannot browse raw storage, export files, or change policies, and its attempt to call an unlisted tool shows up in the gateway log as a denial. A recommended follow-up action requires human approval that shows the exact statement to be run.

The practical checklist that makes that day repeatable:

• Name the boundary in writing: network isolation level, ingress and egress paths, identity sources, transfer mechanisms, and patch import process.
• Choose storage that supports durability, encryption with internal keys, request audit logging, versioning, and a backup path in a separate failure domain.
• Deploy the catalog inside the boundary with a highly available database, and decide explicitly between static scoped keys and remote signing for credential vending.
• Run query engines close to the data, with federation limited to internal sources and egress default-deny.
• Build semantic assets for approved use cases, with classification metadata on every metric and view.
• Scope agent tools narrowly through a single gateway, log every call, and test refusal behavior before release.
• Run an offline registry and package mirrors, sign images, review SBOMs at import, and pin versions.
• Test upgrades against table format compatibility, and rehearse the rollback.
• Protect tagged snapshots with retention that exceeds the audit requirement, and verify they survive expiration.
• Test backup and recovery, including catalog state and semantic definitions, before production reliance.

None of that is flashy. That is the point. Secure lakehouses succeed when strong controls become ordinary work rather than emergency ceremony.

Where This Is Heading

Private and air-gapped lakehouses will become more important as AI reaches regulated and mission-critical environments. The organizations with the strictest controls still need modern analytics. They still need open tables, semantic context, and agents that help without bypassing policy, and they are not going to accept a five year feature gap in exchange for isolation.

Apache Iceberg provides a strong table foundation, but the secure lakehouse is much more than a table format. It is a coordinated system of storage, catalog, query, semantics, identity, audit, operations, and agent controls. The interesting engineering work is in the seams: what happens to credential vending when STS disappears, how a catalog database gets high availability inside an enclave, how snapshot retention is protected long enough for an audit, how an agent gets useful access without a route to the internet.

Secure organizations still need openness and performance. They cannot solve every problem by centralizing data in a public SaaS warehouse, and they should not have to choose between governance and usable analytics. Open Iceberg tables preserve portability. Query federation reaches internal systems without unnecessary copying. Semantic layers give agents and humans approved definitions. Acceleration keeps governed access fast enough that nobody asks for an extract. Agentic interfaces work safely when they call scoped tools over trusted assets through a gateway that logs them.

The conclusion is not that security means no innovation. The conclusion is that innovation has to live inside the security model, designed for the boundary rather than retrofitted to it.

For more of my thinking on data lakehouse and AI architecture, explore my books at books.alexmerced.com. To try a modern Agentic Lakehouse experience, visit dremio.com/get-started.