Redpanda is a streaming data platform that implements the Apache Kafka protocol, so existing Kafka producers, consumers, and many Kafka tools work against it without code changes. It is built by Redpanda Data, written in C++ on a thread-per-core design, and ships as a single binary with no JVM and no separate coordination service.
In a data lakehouse, Redpanda plays the same role Kafka does: it is the durable log that carries events and change data capture streams (for example from Debezium) before they land in Apache Iceberg tables.
How Redpanda Works
- Kafka API compatibility. Topics, partitions, consumer groups, and offsets behave as Kafka clients expect.
- Raft replication. Each partition is replicated with the Raft consensus protocol, and cluster metadata is managed inside Redpanda itself rather than by ZooKeeper.
- Built-in extras. A schema registry and HTTP proxy run inside the broker. Redpanda Connect (formerly Benthos, acquired in 2024) handles connector-style pipelines.
- Delivery guarantees. Like Kafka, it supports idempotent producers and transactions for exactly-once processing within the streaming system.
Iceberg Topics: Streams as Lakehouse Tables
Redpanda’s Iceberg Topics feature writes topic data to Iceberg tables directly from the broker, so you do not need a separate sink connector. You turn it on per topic with redpanda.iceberg.mode:
key_valuewrites a table with record metadata (including the key) and the raw value as binary.value_schema_id_prefixandvalue_schema_latestuse a schema from Schema Registry to create typed columns.
Redpanda can track tables with its own object storage catalog or register them in an external Iceberg REST catalog such as Apache Polaris. Once registered, engines such as Spark, Trino, Dremio, Snowflake, and ClickHouse can query the tables. Iceberg Topics require a Redpanda Enterprise license.
Redpanda vs Apache Kafka for Lakehouse Ingestion
| Redpanda | Apache Kafka | |
|---|---|---|
| Implementation | C++, single binary | Java/Scala on the JVM |
| Coordination | Built-in Raft | KRaft (ZooKeeper removed in Kafka 4.0) |
| License | Community Edition under the Business Source License; enterprise features under a commercial license | Apache License 2.0, governed by the ASF |
| Path to Iceberg | Iceberg Topics in the broker (Enterprise) or a sink such as Kafka Connect | Iceberg Kafka Connect sink, Flink, or Spark |
| Ecosystem | Kafka-compatible clients, plus Redpanda Connect | Largest ecosystem and widest vendor support |
The license difference matters for some teams: the Business Source License forbids offering Redpanda as a commercial streaming service to others and converts each code change to Apache 2.0 four years after it is merged. Teams that require an ASF-governed project tend to choose Kafka or Apache Pulsar.
Broker-side Iceberg writes trade flexibility for simplicity. A Flink or Spark job can clean, join, and merge CDC data before committing, while Iceberg Topics mostly land raw events as a bronze layer. Frequent small commits also mean you still need compaction and snapshot expiration. For the tradeoffs across every path, read The State of Streaming to Apache Iceberg in July 2026 and Kafka Connect to Iceberg: How the Commit Actually Works.
Frequently Asked Questions
Is Redpanda a drop-in replacement for Kafka?
For most client applications, yes, because it speaks the Kafka protocol. Test any tooling that depends on Kafka internals or specific broker configurations before migrating.
Is Redpanda open source?
The Community Edition is source-available under the Business Source License, not an OSI-approved open source license. Enterprise features, including Iceberg Topics and tiered storage, need a commercial license.
Do I still need a stream processor?
Often. Redpanda moves and stores events. Transformations, joins, and upserts into curated Iceberg tables are usually done in Apache Flink or Apache Spark.
Learn More
To learn more about the Data Lakehouse, read The Lakehouse Built for Everyone by Alex Merced. You can find this and other books by Alex Merced at books.alexmerced.com.