What is Data Serialization?

What is Data Serialization?

Introduction to Data Serialization

In distributed systems, data rarely stays in one place. A Python microservice might need to send user data to a Java-based processing engine. A frontend React application might need to request pricing data from a Golang backend.

However, computer programs store data in memory using language-specific data structures (like Python dictionaries or Java objects). You cannot simply send a Python dictionary over a network cable to a Java server; the Java server would not know how to read it.

Data Serialization is the process of translating these complex, in-memory data structures into a standardized format (usually a continuous stream of bytes) that can be easily transmitted over a network or saved to a disk. When the receiving system gets the byte stream, it performs the reverse operation, Deserialization, reconstructing the bytes back into a usable object in its own language.

Human-Readable vs. Binary Serialization

Serialization formats broadly fall into two categories, each with distinct architectural trade-offs.

1. Human-Readable Formats (JSON, XML, YAML)

These formats serialize data into plain text.

• JSON (JavaScript Object Notation) is the most common format for web APIs. It is simple, widely supported, and language-agnostic.
• Pros: Excellent for debugging. A developer can easily read a JSON payload to see what went wrong. Very flexible (Schema-less).
• Cons: Inefficient. Storing the number 1,000,000 as a string takes 7 bytes, plus the bytes required for quotation marks and keys. They use more network bandwidth and require more CPU work to parse text into numbers.

2. Binary Formats (Protobuf, Avro, Thrift)

These formats serialize data into compact binary streams that are not human-readable.

• Protocol Buffers (Protobuf), created by Google, is the default serialization format for gRPC, a popular choice for high-performance microservice communication.
• Pros: Fast and compact. The number 1,000,000 is encoded in a few bytes. Parsing is much faster than parsing text.
• Cons: Unreadable to humans. Requires a predefined schema (a .proto file) to know how to decode the binary stream. Without the schema, the field meanings are lost.

Serialization in Big Data: Apache Avro and Parquet

While JSON is fine for sending a single user profile to a web browser, it is a poor fit for big data. If you serialize terabytes of data as JSON and store it in Amazon S3, you will pay more for storage and queries will spend much of their time parsing text.

Big Data systems rely mostly on specialized binary serialization formats.

Apache Avro

Avro is a row-based binary format. It uses schema-based serialization. In Avro data files, the schema (written in JSON) is embedded in the file alongside the binary data; in Kafka, the schema is usually stored in a schema registry and referenced by ID. Avro is widely used for Streaming Data (like Apache Kafka). Because it serializes row by row, it is efficient for writing single events as they occur.

Apache Parquet

Parquet is a columnar binary format. While Avro serializes data horizontally, Parquet serializes data vertically. It groups all the values for a specific column together, allowing for effective encoding and compression (like Dictionary Encoding). Parquet is the most common file format for Historical Batch Analytics. It is a poor fit for writing single rows, but well suited to analytical queries over large datasets.

Zero-Copy Data Exchange: Apache Arrow

Historically, even binary formats carried a CPU cost. When Spark (Java) read a Parquet file, it had to deserialize the binary disk format into Java objects. When it sent that data to Pandas (Python), it had to serialize it again.

Apache Arrow changed the picture by proposing an alternative: What if we didn’t serialize at all?

Arrow defines a standardized, language-independent columnar memory format. When data is formatted in Arrow, code in different languages (for example, Java and Python in the same process, or separate processes using shared memory) can read the same buffers directly. Data can be passed between languages and systems without serialization or deserialization, and often without copying (Zero-Copy).

Conclusion

Serialization is the invisible bridge connecting all modern software. Choosing the right format affects the speed, cost, and efficiency of your architecture. Use JSON for web APIs where human readability is paramount; use Protobuf/gRPC for high-speed microservices; use Avro for Kafka streaming; and use Parquet/Arrow for analytical data lakehouses. Understanding these trade-offs is fundamental to building scalable data systems.

Learn More

To learn more about the Data Lakehouse, read The Lakehouse Built for Everyone by Alex Merced. You can find this and other books by Alex Merced at books.alexmerced.com.