Building a High-Performance Binary Serialization Format with In-Place Modification - Hamish Morrison

https://cppcon.org
---

Building a High-Performance Binary Serialization Format with In-Place Modification - Hamish Morrison - CppCon 2025
---

There are a huge number of binary serialization formats out there: protobufs, Thrift, Avro, MessagePack, flatbuffers, BSON, CBOR, and many more. How do they differ? Which one should you use? How fast are they? Codegen vs non-codegen? Schema-based or schemaless? Zero-copy or not?

Within our systems we use many different serialization formats for encoding market data. These formats range from the simple, to more complex tagged formats with compression dictionaries. We recently needed an efficient schema-less data format with support for arrays and maps, and -- critically -- the ability to mutate the data quickly in-place, without re-encoding the whole buffer every time. Essentially, a compact, binary version of JSON, with in-place modification.

This talk walks through the design and implementation of this binary serialization format. We'll cover:

Creating a simple binary serialization using tags and values
Implementing arrays using offset tables and indirection
Using SIMD for fast searches in offset tables
Building more complex data structures in contiguous buffers, in order to implement maps
Minimizing copies during in-place updates
Benchmarking against other serialization formats and standard library containers
Exploring the trade-offs of our format

Throughout the talk we will be developing an ergonomic C++ interface for the serialization format, allowing users to interact with the encoded data similarly to how they would interact with ordinary vectors and maps.

While the focus is on serialization, the talk should be interesting to anyone who is interested in data structures and high-performance computing.


---

Slides: https://github.com/CppCon/CppCon2025/...

Work at Hudson River Trading (HRT): https://tinyurl.com/safxfctf
---

Hamish Morrison

Hamish Morrison is a software engineer at Bloomberg where he works on high performance market data processing systems.
---


CppCon is the annual, week-long face-to-face gathering for the entire C++ community. The conference is organized by the C++ community for the community. You will enjoy inspirational talks and a friendly atmosphere designed to help attendees learn from each other, meet interesting people, and generally have a stimulating experience. Taking place this year in Aurora, Colorado, near the Denver airport, and including multiple diverse tracks, the conference will appeal to anyone from C++ novices to experts.
Annual CppCon Conference - https://www.cppcon.org
  / cppcon  
https://x.com/cppcon
  / cppconference  
  / cppcon  
https://mastodon.social/@CppCon
---

Videos Filmed & Edited by Bash Films: http://www.BashFilms.com
YouTube Channel Managed by Digital Medium Ltd: https://events.digital-medium.co.uk
---

#cpp #cplusplus #programming #coding #softwareengineering