Datalake Rock Paper Scissors: Iceberg + Flink or Iceberg + Spark? | Current 2023

Bloomberg uses Apache Kafka® and Apache Iceberg® as core elements in their real-time data pipelines and storage sinks. In this talk, Sitarama Chekuri and Ben de Vera share their lessons learned testing both Apache Flink® and Apache Spark® to ingest data from Kafka into their Iceberg datalake at near-real-time speeds. They compare and contrast the two technologies with regard to functionality, performance, fault-tolerance, scaling, and resource utilization.

CHAPTERS
00:00 - Intro
01:06 - Context on Bloomberg and speakers
03:14 - Motivation
05:41 - Technology overview
16:34 - Performance comparison
32:13 - Scale to multiple applications
35:10 - Summary

Speakers: Sitarama Chekuri and Ben de Vera

--

ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion – designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.

#current2023 #apachekafka #kafka #confluent