Error Handling via Dead Letter Queue in Apache Kafka

In this video, I dive into the critical topic of error handling in data streaming pipelines using Apache Kafka. Discover the power of Dead Letter Queues (DLQ) and learn best practices for implementing reliable error handling in your Kafka infrastructure.

I explore multiple approaches, including:
Custom DLQ implementations
Kafka Streams and Kafka Connect
Real-world case studies from Uber, Santander Bank and Robinhood

Whether you’re building ETL pipelines, real-time fraud detection, or scalable data processing applications, mastering error handling ensures your data streaming architecture is reliable and resilient.

Table of Contents:

00:00 - Intro
00:29 - Dead Letter Queue
01:22 - Dead Letter Queue and Apache Kafka
03:09 - Data Contracts with Schema Registry
05:36 - DLQ in Kafka Connect
07:06 - DLQ in Kafka Streams
07:30 - Consuming from a DLQ
10:05 - Real World Examples for Kafka DLQs
11:43 - Stay in Touch

Don’t forget to like, comment, and subscribe for more in-depth data streaming content. Let us know your thoughts or questions in the comments section below!

More details about error handling with Apache Kafka and related real world success stories:
https://www.kai-waehner.de/blog/2022/...

Stay in touch via
Blog: https://www.kai-waehner.de
LinkedIn:   / kaiwaehner  
X (Twitter):   / kaiwaehner  
Newsletter: https://kai-waehner.de/news