Olena Kutsenko – Mastering real-time anomaly detection with open source tools

More: https://2025.berlinbuzzwords.de/sessi...

Speaker: Olena Kutsenko

With data moving faster than ever, detecting problems as they happen is crucial. This talk covers how to build a real-time anomaly detection system using Apache Kafka for streaming, Apache Flink for processing, and AI for pattern recognition. Plus, we’ll explore Apache Iceberg for storing historical data to refine models.

Detecting problems as they happen is essential in today’s fast-moving world. This talk shows how to build a simple, powerful system for real-time anomaly detection. We’ll use Apache Kafka for streaming data, Apache Flink for processing it, and AI to find unusual patterns. Whether it’s spotting fraud, monitoring systems, or tracking IoT devices, this solution is flexible and reliable.

First, we’ll explain how Kafka helps collect and manage fast-moving data. Then, we’ll show how Flink processes this data in real time to detect events as they happen. We’ll also explore how to add AI to the pipeline, using pre-trained models to find anomalies with high accuracy. Finally, we’ll look at how Apache Iceberg can store past data for analysis and model improvements. Combining real-time detection with historical data makes the system smarter and more effective over time.

This talk includes clear examples and practical steps to help you build your own pipeline. It’s perfect for anyone who wants to learn how to use open-source tools to spot problems in real-time data streams.

###

Follow us on Social Media and join the Community!

Mastodon: https://floss.social/@BerlinBuzzwords
LinkedIn:   / berlin-buzzwords  
Website: https://berlinbuzzwords.de
Mail: [email protected]

Berlin Buzzwords is an event by Plain Schwarz – https://plainschwarz.com