ksqlDB HOWTO: Split and Merge Kafka Topics

Using ksqlDB you can split streams of data in Apache Kafka based on values in a field. You can also merge separate streams of data together into one.

ksqlDB uses SQL to describe the stream processing that you want to do. For example:

Splitting a stream:

CREATE STREAM ORDERS_UK
AS SELECT * FROM ORDERS WHERE COUNTRY='UK';

CREATE STREAM ORDERS_OTHER
AS SELECT * FROM ORDERS WHERE COUNTRY!='UK';

Merging streams

CREATE STREAM INVENTORY_COMBINED
AS SELECT 'WH1' AS SOURCE, * FROM INVENTORY_WH1;

CREATE STREAM INVENTORY_COMBINED
AS SELECT 'WH2' AS SOURCE, * FROM INVENTORY_WH2;

-----

💾 Run ksqlDB yourself: https://ksqldb.io?utm_source=youtube&...
☁️ Use ksqlDB as a managed service: https://www.confluent.io/confluent-cl...

👾 Demo code: https://github.com/confluentinc/demo-...

🤔 Questions? Join the Confluent Community at https://confluent.io/community/ask-th...

-----

⏱ Time codes

00:00:00 Introduction
00:00:36 Listing topics in Confluent Control Center
00:00:41 Viewing the messages in a topic in Confluent Control Center
00:01:28 Creating a ksqlDB stream on existing topics in Apache Kafka
00:02:05 Querying a stream in ksqlDB from Confluent Control Center
00:02:47 Merging streams of data in ksqlDB
00:03:18 Create the target stream with CREATE STREAM
00:03:41 Adding metadata fields
00:03:49 Creating unique fields by concatenating values
00:04:13 Setting the partitioning key
00:04:33 Add new streams into the target stream with INSERT INTO
00:05:24 Inspecting running ksqlDB queries with Confluent Control Center
00:05:53 Querying the merged stream
00:06:27 Merging streams - recap
00:07:08 Splitting a Kafka topic into separate streams with ksqlDB
00:09:00 Splitting a stream - recap