🎄Twelve Days of SMT 🎄 - Day 8: TimestampConverter

The TimestampConverter Single Message Transform lets you work with timestamp fields in Kafka messages. You can convert a string into a native Timestamp type (or Date or Time), as well as Unix epoch - and the same in reverse too.

This is really useful to make sure that data ingested into Kafka is correctly stored as a Timestamp (if it is one), and also enables you to write a Timestamp out to a sink connector in a string format that you choose.

The TimestampConverter takes three arguments; the name of the field holding the timestamp, the data type to which you want to convert it, and the format of the timestamp to parse (if reading a string and target.type is unix/timestamp/date/time) or write (if you’re writing a string and target.type is string). The timestamp format is based on SimpleDateFormat.

"transforms" : "convertTS",
"transforms.convertTS.type" : "org.apache.kafka.connect.transforms.TimestampConverter$Value",
"transforms.convertTS.field" : "txn_date",
"transforms.convertTS.format" : "EEE MMM dd HH:mm:ss zzz yyyy",
"transforms.convertTS.target.type": "Timestamp"

---

⏱️Time codes:

00:00:00 Brief introduction
00:01:16 Kafkacat - the swiss-army knife of Kafka commandline tools
00:02:45 Inspecting the schema for a topic in the Schema Registry
00:04:56 Setting up the basic sink connector (no transform)
00:05:38 Using the TimestampConverter to write a string to a timestamp in a database table
00:08:56 Write the date portion of a string timestamp to a Date type in target sink connector
00:10:54 Write the date portion of a string timestamp to a Date type in target sink connector
00:11:43 Converting a string timestamp into unix timestamp (epoch)
00:14:10 Using the TimestampConverter with a Source connector
00:17:00 A brief tangent about useful bash keyboard shortcuts
00:18:19 Converting at ingest vs egress
00:18:50 Recap

---

👾 Demo code and details: https://github.com/confluentinc/demo-...

🗒️ SMT reference: https://docs.confluent.io/platform/cu...

Learn more about Kafka Connect here:
🏃‍♂️ Quick:    • Kafka Connect in 60 seconds  
🚶 More detail: https://rmoff.dev/kafka-connect-zero-...

---

☁️ Confluent Cloud: https://confluent.cloud/signup?utm_so...

💾Download Confluent Platform: https://www.confluent.io/download/?ut...

📺 Kafka Connect connector deep-dives:    • Kafka Connect  

✍️Kafka Connect documentation: https://docs.confluent.io/current/con...

🧩Confluent Hub: https://www.confluent.io/hub/?utm_sou...