Ch. 3 Is Kafka What You Think It Is?: Designing Event-Driven Systems Concepts & Patterns for Stre...

Full playlist:    • Designing Event-Driven Systems Concepts an...  
CHAPTER 3 Is Kafka What You Think It Is? Designing Event-Driven Systems Concepts & Patterns for Streaming Services with Apache Kafka
CHAPTER 3 Is Kafka What You Think It Is?
The pdf version is available here. https://assets.confluent.io/m/7a91acf...
CHAPTER 3 Is Kafka What You Think It Is?
There is an old parable about an elephant and a group of blind men.
None of the men had come across an elephant before.
One blind man approaches the leg and declares, “It’s like a tree.” Another man approaches the tail and declares, “It’s like a rope.” A third approaches the trunk and declares, “It’s like a snake.” So each blind man senses the elephant from his particular point of view, and comes to a subtly different conclusion as to what an elephant is.
Of course the elephant is like all these things, but it is really just an elephant!
Likewise, when people learn about Kafka they often see it from a certain view‐ point.
These perspectives are usually accurate, but highlight only some subsec‐ tion of the whole platform.
In this chapter we look at some common points of view.
Kafka Is Like REST but Asynchronous?
Kafka provides an asynchronous protocol for connecting programs together, but it is undoubtedly a bit different from, say, TCP (transmission control protocol), HTTP, or an RPC protocol.
The difference is the presence of a broker.
A broker is a separate piece of infrastructure that broadcasts messages to any programs that are interested in them, as well as storing them for as long as is needed.
So it’s per‐ fect for streaming or fire-and-forget messaging.
Other use cases sit further from its home ground.
A good example is request- response.
Say you have a service for querying customer information.
So you call a getCustomer() method, passing a CustomerId, and get a document describing a customer in the reply.
You can build this type of request-response interaction with Kafka using two topics: one that transports the request and one that trans‐ ports the response.
People build systems like this, but in such cases the broker doesn’t contribute all that much.
There is no requirement for broadcast.
There ismore than data transfer and storage, provided at scale and high availability.
This is emphasized by the core mantra of event-driven services: Centralize an immut‐ able stream of facts.
Decentralize the freedom to act, adapt, and change.
Kafka Is Like a Database?
Some people like to compare Kafka to a database.
It certainly comes with similar features.
It provides storage; production topics with hundreds of terabytes are not uncommon.
It has a SQL interface that lets users define queries and execute them over the data held in the log.
These can be piped into views that users can query directly.
It also supports transactions.
These are all things that sound quite “databasey” in nature!
So many of the elements of a traditional database are there, but if anything, Kafka is a database inside out (see “A Database Inside Out” on page 87 in Chapter 9), a tool for storing data, processing it in real time, and creating views.
And while you are perfectly entitled to put a dataset in Kafka, run a KSQL query over it, and get an answer—much like you might in a traditional database—KSQL and Kafka Streams are optimized for continual computation rather than batch processing.
So while the analogy is not wholly inaccurate, it is a little off the mark.
Kafka is designed to move data, operating on that data as it does so.
It’s about real-time processing first, long-term storage second.
What Is Kafka Really?
A Streaming Platform As Figure 3-1 illustrates, Kafka is a streaming platform.
At its core sits a cluster of Kafka brokers (discussed in detail in Chapter 4).
You can interact with the cluster through a wide range of client APIs in Go, Scala, Python, REST, and more.
There are two APIs for stream processing: Kafka Streams and KSQL (which we discuss in Chapter 14).
These are database engines for data in flight, allowing users to filter streams, join them together, aggregate, store state, and run arbi‐ trary functions over the evolving dataflow.
These APIs can be stateful, which means they can hold data tables much like a regular database (see “Making Serv‐ ices Stateful” on page 47 in Chapter 6).
The third API is Connect.
This has a whole ecosystem of connectors that inter‐ face with different types of database or other endpoints, both to pull data from and push data to Kafka.
Finally there is a suite of utilities—such as Replicator and Mirror Maker, which tie disparate clusters together, and the Schema Registry, which validates and manages schemas—applied to messages passed through Kafka and a number of other tools in the Confluent platform.
A streaming platform brings these tools together wi