Apache Kafka

https://kafka.apache.org/intro

Apache Kafka® is a distributed streaming platform.

A streaming platform has three key capabilities:

Publisher/Subscriber, Observer pattern, Message queues.

First a few concepts:

Kafka has four core APIs:

Topics in Kafka are always multi-subscriber; that is, a topic can have zero, one, or many consumers that subscribe to the data written to it.

How does Kafka's notion of streams compare to a traditional enterprise messaging system? Messaging traditionally has two models: queuing and publish-subscribe. In a queue, a pool of consumers may read from a server and each record goes to one of them; in publish-subscribe the record is broadcast to all consumers. By having a notion of parallelism—the partition—within the topics, Kafka is able to provide both ordering guarantees and load balancing over a pool of consumer processes.

https://kafka.apache.org/uses

Kafka works well as a replacement for a more traditional message broker. Kafka is comparable to traditional messaging systems such as ActiveMQ or RabbitMQ.

https://www.upsolver.com/blog/kafka-versus-rabbitmq-architecture-performance-use-case

Kafka uses a pull model, the Kafka broker waits for the consumer to ask for data.

Kafka provides message ordering (stream/streaming).

Kafka is a log, which means that it retains messages by default.

JMS client

https://docs.confluent.io/current/clients/kafka-jms-client/index.html

JMS is a widely used messaging API that is included as part of the Java Platform, Enterprise Edition. Confluent JMS Client (kafka-jms-client) is an implementation of the JMS 1.1 provider interface that allows Apache Kafka® or Confluent Platform to be used as a JMS message broker.

Kafka topics can mimic the behavior of either topics or queues in the traditional messaging system sense. Both JMS messaging models are supported: Publish/Subscribe (Topics), Point-to-Point (Queues)

Example

   1 wget http://mirrors.up.pt/pub/apache/kafka/2.3.0/kafka_2.11-2.3.0.tgz
   2 tar xvzf kafka_2.11-2.3.0.tgz 
   3 cd kafka_2.11-2.3.0/
   4 # single-node ZooKeeper instance (port 2181)
   5 bin/zookeeper-server-start.sh config/zookeeper.properties
   6 # new tab ....
   7 cd kafka_2.11-2.3.0/
   8 bin/kafka-server-start.sh config/server.properties # listens port 9092
   9 # create topic
  10 bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
  11 # check topics
  12 bin/kafka-topics.sh --list --bootstrap-server localhost:9092
  13 # send messages to topic
  14 bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
  15 >hello
  16 >test
  17 # consume messages
  18 bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
  19 # https://pypi.org/project/kafka/
  20 apt install python-pip # as root
  21 pip install kafka
  22 # https://pypi.org/project/kafka/
  23 

   1 #producer.py
   2 from kafka import KafkaProducer
   3 producer = KafkaProducer(bootstrap_servers='localhost:9092',compression_type='gzip' )
   4 for i in range(5):
   5     producer.send('test', b'some_message_bytes [%d]'%(i))
   6 producer.flush()

   1 #consumer.py
   2 from kafka import KafkaConsumer
   3 consumer = KafkaConsumer('test',bootstrap_servers="localhost:9092")
   4 for msg in consumer:
   5     print("%s %d %s"%(msg.topic, msg.timestamp, msg.value))

Create queue adder for 2 consumers

Amount of partitions equals the amount of consumers.

   1 #consumer_adder.py
   2 from kafka import KafkaConsumer
   3 import json
   4 import sys
   5 
   6 topic='adder'
   7 consumer = KafkaConsumer('%s-%s'%(topic,sys.argv[1]),bootstrap_servers="localhost:9092")
   8 print consumer.partitions_for_topic(topic)
   9 
  10 for msg in consumer:
  11     vals = json.loads(msg.value)
  12     print("%s %d %s sum: %d"%(msg.topic, msg.timestamp, msg.value, vals['op1']+vals['op2']  ))

   1 #producer_adder.py
   2 from kafka import KafkaProducer
   3 import json
   4 producer = KafkaProducer(bootstrap_servers='localhost:9092',compression_type='gzip' )
   5 topic='adder'
   6 parts = producer.partitions_for(topic)
   7 amount_partitions = len(parts)
   8 
   9 for i in range(10000):
  10     vals = {'op1':i,'op2':i}
  11     #print('adder-%d'%(i%2))
  12     producer.send('%s-%d'%(topic,i%amount_partitions), value=b'%s'%( json.dumps(vals) )  )

List topics using zookeeper

ZooKeeper is a high-performance coordination service for distributed applications. The name space provided by ZooKeeper is much like that of a standard file system.

bin/zookeeper-shell.sh localhost:2181
ls /config/topics
[adder-0, adder, adder-1, test, __consumer_offsets]
quit

   1 from kazoo.client import KazooClient
   2 zk = KazooClient(hosts='127.0.0.1:2181')
   3 zk.start()
   4 zk.ensure_path('/config/topics')
   5 # True
   6 zk.get_children("/config/topics")
   7 #[u'adder-0', u'adder', u'adder-1', u'test', u'__consumer_offsets']
   8 zk.stop()
   9 quit()

Spring kafka

Dockerfile

   1 FROM eclipse-temurin:17-jdk-alpine
   2 ENV PATH=/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/kafka_2.12-3.4.0/bin/
   3 RUN apk add --update --no-cache curl wget nano vim bash gcompat
   4 RUN cd ~ && wget https://downloads.apache.org/kafka/3.4.0/kafka_2.12-3.4.0.tgz && tar xvzf kafka_2.12-3.4.0.tgz
   5 CMD ["/bin/bash","/mnt/start-servers.sh"]

start-servers.sh

   1 #!/bin/bash
   2 nohup /root/kafka_2.12-3.4.0/bin/zookeeper-server-start.sh /root/kafka_2.12-3.4.0/config/zookeeper.properties &
   3 nohup /root/kafka_2.12-3.4.0/bin/kafka-server-start.sh /root/kafka_2.12-3.4.0/config/server.properties &
   4 cat

Build kafka container

   1 docker build -t kafka-test-image .
   2 docker run --rm -dit --name kafka-test -p 2181:2181 -p 9092:9092 -v $PWD:/mnt/ kafka-test-image
   3 mvn clean install
   4 docker cp target/demo-0.0.1-SNAPSHOT.jar kafka-test:/tmp/
   5 docker exec -it kafka-test bash

pom.xml

   1 <?xml version="1.0" encoding="UTF-8"?>
   2 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   3         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
   4         <modelVersion>4.0.0</modelVersion>
   5         <parent>
   6                 <groupId>org.springframework.boot</groupId>
   7                 <artifactId>spring-boot-starter-parent</artifactId>
   8                 <version>3.1.0</version>
   9                 <relativePath/> <!-- lookup parent from repository -->
  10         </parent>
  11         <groupId>com.example</groupId>
  12         <artifactId>demo</artifactId>
  13         <version>0.0.1-SNAPSHOT</version>
  14         <name>demo</name>
  15         <description>Demo project for Spring Boot</description>
  16         <properties>
  17                 <java.version>17</java.version>
  18         </properties>
  19         <dependencies>
  20                 <dependency>
  21                         <groupId>org.springframework.boot</groupId>
  22                         <artifactId>spring-boot-starter</artifactId>
  23                 </dependency>
  24                 <dependency>
  25                         <groupId>org.springframework.kafka</groupId>
  26                         <artifactId>spring-kafka</artifactId>
  27                 </dependency>
  28                 <dependency>
  29                     <groupId>org.springframework.boot</groupId>
  30                     <artifactId>spring-boot-starter-web</artifactId>
  31                 </dependency>
  32         </dependencies>
  33         <build>
  34                 <plugins>
  35                         <plugin>
  36                                 <groupId>org.springframework.boot</groupId>
  37                                 <artifactId>spring-boot-maven-plugin</artifactId>
  38                         </plugin>
  39                 </plugins>
  40         </build>
  41 </project>

application.properties

   1 spring.kafka.bootstrap-servers=127.0.0.1:9092
   2 spring.kafka.consumer.group-id=group-id

DemoApplication.java

   1 // src/main/java/com/example/demo/DemoApplication.java
   2 package com.example.demo;
   3 
   4 import org.springframework.boot.SpringApplication;
   5 import org.springframework.boot.autoconfigure.SpringBootApplication;
   6 
   7 @SpringBootApplication
   8 public class DemoApplication {
   9 
  10         public static void main(String[] args) {
  11                 SpringApplication.run(DemoApplication.class, args);
  12         }
  13 }

DemoController.java

   1 // src/main/java/com/example/demo/DemoController.java
   2 package com.example.demo;
   3 
   4 import org.springframework.kafka.core.KafkaTemplate;
   5 import org.springframework.stereotype.Controller;
   6 import org.springframework.web.bind.annotation.GetMapping;
   7 import org.springframework.web.bind.annotation.PathVariable;
   8 import org.springframework.web.bind.annotation.ResponseBody;
   9 
  10 @Controller
  11 public class DemoController {
  12     private KafkaTemplate<String, String> kafkaTemplate;
  13 
  14     public DemoController(KafkaTemplate<String, String> kafkaTemplate) {
  15         this.kafkaTemplate = kafkaTemplate;
  16     }
  17 
  18     @GetMapping("/uppercase/{text}")
  19     @ResponseBody
  20     public String uppercase(@PathVariable String text) {
  21         String message = String.format("text to be sent in uppercase %s", text);
  22         kafkaTemplate.send(KafkaTopicConfig.TOPIC_TASK, message);
  23         return message;
  24     }
  25 
  26 }

KafkaConsumerConfig.java

   1 // src/main/java/com/example/demo/KafkaConsumerConfig.java
   2 package com.example.demo;
   3 
   4 import java.util.HashMap;
   5 import java.util.Map;
   6 import org.apache.kafka.clients.consumer.ConsumerConfig;
   7 import org.apache.kafka.common.serialization.StringDeserializer;
   8 import org.springframework.beans.factory.annotation.Value;
   9 import org.springframework.context.annotation.Bean;
  10 import org.springframework.context.annotation.Configuration;
  11 import org.springframework.kafka.annotation.EnableKafka;
  12 import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
  13 import org.springframework.kafka.core.ConsumerFactory;
  14 import org.springframework.kafka.core.DefaultKafkaConsumerFactory;
  15 
  16 @EnableKafka
  17 @Configuration
  18 public class KafkaConsumerConfig {
  19     @Value(value = "${spring.kafka.bootstrap-servers}")
  20     private String bootstrapAddress;
  21     @Value(value = "${spring.kafka.consumer.group-id}")
  22     private String groupId;
  23 
  24     @Bean
  25     public ConsumerFactory<String, String> consumerFactory() {
  26         Map<String, Object> props = new HashMap<>();
  27         props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
  28         props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
  29         props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
  30         props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
  31         return new DefaultKafkaConsumerFactory<>(props);
  32     }
  33 
  34     @Bean
  35     public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
  36         ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
  37         factory.setConsumerFactory(consumerFactory());
  38         return factory;
  39     }    
  40 }

KafkaMessageListener.java

   1 // src/main/java/com/example/demo/KafkaMessageListener.java
   2 package com.example.demo;
   3 
   4 import org.slf4j.Logger;
   5 import org.slf4j.LoggerFactory;
   6 import org.springframework.kafka.annotation.KafkaListener;
   7 import org.springframework.kafka.listener.MessageListener;
   8 import org.springframework.stereotype.Component;
   9 
  10 @Component
  11 public class KafkaMessageListener {
  12     private Logger logger;
  13 
  14     public KafkaMessageListener() {
  15         this.logger = LoggerFactory.getLogger(MessageListener.class);
  16         this.logger.info("Created rest MessageListener");
  17     }
  18 
  19     @KafkaListener(topics = KafkaTopicConfig.TOPIC_TASK)
  20     public void listen(String message) {
  21         System.out.println("Received Message in topicTask: " + message + " in uppercase " + message.toUpperCase());
  22     }
  23 }

KafkaProducerConfig.java

   1 // src/main/java/com/example/demo/KafkaProducerConfig.java
   2 package com.example.demo;
   3 
   4 import java.util.HashMap;
   5 import java.util.Map;
   6 import org.apache.kafka.clients.producer.ProducerConfig;
   7 import org.apache.kafka.common.serialization.StringSerializer;
   8 import org.springframework.beans.factory.annotation.Value;
   9 import org.springframework.context.annotation.Bean;
  10 import org.springframework.context.annotation.Configuration;
  11 import org.springframework.kafka.core.DefaultKafkaProducerFactory;
  12 import org.springframework.kafka.core.KafkaTemplate;
  13 import org.springframework.kafka.core.ProducerFactory;
  14 
  15 @Configuration
  16 public class KafkaProducerConfig {
  17     @Value(value = "${spring.kafka.bootstrap-servers}")
  18     private String bootstrapAddress;
  19 
  20     @Bean
  21     public ProducerFactory<String, String> producerFactory() {
  22         Map<String, Object> configProps = new HashMap<>();
  23         configProps.put(
  24                 ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
  25                 bootstrapAddress);
  26         configProps.put(
  27                 ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
  28                 StringSerializer.class);
  29         configProps.put(
  30                 ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
  31                 StringSerializer.class);
  32         return new DefaultKafkaProducerFactory<>(configProps);
  33     }
  34 
  35     @Bean
  36     public KafkaTemplate<String, String> kafkaTemplate() {
  37         return new KafkaTemplate<>(producerFactory());
  38     }
  39 }

KafkaTopicConfig.java

   1 // src/main/java/com/example/demo/KafkaTopicConfig.java
   2 package com.example.demo;
   3 
   4 import java.util.HashMap;
   5 import java.util.Map;
   6 
   7 import org.apache.kafka.clients.admin.AdminClientConfig;
   8 import org.apache.kafka.clients.admin.NewTopic;
   9 import org.springframework.beans.factory.annotation.Value;
  10 import org.springframework.context.annotation.Bean;
  11 import org.springframework.context.annotation.Configuration;
  12 import org.springframework.kafka.core.KafkaAdmin;
  13 
  14 @Configuration
  15 public class KafkaTopicConfig {
  16     public static final String TOPIC_TASK = "topicTask";
  17     @Value(value = "${spring.kafka.bootstrap-servers}")
  18     private String bootstrapAddress;
  19 
  20     @Bean
  21     public KafkaAdmin kafkaAdmin() {
  22         Map<String, Object> configs = new HashMap<>();
  23         configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
  24         return new KafkaAdmin(configs);
  25     }
  26 
  27     @Bean
  28     public NewTopic topicTask() {
  29         return new NewTopic(TOPIC_TASK, 1, (short) 1);
  30     }
  31 }

ApacheKafka (last edited 2026-02-21 16:01:46 by vitor)