Apache Kafka is a fast and scalable messaging queue, capable of handeling heavy loads in context of read and write, i.e. IO-related stuff. You can find more about Kafka on http://kafka.apache.org/. Apache Kafka requires a running Zookeeper instance, which is used for reliable distributed cordination. Please find more about Zookeeper on https://zookeeper.apache.org/.
Downloading the Required Files
- Download and extract Zookeeper using 7-zip from http://zookeeper.apache.org/releases.html
- Download and extract Kafka using 7-zip from http://kafka.apache.org/downloads.html
1. Goto your Zookeeper config directory. For me its C:\zookeeper-3.4.7\conf
2. Rename file “zoo_sample.cfg” to “zoo.cfg”
3. Open zoo.cfg in any text editor like notepad++.
4. Find & edit dataDir=/tmp/zookeeper to :\zookeeper-3.4.7\data
5. Add entry in System Environment Variables : ZOOKEEPER_HOME = C:\zookeeper-3.4.7
6. Edit System Variable named “Path” add ;%ZOOKEEPER_HOME%\bin;
7. You can change the default Zookeeper port in zoo.cfg file (Default port 2181).
7. Run Zookeeper by opening a new cmd and type :
Setting Up Kafka
1. Go to your Kafka config directory. For me its C:\kafka_2.11-0.9.0.0\config
2. Edit file “server.properties”
3. Find & edit line “log.dirs=/tmp/kafka-logs” to “log.dirs=/kafka_2.11-0.9.0.0/kafka-logs”.
4. If your Zookeeper is running on some other machine or cluster you can edit “zookeeper.connect:2181” to your custom IP and port. For this demo we are using same machine so no need to change. Also Kafka port & broker.id are configurable in this file. Leave other settings as it is.
5. Your Kafka will run on default port 9092 & connect to zookeeper’s default port which is 2181.
Running Kafka Server
Important: Please ensure that your Zookeeper instance is up and running before starting a Kafka server.
1. Go to your Kafka installation directory C:\kafka_2.11-0.9.0.0\
2. Open a command prompt here by pressing Shift + right click and choose“Open command window here” option)
3. C:\kafka_2.11-0.9.0.0> Now type :
.\bin\windows\kafka-server-start.bat .\config\server.properties and press Enter.
5. Now your Kafka is up and running, you can create topics to store messages. Also we can produce or consume data from Java or Scala code or directly from the command prompt.
1. Now create a topic with name “test” and replication factor 1, as we have only one Kafka server running. If you have a cluster with more than 1 Kafka server running, you can increase the replication-factor accordingly which will increase the data availability and act like a fault-tolerant system.
2. Open a new command prompt in the location C:\kafka_2.11-0.9.0.0\bin\windows
3. C:\kafka_2.11-0.9.0.0\bin\windows> Type following command and hit Enter:
kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test and press Enter.
Creating a Producer and Consumer to Test Server
1. Open a new command prompt in the location C:\kafka_2.11-0.9.0.0\bin\windows
2. C:\kafka_2.11-0.9.0.0\bin\windows> To start a producer type the following command:
kafka-console-producer.bat --broker-list localhost:9092 --topic test and press Enter.
3. Again open a new command prompt in the same location as C:\kafka_2.11-0.9.0.0\bin\windows
4. C:\kafka_2.11-0.9.0.0\bin\windows> Now start a consumer by typing the following command:
kafka-console-consumer.bat --zookeeper localhost:2181 --topic test
5. Now you will have two command prompts.
6. Now type anything in the producer command prompt and press Enter, and you should be able to see the message in the other consumer command prompt.
7. If you are able to push & see your messages in the consumer side, you are done with Kafka setup.
Some Other Useful Commands
kafka-topics.bat --list --zookeeper localhost:2181
kafka-topics.bat --describe --zookeeper localhost:2181 --topic [Topic Name]
Read messages from beginning:
kafka-console-consumer.bat --zookeeper localhost:2181 --topic [Topic Name] --from-beginning
kafka-run-class.bat kafka.admin.TopicCommand --delete --topic [Topic Name To Delete] --zookeeper localhost:2181