How to Install and Configure Apache Kafka on Ubuntu
Apache Kafka is a distributed streaming platform that is capable of handling trillions of events a day. It is widely used for building real-time data pipelines and streaming applications. In this article, we’ll guide you through the process of installing and configuring Apache Kafka on Ubuntu.
Prerequisites
Before we begin, ensure that you have the following:
- A running Ubuntu server (Ubuntu 18.04 or later is recommended).
- Java Development Kit (JDK) installed on your system, as Kafka is written in Java.
- Sufficient system resources: A minimum of 1 GB of RAM and 2 CPU cores is recommended for testing.
Step 1: Install Java
Apache Kafka requires Java to run. You can install the OpenJDK package using the following commands:
sudo apt update
sudo apt install openjdk-11-jdk
After installation, verify the Java installation by running:
java -version
You should see an output indicating the installed Java version.
Step 2: Download Apache Kafka
Next, download the latest version of Apache Kafka from the official website. You can find the latest stable release on the Apache Kafka Downloads page.
Use the following commands to download Kafka:
wget https://downloads.apache.org/kafka/3.5.1/kafka_2.13-3.5.1.tgz
(Replace 3.5.1 and 2.13 with the latest version available.)
Once the download is complete, extract the tar file:
tar -xzf kafka_2.13-3.5.1.tgz
Move into the Kafka directory:
cd kafka_2.13-3.5.1
Step 3: Start Zookeeper
Apache Kafka uses Zookeeper to manage distributed brokers. You need to start Zookeeper before starting Kafka. Kafka comes with a built-in Zookeeper instance that you can use for development.
Run the following command to start Zookeeper:
bin/zookeeper-server-start.sh config/zookeeper.properties
Leave this terminal open, as Zookeeper needs to run in the background.
Step 4: Start Kafka Server
Now that Zookeeper is running, you can start the Kafka server. Open a new terminal window and navigate to the Kafka directory again. Then, run the following command:
bin/kafka-server-start.sh config/server.properties
Your Kafka server should now be running. You can check the logs to ensure there are no errors:
tail -f logs/server.log
Step 5: Create a Kafka Topic
Now that your Kafka server is up and running, you can create a topic to send and receive messages. Use the following command to create a new topic named test-topic:
bin/kafka-topics.sh –create –topic test-topic –bootstrap-server localhost:9092 –partitions 1 –replication-factor 1
You should see a confirmation message indicating that the topic has been created.
Step 6: Send Messages to the Topic
You can produce messages to your topic using the following command:
bin/kafka-console-producer.sh –topic test-topic –bootstrap-server localhost:9092
Once you run this command, you can start typing messages. Press Enter after each message to send it.
Step 7: Consume Messages from the Topic
To read the messages you produced, open another terminal window and run the following command:
bin/kafka-console-consumer.sh –topic test-topic –from-beginning –bootstrap-server localhost:9092
You should see the messages you sent previously displayed in this terminal.
Step 8: Configure Kafka
Kafka can be configured by modifying the server.properties file located in the config directory. Here are some important configurations you might want to consider:
- Broker ID: Each broker in a Kafka cluster must have a unique ID. You can set it using the broker.id property.
- Listeners: By default, Kafka listens on localhost. To make it accessible from other machines, modify the listeners property to include the server’s IP address.
- Log directories: Configure the log directory with the log.dirs property. Ensure that this directory has sufficient disk space.
After making changes, restart the Kafka server for the changes to take effect:
bin/kafka-server-stop.sh
bin/kafka-server-start.sh config/server.properties
Step 9: Set Up Kafka as a Systemd Service (Optional)
To run Kafka as a service on your Ubuntu system, create a systemd service file. Create a new file called kafka.service in the /etc/systemd/system/ directory:
sudo nano /etc/systemd/system/kafka.service
Add the following configuration:
[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
After=zookeeper.service[Service]
Type=simple
User=your-username
ExecStart=/path/to/kafka/bin/kafka-server-start.sh /path/to/kafka/config/server.properties
ExecStop=/path/to/kafka/bin/kafka-server-stop.sh
Restart=on-failure[Install]
WantedBy=multi-user.target
Make sure to replace /path/to/kafka with the actual path where Kafka is installed and your-username with the user that should run Kafka.
After creating the file, reload the systemd daemon and start the Kafka service:
sudo systemctl daemon-reload
sudo systemctl start kafka
sudo systemctl enable kafka
Thank you for visiting our page! Don’t forget to check out our other article through the link below to enhance your Linux skills. Also, be sure to read our guide on How to Monitor System Resources with Netdata on Linux! 🙂
