How to Install and Configure Kafka Manager on Ubuntu
Kafka Manager is an open-source tool developed by Yahoo that provides a user-friendly web interface to manage and monitor Apache Kafka clusters. It simplifies tasks like monitoring broker statistics, managing topics, partitions, and consumer groups. In this guide, we will walk you through how to install and configure Kafka Manager on an Ubuntu server.
Prerequisites
Before installing Kafka Manager, ensure you have the following prerequisites set up on your system:
- Ubuntu 20.04 or later.
- Apache Kafka installed and running on your server. If Kafka is not installed, you can follow the official Kafka installation guide to set it up.
- Java Development Kit (JDK) installed, as Kafka Manager requires Java to run.
To verify that you have Java installed, run:
java -version
If Java is not installed, you can install it using the following command:
sudo apt update
sudo apt install openjdk-11-jdk -y
Step 1: Install Kafka Manager
Kafka Manager is a Java application, so you will need to download it from the official GitHub repository and run it on your server.
1.1 Download Kafka Manager
First, clone the Kafka Manager repository from GitHub:
git clone https://github.com/yahoo/kafka-manager.git
Navigate to the cloned directory:
cd kafka-manager
1.2 Build Kafka Manager
Kafka Manager uses sbt (Scala Build Tool) to build and manage its dependencies. To install sbt, first add the sbt repository:
echo “deb https://repo.scala-sbt.org/scalasbt/debian all main” | sudo tee /etc/apt/sources.list.d/sbt.list
echo “deb https://repo.scala-sbt.org/scalasbt/debian /” | sudo tee /etc/apt/sources.list.d/sbt_dev.list
sudo apt-key adv –keyserver hkp://keyserver.ubuntu.com:80 –recv-keys 99E82A75642AC823
sudo apt update
sudo apt install sbt -y
Now, build the Kafka Manager project:
sbt clean dist
This command compiles the project and creates a distribution package.
1.3 Extract and Configure Kafka Manager
After the build process is complete, navigate to the target/universal directory and extract the Kafka Manager package:
cd target/universal
unzip kafka-manager-*.zip
Once extracted, navigate into the extracted directory:
cd kafka-manager-*
Step 2: Configure Kafka Manager
Before starting Kafka Manager, you need to configure it to connect to your Kafka cluster.
2.1 Edit the Configuration File
Kafka Manager uses the application.conf file for its configuration. You will need to modify this file to specify the Kafka clusters it should manage.
First, open the configuration file for editing:
nano conf/application.conf
Add the following configurations to the file:
kafka-manager.zkhosts=”localhost:2181″
This specifies the ZooKeeper host where Kafka is running. Replace localhost:2181 with the actual address of your ZooKeeper server if it’s running on a different machine.
2.2 Enable Authentication (Optional)
If you want to secure Kafka Manager with authentication, you can enable basic authentication in the application.conf file. Add the following lines:
kafka-manager.basicAuthentication.enabled=true
kafka-manager.basicAuthentication.username=”admin”
kafka-manager.basicAuthentication.password=”admin_password”
Replace admin and admin_password with your desired credentials.
Step 3: Start Kafka Manager
Once you’ve configured Kafka Manager, you can start it using the following command:
./bin/kafka-manager
Kafka Manager will start and listen on port 9000 by default. You can access the web interface by visiting:
http://:9000
Step 4: Access the Kafka Manager Web Interface
Once Kafka Manager is running, open your web browser and navigate to the IP address of your server with port 9000, such as:
http://:9000
You will be presented with the Kafka Manager dashboard. From here, you can perform the following tasks:
- Add Kafka clusters.
- Manage topics and partitions.
- Monitor brokers, consumer groups, and cluster statistics.
4.1 Add a Kafka Cluster
To manage your Kafka cluster, you need to add it to Kafka Manager. From the dashboard:
- Click on Cluster -> Add Cluster.
- Provide the required details such as:
- Cluster Name: A name to identify your Kafka cluster.
- ZooKeeper Hosts: The address of your ZooKeeper instance (e.g., localhost:2181).
- Kafka Version: The version of Kafka you are using.
- Enable JMX Polling: Optional, if you want Kafka Manager to retrieve JMX metrics.
- Click Save to add the cluster.
- Kafka Manager will now connect to the cluster and allow you to monitor and manage it through the web interface.
Step 5: Managing Kafka Clusters with Kafka Manager
Once your Kafka cluster is added, you can perform several management tasks using Kafka Manager:
- Monitor Brokers: Get an overview of brokers in your Kafka cluster, including the state of each broker.
- Manage Topics: Create, delete, and modify topics. You can also view topic details like partitions and replica distribution.
- Monitor Consumer Groups: Track consumer lag and the state of consumer groups.
- View Logs: View logs related to Kafka operations and debugging.
Step 6: Running Kafka Manager as a Service
To ensure that Kafka Manager runs continuously, even after a system reboot, you can configure it to run as a systemd service.
Create a service file:
sudo nano /etc/systemd/system/kafka-manager.service
Add the following content:
[Unit]
Description=Kafka Manager
After=network.target[Service]
User=
ExecStart=/path/to/kafka-manager/bin/kafka-manager
Restart=on-failure[Install]
WantedBy=multi-user.target
Replace with your actual username and update the ExecStart path to point to your Kafka Manager installation directory.
Then, reload the systemd daemon and start Kafka Manager:
sudo systemctl daemon-reload
sudo systemctl start kafka-manager
sudo systemctl enable kafka-manager
Kafka Manager will now run as a service and start automatically on system reboot.
We appreciate your visit to our page! If you’re interested in exploring more articles about Linux systems and using Rundeck for job scheduling, feel free to check out the links below.
How to Install and Use Rundeck for Job Scheduling on Ubuntu
Additionally, by renting a server from our site, you can conduct your tests in a reliable and efficient environment, allowing you to enhance your skills more quickly. Keep up the great work! 🙂
