How to install Apache Kafka locally or VMWare or Amazon EC2
This is a guide to installing Apache Kafka locally or VMWare or Amazon EC2 instance. Apache Kafka installation is very straight forward. This installation is based on building kafka from source (there might be a better way of installing kafka, through an existing rpm maybe)
Requirements
- java 1.7 (assumed installed)
- centos 6.5 (assumed installed)
- zookeeper-3.4.5 (assumed installed)
- supervisord (assumed installed)
- kafka 0.8.1.1 (http://www.mirrorservice.org/sites/ftp.apache.org/kafka)
- sbt
Apache Kafka Source
Get the latest version of Apache Kafka from from the mirrorservice
1 2 3 4 5 | wget http://www.mirrorservice.org/sites/ftp.apache.org/kafka/0.8.1.1/kafka-0.8.1.1-src.tgz tar -zxvf kafka-0.8.1.1-src.tgz cd kafka-0.8.1.1-src |
Building Apache Kafka Source
Kafka is written in scala so we need sbt (this is the scala equivalent of maven).
1 2 3 4 5 | wget http://scalasbt.artifactoryonline.com/scalasbt/sbt-native-packages/org/scala-sbt/sbt//0.12.3/sbt.rpm yum localinstall sbt.rpm sbt --version |
We now build and package kafka from source using sbt
1 2 3 4 5 6 | ./sbt update ./sbt package ./sbt assembly-package-dependency mv kafka-0.8.1.1-src kafka-0.8.1.1 |
Environment Variables
Now update your bash profile with some kafka specific environment variables so that these are available to us on restarts or logins
1 2 3 4 5 6 7 | vim ~/.bashrc export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk.x86_64 export KAFKA_HOME=/opt/kafka-0.8.1.1-src export KAFKA=$KAFKA_HOME/bin export KAFKA_CONFIG=$KAFKA_HOME/config |
Apache Zookeeper Configuration
Only perform this statement if we are running zookeeper locally, for connecting to Zk cluster DO NOT perform this line, but change config in the next line
1 2 3 | $KAFKA/zookeeper-server-start.sh $KAFKA_CONFIG/zookeeper.properties |
For connecting to Zk cluster edit and change a few properties
1 2 3 4 5 6 | vim $KAFKA_CONFIG/server.properties broker.id (this should be a unique integer for each broker in the broker cluster) zookeeper.connect (this should be the address of the Zk server) log.dirs |
For more configuration options and details you can checkout the configuration page
Starting Apache Kafka
Finally start (this can also be put under supervisord)
1 2 3 | $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties |
Apache Kafka Cluster Setup
There are multiple ways a kafka cluster can be setup namely
1 2 3 4 5 6 | single node - single broker. single node - multipe brokers (different server.properties files) multiple nodes - single brokers. multiple nodes - multiple brokers. |
Also remember to set the appropriate iptable rules to accept connections on port 9092.
1 2 3 4 5 | $iptables -A INPUT -i eth0 -p tcp --dport 9092 -j ACCEPT $service iptables save $service iptables restart |
Assuming that supervisord is installed on your machine, you can run kafka within supervisord as follows. First stop the supervisord service
1 2 3 4 | $service supervisord stop Stopping supervisord: [ OK ] |
Edit the supervisord config and add the following startup command for kafka
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | $vim /etc/supervisord.conf [program:kafka] command=/opt/kafka-0.8.0/bin/kafka-server-start.sh /opt/kafka-0.8.0/config/server.properties autostart=true autorestart=true startsecs=1 startretries=999 redirect_stderr=false stdout_logfile=/var/log/kafka-out stdout_logfile_maxbytes=10MB stdout_logfile_backups=10 stdout_events_enabled=true stderr_logfile=/var/log/kafka-err stderr_logfile_maxbytes=100MB stderr_logfile_backups=10 stderr_events_enabled=true |
then
1 2 3 4 5 | $chkconfig supervisord on $service supervisord start Starting supervisord: [ OK ] |
Check whether kafka has started successfully by using
1 2 3 4 | $supervisorctl status kafka_supervisor RUNNING pid 2476, uptime 0:03:01 |
NOTE: You might get some intermediary messages while kafka is coming up such as, this could mean there is some issue, you should check the supervisor logs
1 2 3 4 5 6 | $supervisorctl status kafka STARTING $supervisorctl status kafka BACKOFF Bad exit code 1 |