Installing Apache Kafka on Centos 6.5

kafka freedom
Standard

How to install Apache Kafka locally or VMWare or Amazon EC2

This is a guide to installing Apache Kafka locally or VMWare or Amazon EC2 instance. Apache Kafka installation is very straight forward. This installation is based on building kafka from source (there might be a better way of installing kafka, through an existing rpm maybe)

Requirements

  • java 1.7 (assumed installed)
  • centos 6.5 (assumed installed)
  • zookeeper-3.4.5 (assumed installed)
  • supervisord (assumed installed)
  • kafka 0.8.1.1 (http://www.mirrorservice.org/sites/ftp.apache.org/kafka)
  • sbt

Apache Kafka Source

Get the latest version of Apache Kafka from from the mirrorservice

Building Apache Kafka Source

Kafka is written in scala so we need sbt (this is the scala equivalent of maven).

We now build and package kafka from source using sbt

Environment Variables

Now update your bash profile with some kafka specific environment variables so that these are available to us on restarts or logins

Apache Zookeeper Configuration

Only perform this statement if we are running zookeeper locally, for connecting to Zk cluster DO NOT perform this line, but change config in the next line

For connecting to Zk cluster edit and change a few properties

For more configuration options and details you can checkout the configuration page

Starting Apache Kafka

Finally start (this can also be put under supervisord)

Apache Kafka Cluster Setup

There are multiple ways a kafka cluster can be setup namely

Also remember to set the appropriate iptable rules to accept connections on port 9092.

Assuming that supervisord is installed on your machine, you can run kafka within supervisord as follows. First stop the supervisord service

Edit the supervisord config and add the following startup command for kafka

then

Check whether kafka has started successfully by using

NOTE: You might get some intermediary messages while kafka is coming up such as, this could mean there is some issue, you should check the supervisor logs