How to delete a topic in Apache Kafka Message Broker 0.9.x?

Deleting a topic is relevant only in development or testing environments. DO NOT enable this setting in production.

To delete a topic (associated with a message queue in other systems). you need the following
1. The zookeeper ensemble that Kafka clusters use.
2. Enable delete of topic in the server.properties namely

delete.topic.enable=true

Refer to How to setup standalone instance of Apache Kafka 0.9.0.1 on localhost for Mac OS X? for enabling this setting.

For a kafka server cluster installation with a zookeeper ensemble. Refer How to setup a 2 Node Apache Kafka 0.9.0.1 cluster in CentOS 7?

Navigate to any node of Kafka installation instance namely

cd /usr/local/kafka/kafka_2.11-0.9.0.1

bin/kafka-topics.sh --zookeeper ZKNODE1:2181,ZKNODE2:2181,ZKNODE3:2181  --delete --topic topicName

For a local Kafka install. Refer to How to setup standalone instance of Apache Kafka 0.9.0.1 on localhost for Mac OS X?

Navigate to installation instance namely

cd /usr/local/kafka/kafka_2.11-0.9.0.1

bin/kafka-topics.sh --zookeeper yourmac.local:2181 --delete --topic topicName

Note: Make sure that you have killed all consumers before your delete the topic. Kafka would take anywhere between 2 seconds to a minute to delete a topic. When the delete command is issued it would just mark the topic for deletion.

How to read messages in a topic from Apache Kafka Message Broker 0.9.x?

Sometime we need to quickly check what messages are present in Apache Kafka topic. Apache Kafka provides a default consumer shell for reading messages off a topic. Apache Kafka does not allow you to read a message by message Id or partition key. You can only read from the beginning or from the last position that was read which is automatically maintained in Zookeeper.

The reader is like an application waiting to read messages and would read continuously as long as you don't kill the session in the console. i.e a Kafka producer can produce 100 messages in time t1, you would see all the 100 messages printed in the console, if now there are another 10 messages at time t2 as long as the consumer is running you would now see only the next 10 messages.

Kafka consumers has a concept of offset. i.e the last position of the messages that it has read. This offset is maintained in Apache Zookeeper. Since Kafka supports multiple partitions an offset is maintained for each partition.

Apache Kafka is not like most other message queue systems where a message can be read only by one consumer and the message is removed after reading. Kafka allows multiple consumers to read from the same topic. Its the responsibility of each consumer to keep track of what each has read. The default Kafka installation keeps messages for 7 Days after which they are removed from the topic.

Every time a Kafka consumer shell is invoked it maintains the offset in Zookeeper. Apache Kafka supports two types of messages, String and Binary.  The content of the message alone is printed to the console without the partition key or from which partition it was read. Its good for String message types.

To read message  topic (associated with a message queue in other systems). you need the following

1. The zookeeper ensemble that Kafka clusters use.

For a kafka server cluster installation with a zookeeper ensemble. Refer How to setup a 2 Node Apache Kafka 0.9.0.1 cluster in CentOS 7?

Navigate to any node of Kafka installation instance namely

cd /usr/local/kafka/kafka_2.11-0.9.0.1

bin/kafka-console-consumer.sh --zookeeper ZKNODE1:2181,ZKNODE2:2181,ZKNODE3:2181 --topic topicName --from-beginning

To exit reading you would need to kill the process.

For CentOS7
To stop reading press ctrl + C to exit to the shell 
Kafka would now print the number of messages it has read. It would always be 1 more than the messages that your producer has put inside the topic.

ps -aux | grep kafka
to view the consumer processes and note the process id
kill processid or force kill using kill -9 processid

For a local Kafka install. Refer to How to setup standalone instance of Apache Kafka 0.9.0.1 on localhost for Mac OS X?

Navigate to installation instance namely

cd /usr/local/kafka/kafka_2.11-0.9.0.1

bin/kafka-console-consumer.sh --zookeeper yourmac.local:2181 --topic yourTopic --from-beginning

To exit
For Mac
To stop reading press ctrl + C to exit to the shell 
Kafka would now print the number of messages it has read. It would always be 1 more than the messages that your producer has put inside the topic.

ps -a | grep kafka
to view the consumer processes and note the process id
kill processid or force kill using kill -9 processid

Note: if there are no messages, the consumer would wait do not assuming no messages are coming check your producer to verify that its correctly sending to the topic.



How to create a topic in Apache Kafka Message Broker 0.9.x?

To create a topic (associated with a message queue in other systems). you need the following
1. The zookeeper ensemble that Kafka clusters use.

For a kafka server cluster installation with a zookeeper ensemble. Refer How to setup a 2 Node Apache Kafka 0.9.0.1 cluster in CentOS 7?

Navigate to any node of Kafka installation instance namely

cd /usr/local/kafka/kafka_2.11-0.9.0.1

bin/kafka-topics.sh --create --zookeeper ZKNODE1:2181,ZKNODE2:2181,ZKNODE3:2181 --replication-factor 2 --partitions 8 --topic topicname

This assumes that you have a minimum 2 node cluster. If you setup more than 2 you can increase the replication factor correspondingly. The partition is the number of concurrent reads that you would like to perform form your application. In order to better utilize the partition you would need to understand partition key which we would cover int the later lessons.

For a local Kafka install. Refer to How to setup standalone instance of Apache Kafka 0.9.0.1 on localhost for Mac OS X?

Navigate to installation instance namely

cd /usr/local/kafka/kafka_2.11-0.9.0.1

bin/kafka-topics.sh --create --zookeeper yourmac.local:2181 --replication-factor 1 --partitions 4 --topic topicname