Home | 簡體中文 | 繁體中文 | 雜文 | 打賞(Donations) | ITEYE 博客 | OSChina 博客 | Facebook | Linkedin | 知乎專欄 | Search | About

33.5. Apache Kafka is a distributed publish-subscribe messaging system

http://kafka.apache.org/

33.5.1. 安裝 Kafka用於開發與測試環境

如果你是開發或測試環境使用,可以使用內置 zookeeper

cd /usr/local/src
wget http://apache.communilink.net/kafka/0.10.2.0/kafka_2.12-0.10.2.0.tgz
tar zxvf kafka_2.12-0.10.2.0.tgz
mv kafka_2.12-0.10.2.0 /srv/
cp /srv/kafka_2.12-0.10.2.0/config/server.properties{,.original}
echo "advertised.host.name=localhost" >> /srv/kafka_2.12-0.10.2.0/config/server.properties
ln -s /srv/kafka_2.12-0.10.2.0 /srv/kafka
/srv/kafka/bin/zookeeper-server-start.sh config/zookeeper.properties
/srv/kafka/bin/kafka-server-start.sh /srv/kafka/config/server.properties		
		

啟動 Kafka 服務

/srv/kafka/bin/zookeeper-server-start.sh -daemon config/zookeeper.properties
/srv/kafka/bin/kafka-server-start.sh -daemon /srv/kafka/config/server.properties
		

-daemon 表示守護進程方式在後台啟動

停止 Kafka 服務

/srv/kafka/bin/kafka-server-stop.sh
/srv/kafka/bin/zookeeper-server-stop.sh
		

33.5.2. 安裝 Kafka 適用於 IDC

如果是生產環境安裝腳本如下,獨立安裝zookeeper.

#!/bin/bash

cd /usr/local/src
wget http://apache.communilink.net/zookeeper/zookeeper-3.4.9/zookeeper-3.4.9.tar.gz
tar zxvf zookeeper-3.4.9.tar.gz
cp zookeeper-3.4.9/conf/zoo_sample.cfg zookeeper-3.4.9/conf/zoo.cfg
vim zookeeper-3.4.9/conf/zoo.cfg
mv zookeeper-3.4.9 /srv/
ln -s /srv/zookeeper-3.4.9 /srv/zookeeper
#cd zookeeper-3.4.9
/srv/zookeeper/bin/zkServer.sh start

cd /usr/local/src
wget http://apache.communilink.net/kafka/0.10.2.0/kafka_2.12-0.10.2.0.tgz
tar zxvf kafka_2.12-0.10.2.0.tgz
mv kafka_2.12-0.10.2.0 /srv/
cp /srv/kafka_2.12-0.10.2.0/config/server.properties{,.original}
echo "advertised.host.name=localhost" >> /srv/kafka_2.12-0.10.2.0/config/server.properties
ln -s /srv/kafka_2.12-0.10.2.0 /srv/kafka
/srv/kafka/bin/kafka-server-start.sh /srv/kafka/config/server.properties
		

啟動 zookeeper

$ /srv/zookeeper/bin/zkServer.sh start		
		

停止 zookeeper

$ /srv/zookeeper/bin/zkServer.sh stop
ZooKeeper JMX enabled by default
Using config: /srv/zookeeper/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED			
		

33.5.3. Kafka 日誌

查看 server 日誌

tailf /srv/kafka/logs/server.log
		

33.5.4. 測試 Kafka

$ cd /srv/kafka		
		

創建Topic

$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test	
Created topic "test".
		

查看Topic

$ bin/kafka-topics.sh --list --zookeeper localhost:2181
test
		

啟動Producer 生產消息

$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
This is a message
This is another message
		

啟動Consumer 消費消息

$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
This is a message
This is another message
		

33.5.5. 配置 Kafka

33.5.5.1. 外網訪問

預設 kafka對localhost提供訪問,如果開放外面的IP進來你需要配置 config/server.properties

listeners = PLAINTEXT://147.189.135.55:9092
			

以及

advertised.host.name=147.189.135.55
			

33.5.5.2. group.id

查看 group.id 配置

# cat config/consumer.properties  | grep "group\.id"
group.id=test-consumer-group
			

33.5.6. 管理 Kafka

進入控制台

bin/zookeeper-shell.sh localhost:2181
		

刪除Topic

$ /srv/kafka/bin/kafka-run-class.sh kafka.admin.TopicCommand --delete --topic kafkatopic --zookeeper localhost:2181
		

查看Topic 的 offset

$ /srv/kafka/bin/kafka-consumer-offset-checker.sh  --zookeeper localhost:2181 --topic kafkatopic --group consumer	
		

33.5.7. FAQ

33.5.7.1. WARN Error while fetching metadata with correlation id 1 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

解決方法

			
echo "advertised.host.name=localhost" >> /srv/kafka/config/server.properties