Home | 簡體中文 | 繁體中文 | 雜文 | 打賞(Donations) | OSChina 博客 | Facebook | Linkedin | 知乎專欄 | Search | About

142.5. Apache Kafka is a distributed publish-subscribe messaging system

http://kafka.apache.org/

142.5.1. 安裝 Kafka

142.5.1.1. 安裝 Kafka用於開發與測試環境

如果你是開發或測試環境使用,可以使用內置 zookeeper

			
			
cd /usr/local/src
wget http://apache.communilink.net/kafka/0.10.2.0/kafka_2.12-0.10.2.0.tgz
tar zxvf kafka_2.12-0.10.2.0.tgz
mv kafka_2.12-0.10.2.0 /srv/
cp /srv/kafka_2.12-0.10.2.0/config/server.properties{,.original}
echo "advertised.host.name=localhost" >> /srv/kafka_2.12-0.10.2.0/config/server.properties
ln -s /srv/kafka_2.12-0.10.2.0 /srv/kafka
/srv/kafka/bin/zookeeper-server-start.sh config/zookeeper.properties
/srv/kafka/bin/kafka-server-start.sh /srv/kafka/config/server.properties
					
			

啟動 Kafka 服務

			
/srv/kafka/bin/zookeeper-server-start.sh -daemon /srv/kafka/config/zookeeper.properties
/srv/kafka/bin/kafka-server-start.sh -daemon /srv/kafka/config/server.properties
			
			

-daemon 表示守護進程方式在後台啟動

停止 Kafka 服務

/srv/kafka/bin/kafka-server-stop.sh
/srv/kafka/bin/zookeeper-server-stop.sh
			

142.5.1.2. 安裝 Kafka 適用於 IDC

如果是生產環境安裝腳本如下,獨立安裝zookeeper.

			
#!/bin/bash

cd /usr/local/src
wget http://apache.communilink.net/zookeeper/zookeeper-3.4.9/zookeeper-3.4.9.tar.gz
tar zxvf zookeeper-3.4.9.tar.gz
cp zookeeper-3.4.9/conf/zoo_sample.cfg zookeeper-3.4.9/conf/zoo.cfg
vim zookeeper-3.4.9/conf/zoo.cfg
mv zookeeper-3.4.9 /srv/
ln -s /srv/zookeeper-3.4.9 /srv/zookeeper
#cd zookeeper-3.4.9
/srv/zookeeper/bin/zkServer.sh start

cd /usr/local/src
wget http://apache.communilink.net/kafka/0.10.2.0/kafka_2.12-0.10.2.0.tgz
tar zxvf kafka_2.12-0.10.2.0.tgz
mv kafka_2.12-0.10.2.0 /srv/
cp /srv/kafka_2.12-0.10.2.0/config/server.properties{,.original}
echo "advertised.host.name=localhost" >> /srv/kafka_2.12-0.10.2.0/config/server.properties
ln -s /srv/kafka_2.12-0.10.2.0 /srv/kafka
/srv/kafka/bin/kafka-server-start.sh /srv/kafka/config/server.properties
			
			

啟動 zookeeper

			
$ /srv/zookeeper/bin/zkServer.sh start
					
			

停止 zookeeper

			
$ /srv/zookeeper/bin/zkServer.sh stop
ZooKeeper JMX enabled by default
Using config: /srv/zookeeper/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
						
			

142.5.1.3. Kafka 日誌

查看 server 日誌

			
tailf /srv/kafka/logs/server.log			
			
			

142.5.1.4. 檢查 Kafka 綫程

使用 jps 命令監控 Kafka 綫程是否正確啟動。

			
root@netkiller /srv/kafka/logs % jps | grep Kafka
32246 Kafka			
			
			

142.5.2. 測試 Kafka

$ cd /srv/kafka		
		

創建Topic

$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test	
Created topic "test".
		

查看Topic

$ bin/kafka-topics.sh --list --zookeeper localhost:2181
test
		

啟動Producer 生產消息

$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
This is a message
This is another message
		

啟動Consumer 消費消息

$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
This is a message
This is another message
		

142.5.3. 配置 Kafka

142.5.3.1. 外網訪問

預設 kafka對localhost提供訪問,如果開放外面的IP進來你需要配置 config/server.properties

listeners = PLAINTEXT://147.189.135.55:9092
			

以及

advertised.host.name=147.189.135.55
			

142.5.3.2. group.id

查看 group.id 配置

# cat config/consumer.properties  | grep "group\.id"
group.id=test-consumer-group
			

142.5.4. 管理 Kafka

進入控制台

bin/zookeeper-shell.sh localhost:2181
		

刪除Topic

$ /srv/kafka/bin/kafka-run-class.sh kafka.admin.TopicCommand --delete --topic kafkatopic --zookeeper localhost:2181
		

查看Topic 的 offset

$ /srv/kafka/bin/kafka-consumer-offset-checker.sh  --zookeeper localhost:2181 --topic kafkatopic --group consumer	
		

142.5.5. FAQ

142.5.5.1. WARN Error while fetching metadata with correlation id 1 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

解決方法

			
echo "advertised.host.name=localhost" >> /srv/kafka/config/server.properties
			
			

142.5.5.2. Error while executing topic command : Replication factor: 1 larger than available brokers: 0.

			
root@VM_7_221_centos /srv/kafka % bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Error while executing topic command : Replication factor: 1 larger than available brokers: 0.
[2017-11-26 10:55:11,532] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
 (kafka.admin.TopicCommand$)			
			
			

檢查 broker.id 配置 broker.id 必須大於 0

			
root@netkiller /srv/kafka % cat config/server.properties | grep broker.id
broker.id=1			
			
			

142.5.5.3. WARN Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

Kafka 在防火牆後面,防火牆上面配置 NAT 規則映射到伺服器

			
# bind 任何IP地址
listeners=PLAINTEXT://:9092
# Wan IP 地址
advertised.host.name=223.207.161.225
			
			
[提示]提示

修改 advertised.host.name 後要刪除 /tmp/kafka-logs 中的日誌檔案,否則無論如何你你都難以配置成功

			
rm -rf /tmp/kafka-logs