33.5. Apache Kafka is a distributed publish-subscribe messaging system

33.5. Apache Kafka is a distributed publish-subscribe messaging system
上一頁	第 33 章 Message Queuing & RPC	下一頁

http://kafka.apache.org/

33.5.1. 安裝 Kafka用於開發與測試環境

如果你是開發或測試環境使用，可以使用內置 zookeeper

cd /usr/local/src
wget http://apache.communilink.net/kafka/0.10.2.0/kafka_2.12-0.10.2.0.tgz
tar zxvf kafka_2.12-0.10.2.0.tgz
mv kafka_2.12-0.10.2.0 /srv/
cp /srv/kafka_2.12-0.10.2.0/config/server.properties{,.original}
echo "advertised.host.name=localhost" >> /srv/kafka_2.12-0.10.2.0/config/server.properties
ln -s /srv/kafka_2.12-0.10.2.0 /srv/kafka
/srv/kafka/bin/zookeeper-server-start.sh config/zookeeper.properties
/srv/kafka/bin/kafka-server-start.sh /srv/kafka/config/server.properties

啟動 Kafka 服務

/srv/kafka/bin/zookeeper-server-start.sh -daemon config/zookeeper.properties
/srv/kafka/bin/kafka-server-start.sh -daemon /srv/kafka/config/server.properties

-daemon 表示守護進程方式在後台啟動

停止 Kafka 服務

/srv/kafka/bin/kafka-server-stop.sh
/srv/kafka/bin/zookeeper-server-stop.sh

33.5.2. 安裝 Kafka 適用於 IDC

如果是生產環境安裝腳本如下，獨立安裝zookeeper.

#!/bin/bash

cd /usr/local/src
wget http://apache.communilink.net/zookeeper/zookeeper-3.4.9/zookeeper-3.4.9.tar.gz
tar zxvf zookeeper-3.4.9.tar.gz
cp zookeeper-3.4.9/conf/zoo_sample.cfg zookeeper-3.4.9/conf/zoo.cfg
vim zookeeper-3.4.9/conf/zoo.cfg
mv zookeeper-3.4.9 /srv/
ln -s /srv/zookeeper-3.4.9 /srv/zookeeper
#cd zookeeper-3.4.9
/srv/zookeeper/bin/zkServer.sh start

cd /usr/local/src
wget http://apache.communilink.net/kafka/0.10.2.0/kafka_2.12-0.10.2.0.tgz
tar zxvf kafka_2.12-0.10.2.0.tgz
mv kafka_2.12-0.10.2.0 /srv/
cp /srv/kafka_2.12-0.10.2.0/config/server.properties{,.original}
echo "advertised.host.name=localhost" >> /srv/kafka_2.12-0.10.2.0/config/server.properties
ln -s /srv/kafka_2.12-0.10.2.0 /srv/kafka
/srv/kafka/bin/kafka-server-start.sh /srv/kafka/config/server.properties

啟動 zookeeper

$ /srv/zookeeper/bin/zkServer.sh start

停止 zookeeper

$ /srv/zookeeper/bin/zkServer.sh stop
ZooKeeper JMX enabled by default
Using config: /srv/zookeeper/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED

33.5.3. Kafka 日誌

查看 server 日誌

tailf /srv/kafka/logs/server.log

33.5.4. 測試 Kafka

$ cd /srv/kafka

創建Topic

$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test	
Created topic "test".

查看Topic

$ bin/kafka-topics.sh --list --zookeeper localhost:2181
test

啟動Producer 生產消息

$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
This is a message
This is another message

啟動Consumer 消費消息

$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
This is a message
This is another message

33.5.5. 配置 Kafka

33.5.5.1. 外網訪問

預設 kafka對localhost提供訪問，如果開放外面的IP進來你需要配置 config/server.properties

listeners = PLAINTEXT://147.189.135.55:9092

以及

advertised.host.name=147.189.135.55

33.5.5.2. group.id

查看 group.id 配置

# cat config/consumer.properties  | grep "group\.id"
group.id=test-consumer-group

33.5.6. 管理 Kafka

進入控制台

bin/zookeeper-shell.sh localhost:2181

刪除Topic

$ /srv/kafka/bin/kafka-run-class.sh kafka.admin.TopicCommand --delete --topic kafkatopic --zookeeper localhost:2181

查看Topic 的 offset

$ /srv/kafka/bin/kafka-consumer-offset-checker.sh  --zookeeper localhost:2181 --topic kafkatopic --group consumer

33.5.7. FAQ

33.5.7.1. WARN Error while fetching metadata with correlation id 1 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

解決方法

			
echo "advertised.host.name=localhost" >> /srv/kafka/config/server.properties

上一頁	上一級	下一頁
33.4. Gearman	起始頁	33.6. Celery