Rootop 服务器运维与web架构

fluentd发送日志到kafka失败 Failed to connect to localhost:9092: Connection refused

# td-agent配置的broker地址

[root@abc td-agent]# cat /etc/td-agent/td-agent.conf
略
# 指定kafka地址及端口
brokers 192.168.10.74:9092
略

原本以为这样配置后,fluentd接收到的日志会直接发送到kafka。
(手动启动生产者,消费者,插入内容发现是可以消费的。)

# 但是看fluentd日志发现问题:

[root@abc td-agent]# tail -f /var/log/td-agent/td-agent.log
2019-08-21 14:30:05 +0800 [info]: #0 Fetching cluster metadata from kafka://192.168.10.74:9092
2019-08-21 14:30:05 +0800 [info]: #0 Discovered cluster metadata; nodes: localhost:9092 (node_id=0)

第一步从指定的地址获取集群metadata信息,第二步查找节点,发现地址是localhost:9092,一直没找到这个localhost地址是从哪里配置的。
然后fluentd发送日志到这个地址总是失败。
# 看下昨天排查问题时的日志:

2019-08-20 19:35:31 +0000 [info]: #0 New topics added to target list: test
2019-08-20 19:35:31 +0000 [info]: #0 Fetching cluster metadata from kafka://192.168.10.74:9092
2019-08-20 19:35:31 +0000 [info]: #0 Discovered cluster metadata; nodes: localhost:9092 (node_id=0)
2019-08-20 19:35:31 +0000 [info]: #0 Sending 7 messages to localhost:9092 (node_id=0)
2019-08-20 19:35:31 +0000 [error]: #0 [produce] Failed to connect to localhost:9092: Connection refused - connect(2) for [::1]:9092
2019-08-20 19:35:31 +0000 [error]: #0 Could not connect to broker localhost:9092 (node_id=0): Connection refused - connect(2) for [::1]:9092
2019-08-20 19:35:31 +0000 [warn]: #0 Failed to send all messages; attempting retry 1 of 2 after 1s
2019-08-20 19:35:31.833125548 +0000 fluent.info: {"message":"New topics added to target list: test"}
2019-08-20 19:35:31.833243919 +0000 fluent.info: {"message":"Fetching cluster metadata from kafka://192.168.10.74:9092"}
2019-08-20 19:35:31.836307791 +0000 fluent.info: {"message":"Discovered cluster metadata; nodes: localhost:9092 (node_id=0)"}
2019-08-20 19:35:31.836867642 +0000 fluent.info: {"message":"Sending 7 messages to localhost:9092 (node_id=0)"}

# 觉得这个地址还是需要在kafka配置文件配置,终于找到一个参数。

[root@abc config]# vi /usr/local/kafka/config/server.properties
listeners=PLAINTEXT://192.168.10.74:9092

保存退出。

去zookeeper命令行里删除一个key。

[root@abc bin]# ./zookeeper-shell.sh localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is disabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
get /brokers/ids/0 # 查看 
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://192.168.10.74:9092"],"jmx_port":-1,"host":"192.168.10.74","timestamp":"1566369132247","port":9092,"version":4}
cZxid = 0x2c7
ctime = Wed Aug 21 14:32:12 CST 2019
mZxid = 0x2c7
mtime = Wed Aug 21 14:32:12 CST 2019
pZxid = 0x2c7
cversion = 0
dataVersion = 1
aclVersion = 0
ephemeralOwner = 0x100055b01b60002
dataLength = 196
numChildren = 0
rmr /brokers/ids/0 # 删除

(我之前endpoints和host的值都是localhost,所以要删除,让kafka重新写入)

# 获取键值
get /brokers/ids/0
# 删除键
rmr /brokers/ids/0

重启zookeeper,重启kafka。

再查看fluentd日志,这次获取的nodes地址对了。

2019-08-21 14:32:38.014038626 +0800 fluent.info: {"message":"Fetching cluster metadata from kafka://192.168.10.74:9092"}
2019-08-21 14:32:38.018237952 +0800 fluent.info: {"message":"Discovered cluster metadata; nodes: 192.168.10.74:9092 (node_id=0)"}

消息终于发送到kafka了,而且开启一个消费者也能看到内容了。

原创文章,转载请注明。本文链接地址: https://www.rootop.org/pages/4506.html

作者:Venus

专注于 服务器运维与web架构 E-mail:venus#rootop.org

评论已关闭。