flume吧 关注:1,138贴子:311
  • 1回复贴,共1

Flume监听文件到kafka,文件通道,kafka消费重复问题

只看楼主收藏回复

#me the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /usr/local/test1.txt
a1.sources.r1.fileHeader = true
a1.sources.r1.deserializer.outputCharset=UTF-8
# Describe the sink
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.brokerList = c100:9092,c200:9092,c300:9092
a1.sinks.k1.topic = test5
a1.sinks.k1.requiredAcks = 1
a1.sinks.k1.batchSize = 20
a1.sinks.k1.channel = memcnl
#a1.sinks.k1.type = avro
#a1.sinks.k1.hostname = 192.168.28.134
#a1.sinks.k1.port = 44444
# Use a channel which buffers events in memory
a1.channels.c1.type = file
a1.channels.c1.checkpointDir = /usr/local/chk
a1.channels.c1.dataDirs = /usr/local/flumeData
a1.channels.c1.checkpointOnClose = true
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
把flume停掉,重启后,数据会重复,怎么解决呢


IP属地:吉林1楼2018-10-26 12:52回复
    tail每次都重新读了吧


    IP属地:北京2楼2019-04-17 13:46
    回复