kafka connect应用之：文本数据的接入接出

source数据源配置：

vagrant@vagrant-ubuntu-trusty-64:/mnt/etc$ cat connect-file-source.properties

# Licensed to the Apache Software Foundation (ASF) under one or more

# contributor license agreements. See the NOTICE file distributed with

# this work for additional information regarding copyright ownership.

# The ASF licenses this file to You under the Apache License, Version 2.0

# (the "License"); you may not use this file except in compliance with

# the License. You may obtain a copy of the License at

#

# http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

name=local-file-source

connector.class=FileStreamSource

tasks.max=1

file=test.txt

topic=file_connect_test

sink到文件的配置：

vagrant@vagrant-ubuntu-trusty-64:/mnt/etc$ cat connect-file-sink.properties

# Licensed to the Apache Software Foundation (ASF) under one or more

# contributor license agreements. See the NOTICE file distributed with

# this work for additional information regarding copyright ownership.

# The ASF licenses this file to You under the Apache License, Version 2.0

# (the "License"); you may not use this file except in compliance with

# the License. You may obtain a copy of the License at

#

# http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

name=local-file-sink

connector.class=FileStreamSink

tasks.max=1

topics=file_connect_test

connect-standalone ：加载配置文件

vagrant@vagrant-ubuntu-trusty-64:~$ cat conn_file.sh

nohup connect-standalone /mnt/etc/connect-avro-standalone.properties /mnt/etc/connect-file-source.properties /mnt/etc/connect-file-sink.properties > conn_file.log 2>&1 &

tail -f conn_file.log

验证：kafka消费：

vagrant@vagrant-ubuntu-trusty-64:~$ kafka-console-consumer --zookeeper localhost:2181 --topic file_connect_test --from-beginning

dadada

log line 1

log line 2

log line 3

log line 1

log line 2

log line 3

0log line 1 for nico demo

0log line 2 for nico demo

验证：sink到文件：

log line 91 for nico demo

log line 92 for nico demo

log line 93 for nico demo

log line 94 for nico demo

log line 95 for nico demo

log line 96 for nico demo

log line 97 for nico demo

log line 98 for nico demo

log line 99 for nico demo

注意事项：

properties文件中的 topic 要一致：source 文件和 sink文件要对应。

properties文件中，如果不指定绝对路径，默认的根目录是 ~/ , 本文中是

vagrant@vagrant-ubuntu-trusty-64:~$ cd

vagrant@vagrant-ubuntu-trusty-64:~$ pwd

/home/vagrant

vagrant@vagrant-ubuntu-trusty-64:~$

参考资料：

[1] https://gerardnico.com/dit/kafka/connector/file-source

[2] https://docs.confluent.io/current/connect/connect-filestream/filestream_connector.html