source数据源配置:
vagrant@vagrant-ubuntu-trusty-64:/mnt/etc$ cat connect-file-source.properties
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name=local-file-source
connector.class=FileStreamSource
tasks.max=1
file=test.txt
topic=file_connect_test
sink到文件的配置:
vagrant@vagrant-ubuntu-trusty-64:/mnt/etc$ cat connect-file-sink.properties
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name=local-file-sink
connector.class=FileStreamSink
tasks.max=1
topics=file_connect_test
connect-standalone :加载配置文件
vagrant@vagrant-ubuntu-trusty-64:~$ cat conn_file.sh
nohup connect-standalone /mnt/etc/connect-avro-standalone.properties /mnt/etc/connect-file-source.properties /mnt/etc/connect-file-sink.properties > conn_file.log 2>&1 &
tail -f conn_file.log
验证:kafka消费:
vagrant@vagrant-ubuntu-trusty-64:~$ kafka-console-consumer --zookeeper localhost:2181 --topic file_connect_test --from-beginning
dadada
log line 1
log line 2
log line 3
log line 1
log line 2
log line 3
0log line 1 for nico demo
0log line 2 for nico demo
验证:sink到文件:
log line 91 for nico demo
log line 92 for nico demo
log line 93 for nico demo
log line 94 for nico demo
log line 95 for nico demo
log line 96 for nico demo
log line 97 for nico demo
log line 98 for nico demo
log line 99 for nico demo
注意事项:
properties文件中的 topic 要一致:source 文件和 sink文件要对应。
properties文件中,如果不指定绝对路径,默认的根目录是 ~/ , 本文中是
vagrant@vagrant-ubuntu-trusty-64:~$ cd
vagrant@vagrant-ubuntu-trusty-64:~$ pwd
/home/vagrant
vagrant@vagrant-ubuntu-trusty-64:~$
参考资料:
[1] https://gerardnico.com/dit/kafka/connector/file-source
[2] https://docs.confluent.io/current/connect/connect-filestream/filestream_connector.html
领取专属 10元无门槛券
私享最新 技术干货