cloudera 第四天 Flume

Last login: Thu Dec 27 05:55:50 2018 from 192.168.6.1

Creating an empty configuration

For the sake of this tutorial, you won't need to actually execute steps 1 or 2, as we have included the configuration and the schema file in your cluster already. They can be reviewed by exploring /opt/examples/flume/solr_configs.

If you were doing this on your own, you would generate the configs by executing the following command:
#为本教程创建一个空配置，您不需要实际执行步骤1或2，
#因为我们已经在集群中包含了配置和模式文件。
#可以通过探索/opt/example /flume/solr_configs来查看它们。
#如果您独自完成这项工作，您可以通过执行以下命令来生成配置:

[root@quickstart ~]# solrctl --zk quickstart:2181/solr instancedir --generate solr_configs

You don't need to do this for this tutorial. We have already generated the configuration for you. This instruction is here in case you want to create your own index.

The result of this command would be a skeleton configuration that you could then customize to your liking. The primary thing that you would ordinarily be customizing is the conf/schema.xml, which we cover in the next step.
Edit your schema

As mentioned previously, we have already generated the configuration files for you. You can view the modified sample schema here.

The most common area that you would be interested in is the <fields></fields> section. From this area you can define the fields that are present and searchable in your index.
Uploading your configuration

#本教程不需要这样做。我们已经为您生成了配置。
#如果您希望创建自己的索引，可以在这里使用此指令。
#这个命令的结果将是一个框架配置，然后您可以根据自己的喜好进行定制。
#您通常要定制的主要内容是conf/schema。xml，我们将在下一个步骤中介绍。
#编辑你的模式
#如前所述，我们已经为您生成了配置文件。
#您可以在这里查看修改后的示例模式。
#你最感兴趣的领域是什么

[root@quickstart ~]# cd /opt/examples/flume
[root@quickstart flume]#  solrctl --zk quickstart:2181/solr instancedir --create live_logs ./solr_configs
Uploading configs from ./solr_configs/conf to quickstart:2181/solr. This may take up to a minute.


Creating your collection
#创建你的集合

[root@quickstart flume]# solrctl --zk quickstart:2181/solr collection --create live_logs -s 1

You can verify that you successfully created your collection in Solr by going to Hue, and clicking Search in the top menu
#您可以通过转到Hue并单击顶部菜单中的Search来验证是否成功地在Solr中创建了集合
[root@quickstart flume]# 

#您的Cloudera Live集群有一个日志生成器，用于示例数据。
#通过运行以下命令启动日志生成器:

[root@quickstart ~]# start_logs
[root@quickstart ~]# tail_logs
195.205.97.205 - - [03/Jan/2019:08:07:23 -0800] "GET /departments HTTP/1.1" 200 1272 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"
211.12.138.198 - - [03/Jan/2019:08:07:24 -0800] "GET /departments HTTP/1.1" 200 2194 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0"
100.27.86.204 - - [03/Jan/2019:08:07:16 -0800] "GET /categories/mlb%20players/products HTTP/1.1" 200 2214 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"
85.39.147.56 - - [03/Jan/2019:08:07:17 -0800] "GET /departments HTTP/1.1" 200 324 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.77.4 (KHTML, like Gecko) Version/7.0.5 Safari/537.77.4"
110.91.75.175 - - [03/Jan/2019:08:07:18 -0800] "GET /product/1145 HTTP/1.1" 200 196 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.76.4 (KHTML, like Gecko) Version/7.0.4 Safari/537.76.4"
218.75.175.52 - - [03/Jan/2019:08:07:19 -0800] "GET /departments HTTP/1.1" 200 429 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0"
140.117.68.194 - - [03/Jan/2019:08:07:20 -0800] "GET /department/outdoors/categories HTTP/1.1" 200 1500 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"
145.216.153.36 - - [03/Jan/2019:08:07:21 -0800] "GET /product/1243 HTTP/1.1" 200 264 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"
10.179.9.112 - - [03/Jan/2019:08:07:22 -0800] "GET /departments HTTP/1.1" 200 1779 "-" "Mozilla/5.0 (Windows NT 6.1; rv:30.0) Gecko/20100101 Firefox/30.0"
163.133.60.127 - - [03/Jan/2019:08:07:23 -0800] "GET /department/golf/products HTTP/1.1" 200 1640 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"
188.9.126.123 - - [03/Jan/2019:08:07:24 -0800] "GET /department/golf/categories HTTP/1.1" 200 1881 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"
85.39.147.56 - - [03/Jan/2019:08:07:25 -0800] "GET /departments HTTP/1.1" 200 1451 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.77.4 (KHTML, like Gecko) Version/7.0.5 Safari/537.77.4"
14.31.248.176 - - [03/Jan/2019:08:07:26 -0800] "GET /department/footwear/categories HTTP/1.1" 200 689 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
173.236.144.147 - - [03/Jan/2019:08:07:27 -0800] "GET /departments HTTP/1.1" 200 2115 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.77.4 (KHTML, like Gecko) Version/7.0.5 Safari/537.77.4"
108.81.243.150 - - [03/Jan/2019:08:07:28 -0800] "GET /departments HTTP/1.1" 200 1399 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
^Z
[1]+  Stopped                 tail_logs
[root@quickstart ~]# stop_logs

执行以下命令启动水槽代理:
[root@quickstart ~]$ flume-ng agent \
    --conf /opt/examples/flume/conf \
    --conf-file /opt/examples/flume/conf/flume.conf \
    --name agent1 \
    -Dflume.root.logger=DEBUG,INFO,console