前往小程序，Get更优阅读体验！
立即前往
文档建议反馈控制台
首页
学习
活动
专区
工具
TVP
最新优惠活动
文章/答案/技术大牛
发布
首页
学习
活动
专区
工具
TVP最新优惠活动
返回腾讯云官网
社区首页 >专栏 >ElasticSearch 6.x 学习笔记：4.IK分词器插件

ElasticSearch 6.x 学习笔记：4.IK分词器插件

作者头像
拓荒者
发布于 2019-03-11 16:06:56
1.1K0
发布于 2019-03-11 16:06:56
举报
文章被收录于专栏：运维经验分享运维经验分享

4.1 elasticsearch-analysis-ik 6.1.1

（1）源码  https://github.com/medcl/elasticsearch-analysis-ik

这里写图片描述
这里写图片描述

（2）releases  https://github.com/medcl/elasticsearch-analysis-ik/releases

这里写图片描述
这里写图片描述

（3）复制zip地址  https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.1.1/elasticsearch-analysis-ik-6.1.1.zip

4.2 安装插件

（1）elasticsearch-plugin

代码语言：javascript
复制
[es@node1 elasticsearch-6.1.1]$ bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.1.1/elasticsearch-analysis-ik-6.1.1.zip
-> Downloading https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.1.1/elasticsearch-analysis-ik-6.1.1.zip
[=================================================] 100%   
-> Installed analysis-ik
[es@node1 elasticsearch-6.1.1]$ ll plugins/
total 0
drwxr-xr-x 2 es es 199 Jan  7 08:52 analysis-ik
[es@node1 elasticsearch-6.1.1]$

（2）查看目录

代码语言：javascript
复制
[es@node1 elasticsearch-6.1.1]$ ll plugins/analysis-ik/
total 1420
-rw-r--r-- 1 es es 263965 Jan  7 08:52 commons-codec-1.9.jar
-rw-r--r-- 1 es es  61829 Jan  7 08:52 commons-logging-1.2.jar
-rw-r--r-- 1 es es  51658 Jan  7 08:52 elasticsearch-analysis-ik-6.1.1.jar
-rw-r--r-- 1 es es 736658 Jan  7 08:52 httpclient-4.5.2.jar
-rw-r--r-- 1 es es 326724 Jan  7 08:52 httpcore-4.4.4.jar
-rw-r--r-- 1 es es   2666 Jan  7 08:52 plugin-descriptor.properties
[es@node1 elasticsearch-6.1.1]$

4.3 重启elasticsearch

代码语言：javascript
复制
[es@node1 elasticsearch-6.1.1]$ bin/elasticsearch
[2018-01-07T09:01:17,283][INFO ][o.e.n.Node               ] [] initializing ...
[2018-01-07T09:01:17,421][INFO ][o.e.e.NodeEnvironment    ] [cNWkQjt] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [14.3gb], net total_space [21.9gb], types [rootfs]
[2018-01-07T09:01:17,422][INFO ][o.e.e.NodeEnvironment    ] [cNWkQjt] heap size [1007.3mb], compressed ordinary object pointers [true]
[2018-01-07T09:01:17,484][INFO ][o.e.n.Node               ] node name [cNWkQjt] derived from node ID [cNWkQjt9SzKFNtyx8IIu-A]; set [node.name] to override
[2018-01-07T09:01:17,484][INFO ][o.e.n.Node               ] version[6.1.1], pid[3445], build[bd92e7f/2017-12-17T20:23:25.338Z], OS[Linux/3.10.0-514.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_112/25.112-b15]
[2018-01-07T09:01:17,485][INFO ][o.e.n.Node               ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/opt/elasticsearch-6.1.1, -Des.path.conf=/opt/elasticsearch-6.1.1/config]
[2018-01-07T09:01:19,000][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [aggs-matrix-stats]
[2018-01-07T09:01:19,000][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [analysis-common]
[2018-01-07T09:01:19,000][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [ingest-common]
[2018-01-07T09:01:19,001][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [lang-expression]
[2018-01-07T09:01:19,001][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [lang-mustache]
[2018-01-07T09:01:19,001][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [lang-painless]
[2018-01-07T09:01:19,001][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [mapper-extras]
[2018-01-07T09:01:19,001][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [parent-join]
[2018-01-07T09:01:19,002][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [percolator]
[2018-01-07T09:01:19,002][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [reindex]
[2018-01-07T09:01:19,002][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [repository-url]
[2018-01-07T09:01:19,002][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [transport-netty4]
[2018-01-07T09:01:19,002][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [tribe]
[2018-01-07T09:01:19,003][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded plugin [analysis-ik]
[2018-01-07T09:01:21,678][INFO ][o.e.d.DiscoveryModule    ] [cNWkQjt] using discovery type [zen]
[2018-01-07T09:01:22,567][INFO ][o.e.n.Node               ] initialized
[2018-01-07T09:01:22,568][INFO ][o.e.n.Node               ] [cNWkQjt] starting ...
[2018-01-07T09:01:22,803][INFO ][o.e.t.TransportService   ] [cNWkQjt] publish_address {192.168.80.131:9300}, bound_addresses {192.168.80.131:9300}
[2018-01-07T09:01:22,837][INFO ][o.e.b.BootstrapChecks    ] [cNWkQjt] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2018-01-07T09:01:25,940][INFO ][o.e.c.s.MasterService    ] [cNWkQjt] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {cNWkQjt}{cNWkQjt9SzKFNtyx8IIu-A}{Xvho5gpPTuavakz227C_uA}{192.168.80.131}{192.168.80.131:9300}
[2018-01-07T09:01:25,949][INFO ][o.e.c.s.ClusterApplierService] [cNWkQjt] new_master {cNWkQjt}{cNWkQjt9SzKFNtyx8IIu-A}{Xvho5gpPTuavakz227C_uA}{192.168.80.131}{192.168.80.131:9300}, reason: apply cluster state (from master [master {cNWkQjt}{cNWkQjt9SzKFNtyx8IIu-A}{Xvho5gpPTuavakz227C_uA}{192.168.80.131}{192.168.80.131:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-01-07T09:01:25,993][INFO ][o.e.h.n.Netty4HttpServerTransport] [cNWkQjt] publish_address {192.168.80.131:9200}, bound_addresses {192.168.80.131:9200}
[2018-01-07T09:01:25,993][INFO ][o.e.n.Node               ] [cNWkQjt] started
[2018-01-07T09:01:26,077][INFO ][o.w.a.d.Monitor          ] try load config from /opt/elasticsearch-6.1.1/config/analysis-ik/IKAnalyzer.cfg.xml
[2018-01-07T09:01:26,799][INFO ][o.e.g.GatewayService     ] [cNWkQjt] recovered [2] indices into cluster_state
[2018-01-07T09:01:27,526][INFO ][o.e.c.r.a.AllocationService] [cNWkQjt] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[test][2], [test][0]] ...]).

4.3 测试IK中文分词器的基本功能

（1）ik_smart  其中pretty本意”漂亮的”，表示以美观的形式打印出JSON格式响应。

代码语言：javascript
复制
GET _analyze?pretty
{
  "analyzer": "ik_smart",
  "text":"安徽省长江流域"
}

分词结果

代码语言：javascript
复制
{
  "tokens": [
    {
      "token": "安徽省",
      "start_offset": 0,
      "end_offset": 3,
      "type": "CN_WORD",
      "position": 0
    },
    {
      "token": "长江流域",
      "start_offset": 3,
      "end_offset": 7,
      "type": "CN_WORD",
      "position": 1
    }
  ]
}

这里写图片描述
这里写图片描述

（2）ik_max_word

代码语言：javascript
复制
GET _analyze?pretty
{
  "analyzer": "ik_max_word",
  "text":"安徽省长江流域"
}

分词结果

代码语言：javascript
复制
{
  "tokens": [
    {
      "token": "安徽省",
      "start_offset": 0,
      "end_offset": 3,
      "type": "CN_WORD",
      "position": 0
    },
    {
      "token": "安徽",
      "start_offset": 0,
      "end_offset": 2,
      "type": "CN_WORD",
      "position": 1
    },
    {
      "token": "省长",
      "start_offset": 2,
      "end_offset": 4,
      "type": "CN_WORD",
      "position": 2
    },
    {
      "token": "长江流域",
      "start_offset": 3,
      "end_offset": 7,
      "type": "CN_WORD",
      "position": 3
    },
    {
      "token": "长江",
      "start_offset": 3,
      "end_offset": 5,
      "type": "CN_WORD",
      "position": 4
    },
    {
      "token": "江流",
      "start_offset": 4,
      "end_offset": 6,
      "type": "CN_WORD",
      "position": 5
    },
    {
      "token": "流域",
      "start_offset": 5,
      "end_offset": 7,
      "type": "CN_WORD",
      "position": 6
    }
  ]
}

这里写图片描述
这里写图片描述

（3）新词

代码语言：javascript
复制
GET _analyze?pretty
{
  "analyzer": "ik_smart",
  "text": "王者荣耀"
}

分词结果

代码语言：javascript
复制
{
  "tokens": [
    {
      "token": "王者",
      "start_offset": 0,
      "end_offset": 2,
      "type": "CN_WORD",
      "position": 0
    },
    {
      "token": "荣耀",
      "start_offset": 2,
      "end_offset": 4,
      "type": "CN_WORD",
      "position": 1
    }
  ]
}

这里写图片描述
这里写图片描述

4.4 扩展字典

（1）查看已有词典

代码语言：javascript
复制
[es@node1 analysis-ik]$ pwd
/opt/elasticsearch-6.1.1/config/analysis-ik
[es@node1 analysis-ik]$ ll
total 8260
-rw-rw---- 1 es bigdata 5225922 Jan  7 08:52 extra_main.dic
-rw-rw---- 1 es bigdata   63188 Jan  7 08:52 extra_single_word.dic
-rw-rw---- 1 es bigdata   63188 Jan  7 08:52 extra_single_word_full.dic
-rw-rw---- 1 es bigdata   10855 Jan  7 08:52 extra_single_word_low_freq.dic
-rw-rw---- 1 es bigdata     156 Jan  7 08:52 extra_stopword.dic
-rw-rw---- 1 es bigdata     625 Jan  7 08:52 IKAnalyzer.cfg.xml
-rw-rw---- 1 es bigdata 3058510 Jan  7 08:52 main.dic
-rw-rw---- 1 es bigdata     123 Jan  7 08:52 preposition.dic
-rw-rw---- 1 es bigdata    1824 Jan  7 08:52 quantifier.dic
-rw-rw---- 1 es bigdata     164 Jan  7 08:52 stopword.dic
-rw-rw---- 1 es bigdata     192 Jan  7 08:52 suffix.dic
-rw-rw---- 1 es bigdata     752 Jan  7 08:52 surname.dic
[es@node1 analysis-ik]$

（2）自定义词典

代码语言：javascript
复制
[es@node1 analysis-ik]$ mkdir custom
[es@node1 analysis-ik]$ vi custom/new_word.dic
[es@node1 analysis-ik]$ cat custom/new_word.dic 
老铁
王者荣耀
洪荒之力
共有产权房
一带一路
[es@node1 analysis-ik]$

（3）更新配置

代码语言：javascript
复制
[es@node1 analysis-ik]$ vi IKAnalyzer.cfg.xml 
[es@node1 analysis-ik]$ cat IKAnalyzer.cfg.xml 
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
    <comment>IK Analyzer 扩展配置</comment>
    <!--用户可以在这里配置自己的扩展字典 -->
    <entry key="ext_dict">custom/new_word.dic</entry>
     <!--用户可以在这里配置自己的扩展停止词字典-->
    <entry key="ext_stopwords"></entry>
    <!--用户可以在这里配置远程扩展字典 -->
    <!-- <entry key="remote_ext_dict">words_location</entry> -->
    <!--用户可以在这里配置远程扩展停止词字典-->
    <!-- <entry key="remote_ext_stopwords">words_location</entry> -->
</properties>
[es@node1 analysis-ik]$

（4）重启elasticsearch

代码语言：javascript
复制
[es@node1 elasticsearch-6.1.1]$ bin/elasticsearch
[2018-01-07T10:00:23,032][INFO ][o.e.n.Node               ] [] initializing ...
[2018-01-07T10:00:23,170][INFO ][o.e.e.NodeEnvironment    ] [cNWkQjt] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [14.3gb], net total_space [21.9gb], types [rootfs]
[2018-01-07T10:00:23,171][INFO ][o.e.e.NodeEnvironment    ] [cNWkQjt] heap size [1007.3mb], compressed ordinary object pointers [true]
[2018-01-07T10:00:23,209][INFO ][o.e.n.Node               ] node name [cNWkQjt] derived from node ID [cNWkQjt9SzKFNtyx8IIu-A]; set [node.name] to override
[2018-01-07T10:00:23,210][INFO ][o.e.n.Node               ] version[6.1.1], pid[3574], build[bd92e7f/2017-12-17T20:23:25.338Z], OS[Linux/3.10.0-514.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_112/25.112-b15]
[2018-01-07T10:00:23,210][INFO ][o.e.n.Node               ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/opt/elasticsearch-6.1.1, -Des.path.conf=/opt/elasticsearch-6.1.1/config]
[2018-01-07T10:00:24,717][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [aggs-matrix-stats]
[2018-01-07T10:00:24,717][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [analysis-common]
[2018-01-07T10:00:24,718][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [ingest-common]
[2018-01-07T10:00:24,718][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [lang-expression]
[2018-01-07T10:00:24,718][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [lang-mustache]
[2018-01-07T10:00:24,718][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [lang-painless]
[2018-01-07T10:00:24,718][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [mapper-extras]
[2018-01-07T10:00:24,719][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [parent-join]
[2018-01-07T10:00:24,719][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [percolator]
[2018-01-07T10:00:24,719][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [reindex]
[2018-01-07T10:00:24,719][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [repository-url]
[2018-01-07T10:00:24,719][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [transport-netty4]
[2018-01-07T10:00:24,720][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded module [tribe]
[2018-01-07T10:00:24,720][INFO ][o.e.p.PluginsService     ] [cNWkQjt] loaded plugin [analysis-ik]
[2018-01-07T10:00:27,866][INFO ][o.e.d.DiscoveryModule    ] [cNWkQjt] using discovery type [zen]
[2018-01-07T10:00:28,794][INFO ][o.e.n.Node               ] initialized
[2018-01-07T10:00:28,795][INFO ][o.e.n.Node               ] [cNWkQjt] starting ...
[2018-01-07T10:00:29,047][INFO ][o.e.t.TransportService   ] [cNWkQjt] publish_address {192.168.80.131:9300}, bound_addresses {192.168.80.131:9300}
[2018-01-07T10:00:29,093][INFO ][o.e.b.BootstrapChecks    ] [cNWkQjt] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2018-01-07T10:00:32,210][INFO ][o.e.c.s.MasterService    ] [cNWkQjt] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {cNWkQjt}{cNWkQjt9SzKFNtyx8IIu-A}{N6t0NiDmQp2vlrbx-FtcUQ}{192.168.80.131}{192.168.80.131:9300}
[2018-01-07T10:00:32,217][INFO ][o.e.c.s.ClusterApplierService] [cNWkQjt] new_master {cNWkQjt}{cNWkQjt9SzKFNtyx8IIu-A}{N6t0NiDmQp2vlrbx-FtcUQ}{192.168.80.131}{192.168.80.131:9300}, reason: apply cluster state (from master [master {cNWkQjt}{cNWkQjt9SzKFNtyx8IIu-A}{N6t0NiDmQp2vlrbx-FtcUQ}{192.168.80.131}{192.168.80.131:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-01-07T10:00:32,285][INFO ][o.e.h.n.Netty4HttpServerTransport] [cNWkQjt] publish_address {192.168.80.131:9200}, bound_addresses {192.168.80.131:9200}
[2018-01-07T10:00:32,286][INFO ][o.e.n.Node               ] [cNWkQjt] started
[2018-01-07T10:00:32,326][INFO ][o.w.a.d.Monitor          ] try load config from /opt/elasticsearch-6.1.1/config/analysis-ik/IKAnalyzer.cfg.xml
[2018-01-07T10:00:32,905][INFO ][o.w.a.d.Monitor          ] [Dict Loading] custom/new_word.dic
[2018-01-07T10:00:33,279][INFO ][o.e.g.GatewayService     ] [cNWkQjt] recovered [2] indices into cluster_state
[2018-01-07T10:00:34,092][INFO ][o.e.c.r.a.AllocationService] [cNWkQjt] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[test][3]] ...]).

从输出信息中可以看到

代码语言：javascript
复制
[Dict Loading] custom/new_word.dic

说明自定义词典已经加载了。

（5）重启Kibana  重启Kibana后，从新执行下面命令

代码语言：javascript
复制
GET _analyze?pretty
{
  "analyzer": "ik_smart",
  "text":"王者荣耀"
}

分词结果

代码语言：javascript
复制
{
  "tokens": [
    {
      "token": "王者荣耀",
      "start_offset": 0,
      "end_offset": 4,
      "type": "CN_WORD",
      "position": 0
    }
  ]
}
这里写图片描述
这里写图片描述
本文参与 腾讯云自媒体分享计划，分享自作者个人站点/博客。
原始发表：2018/03/15 ，如有侵权请联系 cloudcommunity@tencent.com 删除
Elasticsearch Service
https
github
网络安全
git

本文分享自 作者个人站点/博客 前往查看

如有侵权，请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体分享计划  ，欢迎热爱写作的你一起参与！

Elasticsearch Service
https
github
网络安全
git
评论
登录后参与评论
0 条评论
热度
最新
登录 后参与评论
推荐阅读
LV.
文章
0
获赞
0
目录
  • 4.1 elasticsearch-analysis-ik 6.1.1
  • 4.2 安装插件
  • 4.3 重启elasticsearch
  • 4.3 测试IK中文分词器的基本功能
  • 4.4 扩展字典
相关产品与服务
Elasticsearch Service
腾讯云 Elasticsearch Service（ES）是云端全托管海量数据检索分析服务，拥有高性能自研内核，集成X-Pack。ES 支持通过自治索引、存算分离、集群巡检等特性轻松管理集群，也支持免运维、自动弹性、按需使用的 Serverless 模式。使用 ES 您可以高效构建信息检索、日志分析、运维监控等服务，它独特的向量检索还可助您构建基于语义、图像的AI深度应用。
免费体验产品介绍产品文档
ES特惠专场，首月1折秒杀，新客首购6折起！
领券

腾讯云开发者

扫码关注腾讯云开发者

扫码关注腾讯云开发者

领取腾讯云代金券

热门产品

热门推荐

更多推荐

Copyright © 2013 - 2024 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有 

深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059 深公网安备号 44030502008569

腾讯云计算（北京）有限责任公司 京ICP证150476号 |  京ICP备11018762号 | 京公网安备号11010802020287

问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档

Copyright © 2013 - 2024 Tencent Cloud.

All Rights Reserved. 腾讯云 版权所有

登录 后参与评论