【手把手教你全文检索】Lucene索引的【增、删、改、查】

前言   搞检索的,应该多少都会了解Lucene一些,它开源而且简单上手,官方API足够编写些小DEMO。并且根据倒排索引,实现快速检索。本文就简单的实现增量添加索引,删除索引,通过关键字查询,以及更新索引等操作。   目前博猪使用的不爽的地方就是,读取文件内容进行全文检索时,需要自己编写读取过程(这个solr免费帮我们实现)。而且创建索引的过程比较慢,还有很大的优化空间,这个就要细心下来研究了。

  创建索引

  Lucene在进行创建索引时,根据前面一篇博客,已经讲完了大体的流程,这里再简单说下:

1 Directory directory = FSDirectory.open("/tmp/testindex");
2 IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_CURRENT, analyzer);
3 IndexWriter iwriter = new IndexWriter(directory, config);
4 Document doc = new Document();
5 String text = "This is the text to be indexed.";
6 doc.add(new Field("fieldname", text, TextField.TYPE_STORED)); iwriter.close();

1 创建Directory,获取索引目录

  2 创建词法分析器,创建IndexWriter对象

  3 创建document对象,存储数据

  4 关闭IndexWriter,提交

 1 /**
 2      * 建立索引
 3      * 
 4      * @param args
 5      */
 6     public static void index() throws Exception {
 7         
 8         String text1 = "hello,man!";
 9         String text2 = "goodbye,man!";
10         String text3 = "hello,woman!";
11         String text4 = "goodbye,woman!";
12         
13         Date date1 = new Date();
14         analyzer = new StandardAnalyzer(Version.LUCENE_CURRENT);
15         directory = FSDirectory.open(new File(INDEX_DIR));
16 
17         IndexWriterConfig config = new IndexWriterConfig(
18                 Version.LUCENE_CURRENT, analyzer);
19         indexWriter = new IndexWriter(directory, config);
20 
21         Document doc1 = new Document();
22         doc1.add(new TextField("filename", "text1", Store.YES));
23         doc1.add(new TextField("content", text1, Store.YES));
24         indexWriter.addDocument(doc1);
25         
26         Document doc2 = new Document();
27         doc2.add(new TextField("filename", "text2", Store.YES));
28         doc2.add(new TextField("content", text2, Store.YES));
29         indexWriter.addDocument(doc2);
30         
31         Document doc3 = new Document();
32         doc3.add(new TextField("filename", "text3", Store.YES));
33         doc3.add(new TextField("content", text3, Store.YES));
34         indexWriter.addDocument(doc3);
35         
36         Document doc4 = new Document();
37         doc4.add(new TextField("filename", "text4", Store.YES));
38         doc4.add(new TextField("content", text4, Store.YES));
39         indexWriter.addDocument(doc4);
40         
41         indexWriter.commit();
42         indexWriter.close();
43 
44         Date date2 = new Date();
45         System.out.println("创建索引耗时:" + (date2.getTime() - date1.getTime()) + "ms\n");
46     }

  增量添加索引

  Lucene拥有增量添加索引的功能,在不会影响之前的索引情况下,添加索引,它会在何时的时机,自动合并索引文件。

 1 /**
 2      * 增加索引
 3      * 
 4      * @throws Exception
 5      */
 6     public static void insert() throws Exception {
 7         String text5 = "hello,goodbye,man,woman";
 8         Date date1 = new Date();
 9         analyzer = new StandardAnalyzer(Version.LUCENE_CURRENT);
10         directory = FSDirectory.open(new File(INDEX_DIR));
11 
12         IndexWriterConfig config = new IndexWriterConfig(
13                 Version.LUCENE_CURRENT, analyzer);
14         indexWriter = new IndexWriter(directory, config);
15 
16         Document doc1 = new Document();
17         doc1.add(new TextField("filename", "text5", Store.YES));
18         doc1.add(new TextField("content", text5, Store.YES));
19         indexWriter.addDocument(doc1);
20 
21         indexWriter.commit();
22         indexWriter.close();
23 
24         Date date2 = new Date();
25         System.out.println("增加索引耗时:" + (date2.getTime() - date1.getTime()) + "ms\n");
26     }

  删除索引

  Lucene也是通过IndexWriter调用它的delete方法,来删除索引。我们可以通过关键字,删除与这个关键字有关的所有内容。如果仅仅是想要删除一个文档,那么最好就顶一个唯一的ID域,通过这个ID域,来进行删除操作。

 1 /**
 2      * 删除索引
 3      * 
 4      * @param str 删除的关键字
 5      * @throws Exception
 6      */
 7     public static void delete(String str) throws Exception {
 8         Date date1 = new Date();
 9         analyzer = new StandardAnalyzer(Version.LUCENE_CURRENT);
10         directory = FSDirectory.open(new File(INDEX_DIR));
11 
12         IndexWriterConfig config = new IndexWriterConfig(
13                 Version.LUCENE_CURRENT, analyzer);
14         indexWriter = new IndexWriter(directory, config);
15         
16         indexWriter.deleteDocuments(new Term("filename",str));  
17         
18         indexWriter.close();
19         
20         Date date2 = new Date();
21         System.out.println("删除索引耗时:" + (date2.getTime() - date1.getTime()) + "ms\n");
22     }

  更新索引

  Lucene没有真正的更新操作,通过某个fieldname,可以更新这个域对应的索引,但是实质上,它是先删除索引,再重新建立的。

 1 /**
 2      * 更新索引
 3      * 
 4      * @throws Exception
 5      */
 6     public static void update() throws Exception {
 7         String text1 = "update,hello,man!";
 8         Date date1 = new Date();
 9          analyzer = new StandardAnalyzer(Version.LUCENE_CURRENT);
10          directory = FSDirectory.open(new File(INDEX_DIR));
11 
12          IndexWriterConfig config = new IndexWriterConfig(
13                  Version.LUCENE_CURRENT, analyzer);
14          indexWriter = new IndexWriter(directory, config);
15          
16          Document doc1 = new Document();
17         doc1.add(new TextField("filename", "text1", Store.YES));
18         doc1.add(new TextField("content", text1, Store.YES));
19         
20         indexWriter.updateDocument(new Term("filename","text1"), doc1);
21         
22          indexWriter.close();
23          
24          Date date2 = new Date();
25          System.out.println("更新索引耗时:" + (date2.getTime() - date1.getTime()) + "ms\n");
26     }

  通过索引查询关键字

  Lucene的查询方式有很多种,这里就不做详细介绍了。它会返回一个ScoreDoc的集合,类似ResultSet的集合,我们可以通过域名获取想要获取的内容。

 1 /**
 2      * 关键字查询
 3      * 
 4      * @param str
 5      * @throws Exception
 6      */
 7     public static void search(String str) throws Exception {
 8         directory = FSDirectory.open(new File(INDEX_DIR));
 9         analyzer = new StandardAnalyzer(Version.LUCENE_CURRENT);
10         DirectoryReader ireader = DirectoryReader.open(directory);
11         IndexSearcher isearcher = new IndexSearcher(ireader);
12 
13         QueryParser parser = new QueryParser(Version.LUCENE_CURRENT, "content",analyzer);
14         Query query = parser.parse(str);
15 
16         ScoreDoc[] hits = isearcher.search(query, null, 1000).scoreDocs;
17         for (int i = 0; i < hits.length; i++) {
18             Document hitDoc = isearcher.doc(hits[i].doc);
19             System.out.println(hitDoc.get("filename"));
20             System.out.println(hitDoc.get("content"));
21         }
22         ireader.close();
23         directory.close();
24     }

  全部代码

  1 package test;
  2 
  3 import java.io.File;
  4 import java.util.Date;
  5 import java.util.List;
  6 
  7 import org.apache.lucene.analysis.Analyzer;
  8 import org.apache.lucene.analysis.standard.StandardAnalyzer;
  9 import org.apache.lucene.document.Document;
 10 import org.apache.lucene.document.LongField;
 11 import org.apache.lucene.document.TextField;
 12 import org.apache.lucene.document.Field.Store;
 13 import org.apache.lucene.index.DirectoryReader;
 14 import org.apache.lucene.index.IndexWriter;
 15 import org.apache.lucene.index.IndexWriterConfig;
 16 import org.apache.lucene.index.Term;
 17 import org.apache.lucene.queryparser.classic.QueryParser;
 18 import org.apache.lucene.search.IndexSearcher;
 19 import org.apache.lucene.search.Query;
 20 import org.apache.lucene.search.ScoreDoc;
 21 import org.apache.lucene.store.Directory;
 22 import org.apache.lucene.store.FSDirectory;
 23 import org.apache.lucene.util.Version;
 24 
 25 public class TestLucene {
 26     // 保存路径
 27     private static String INDEX_DIR = "D:\\luceneIndex";
 28     private static Analyzer analyzer = null;
 29     private static Directory directory = null;
 30     private static IndexWriter indexWriter = null;
 31 
 32     public static void main(String[] args) {
 33         try {
 34 //            index();
 35             search("man");
 36 //            insert();
 37 //            delete("text5");
 38 //            update();
 39         } catch (Exception e) {
 40             e.printStackTrace();
 41         }
 42     }
 43     /**
 44      * 更新索引
 45      * 
 46      * @throws Exception
 47      */
 48     public static void update() throws Exception {
 49         String text1 = "update,hello,man!";
 50         Date date1 = new Date();
 51          analyzer = new StandardAnalyzer(Version.LUCENE_CURRENT);
 52          directory = FSDirectory.open(new File(INDEX_DIR));
 53 
 54          IndexWriterConfig config = new IndexWriterConfig(
 55                  Version.LUCENE_CURRENT, analyzer);
 56          indexWriter = new IndexWriter(directory, config);
 57          
 58          Document doc1 = new Document();
 59         doc1.add(new TextField("filename", "text1", Store.YES));
 60         doc1.add(new TextField("content", text1, Store.YES));
 61         
 62         indexWriter.updateDocument(new Term("filename","text1"), doc1);
 63         
 64          indexWriter.close();
 65          
 66          Date date2 = new Date();
 67          System.out.println("更新索引耗时:" + (date2.getTime() - date1.getTime()) + "ms\n");
 68     }
 69     /**
 70      * 删除索引
 71      * 
 72      * @param str 删除的关键字
 73      * @throws Exception
 74      */
 75     public static void delete(String str) throws Exception {
 76         Date date1 = new Date();
 77         analyzer = new StandardAnalyzer(Version.LUCENE_CURRENT);
 78         directory = FSDirectory.open(new File(INDEX_DIR));
 79 
 80         IndexWriterConfig config = new IndexWriterConfig(
 81                 Version.LUCENE_CURRENT, analyzer);
 82         indexWriter = new IndexWriter(directory, config);
 83         
 84         indexWriter.deleteDocuments(new Term("filename",str));  
 85         
 86         indexWriter.close();
 87         
 88         Date date2 = new Date();
 89         System.out.println("删除索引耗时:" + (date2.getTime() - date1.getTime()) + "ms\n");
 90     }
 91     /**
 92      * 增加索引
 93      * 
 94      * @throws Exception
 95      */
 96     public static void insert() throws Exception {
 97         String text5 = "hello,goodbye,man,woman";
 98         Date date1 = new Date();
 99         analyzer = new StandardAnalyzer(Version.LUCENE_CURRENT);
100         directory = FSDirectory.open(new File(INDEX_DIR));
101 
102         IndexWriterConfig config = new IndexWriterConfig(
103                 Version.LUCENE_CURRENT, analyzer);
104         indexWriter = new IndexWriter(directory, config);
105 
106         Document doc1 = new Document();
107         doc1.add(new TextField("filename", "text5", Store.YES));
108         doc1.add(new TextField("content", text5, Store.YES));
109         indexWriter.addDocument(doc1);
110 
111         indexWriter.commit();
112         indexWriter.close();
113 
114         Date date2 = new Date();
115         System.out.println("增加索引耗时:" + (date2.getTime() - date1.getTime()) + "ms\n");
116     }
117     /**
118      * 建立索引
119      * 
120      * @param args
121      */
122     public static void index() throws Exception {
123         
124         String text1 = "hello,man!";
125         String text2 = "goodbye,man!";
126         String text3 = "hello,woman!";
127         String text4 = "goodbye,woman!";
128         
129         Date date1 = new Date();
130         analyzer = new StandardAnalyzer(Version.LUCENE_CURRENT);
131         directory = FSDirectory.open(new File(INDEX_DIR));
132 
133         IndexWriterConfig config = new IndexWriterConfig(
134                 Version.LUCENE_CURRENT, analyzer);
135         indexWriter = new IndexWriter(directory, config);
136 
137         Document doc1 = new Document();
138         doc1.add(new TextField("filename", "text1", Store.YES));
139         doc1.add(new TextField("content", text1, Store.YES));
140         indexWriter.addDocument(doc1);
141         
142         Document doc2 = new Document();
143         doc2.add(new TextField("filename", "text2", Store.YES));
144         doc2.add(new TextField("content", text2, Store.YES));
145         indexWriter.addDocument(doc2);
146         
147         Document doc3 = new Document();
148         doc3.add(new TextField("filename", "text3", Store.YES));
149         doc3.add(new TextField("content", text3, Store.YES));
150         indexWriter.addDocument(doc3);
151         
152         Document doc4 = new Document();
153         doc4.add(new TextField("filename", "text4", Store.YES));
154         doc4.add(new TextField("content", text4, Store.YES));
155         indexWriter.addDocument(doc4);
156         
157         indexWriter.commit();
158         indexWriter.close();
159 
160         Date date2 = new Date();
161         System.out.println("创建索引耗时:" + (date2.getTime() - date1.getTime()) + "ms\n");
162     }
163 
164     /**
165      * 关键字查询
166      * 
167      * @param str
168      * @throws Exception
169      */
170     public static void search(String str) throws Exception {
171         directory = FSDirectory.open(new File(INDEX_DIR));
172         analyzer = new StandardAnalyzer(Version.LUCENE_CURRENT);
173         DirectoryReader ireader = DirectoryReader.open(directory);
174         IndexSearcher isearcher = new IndexSearcher(ireader);
175 
176         QueryParser parser = new QueryParser(Version.LUCENE_CURRENT, "content",analyzer);
177         Query query = parser.parse(str);
178 
179         ScoreDoc[] hits = isearcher.search(query, null, 1000).scoreDocs;
180         for (int i = 0; i < hits.length; i++) {
181             Document hitDoc = isearcher.doc(hits[i].doc);
182             System.out.println(hitDoc.get("filename"));
183             System.out.println(hitDoc.get("content"));
184         }
185         ireader.close();
186         directory.close();
187     }
188 }

  参考资料

  http://www.cnblogs.com/xing901022/p/3933675.html

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏武军超python专栏

2018年7月21日python中的加密和解密

·在函数调用执行过程中: 如果出现return,return中的函数执行完则本函数就运行结束,return下面的语句不会再继续执行,所以return使 用时...

29050
来自专栏小樱的经验随笔

凯撒密码加解密及破解实现原理

概念及原理 根据百度百科上的解释,凯撒密码是一种古老的加密算法。 密码的使用最早可以追溯到古罗马时期,《高卢战记》有描述恺撒曾经使用密码来传递信息,即所谓的“恺...

42660
来自专栏蘑菇先生的技术笔记

qt5中信号和槽的新语法

31470
来自专栏葬爱家族

RxJava再回首

很早前就看了RxJava,当时就觉得好牛掰,但是公司项目一直没有用起来,知识不用就会忘,前段时间突然要写RxJava,发现已经不会写了。所以今天再回头整理一下R...

22610
来自专栏cloudskyme

一步一步学lucene——(第三步:索引篇)

在前面概要的了解了lucene的内容下面就深入一下lucene的各个模块。这里我们主要深入一下lucene的索引,就是如何构建索引的过程及概念。 lucene与...

41560
来自专栏码神联盟

Shiro系列 | 《Shiro开发详细教程》第五章:Shiro编码加密

在涉及到密码存储问题上,应该加密或者生成密码摘要存储,而不是存储明文密码。为避免数据泄露对用户造成很大的损失,应该加密或者生成不可逆的摘要方式存储。

12720
来自专栏刘晓杰

observeOn()与subscribeOn()的区别

32580
来自专栏小鄧子的技术博客专栏

【译】对RxJava中.repeatWhen()和.retryWhen()操作符的思考

第一次见到.repeatWhen()和.retryWhen()这两个操作符的时候就非常困惑了。不得不说,它们绝对是“最令人困惑弹珠图”的有力角逐者。

13620
来自专栏程序员Gank

【译】对RxJava中-repeatWhen()和-retryWhen()操作符的思考

第一次见到.repeatWhen()和.retryWhen()这两个操作符的时候就非常困惑了。不得不说,它们绝对是“最令人困惑弹珠图”的有力角逐者。

29830
来自专栏编程之旅

iOS开发——32位MD5加密

在IOS开发过程中,为了保证数据的安全,我们通常要采取一些加密方法,常见的加密有Base64加密和MD5加密。Base64加密是可逆的,MD5加密目前来说一般是...

18920

扫码关注云+社区

领取腾讯云代金券