blocks|key|676985|text|通常，两个文档之间的余弦相似度被用作文档的相似性度量。在Java语言中，您可以使用Lucene+(如果您的集合非常大)或LingPipe来完成此任务。基本概念是计算每个文档中的术语，并计算术语向量的点积。与这种通用方法相比，这些库确实提供了几个改进，例如使用反向文档频率和计算tf-idf向量。如果你想做一些copmlex的事情，LingPipe还提供了计算文档之间的LSA相似度的方法，这比余弦相似度提供了更好的结果。对于Python，您可以使用NLTK。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|676986|entityMap|0|LINK|mutability|MUTABLE|url|http://lucene.apache.org/java/docs/index.html|1|http://alias-i.com/lingpipe/|2|http://www.nltk.org/^0|15|6|0|1O|8|1|68|4|2|0^^$0|@$1|2|3|4|5|6|7|P|8|@]|9|@$A|Q|B|R|1|S]|$A|T|B|U|1|V]|$A|W|B|X|1|Y]]|C|$]]|$1|D|3|-4|5|6|7|Z|8|@]|9|@]|C|$]]]|E|$F|$5|G|H|I|C|$J|K]]|L|$5|G|H|I|C|$J|M]]|N|$5|G|H|I|C|$J|O]]]]

Generally a cosine similarity between two documents is used as a similarity measure of documents. In Java, you can use <a href="http://lucene.apache.org/java/docs/index.html">Lucene</a> (if your collection is pretty large) or <a href="http://alias-i.com/lingpipe/">LingPipe</a> to do this. The basic concept would be to count the terms in every document and calculate the dot product of the term vectors. The libraries do provide several improvements over this general approach, e.g. using inverse document frequencies and calculating tf-idf vectors. If you are looking to do something copmlex, LingPipe also provides methods to calculate LSA similarity between documents which gives better results than cosine similarity. 
For Python, you can use <a href="http://www.nltk.org/">NLTK</a>.

blocks|key|677043|text|这里有一个小应用程序可以让你入门。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|677044|import+difflib+as+dl

a+=+file('file').read()
b+=+file('file1').read()

sim+=+dl.get_close_matches

s+=+0
wa+=+a.split()
wb+=+b.split()

for+i+in+wa:
++++if+sim(i,+wb):
++++++++s+%2B=+1

n+=+float(s)+/+float(len(wa))
print+'%25d%25%25+similarity'+%25+int(n+*+100)|code-block|syntax|javascript|677045|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Here's a little app to get you started...

<pre><code>import difflib as dl

a = file('file').read()
b = file('file1').read()

sim = dl.get_close_matches

s = 0
wa = a.split()
wb = b.split()

for i in wa:
 if sim(i, wb):
 s += 1

n = float(s) / float(len(wa))
print '%d%% similarity' % int(n * 100)
</code></pre>

blocks|key|461950|text|您可能想要尝试这个余弦文档相似度http://www.scurtu.it/documentSimilarity.html的在线服务|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|461951|import+urllib,urllib2
import+json
API_URL="http://www.scurtu.it/apis/documentSimilarity"
inputDict={}
inputDict['doc1']='Document+with+some+text'
inputDict['doc2']='Other+document+with+some+text'
params+=+urllib.urlencode(inputDict)++++
f+=+urllib2.urlopen(API_URL,+params)
response=+f.read()
responseObject=json.loads(response)++
print+responseObject|code-block|syntax|javascript|461952|entityMap|0|LINK|mutability|MUTABLE|url|http://www.scurtu.it/documentSimilarity.html^0|G|18|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@$A|R|B|S|1|T]]|C|$]]|$1|D|3|E|5|F|7|U|8|@]|9|@]|C|$G|H]]|$1|I|3|-4|5|6|7|V|8|@]|9|@]|C|$]]]|J|$K|$5|L|M|N|C|$O|P]]]]

You might want to try this online service for cosine document similarity <a href="http://www.scurtu.it/documentSimilarity.html" rel="nofollow">http://www.scurtu.it/documentSimilarity.html</a>

<pre><code>import urllib,urllib2
import json
API_URL="http://www.scurtu.it/apis/documentSimilarity"
inputDict={}
inputDict['doc1']='Document with some text'
inputDict['doc2']='Other document with some text'
params = urllib.urlencode(inputDict) 
f = urllib2.urlopen(API_URL, params)
response= f.read()
responseObject=json.loads(response) 
print responseObject
</code></pre>

blocks|key|677098|text|与@larsman相同，但经过了一些预处理|type|unstyled|depth|inlineStyleRanges|entityRanges|data|677099|import+nltk,+string
from+sklearn.feature_extraction.text+import+TfidfVectorizer

nltk.download('punkt')+#+if+necessary...


stemmer+=+nltk.stem.porter.PorterStemmer()
remove_punctuation_map+=+dict((ord(char),+None)+for+char+in+string.punctuation)

def+stem_tokens(tokens):
++++return+[stemmer.stem(item)+for+item+in+tokens]

'''remove+punctuation,+lowercase,+stem'''
def+normalize(text):
++++return+stem_tokens(nltk.word_tokenize(text.lower().translate(remove_punctuation_map)))

vectorizer+=+TfidfVectorizer(tokenizer=normalize,+stop_words='english')

def+cosine_sim(text1,+text2):
++++tfidf+=+vectorizer.fit_transform([text1,+text2])
++++return+((tfidf+*+tfidf.T).A)[0,1]


print+cosine_sim('a+little+bird',+'a+little+bird')
print+cosine_sim('a+little+bird',+'a+little+bird+chirps')
print+cosine_sim('a+little+bird',+'a+big+dog+barks')|code-block|syntax|javascript|677100|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Identical to @larsman, but with some preprocessing

<pre class="lang-py prettyprint-override"><code>import nltk, string
from sklearn.feature_extraction.text import TfidfVectorizer

nltk.download('punkt') # if necessary...


stemmer = nltk.stem.porter.PorterStemmer()
remove_punctuation_map = dict((ord(char), None) for char in string.punctuation)

def stem_tokens(tokens):
 return [stemmer.stem(item) for item in tokens]

'''remove punctuation, lowercase, stem'''
def normalize(text):
 return stem_tokens(nltk.word_tokenize(text.lower().translate(remove_punctuation_map)))

vectorizer = TfidfVectorizer(tokenizer=normalize, stop_words='english')

def cosine_sim(text1, text2):
 tfidf = vectorizer.fit_transform([text1, text2])
 return ((tfidf * tfidf.T).A)[0,1]


print cosine_sim('a little bird', 'a little bird')
print cosine_sim('a little bird', 'a little bird chirps')
print cosine_sim('a little bird', 'a big dog barks')
</code></pre>

blocks|key|677145|text|这是一个老问题，但我发现使用Spacy可以很容易地做到这一点。一旦读取了文档，就可以使用简单的similarity接口来查找文档向量之间的余弦相似度。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|677146|首先安装软件包并下载模型：|677147|pip+install+spacy
python+-m+spacy+download+en_core_web_sm|code-block|syntax|javascript|677148|然后像这样使用：|677149|import+spacy
nlp+=+spacy.load('en_core_web_sm')
doc1+=+nlp(u'Hello+hi+there!')
doc2+=+nlp(u'Hello+hi+there!')
doc3+=+nlp(u'Hey+whatsup?')

print+(doc1.similarity(doc2))+#+0.999999954642
print+(doc2.similarity(doc3))+#+0.699032527716
print+(doc1.similarity(doc3))+#+0.699032527716|677150|entityMap|0|LINK|mutability|MUTABLE|url|https://spacy.io/usage/vectors-similarity^0|1B|A|E|5|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|Y|8|@$9|Z|A|10|B|C]]|D|@$9|11|A|12|1|13]]|E|$]]|$1|F|3|G|5|6|7|14|8|@]|D|@]|E|$]]|$1|H|3|I|5|J|7|15|8|@]|D|@]|E|$K|L]]|$1|M|3|N|5|6|7|16|8|@]|D|@]|E|$]]|$1|O|3|P|5|J|7|17|8|@]|D|@]|E|$K|L]]|$1|Q|3|-4|5|6|7|18|8|@]|D|@]|E|$]]]|R|$S|$5|T|U|V|E|$W|X]]]]

It's an old question, but I found this can be done easily with <a href="https://spacy.io/usage/vectors-similarity" rel="noreferrer">Spacy</a>. Once the document is read, a simple api <code>similarity</code> can be used to find the cosine similarity between the document vectors.
Start by installing the package and downloading the model:
<pre><code>pip install spacy
python -m spacy download en_core_web_sm
</code></pre>
Then use like so:
<pre class="lang-py prettyprint-override"><code>import spacy
nlp = spacy.load('en_core_web_sm')
doc1 = nlp(u'Hello hi there!')
doc2 = nlp(u'Hello hi there!')
doc3 = nlp(u'Hey whatsup?')

print (doc1.similarity(doc2)) # 0.999999954642
print (doc2.similarity(doc3)) # 0.699032527716
print (doc1.similarity(doc3)) # 0.699032527716
</code></pre>

blocks|key|462084|text|如果你对测量两段文本的语义相似性更感兴趣，我建议你看看this+gitlab+project。你可以将它作为服务器运行，还有一个预建的模型，你可以很容易地使用它来测量两段文本的相似度；即使它主要是为了测量两个句子的相似度而训练的，你仍然可以在你的case.It中使用它，但是你可以将它作为RESTful服务运行。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|462085|另一个选择也是DKPro+Similarity，这是一个具有各种算法的库，用于测量文本的相似度。然而，它也是用java编写的。|462086|代码示例：|462087|//+this+similarity+measure+is+defined+in+the+dkpro.similarity.algorithms.lexical-asl+package
//+you+need+to+add+that+to+your+.pom+to+make+that+example+work
//+there+are+some+examples+that+should+work+out+of+the+box+in+dkpro.similarity.example-gpl+
TextSimilarityMeasure+measure+=+new+WordNGramJaccardMeasure(3);++++//+Use+word+trigrams

String[]+tokens1+=+"This+is+a+short+example+text+.".split("+");+++
String[]+tokens2+=+"A+short+example+text+could+look+like+that+.".split("+");

double+score+=+measure.getSimilarity(tokens1,+tokens2);

System.out.println("Similarity:+"+%2B+score);|code-block|syntax|javascript|462088|entityMap|0|LINK|mutability|MUTABLE|url|https://gitlab.com/mysilver/semantic-text-similarity|1|https://dkpro.github.io/dkpro-similarity/^0|R|J|0|0|7|G|1|0|0|0^^$0|@$1|2|3|4|5|6|7|W|8|@]|9|@$A|X|B|Y|1|Z]]|C|$]]|$1|D|3|E|5|6|7|10|8|@]|9|@$A|11|B|12|1|13]]|C|$]]|$1|F|3|G|5|6|7|14|8|@]|9|@]|C|$]]|$1|H|3|I|5|J|7|15|8|@]|9|@]|C|$K|L]]|$1|M|3|-4|5|6|7|16|8|@]|9|@]|C|$]]]|N|$O|$5|P|Q|R|C|$S|T]]|U|$5|P|Q|R|C|$S|V]]]]

If you are more interested in measuring semantic similarity of two pieces of text, I suggest take a look at <a href="https://gitlab.com/mysilver/semantic-text-similarity" rel="nofollow noreferrer">this gitlab project</a>. You can run it as a server, there is also a pre-built model which you can use easily to measure the similarity of two pieces of text; even though it is mostly trained for measuring the similarity of two sentences, you can still use it in your case.It is written in java but you can run it as a RESTful service. 

Another option also is <a href="https://dkpro.github.io/dkpro-similarity/" rel="nofollow noreferrer">DKPro Similarity</a> which is a library with various algorithm to measure the similarity of texts. However, it is also written in java. 

code example:

<pre><code>// this similarity measure is defined in the dkpro.similarity.algorithms.lexical-asl package
// you need to add that to your .pom to make that example work
// there are some examples that should work out of the box in dkpro.similarity.example-gpl 
TextSimilarityMeasure measure = new WordNGramJaccardMeasure(3); // Use word trigrams

String[] tokens1 = "This is a short example text .".split(" "); 
String[] tokens2 = "A short example text could look like that .".split(" ");

double score = measure.getSimilarity(tokens1, tokens2);

System.out.println("Similarity: " + score);
</code></pre>

blocks|key|677252|text|为了用非常少的数据集找到句子相似度并获得高精度，你可以使用下面的python包，它使用预训练的BERT模型，|type|unstyled|depth|inlineStyleRanges|entityRanges|data|677253|pip+install+similar-sentences|code-block|syntax|javascript|677254|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

To find sentence similarity with very less dataset and to get high accuracy you can use below python package which is using pre-trained BERT models,

<pre><code>pip install similar-sentences
</code></pre>

blocks|key|677312|text|我结合了@FredFoo和@Renaud的答案。我的解决方案是能够在@FredFoo的文本语料库上应用@Renaud的预处理，然后在相似度大于0的地方显示成对的相似度。我首先通过安装python和pip在Windows上运行了这段代码。pip是作为python的一部分安装的，但您可能必须显式地重新运行安装包，选择修改，然后选择pip。我使用命令行执行保存在文件"similarity.py“中的python代码。我必须执行以下命令：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|677313|>set+PYTHONPATH=%25PYTHONPATH%25;C:\_location_of_python_lib_
>python+-m+pip+install+sklearn
>python+-m+pip+install+nltk
>py+similarity.py|code-block|syntax|javascript|677314|similarity.py的代码如下：|677315|from+sklearn.feature_extraction.text+import+TfidfVectorizer
import+nltk,+string
import+numpy+as+np
nltk.download('punkt')+#+if+necessary...

stemmer+=+nltk.stem.porter.PorterStemmer()
remove_punctuation_map+=+dict((ord(char),+None)+for+char+in+string.punctuation)

def+stem_tokens(tokens):
++++return+[stemmer.stem(item)+for+item+in+tokens]

def+normalize(text):
++++return+stem_tokens(nltk.word_tokenize(text.lower().translate(remove_punctuation_map)))

corpus+=+["I'd+like+an+apple",+
+++++++++++"An+apple+a+day+keeps+the+doctor+away",+
+++++++++++"Never+compare+an+apple+to+an+orange",+
+++++++++++"I+prefer+scikit-learn+to+Orange",+
+++++++++++"The+scikit-learn+docs+are+Orange+and+Blue"]++

vect+=+TfidfVectorizer(tokenizer=normalize,+stop_words='english')
tfidf+=+vect.fit_transform(corpus)+++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
pairwise_similarity+=+tfidf+*+tfidf.T

#view+the+pairwise+similarities+
print(pairwise_similarity)

#check+how+a+string+is+normalized
print(normalize("The+scikit-learn+docs+are+Orange+and+Blue"))|677316|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|N|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|O|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

I am combining the solutions from answers of @FredFoo and @Renaud. My solution is able to apply @Renaud's preprocessing on the text corpus of @FredFoo and then display pairwise similarities where the similarity is greater than 0. I ran this code on Windows by installing python and pip first. pip is installed as part of python but you may have to explicitly do it by re-running the installation package, choosing modify and then choosing pip. I use the command line to execute my python code saved in a file &quot;similarity.py&quot;. I had to execute the following commands:
<pre><code>&gt;set PYTHONPATH=%PYTHONPATH%;C:\_location_of_python_lib_
&gt;python -m pip install sklearn
&gt;python -m pip install nltk
&gt;py similarity.py
</code></pre>
The code for similarity.py is as follows:
<pre><code>from sklearn.feature_extraction.text import TfidfVectorizer
import nltk, string
import numpy as np
nltk.download('punkt') # if necessary...

stemmer = nltk.stem.porter.PorterStemmer()
remove_punctuation_map = dict((ord(char), None) for char in string.punctuation)

def stem_tokens(tokens):
 return [stemmer.stem(item) for item in tokens]

def normalize(text):
 return stem_tokens(nltk.word_tokenize(text.lower().translate(remove_punctuation_map)))

corpus = [&quot;I'd like an apple&quot;, 
 &quot;An apple a day keeps the doctor away&quot;, 
 &quot;Never compare an apple to an orange&quot;, 
 &quot;I prefer scikit-learn to Orange&quot;, 
 &quot;The scikit-learn docs are Orange and Blue&quot;] 

vect = TfidfVectorizer(tokenizer=normalize, stop_words='english')
tfidf = vect.fit_transform(corpus) 
 
pairwise_similarity = tfidf * tfidf.T

#view the pairwise similarities 
print(pairwise_similarity)

#check how a string is normalized
print(normalize(&quot;The scikit-learn docs are Orange and Blue&quot;))
</code></pre>

I am looking at working on an NLP project, in any programming language (though Python will be my preference).

I want to take two documents and determine how similar they are.

How to compute the similarity between two text documents?

服务器

Windows

Python

Java

我正在考虑在一个NLP项目上工作，使用任何编程语言(尽管Python将是我的首选)。我想要获取两个文档，并确定它们有多相似。

问如何计算两个文本文档之间的相似度？
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何计算两个文本文档之间的相似度？EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何计算两个文本文档之间的相似度？
EN