entityMap|0|type|LINK|mutability|MUTABLE|data|url|https://github.com/RaRe-Technologies/gensim/blob/460dc1cb9921817f71b40b412e11a6d413926472/gensim/models/phrases.py#L663|blocks|key|9cq4m|text|Gensim的Phrases类使用基于相对计数的简单统计分析&一些可调的阈值来决定应该将某些标记对(通常是单词对而不是字符对)提升为单个连接的二元语法。潜在的配对会被赋予一个“分数”，那些分数超过可配置的“阈值”的配对将被合并。|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|bcl4i|即使在其正常的领域中使用，单词，其结果往往不会令人印象深刻的人类评价-遗漏了许多我们认为合乎逻辑的组合，使其他组合，我们将不同的分组。但是，由于有大量的训练数据，并对其参数进行了一些调整，即使其不完美的组合也可能有助于下游模型表现得更好。|3aj0k|对于您的小玩具语料库，根据类的默认值，您希望看到的字符二元语法是不合格的。|8c67a|您可以在以下位置查看用于评分潜在配对的默认original_scorer()代码：|23pjt|d2mea|您可以在您的模型上为您预期的配对之一运行此代码，以查看其得分：|a2lqa|>>>+from+gensim.models.phrases+import+original_scorer
>>>+original_scorer(bigram.vocab[b'G'],+bigram.vocab[b'o'],+
++++++++++++++++++++bigram.vocab[b'G_o'],+len(bigram.vocab),+
++++++++++++++++++++bigram.min_count,+bigram.corpus_word_count)
4.666666666666666|code-block|syntax|javascript|456u2|与10.0的默认bigram.threshold相比，这个'G_o'二元语法不会被选中。|dnhvi|您可以对threshold+(或Phrases的其他参数)进行修补，以获得不同的结果，但最好的值/折衷将取决于您的最终目标。例如：|e92ig|>>>+bigram.threshold+=+4.0
>>>+list(bigram.export_phrases(documents_proc,+out_delimiter=b''))
[(b'Go',+4.666666666666666),
+(b'Go',+4.666666666666666),
+(b'od',+4.666666666666666),
+(b'ha',+7.0),
+(b'ha',+7.0)]
>>>+list(bigram[documents_proc])
[['G_o',+'d'],
+['G_o',+'o_d'],
+['h_a',+'p',+'p',+'y'],
+['h_a',+'n',+'g',+'r',+'y'],
+['p',+'y',+'p',+'i']]|aiagl|因此，使用bigram.threshold=4.0，您想要的'p_y'仍然不在那里-因为考虑到'p'和'y'的数量，它需要更低的threshold。|ao0he^0|7|7|0|0|0|L|H|0|0|3B|0|0|0|0|1|4|8|G|T|5|0|4|9|G|7|0|0|5|K|U|5|1B|3|1F|3|1S|9|0^^$0|$1|$2|3|4|5|6|$7|8]]]|9|@$A|B|C|D|2|E|F|19|G|@$H|1A|I|1B|J|K]]|L|@]|6|$]]|$A|M|C|N|2|E|F|1C|G|@]|L|@]|6|$]]|$A|O|C|P|2|E|F|1D|G|@]|L|@]|6|$]]|$A|Q|C|R|2|E|F|1E|G|@$H|1F|I|1G|J|K]]|L|@]|6|$]]|$A|S|C|8|2|E|F|1H|G|@]|L|@$H|1I|I|1J|A|1K]]|6|$]]|$A|T|C|U|2|E|F|1L|G|@]|L|@]|6|$]]|$A|V|C|W|2|X|F|1M|G|@]|L|@]|6|$Y|Z]]|$A|10|C|11|2|E|F|1N|G|@$H|1O|I|1P|J|K]|$H|1Q|I|1R|J|K]|$H|1S|I|1T|J|K]]|L|@]|6|$]]|$A|12|C|13|2|E|F|1U|G|@$H|1V|I|1W|J|K]|$H|1X|I|1Y|J|K]]|L|@]|6|$]]|$A|14|C|15|2|X|F|1Z|G|@]|L|@]|6|$Y|Z]]|$A|16|C|17|2|E|F|20|G|@$H|21|I|22|J|K]|$H|23|I|24|J|K]|$H|25|I|26|J|K]|$H|27|I|28|J|K]|$H|29|I|2A|J|K]]|L|@]|6|$]]|$A|18|C|-4|2|E|F|2B|G|@]|L|@]|6|$]]]]

Gensim's <code>Phrases</code> class uses a simple statistical analysis based on relative counts &amp; some tunable thresholds to decide some token-pairs (usually word pairs rather than character pairs) should be promoted to a single connected bigram. Potential pairings are given a 'score', and those that score over a configurable 'threshold' are combined.

Even when used in its normal domain, words, its results often won't seem impressive to human evaluation – missing many combinations we'd consider logical, making other combinations we'd group differently. But, with a lot of training data, and some tuning of its parameters, even its imperfect combinations may help downstream models perform better. 

For your tiny toy corpus, the character bigrams you'd like to see aren't qualifying, based on the class defaults.

You can view the default <code>original_scorer()</code> code for scoring potential pairings at: 

<a href="https://github.com/RaRe-Technologies/gensim/blob/460dc1cb9921817f71b40b412e11a6d413926472/gensim/models/phrases.py#L663" rel="noreferrer">https://github.com/RaRe-Technologies/gensim/blob/460dc1cb9921817f71b40b412e11a6d413926472/gensim/models/phrases.py#L663</a>

You can run this code on your model for one of your expected pairings, to see its score:

<pre><code>&gt;&gt;&gt; from gensim.models.phrases import original_scorer
&gt;&gt;&gt; original_scorer(bigram.vocab[b'G'], bigram.vocab[b'o'], 
 bigram.vocab[b'G_o'], len(bigram.vocab), 
 bigram.min_count, bigram.corpus_word_count)
4.666666666666666
</code></pre>

Compared to the default <code>bigram.threshold</code> of <code>10.0</code>, this <code>'G_o'</code> bigram won't make the cut. 

You can tinker with the <code>threshold</code> (or other parameters of <code>Phrases</code>), to get different results, but what values/tradeoffs are best will depend on your ultimate goals. For example:

<pre><code>&gt;&gt;&gt; bigram.threshold = 4.0
&gt;&gt;&gt; list(bigram.export_phrases(documents_proc, out_delimiter=b''))
[(b'Go', 4.666666666666666),
 (b'Go', 4.666666666666666),
 (b'od', 4.666666666666666),
 (b'ha', 7.0),
 (b'ha', 7.0)]
&gt;&gt;&gt; list(bigram[documents_proc])
[['G_o', 'd'],
 ['G_o', 'o_d'],
 ['h_a', 'p', 'p', 'y'],
 ['h_a', 'n', 'g', 'r', 'y'],
 ['p', 'y', 'p', 'i']]
</code></pre>

So, with <code>bigram.threshold=4.0</code>, your desired <code>'p_y'</code> is still not there - as it'd need an even lower <code>threshold</code> given the counts of <code>'p'</code> and <code>'y'</code>.

I want to get bigrams for symbols (letters of words). For example, for words "done" and "dog" I want to be able to find bigram "do". 

I tried to do it using gensim.Phrases, but it doesn`t work for me. 

Here is my code:

<pre><code>from gensim.models import Phrases

documents = ["God", "Good","happy","hangry","pypi"]
documents_proc = [list(doc) for doc in documents]

bigram = Phrases(documents_proc, min_count=1)
trigram = Phrases(bigram[documents_proc], min_count=1)

for sent in documents_proc:
 print(sent, bigram[sent])
 bigrams_ = [b for b in bigram[sent] if b.count('_') == 1]
 trigrams_ = [t for t in trigram[bigram[sent]] if t.count('_') == 2]
 print(bigrams_)
 print(trigrams_)
 print()
</code></pre>

I expected the output of
<code>['Go', 'od', 'ha', 'py']</code>, but there are nothing in the output. 
What am I doing wrong? 

Thank you.

Gensim phrases don`t find some bigrams

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

 我想得到符号(单词的字母)的二元模型。例如，对于单词"done“和"dog”，我希望能够找到二元语法"do“。 我试着用gensim.Phrases写，但对我不起作用。 下面是我的代码： from gensim.models import Phrasesdocuments = ["God", "Good","happ...

Gensim短语找不到一些二元语法-腾讯云开发者社区-腾讯云

问Gensim短语找不到一些二元语法
EN

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Gensim短语找不到一些二元语法EN

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Gensim短语找不到一些二元语法
EN