blocks|key|2408217|text|这是非常c风格的，但很有效。我的想法是跟踪每个文档的“当前”二元语法，确保每个文档只添加一次它们(cur_bigrams+=+set())，并在每个文档之后增加一个全局频率计数器(bigram_freq)+(如果它在当前文档中)。然后从bigram_freq中的信息构建一个新的数据帧，这是跨文档的全局计数器。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|2408218|bigram_freq+=+{}
for+doc+in+df["text_column"]:
++++cur_bigrams+=+set()
++++words+=+doc.split("+")
++++bigrams+=+zip(words,+words[1:])
++++for+bigram+in+bigrams:
++++++++if+bigram+not+in+cur_bigrams:+#+Add+bigram,+but+only+once/doc
++++++++++++cur_bigrams.add(bigram)
++++for+bigram+in+cur_bigrams:
++++++++if+bigram+in+bigram_freq:
++++++++++++bigram_freq[bigram]+%2B=+1
++++++++else:
++++++++++++bigram_freq[bigram]+=+1

result_df+=+pd.DataFrame(columns=["2_gram",+"count"])
row_list+=+[]
for+bigram,+freq+in+bigram_freq.items():
++++row_list.append([bigram[0]+%2B+"+"+%2B+bigram[1],+freq])
for+i+in+range(len(row_list)):
++++result_df.loc[i]+=+row_list[i]

print(result_df)|code-block|syntax|javascript|2408219|输出：|2408220|+++++++++++2_gram+count
0++++++++++a+book+++++3
1++++++++++++is+a+++++3
2+++++++++This+is+++++3
3+++++++++is+read+++++1
4+++++++++that+is+++++1
5+++++++book+that+++++1
6++++++he+doesn't+++++1
7+++++++++this+is+++++1
8++++++++book+but+++++1
9++++++++++but+he+++++1
10+++++think+this+++++1
11++doesn't+think+++++1|2408221|您可能可以使用更多的函数式样式和/或列表理解对代码进行适当的裁剪。我将把它作为练习留给读者。|2408222|entityMap^0|1D|J|2I|B|3A|B|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|S|8|@$9|T|A|U|B|C]|$9|V|A|W|B|C]|$9|X|A|Y|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|Z|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|10|8|@]|D|@]|E|$]]|$1|M|3|N|5|H|7|11|8|@]|D|@]|E|$I|J]]|$1|O|3|P|5|6|7|12|8|@]|D|@]|E|$]]|$1|Q|3|-4|5|6|7|13|8|@]|D|@]|E|$]]]|R|$]]

This is very c-style, but works. Idea is to keep track of the "current" bigrams per document, making sure that they're added only once per doc (<code>cur_bigrams = set()</code>), and after each document, increment a global freq counter (<code>bigram_freq</code>) if it was in the current document. Then build a new dataframe out of the info in <code>bigram_freq</code>, the global counter across documents.

<pre><code>bigram_freq = {}
for doc in df["text_column"]:
 cur_bigrams = set()
 words = doc.split(" ")
 bigrams = zip(words, words[1:])
 for bigram in bigrams:
 if bigram not in cur_bigrams: # Add bigram, but only once/doc
 cur_bigrams.add(bigram)
 for bigram in cur_bigrams:
 if bigram in bigram_freq:
 bigram_freq[bigram] += 1
 else:
 bigram_freq[bigram] = 1

result_df = pd.DataFrame(columns=["2_gram", "count"])
row_list = []
for bigram, freq in bigram_freq.items():
 row_list.append([bigram[0] + " " + bigram[1], freq])
for i in range(len(row_list)):
 result_df.loc[i] = row_list[i]

print(result_df)
</code></pre>

Output:

<pre><code> 2_gram count
0 a book 3
1 is a 3
2 This is 3
3 is read 1
4 that is 1
5 book that 1
6 he doesn't 1
7 this is 1
8 book but 1
9 but he 1
10 think this 1
11 doesn't think 1
</code></pre>

You can probably trim that code down a good bit using a more functional style and/or list comprehensions. I'll leave that as an exercise to the reader.

blocks|key|2619075|text|Pythonic式的回答(写成通用的，所以可以应用到文件/dataframe/任何地方)：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2619076|c=collections.Counter()
for+i+in+fh:
++x+=+i.rstrip().split("+")
++c.update(set(zip(x[:-1],x[1:])))|code-block|syntax|javascript|2619077|现在c保持了每2个字母组的频率。|offset|length|style|CODE|2619078|说明：|BOLD|2619079|2619080|将每一行按空格填充到一个列表中。|ordered-list-item|2619081|然后zip()返回长度为2(2克)的元组上的迭代器。|2619082|为了删除代码，将迭代器馈送到代码集被馈送到<+set()+>d15split+>对象中，该对象跟踪每个元组出现的次数。您需要import+collections才能使用它。|2619083|现在很容易列出计数器的内容或将其转换为您喜欢的任何其他格式(例如，import+collections)。|2619084|2619085|是的，Python很棒。|2619086|entityMap^0|0|0|2|1|0|0|3|0|0|0|2|5|0|N|5|1Q|I|0|X|I|0|0|0^^$0|@$1|2|3|4|5|6|7|14|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|15|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|16|8|@$I|17|J|18|K|L]]|9|@]|A|$]]|$1|M|3|N|5|6|7|19|8|@$I|1A|J|1B|K|O]]|9|@]|A|$]]|$1|P|3|-4|5|6|7|1C|8|@]|9|@]|A|$]]|$1|Q|3|R|5|S|7|1D|8|@]|9|@]|A|$]]|$1|T|3|U|5|S|7|1E|8|@$I|1F|J|1G|K|L]]|9|@]|A|$]]|$1|V|3|W|5|S|7|1H|8|@$I|1I|J|1J|K|L]|$I|1K|J|1L|K|L]]|9|@]|A|$]]|$1|X|3|Y|5|S|7|1M|8|@$I|1N|J|1O|K|L]]|9|@]|A|$]]|$1|Z|3|-4|5|6|7|1P|8|@]|9|@]|A|$]]|$1|10|3|11|5|6|7|1Q|8|@]|9|@]|A|$]]|$1|12|3|-4|5|6|7|1R|8|@]|9|@]|A|$]]]|13|$]]

Pythonic answer (wrote it general so can be applied to a file/dataframe/whatever):

<pre><code>c=collections.Counter()
for i in fh:
 x = i.rstrip().split(" ")
 c.update(set(zip(x[:-1],x[1:])))
</code></pre>

Now <code>c</code> holds the frequency of every 2-gram.

Explanation:

<ol>
<li>Every line is <code>split</code> by whitespace into a list.</li>
<li>Then <code>zip()</code> to return an iterator over tuples of length 2 (the 2 grams).</li>
<li>The iterator is fed into a <code>set()</code> in order to remove redundancies.</li>
<li>Then the set is fed into a <code>collections.Counter()</code> object, which keeps track of how many times each tuple appeared. You need to <code>import collections</code> to use this.</li>
<li>It is easy now to list the contents of the Counter or convert it to any other format you like (e.g. dataframe).</li>
</ol>

Yes, Python is awesome.

I have a pandas data frame and I want to do 2-gram frequency based on one text column.

<pre><code>text_column
This is a book
This is a book that is read
This is a book but he doesn't think this is a book
</code></pre>

The final result would like is a frequency count of 2 gram but the frequency is counting whether a 2 gram is in each document, not a 2-gram count.

So the partial result would be 

<pre><code>2 gram Count
This is 3
a book 3
</code></pre>

"This is" and "a book" appeared in the all 3 texts although there are 2 of each of the 3rd text, since I'm only interested how many documents did such 2 gram appeared, the counting is 3 so not 4.

Any idea how I can do that?

Thank you

Python N_gram frequency count

我有一个熊猫数据框架，我想做基于一个文本列的2-gram频率。text_columnThis is a bookThis is a book that is readThis is a book but he doesn't think this is a book最终结果可能是2克的频率计数，但频率是计算每个文档中是...

问Python N_gram频率计数
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Python N_gram频率计数EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Python N_gram频率计数
EN