## 什么是“Entropy 和信息增益”？内容来源于 Stack Overflow，并遵循CC BY-SA 3.0许可协议进行翻译与使用

• 回答 (6)
• 关注 (0)
• 查看 (164)

Entropy 每个标签的概率之和乘以相同标签的对数概率吗?

### 6 个回答

ends-vowel
[9m,5f]          <--- the [..,..] notation represents the class
/          \            distribution of instances that reached a node
=1          =0
-------     -------
[3m,4f]     [6m,1f]

Entropy_before = - (5/14)*log2(5/14) - (9/14)*log2(9/14) = 0.9403

Entropy_left = - (3/7)*log2(3/7) - (4/7)*log2(4/7) = 0.9852

Entropy_right = - (6/7)*log2(6/7) - (1/7)*log2(1/7) = 0.5917

Entropy_after = 7/14*Entropy_left + 7/14*Entropy_right = 0.7885

Information_Gain = Entropy_before - Entropy_after = 0.1518

、对于文本挖掘分类，可以采取如下步骤：预处理(标记化、蒸化、信息增益特征选择...)。转换为数字(频率或TF-ID)(我认为这是将文本作为只接受数字的算法的输入时理解的关键步骤)，然后用MaxEnt进行分类，确定这只是一个例子。

//Loop over image array elements and count occurrences of each possible
//pixel to pixel difference value. Store these values in prob_array
for j = 0, ysize-1 do \$
for i = 0, xsize-2 do begin
diff = array(i+1,j) - array(i,j)
if diff lt (array_size+1)/2 and diff gt -(array_size+1)/2 then begin
prob_array(diff+(array_size-1)/2) = prob_array(diff+(array_size-1)/2) + 1
endif
endfor

//Convert values in prob_array to probabilities and compute entropy
n = total(prob_array)

entrop = 0
for i = 0, array_size-1 do begin
prob_array(i) = prob_array(i)/n

//Base 2 log of x is Ln(x)/Ln(2). Take Ln of array element
//here and divide final sum by Ln(2)
if prob_array(i) ne 0 then begin
entrop = entrop - prob_array(i)*alog(prob_array(i))
endif
endfor

entrop = entrop/alog(2)

-log pi

π越小，这个值就越大。如果xi变为不可能的两倍，则此值将增加一个固定数量(log(2))。这应该会提醒您在消息中添加一点内容。

I = -Σ pi log(pi)

Red bulb burnt out: pred = 0, pgreen=1, I = -(0 + 0)  = 0
Red and green equiprobable: pred = 1/2, pgreen = 1/2, I = -(2 * 1/2 * log(1/2)) = log(2)
Three colors, equiprobable: pi=1/3, I = -(3 * 1/3 * log(1/3)) = log(3)
Green and red, green twice as likely: pred=1/3, pgreen=2/3, I = -(1/3 log(1/3) + 2/3 log(2/3)) = log(3) - 2/3 log(2)

# 我们如何测量信息?

• 如果事件的概率为1(可预测的)，则该函数给出0
• 如果事件的概率接近于0，则函数应该给出较高的数目。
• 如果发生0.5次事件的概率one bit关于信息的。

I(X) = -log_2(p)