机器学习课程_笔记04

广义线性模型

推导伯努利分布

P(y=1;\phi) = \phi \\ P(y;\phi) = \phi^y (1 - \phi)^{1-y} \\ = exp(log\phi^y(1-\phi)^{1-y}) \\ = exp(ylog\phi + (1-y)log(1-\phi)) \\ = exp(log \frac \phi {1-\phi} y + log(1-\phi)) \\ = b(y)exp(η^T T(y)-a(η)) \quad where \quad b(y)=1, \quad η= log \frac \phi {1-\phi}, \quad T(y)=y, \quad a(η)=-log(1-\phi)

推导高斯分布

N(\mu, \sigma^2) \quad assume \quad \sigma=1 \\ N(\mu, \sigma^2) = \frac 1 {\sqrt{2\pi}} (- \frac 1 2 (y - \mu)^2) \\ = \frac 1 {\sqrt{2\pi}} exp(- \frac 1 2 y^2)exp(\mu y - \frac 1 2 \mu^2) \\ = b(y)exp(η^T T(y)-a(η)) \quad where \quad b(y)=\frac 1 {\sqrt{2\pi}} exp(- \frac 1 2 y^2), \quad η= \mu, \quad T(y)=y, \quad a(η)=\frac 1 2 \mu^2 = \frac 1 2 η^2

指数分布族推导出一个广义线性模型

h_\Theta(X) = ExpFamily[T(y)|X; \Theta] = P(y=1|X;\Theta) = P(y=1;\phi) \\ =\phi \\ = \frac 1 {1+e^{-\mu}} \\ = \frac 1 {1+e^{-\Theta^T X}}

g(η)将自然参数η与y的期望值联系起来，这个函数被称为正则响应函数，而g^{-1}被称为正则关联函数。

多项式分布

y \in {1, \cdots, k} \\ Parameters: \quad \phi_1,\phi_2,\cdots,\phi_k \\ P(y=i) = \phi_i \\ \sum_{i=1}^k P(y=i) = 1 \\ \phi_k = 1- \sum_{i=1}^{k-1} \phi_i \\ Assume \quad Parameters: \quad \phi_1,\phi_2,\cdots,\phi_{k-1} \\ T(1) = \begin{bmatrix}1 \\ 0 \\ \vdots \\ 0 \end{bmatrix}, \quad T(k-1) = \begin{bmatrix}0 \\ 0 \\ \vdots \\ 1 \end{bmatrix}, \quad T(k) = \begin{bmatrix}0 \\ 0 \\ \vdots \\ 0 \end{bmatrix} \in \mathbb R^{k-1} \\ \mathbb{1}\{True\} =1, \mathbb{1}\{False\} =0 \\ T(y)_i = \mathbb{1}\{y=i\} \\ P(y) = \phi^{\mathbb{1}\{y=1\}}\phi^{\mathbb{1}\{y=2\}}\cdots\phi^{\mathbb{1}\{y=k\}}\\ =\phi^{T(y)_{1}}\phi^{T(y)_{2}}\cdots\phi^{T(y)_{k-1}}\phi^{1-\sum_{j=1}^{k-1}T(y)_j} \\ = b(y)exp(η^T T(y)-a(η)) \quad where \quad b(y)=1, \quad η= \begin{bmatrix} log(\frac {\phi_1} {\phi_k}) \\ \vdots \\ log(\frac {\phi_{k-1}} {\phi_k}) \end{bmatrix} \in \mathbb R^{k-1}, \quad T(y)_i = \mathbb{1}\{y=i\}, \quad a(η)=-log(\phi_k)

\phi_i= \frac {e^η_i} {1+ \sum_{j=1}^{k-1} e^{η_j}} \quad Here \quad (i=1, \cdots, k-1) \\ = \frac {e^{\Theta_i^T X}} {1+ \sum_{j=1}^{k-1} e^{\Theta_j^T X}}

h_\Theta(X) = ExpFamily[T(y)|X;\Theta] \\ = ExpFamily\left[ \begin{matrix} \mathbb 1\{j=1\} \\ \vdots \\ \mathbb 1\{j=k-1\} \end{matrix} \arrowvert X; \Theta \right] \\ = \begin{bmatrix} \phi_1 \\ \vdots \\ \phi_{k-1} \end{bmatrix} \\ = \begin{bmatrix} \frac {e^{\Theta_1^T X}} {1+ \sum_{j=1}^{k-1} e^{\Theta_j^T X}} \\ \vdots \\ \frac {e^{\Theta_{k-1}^T X}} {1+ \sum_{j=1}^{k-1} e^{\Theta_j^T X}} \end{bmatrix}

softmax回归的求解过程就可以归纳如下：

Assume \quad y \in \{1, \cdots, k\} \\ You \quad Have: \quad (X^{(1)}, y^{(1)}), \cdots, (X^{(m)}, y^{(m)}) \\ L(\Theta) = \Pi_{i=1}^m P(y^{(i)}|X^{(i)}; \Theta) \\ = \Pi_{i=1}^m \phi_1^{\mathbb 1 \{y^{(i)} = 1\}} \cdots \phi_k^{\mathbb 1 \{y^{(i)} = k\}} \quad Here \quad \begin{bmatrix} \phi_1 \\ \vdots \\ \phi_{k-1} \end{bmatrix} = \begin{bmatrix} \frac {e^{\Theta_1^T X}} {1+ \sum_{j=1}^{k-1} e^{\Theta_j^T X}} \\ \vdots \\ \frac {e^{\Theta_{k-1}^T X}} {1+ \sum_{j=1}^{k-1} e^{\Theta_j^T X}} \end{bmatrix}

188 篇文章40 人订阅

0 条评论

相关文章

37470

86970

19950

41830

SimRank协同过滤推荐算法

在协同过滤推荐算法总结中，我们讲到了用图模型做协同过滤的方法，包括SimRank系列算法和马尔科夫链系列算法。现在我们就对SimRank算法在推荐系统的...

37610

29630

19250

20090

1.6K60