In statistics , a mixture model is a probabilistic model for representing the presence of subpopulations within an overall population, without requiring that an observed data set should identify the sub-population to which an individual observation belongs
Introduce
三硬币模型
假设有 3 枚硬币，分别记作 A , B , C A,B,C A , B , C π , p , q \pi,p,q π , p , q A A A B B B C C C n n n n = 10 n=10 n = 10
1 , 1 , 0 , 1 , 0 , 0 , 1 , 0 , 1 , 1 1,1,0,1,0,0,1,0,1,1 1 , 1 , 0 , 1 , 0 , 0 , 1 , 0 , 1 , 1 假设我们只能观测到最终掷硬币的结果，不能观测中间状态。问如何估计三硬币正面出现的概率，即三硬币模型的参数。
三硬币模型可以写作
P ( y ∣ θ ) = ∑ z P ( y , z ∣ θ ) = ∑ z P ( z ∣ θ ) P ( y ∣ z , θ ) = π p y ( 1 − p ) 1 − y ⏟ P ( y ∣ z , θ ) + ( 1 − π ) ⏟ P ( z ∣ θ ) q y ( 1 − q ) 1 − y ⏟ ∑ \begin{aligned}
P(y \mid \theta) & =\sum_{z} P(y, z \mid \theta)=\sum_{z} P(z \mid \theta) P(y \mid z, \theta) \\
& =\underbrace{\pi \underbrace{p^{y}(1-p)^{1-y}}_{{\color{Green} P(y\mid z,\theta)} } +\underbrace{(1-\pi)}_{{\color{Red} P(z\mid \theta)} } q^{y}(1-q)^{1-y}}_{\sum}
\end{aligned} P ( y ∣ θ ) = z ∑ P ( y , z ∣ θ ) = z ∑ P ( z ∣ θ ) P ( y ∣ z , θ ) = ∑ π P ( y ∣ z , θ ) p y ( 1 − p ) 1 − y + P ( z ∣ θ ) ( 1 − π ) q y ( 1 − q ) 1 − y 在这里，随机变量 y y y 1 1 1 0 0 0 z z z A A A θ = ( π , p , q ) \theta=(\pi,p,q) θ = ( π , p , q )
将观测数据表示为 Y = ( Y 1 , Y 2 , … , Y n ) ⊤ Y=(Y_{1},Y_{2},\dots,Y_{n})^\top Y = ( Y 1 , Y 2 , … , Y n ) ⊤ Z = ( Z 1 , Z 2 , … , Z n ) ⊤ Z=(Z_{1},Z_{2},\dots,Z_{n})^\top Z = ( Z 1 , Z 2 , … , Z n ) ⊤
P ( Y ∣ θ ) = ∑ Z P ( Z ∣ θ ) P ( Y ∣ Z , θ ) \begin{equation}
P(Y \mid \theta)=\sum_{Z} P(Z \mid \theta) P(Y \mid Z, \theta)
\end{equation} P ( Y ∣ θ ) = Z ∑ P ( Z ∣ θ ) P ( Y ∣ Z , θ ) 即
P ( Y ∣ θ ) = ∏ j = 1 n [ π p y j ( 1 − p ) 1 − y j + ( 1 − π ) q y j ( 1 − q ) 1 − y j ] P(Y \mid \theta)=\prod_{j=1}^{n}\left[\pi p^{y_{j}}(1-p)^{1-y_{j}}+(1-\pi) q^{y_{j}}(1-q)^{1-y_{j}}\right] P ( Y ∣ θ ) = j = 1 ∏ n [ π p y j ( 1 − p ) 1 − y j + ( 1 − π ) q y j ( 1 − q ) 1 − y j ] 考虑到求模型参数 θ = ( π , p , q ) \theta=(\pi,p,q) θ = ( π , p , q )
θ ^ = arg max θ log P ( Y ∣ θ ) \begin{equation}
\hat{\theta}=\arg \max _{\theta} \log P(Y \mid \theta)
\end{equation} θ ^ = arg θ max log P ( Y ∣ θ ) 这里的三变量模型实际上就是一种混合模型，最后 y y y K K K p , q p,q p , q π \pi π mixture weights \text{mixture weights} mixture weights
更加一般的，混合模型一般有下面几个特点。
Mixture Model
观测到的 N N N K K K
有 N N N K K K
一个 K K K mixture weights \text{mixture weights} mixture weights 1 1 1
特别的，当各个子分布是高斯（正态）分布时，我们叫 Gaussian mixture model \text{Gaussian mixture model} Gaussian mixture model
p ( y ) = ∑ i = 1 K ϕ i N ( μ i , Σ i ) \begin{equation}
p(\boldsymbol{y})=\sum_{i=1}^{K} \phi_{i} \mathcal{N}\left(\boldsymbol{\mu}_{i}, \boldsymbol{\Sigma}_{i}\right)
\end{equation} p ( y ) = i = 1 ∑ K ϕ i N ( μ i , Σ i ) 其中的 ϕ i \phi_{i} ϕ i i i i ϕ i ≥ 0 , ∑ i = 1 K ϕ i = 1 \phi_{i}\ge 0, \sum_{i=1}^K \phi_{i}=1 ϕ i ≥ 0 , ∑ i = 1 K ϕ i = 1 N ( μ i , Σ i ) \mathcal{N}\left(\boldsymbol{\mu}_{i}, \boldsymbol{\Sigma}_{i}\right) N ( μ i , Σ i ) i i i
类似于上面中的隐随机变量 z z z
我们可以这样想象一个符合混合高斯模型分布的数据点的产生过程：首先以一定的概率从 K K K i i i ϕ i \phi_{i} ϕ i
为了表示选中哪一个高斯分布，我们引入一个二值指示变量 z \boldsymbol{z} z one hot \text{one hot} one hot i i i ϕ i \phi_{i} ϕ i
p ( z i = 1 ) = ϕ i p(z_{i}=1) = \phi_{i} p ( z i = 1 ) = ϕ i 因此，随机变量 z \boldsymbol{z} z
p ( z ) = ∏ i = 1 K ϕ i z i p(\boldsymbol{z}) = \prod^{K}_{i=1} \phi_{i}^{z_{i}} p ( z ) = i = 1 ∏ K ϕ i z i 所以，当已知选中第 i i i y \boldsymbol{y} y
p ( y ∣ z i = 1 ) = N ( y ∣ μ i , Σ i ) p(\boldsymbol{y}\mid z_{i}=1) = \mathcal{N}\left(\boldsymbol{y}\mid \boldsymbol{\mu}_{i}, \boldsymbol{\Sigma}_{i}\right) p ( y ∣ z i = 1 ) = N ( y ∣ μ i , Σ i ) 或者表示为
p ( y ∣ z ) = ∏ i = 1 K N ( y ∣ μ i , Σ i ) z i p(\boldsymbol{y}\mid \boldsymbol{z}) = \prod^{K}_{i=1} \mathcal{N}\left(\boldsymbol{y}\mid \boldsymbol{\mu}_{i}, \boldsymbol{\Sigma}_{i}\right) ^ {z_{i}} p ( y ∣ z ) = i = 1 ∏ K N ( y ∣ μ i , Σ i ) z i 从而，y \boldsymbol{y} y
p ( y ) = ∑ z p ( y ∣ z ) p ( z ) = ∑ z ∏ i = 1 K [ ϕ i N ( y ∣ μ i , Σ i ) ] z i \begin{equation}
p(\boldsymbol{y}) = \sum_{\boldsymbol{z}} p(\boldsymbol{y}\mid \boldsymbol{z}) p(\boldsymbol{z}) = \sum_{{\boldsymbol{z}}} \prod_{i=1}^{K}[\phi_{i}\mathcal{N}(\boldsymbol{y}\mid \boldsymbol{\mu}_{i}, \boldsymbol{\Sigma}_{i})]^{z_{i}}
\end{equation} p ( y ) = z ∑ p ( y ∣ z ) p ( z ) = z ∑ i = 1 ∏ K [ ϕ i N ( y ∣ μ i , Σ i ) ] z i 注意到 z \boldsymbol{z} z one hot \text{one hot} one hot
p ( y ) = ∑ z p ( y ∣ z ) p ( z ) = ∑ i = 1 K ϕ i N ( y ∣ μ i , Σ i ) p(\boldsymbol{y}) = \sum_{\boldsymbol{z}} p(\boldsymbol{y}\mid \boldsymbol{z}) p(\boldsymbol{z}) = \sum_{i=1}^{K}\phi_{i} \mathcal{N}(\boldsymbol{y}\mid\boldsymbol{\mu}_{i}, \boldsymbol{\Sigma}_{i}) p ( y ) = z ∑ p ( y ∣ z ) p ( z ) = i = 1 ∑ K ϕ i N ( y ∣ μ i , Σ i ) ^d1001b
相关资料
