贝叶斯方法与正则项

从贝叶斯角度，正则项等价于引入参数 $w@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@的先验概率分布。常见的L1/L2正则，分别等价于引入先验信息：参数 w <script type="math/tex" id="MathJax-Element-781">w@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@符合均值为0的拉普拉斯分布/高斯分布。 <div class="toc"> <div class="toc"> <ul> <li><a href="javascript:void()" class="outlink" data="/link?url=#贝叶斯方法的参数估计" rel="nofollow" target="_blank">贝叶斯方法的参数估计</a></li> <li><a href="javascript:void()" class="outlink" data="/link?url=#后验概率的展开形式" rel="nofollow" target="_blank">后验概率的展开形式</a></li> <li><a href="javascript:void()" class="outlink" data="/link?url=#参数的先验概率与正则项" rel="nofollow" target="_blank">参数的先验概率与正则项</a></li> <li><a href="javascript:void()" class="outlink" data="/link?url=#模型举例" rel="nofollow" target="_blank">模型举例</a> <ul> <li><a href="javascript:void()" class="outlink" data="/link?url=#逻辑回归" rel="nofollow" target="_blank">逻辑回归</a></li> <li><a href="javascript:void()" class="outlink" data="/link?url=#线性回归" rel="nofollow" target="_blank">线性回归</a></li> </ul> </li> </ul> </div> </div> <h1 id="贝叶斯方法的参数估计">贝叶斯<a href="/tag/fangfa/" target="_blank" class="keywords">方法</a>的参数估计</h1> 贝叶斯<a href="/tag/fangfa/" target="_blank" class="keywords">方法</a>的参数估计，就是通过最大化后验概率来估计模型的参数。 假定模型参数为 w <script type="math/tex" id="MathJax-Element-522">w@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@，数据集为 D <script type="math/tex" id="MathJax-Element-523">D@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@，贝叶斯通过最大化后验概率估计模型参数 w <script type="math/tex" id="MathJax-Element-524">w@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@，即： <div class="MathJax_Display" role="textbox" aria-readonly="true" style="text-align: center;"> w=argmaxwp(w|D)=argmaxwp(w)p(D|w)P(D)=argmaxwp(w)p(D|w) </div> <script type="math/tex; mode=display" id="MathJax-Element-473">w = \arg\max_w p(w|D)= \arg\max_w \frac{p(w) p(D|w) }{P(D)}=\arg\max_w p(w) p(D|w) @H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@ <h1 id="后验概率的展开形式">后验概率的展开形式</h1> 假定如下： <ul> <li>样本独立不相关</li> <li>模型参数独立不相关</li> </ul> <div class="MathJax_Display" role="textbox" aria-readonly="true" style="text-align: center;"> p(w)p(D|w)=@H_798_<a href="/tag/403/" target="_blank" class="keywords">403</a>@@H_778_<a href="/tag/404/" target="_blank" class="keywords">404</a>@∏i=1Kp(wi)@H_778_<a href="/tag/404/" target="_blank" class="keywords">404</a>@∏i=1Np(D@H_102_<a href="/tag/502/" target="_blank" class="keywords">502</a>@i|w)←∑i=1Klogp(wi)+∑i=1Nlogp(Di@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_646@|w) </div> <script type="math/tex; mode=display" id="MathJax-Element-785">\begin{split} p(w)p(D|w) &= \prod_{i=1}^K p(w_i) \prod_{i=1}^N p(D_i|w) \\ &\leftarrow \sum_{i=1}^K \log p(w_i)+ \sum_{i=1}^N \log p(D_i|w) \end{split}@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@ 最新的优化问题为： <div class="MathJax_Display" role="textbox" aria-readonly="true" style="text-align: center;"> w=argminw−∑i=1Klogp(wi)−∑i=1Nlogp(Di|w) </div> <script type="math/tex; mode=display" id="MathJax-Element-531">w = \arg\min_w - \sum_{i=1}^K \log p(w_i)- \sum_{i=1}^N \log p(D_i|w)@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@ <h1 id="参数的先验概率与正则项">参数的先验概率与正则项</h1> 当参数 w <script type="math/tex" id="MathJax-Element-188">w@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@的先验概率满足高斯分布： <div class="MathJax_Display" role="textbox" aria-readonly="true" style="text-align: center;"> p(wi)=N(wi|μ,σ2)=12πσ2−−−−√e−(@H_<a href="/tag/301/" target="_blank" class="keywords">301</a>_1074@wi−μ@H_<a href="/tag/502/" target="_blank" class="keywords">502</a>_1094@)22σ2 </div> <script type="math/tex; mode=display" id="MathJax-Element-353">p(w_i) = N(w_i | \mu,\sigma^2) = \frac{1}{\sqrt{2\pi \sigma^2}} e^{-\frac{(w_i-\mu)^2}{2\sigma^2}}@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@ 优化问题的左项中，如果 w <script type="math/tex" id="MathJax-Element-354">w@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@满足 N(0,12λ) <script type="math/tex" id="MathJax-Element-355">N(0,\frac{1}{2\lambda})@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@： <div class="MathJax_Display" role="textbox" aria-readonly="true" style="text-align: center;"> −∑i=1K@H_209_1<a href="/tag/301/" target="_blank" class="keywords">301</a>@logp(wi)=−∑i=1Klog12πσ2−−−−√+∑i=1K(wi−μ)22σ2=const@H_<a href="/tag/502/" target="_blank" class="keywords">502</a>_1613@+∑i=1K(wi−μ)22σ2=const+λ∑i=1Kw2i </div> <script type="math/tex; mode=display" id="MathJax-Element-329">\begin{split} - \sum_{i=1}^K \log p(w_i) &= - \sum_{i=1}^K \log \frac{1}{\sqrt{2\pi \sigma^2}} + \sum_{i=1}^K \frac{(w_i-\mu)^2}{2\sigma^2} \\ &= const + \sum_{i=1}^K \frac{(w_i-\mu)^2}{2\sigma^2} \\ &= const + \lambda \sum_{i=1}^K w_i^2 \end{split}@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@ 这时候的优化<a href="/tag/hanshu/" target="_blank" class="keywords">函数</a>为： <div class="MathJax_Display" role="textbox" aria-readonly="true" style="text-align: center;"> w=argminwλ∑i=1Kw2i−∑i=1Nlogp(Di|w) </div> <script type="math/tex; mode=display" id="MathJax-Element-575">w = \arg\min_w \lambda \sum_{i=1}^K w_i^2 - \sum_{i=1}^N \log p(D_i|w)@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@ 同样地，参数 w <script type="math/tex" id="MathJax-Element-576">w@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@的先验概率满足均值为0的拉普拉斯分布，有： <div class="MathJax_Display" role="textbox" aria-readonly="true" style="text-align: center;"> w=argminwλ∑i=1K|wi|−∑i=1Nlogp(Di|w) </div> <script type="math/tex; mode=display" id="MathJax-Element-789">w = \arg\min_w \lambda \sum_{i=1}^K |w_i| - \sum_{i=1}^N \log p(D_i|w)@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@ 这说明： <ul> <li>L2正则，等价于参数 w <script type="math/tex" id="MathJax-Element-790">w@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@的先验分布满足均值为0的正态分布</li> <li>L1正则，等价于参数 w <script type="math/tex" id="MathJax-Element-791">w@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@的先验分布满足均值为0的拉普拉斯分布</li> <li>拉普拉斯在0附近突出，周围稀疏，对应容易产生稀疏解的模型</li> </ul> <h1 id="模型举例">模型举例</h1> 以参数 w <script type="math/tex" id="MathJax-Element-613">w@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@的先验概率满足均值为0的高斯分布为例，优化问题为： <div class="MathJax_Display" role="textbox" aria-readonly="true" style="text-align: center;"> w=argminwλ∑i=1Kw2i−∑i=1Nlogp(Di|w) </div> <script type="math/tex; mode=display" id="MathJax-Element-616">w = \arg\min_w \lambda \sum_{i=1}^K w_i^2 - \sum_{i=1}^N \log p(D_i|w)@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@ <h2 id="逻辑回归">逻辑回归</h2> <div class="MathJax_Display" role="textbox" aria-readonly="true" style="text-align: center;"> −∑i=1@H_989_2<a href="/tag/502/" target="_blank" class="keywords">502</a>@Nlogp(Di|w)=−∑i=1Nlogθ(ynwTxn)=∑i=1Nlog(1+exp(−ynwTxn)) </div> <script type="math/tex; mode=display" id="MathJax-Element-649">\begin{split} - \sum_{i=1}^N \log p(D_i|w) &= - \sum_{i=1}^N \log \theta(y_n w^T x_n) \\ &=\sum_{i=1}^N \log (1+\exp(-y_n w^T x_n) ) \end{split}@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@ 所以有： <div class="MathJax_Display" role="textbox" aria-readonly="true" style="text-align: center;"> w=argminwλ∑i=1Kw2i+∑i=1Nlog(1+exp(−ynwTxn@H_<a href="/tag/301/" target="_blank" class="keywords">301</a>_3002@)) </div> @H_706_<a href="/tag/301/" target="_blank" class="keywords">301</a>7@w = \arg\min_w \lambda \sum_{i=1}^K w_i^2 +\sum_{i=1}^N \log (1+\exp(-y_n w^T x_n) )@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@ 总结：逻辑回归，通过贝叶斯法最大化后验概率。在数据的概率满足逻辑<a href="/tag/hanshu/" target="_blank" class="keywords">函数</a>的假设下得到了cross entropy的误差<a href="/tag/hanshu/" target="_blank" class="keywords">函数</a>；在样本独立、模型参数独立、模型参数满足均值为0的高斯分布的假设下获得了L2正则项。 <h2 id="线性回归">线性回归</h2> 线性回归，假设误差满足均值为0的高斯分布，该假设符合一般的规律。 <div class="MathJax_Display" role="textbox" aria-readonly="true" style="text-align: center;"> p(Di|w)=12πσ2−−−−√e−(wT@H_<a href="/tag/502/" target="_blank" class="keywords">502</a>_1094@xi−yi@H_<a href="/tag/502/" target="_blank" class="keywords">502</a>_1094@)22σ2 </div> <script type="math/tex; mode=display" id="MathJax-Element-676">p(D_i|w) = \frac{1}{\sqrt{2\pi \sigma^2}} e^{-\frac{(w^Tx_i-y_i)^2}{2\sigma^2}}@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@ <div class="MathJax_Display" role="textbox" aria-readonly="true" style="text-align: center;"> −∑i=1Nlogp(Di|w)@H_890_3<a href="/tag/403/" target="_blank" class="keywords">403</a>@@H_990_3<a href="/tag/404/" target="_blank" class="keywords">404</a>@=−∑i=1Nlog12πσ2−−−−√e−(wT@H_<a href="/tag/502/" target="_blank" class="keywords">502</a>_1094@xi−yi@H_<a href="/tag/502/" target="_blank" class="keywords">502</a>_1094@)22σ2←∑i=1N(wTxi−yi)2 </div> <script type="math/tex; mode=display" id="MathJax-Element-685">\begin{split} - \sum_{i=1}^N \log p(D_i|w) &= - \sum_{i=1}^N \log \frac{1}{\sqrt{2\pi \sigma^2}} e^{-\frac{(w^Tx_i-y_i)^2}{2\sigma^2}} \\ &\leftarrow \sum_{i=1}^N (w^Tx_i-y_i)^2 \end{split}@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@ 所以有： <div class="MathJax_Display" role="textbox" aria-readonly="true" style="text-align: center;"> w=argminwλ∑i=1Kw2i+∑i=1N(wTxi−yi)2 </div> <script type="math/tex; mode=display" id="MathJax-Element-750">w = \arg\min_w \lambda \sum_{i=1}^K w_i^2 +\sum_{i=1}^N (w^Tx_i-y_i)^2@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_23@ 总结：线性回归，通过贝叶斯法最大化后验概率。在误差为均值0的高斯分布的假设下得到了square error的误差<a href="/tag/hanshu/" target="_blank" class="keywords">函数</a>；在样本独立、模型参数独立、模型参数满足均值为0的高斯分布的假设下获得了L2正则项。 </div></div> <div class="topcard-tags"></div> <ul class="list-group"> <li class="list-group-item"><a href="/regex/358521.html" title="正则表达式在iOS中的运用">上一篇：正则表达式在iOS中的运用</a><a href="/regex/358519.html" title="grep简单正则" class="text-muted pull-right">下一篇：grep简单正则</a> </li> </ul> </div> </div> </div>  <div class="row row-sm"> <div class="col-sm-12 col-md-12 col-lg-12"> <div class="card"> <ins class="adsbygoogle" style="display:block" data-ad-format="autorelaxed" data-ad-client="ca-pub-4605373693034661" data-ad-slot="9144498553"></ins> <script> (adsbygoogle = window.adsbygoogle || []).push({});$

贝叶斯方法与正则项

猜你在找的正则表达式相关文章