文章作者：Tyan
博客：noahsnail.com | CSDN | 简书

注：本文为李沐大神的《动手学深度学习》的课程笔记！

高维线性回归

使用线性函数 y=0.05@H_404_59@+∑pi=10.01@H_404_93@xi+noise $y=0.05+\underset{i=1}{\overset{p}{\sum }}0.01x_i+\text{noise}$$y = 0.05 + \sum_{i = 1}^p 0.01x_i + \text{noise}@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_164@<a href="/tag/shengcheng/" target="_blank" class="keywords">生成</a>数据样本，噪音服从均值0和标准差为0.01的正态分布。</p> <pre class="prettyprint"><span class="hljs-comment"># 导入mxnet</span> <span class="hljs-keyword">import</span> random <span class="hljs-keyword">import</span> mxnet <span class="hljs-keyword">as</span> mx <span class="hljs-comment"># 设置<a href="/tag/suiji/" target="_blank" class="keywords">随机</a>种子</span> random.seed(<span class="hljs-number">2</span>) mx.random.seed(<span class="hljs-number">2</span>) <span class="hljs-keyword">from</span> mxnet <span class="hljs-keyword">import</span> gluon <span class="hljs-keyword">from</span> mxnet <span class="hljs-keyword">import</span> ndarray <span class="hljs-keyword">as</span> nd <span class="hljs-keyword">from</span> mxnet <span class="hljs-keyword">import</span> autograd</pre> <h2 id="生成数据集"><a href="/tag/shengcheng/" target="_blank" class="keywords">生成</a>数据集</h2> <pre class="prettyprint"><span class="hljs-comment"># 训练数据<a href="/tag/shuliang/" target="_blank" class="keywords">数量</a></span> num_train = <span class="hljs-number">20</span> <span class="hljs-comment"># 测试数据<a href="/tag/shuliang/" target="_blank" class="keywords">数量</a></span> num_test = <span class="hljs-number">100</span> <span class="hljs-comment"># 输入数据特征维度</span> num_inputs = <span class="hljs-number">200</span> <span class="hljs-comment"># 实际权重</span> true_w = nd.ones((num_inputs,<span class="hljs-number">1</span>)) * <span class="hljs-number">0.01</span> <span class="hljs-comment"># 实际偏置</span> true_b = <span class="hljs-number">0.05</span></pre> <pre class="prettyprint"><span class="hljs-comment"># <a href="/tag/shengcheng/" target="_blank" class="keywords">生成</a>数据</span> X = nd.random_normal(shape=(num_train + num_test,num_inputs)) y = nd.dot(X,true_w) + true_b <span class="hljs-comment"># <a href="/tag/tianjia/" target="_blank" class="keywords">添加</a><a href="/tag/suiji/" target="_blank" class="keywords">随机</a>噪声</span> y += <span class="hljs-number">0.01</span> * nd.random_normal(shape=y.shape) <span class="hljs-comment"># 训练数据和测试数据</span> X_train,X_test = X[:num_train,:],X[num_train:,:] y_train,y_test = y[:num_train],y[num_train:]</pre> <pre class="prettyprint"><span class="hljs-comment"># 批数据大小</span> batch_size = <span class="hljs-number">1</span> <span class="hljs-comment"># 通过yield进行数据读取</span> <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">data_iter</span><span class="hljs-params">(num_examples)</span>:</span> <span class="hljs-comment"># 产生样本的索引</span> idx = list(range(num_examples)) <span class="hljs-comment"># 将索引<a href="/tag/suiji/" target="_blank" class="keywords">随机</a>打乱</span> random.shuffle(idx) <span class="hljs-comment"># 迭代一个epoch</span> <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> range(<span class="hljs-number">0</span>,num_examples,batch_size): <span class="hljs-comment"># 依次取出样本的索引,这种实现方式在num_examples/batch_size不能整除时也适用</span> j = nd.array(idx[i:min((i + batch_size),num_examples)]) <span class="hljs-comment"># 根据提供的索引取元素</span> <span class="hljs-keyword">yield</span> nd.take(X,j),nd.take(y,j)</pre> <h2 id="初始化模型参数">初始化模型参数</h2> <pre class="prettyprint"><span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">init_params</span><span class="hljs-params">()</span>:</span> <span class="hljs-comment"># <a href="/tag/suiji/" target="_blank" class="keywords">随机</a>初始化权重w</span> w = nd.random_normal(shape=(num_inputs,<span class="hljs-number">1</span>)) <span class="hljs-comment"># 偏置b初始化为0</span> b = nd.zeros((<span class="hljs-number">1</span>,)) <span class="hljs-comment"># w,b放入list里</span> params = [w,b] <span class="hljs-comment"># 需要计算反向传播,<a href="/tag/tianjia/" target="_blank" class="keywords">添加</a><a href="/tag/zidong/" target="_blank" class="keywords">自动</a>求导</span> <span class="hljs-keyword">for</span> param <span class="hljs-keyword">in</span> params: param.attach_grad() <span class="hljs-keyword">return</span> params</pre> <h2 id="l2-范数正则化"><span class="MathJax_Preview" style="color: inherit; display: none;"></span><span class="MathJax" id="MathJax-Element-4-Frame" tabindex="0" data-mathml="<math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;><msub><mi>L</mi><mn>2</mn></msub></math>" role="presentation" style="position: relative;"> <nobr aria-hidden="true"> <span class="math" id="MathJax-Span-50" style="width: 1.182em; display: inline-block;"><span style="display: inline-block; position: relative; width: 0.974em; height: 0px; font-size: 120%;"><span style="position: absolute; clip: rect(1.391em,1000.97em,2.398em,-999.998em); top: -2.151em; left: 0em;"><span class="mrow" id="MathJax-Span-51"><span class="msubsup" id="MathJax-Span-52"><span style="display: inline-block; position: relative; width: 0.974em; height: 0px;"><span style="position: absolute; clip: rect(3.231em,1000.56em,4.099em,-999.998em); top: -3.991em; left: 0em;"><span class="mi" id="MathJax-Span-53" style="font-family: STIXGeneral-Italic;">L<span style="display: inline-block; overflow: hidden; height: 1px; width: 0.002em;"></span></span><span style="display: inline-block; width: 0px; height: 3.995em;"></span></span><span style="position: absolute; top: -3.852em; left: 0.557em;"><span class="mn" id="MathJax-Span-54" style="font-size: 70.7%; font-family: STIXGeneral-Regular;">2</span><span style="display: inline-block; width: 0px; height: 3.995em;"></span></span></span></span></span><span style="display: inline-block; width: 0px; height: 2.155em;"></span></span></span><span style="display: inline-block; overflow: hidden; vertical-align: -0.206em; border-left: 0px solid; width: 0px; height: 1.044em;"></span></span> </nobr><span class="MJX_Assistive_MathML" role="presentation"> <math xmlns="http://www.w3.org/1998/Math/MathML"> <msub> <mi> L </mi> <mn> 2 </mn> </msub> </math></span></span><script type="math/tex" id="MathJax-Element-4">L_2@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_164@ 范数正则化</h2> <p>在训练时最小化<a href="/tag/hanshu/" target="_blank" class="keywords">函数</a>为：<span class="MathJax_Preview" style="color: inherit; display: none;"></span><span class="MathJax" id="MathJax-Element-5-Frame" tabindex="0" data-mathml="<math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;><mtext>loss</mtext><mo>+</mo><mi>&amp;#x03BB;</mi><munder><mo>&amp;#x2211;</mo><mrow class=&quot;MJX-TeXAtom-ORD&quot;><mi>p</mi><mo>&amp;#x2208;</mo><mrow class=&quot;MJX-TeXAtom-ORD&quot;><mtext>params</mtext></mrow></mrow></munder><mo fence=&quot;false&quot; stretchy=&quot;false&quot;>&amp;#x2016;</mo><mi>p</mi><msubsup><mo fence=&quot;false&quot; stretchy=&quot;false&quot;>&amp;#x2016;</mo><mn>2</mn><mn>2</mn></msubsup><mrow class=&quot;MJX-TeXAtom-ORD&quot;><mo>&amp;#x3002;</mo></mrow></math>" role="presentation" style="position: relative;"> <nobr aria-hidden="true"> <span class="math" id="MathJax-Span-55" style="width: 12.398em; display: inline-block;"><span style="display: inline-block; position: relative; width: 10.315em; height: 0px; font-size: 120%;"><span style="position: absolute; clip: rect(1.513em,1010.32em,3.128em,-999.997em); top: -2.497em; left: 0em;"><span class="mrow" id="MathJax-Span-56"><span class="mtext" id="MathJax-Span-57" style="font-family: STIXGeneral-Regular;">loss</span><span class="mo" id="MathJax-Span-58" style="font-family: STIXGeneral-Regular; padding-left: 0.263em;">+</span>@H_856_<a href="/tag/403/" target="_blank" class="keywords">403</a>@λ</span><span class="munderover" id="MathJax-Span-60" style="padding-left: 0.211em;"><span style="display: inline-block; position: relative; width: 3.857em; height: 0px;"><span style="position: absolute; clip: rect(3.076em,-999.997em); top: -4.008em; left: 0em;"><span class="mo" id="MathJax-Span-61" style="font-family: STIXGeneral-Regular; vertical-align: 0.003em;">∑</span><span style="display: inline-block; width: 0px; height: 4.013em;"></span></span><span style="position: absolute; top: -3.695em; left: 0.94em;"><span class="texatom" id="MathJax-Span-62"><span class="mrow" id="MathJax-Span-63"><span class="mi" id="MathJax-Span-64" style="font-size: 70.7%; font-family: STIXGeneral-Italic;">p</span><span class="mo" id="MathJax-Span-65" style="font-size: 70.7%; font-family: STIXGeneral-Regular;">∈</span><span class="texatom" id="MathJax-Span-66"><span class="mrow" id="MathJax-Span-67"><span class="mtext" id="MathJax-Span-68" style="font-size: 70.7%; font-family: STIXGeneral-Regular;">params</span></span></span></span></span><span style="display: inline-block; width: 0px; height: 4.013em;"></span></span></span></span><span class="mo" id="MathJax-Span-69" style="font-family: STIXGeneral-Regular; padding-left: 0.211em;">‖</span><span class="mi" id="MathJax-Span-70" style="font-family: STIXGeneral-Italic;">p</span><span class="msubsup" id="MathJax-Span-71"><span style="display: inline-block; position: relative; width: 0.94em; height: 0px;"><span style="position: absolute; clip: rect(3.18em,4.378em,-999.997em); top: -4.008em; left: 0em;"><span class="mo" id="MathJax-Span-72" style="font-family: STIXGeneral-Regular;">‖</span><span style="display: inline-block; width: 0px; height: 4.013em;"></span></span><span style="position: absolute; clip: rect(3.388em,-999.997em); top: -4.372em; left: 0.523em;"><span class="mn" id="MathJax-Span-73" style="font-size: 70.7%; font-family: STIXGeneral-Regular;">2</span><span style="display: inline-block; width: 0px; height: 4.013em;"></span></span><span style="position: absolute; clip: rect(3.388em,-999.997em); top: -3.695em; left: 0.523em;"><span class="mn" id="MathJax-Span-74" style="font-size: 70.7%; font-family: STIXGeneral-Regular;">2</span><span style="display: inline-block; width: 0px; height: 4.013em;"></span></span></span></span><span class="texatom" id="MathJax-Span-75"><span class="mrow" id="MathJax-Span-76"><span class="mo" id="MathJax-Span-77"><span style="font-family: STIXGeneral,&quot;Arial Unicode MS&quot;,serif; font-size: 83%; font-style: normal; font-weight: normal;">。</span></span></span></span></span><span style="display: inline-block; width: 0px; height: 2.503em;"></span></span></span><span style="display: inline-block; overflow: hidden; vertical-align: -0.622em; border-left: 0px solid; width: 0px; height: 1.628em;"></span></span> </nobr><span class="MJX_Assistive_MathML" role="presentation"> <math xmlns="http://www.w3.org/1998/Math/MathML"> <mtext> loss </mtext> <mo> + </mo> <mi> λ </mi> <munder> <mo> ∑ </mo> <mrow class="MJX-TeXAtom-ORD"> <mi> p </mi> <mo> ∈ </mo> <mrow class="MJX-TeXAtom-ORD"> <mtext> params </mtext> </mrow> </mrow> </munder> <mo fence="false" stretchy="false"> ‖ </mo> <mi> p </mi> <msubsup> <mo fence="false" stretchy="false"> ‖ </mo> <mn> 2 </mn> <mn> 2 </mn> </msubsup> <mrow class="MJX-TeXAtom-ORD"> <mo> 。 </mo> </mrow> </math></span></span><script type="math/tex" id="MathJax-Element-5">\text{loss} + \lambda \sum_{p \in \textrm{params}}\|p\|_2^2。@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_164@</p> <pre class="prettyprint"><span class="hljs-comment"># L2范数</span> <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">L2_penalty</span><span class="hljs-params">(w,b)</span>:</span> <span class="hljs-keyword">return</span> ((w**<span class="hljs-number">2</span>).sum() + b ** <span class="hljs-number">2</span>) / <span class="hljs-number">2</span></pre> <h2 id="定义训练和测试">定义训练和测试</h2> <pre class="prettyprint">%matplotlib inline <span class="hljs-keyword">import</span> matplotlib <span class="hljs-keyword">as</span> mpl mpl.rcParams[<span class="hljs-string">'figure.dpi'</span>]= <span class="hljs-number">120</span> <span class="hljs-keyword">import</span> matplotlib.pyplot <span class="hljs-keyword">as</span> plt <span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np <span class="hljs-comment"># 定义网络</span> <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">net</span><span class="hljs-params">(X,w,b)</span>:</span> <span class="hljs-keyword">return</span> nd.dot(X,w) + b <span class="hljs-comment"># 损失<a href="/tag/hanshu/" target="_blank" class="keywords">函数</a></span> <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">square_loss</span><span class="hljs-params">(yhat,y)</span>:</span> <span class="hljs-keyword">return</span> (yhat - y.reshape(yhat.shape)) ** <span class="hljs-number">2</span> / <span class="hljs-number">2</span> <span class="hljs-comment"># 梯度下降</span> <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">sgd</span><span class="hljs-params">(params,lr,batch_size)</span>:</span> <span class="hljs-keyword">for</span> param <span class="hljs-keyword">in</span> params: param[:] = param - lr * param.grad / batch_size <span class="hljs-comment"># 测试</span> <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">test</span><span class="hljs-params">(net,params,X,y)</span>:</span> <span class="hljs-keyword">return</span> square_loss(net(X,*params),y).mean().asscalar() <span class="hljs-comment"># 训练</span> <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">train</span><span class="hljs-params">(_lambda)</span>:</span> <span class="hljs-comment"># 定义训练的迭代周期</span> epochs = <span class="hljs-number">10</span> <span class="hljs-comment"># 定义学习率</span> learning_rate = <span class="hljs-number">0.005</span> <span class="hljs-comment"># 初始化参数</span> w,b = params = init_params() <span class="hljs-comment"># 训练损失</span> train_loss = [] <span class="hljs-comment"># 测试损失</span> test_loss = [] <span class="hljs-keyword">for</span> epoch <span class="hljs-keyword">in</span> range(epochs): <span class="hljs-keyword">for</span> data,label <span class="hljs-keyword">in</span> data_iter(num_train): <span class="hljs-comment"># 记录梯度</span> <span class="hljs-keyword">with</span> autograd.record(): <span class="hljs-comment"># 计算预测值</span> output = net(data,*params) <span class="hljs-comment"># 计算loss</span> loss = square_loss(output,label) + _lambda * L2_penalty(*params) <span class="hljs-comment"># 反向传播</span> loss.backward() <span class="hljs-comment"># 更新梯度</span> sgd(params,learning_rate,batch_size) <span class="hljs-comment"># 训练损失</span> train_loss.append(test(net,X_train,y_train)) <span class="hljs-comment"># 测试损失</span> test_loss.append(test(net,X_test,y_test)) <span class="hljs-comment"># 绘制损失图像</span> plt.plot(train_loss) plt.plot(test_loss) plt.legend([<span class="hljs-string">'train'</span>,<span class="hljs-string">'test'</span>]) plt.show() <span class="hljs-keyword">return</span> <span class="hljs-string">'learned w[:10]:'</span>,w[:<span class="hljs-number">10</span>].T,<span class="hljs-string">'learned b:'</span>,b</pre> <h1 id="观察过拟合">观察过拟合@H_<a href="/tag/403/" target="_blank" class="keywords">403</a>_677@ <pre class="prettyprint">train(<span class="hljs-number">0</span>)</pre> <pre>('learned w[:10]:',[[ 1.04817176 -0.02568593 0.86764956 0.29322267 0.01006179 -0.56152576 0.38436398 -0.308<a href="/tag/403/" target="_blank" class="keywords">403</a>67 -2.32450151 0.03733355]] &lt;NDArray 1x10 @<a href="/tag/cpu/" target="_blank" class="keywords">cpu</a>(0)&gt;,'learned b:',[ 0.79914856] &lt;NDArray 1 @<a href="/tag/cpu/" target="_blank" class="keywords">cpu</a>(0)&gt;)</pre> <h2 id="使用正则化">使用正则化</h2> <pre class="prettyprint">train(<span class="hljs-number">5</span>)</pre> <pre>('learned w[:10]:',[[ 0.00107633 -0.00052574 0.004<a href="/tag/502/" target="_blank" class="keywords">502</a>33 -0.00110545 -0.0068391 -0.00181657 -0.00530632 0.00512845 -0.00742549 -0.00058495]] &lt;NDArray 1x10 @<a href="/tag/cpu/" target="_blank" class="keywords">cpu</a>(0)&gt;,[ 0.00449432] &lt;NDArray 1 @<a href="/tag/cpu/" target="_blank" class="keywords">cpu</a>(0)&gt;)</pre></div> <div class="topcard-tags"></div> <ul class="list-group"> <li class="list-group-item"><a href="/regex/357618.html" title="正则表达式 – 计算小数点和第一个非零数字之间的前导零">上一篇：正则表达式 – 计算小数点和第一个</a><a href="/regex/357616.html" title="正则表达式核心" class="text-muted pull-right">下一篇：正则表达式核心</a> </li> </ul> </div> </div> </div> <!-- row end --> <div class="row row-sm"> <div class="col-sm-12 col-md-12 col-lg-12"> <div class="card"> <ins class="adsbygoogle" style="display:block" data-ad-format="autorelaxed" data-ad-client="ca-pub-4605373693034661" data-ad-slot="9144498553"></ins> <script> (adsbygoogle = window.adsbygoogle || []).push({});$