第三章、线性神经网络

线性回归（linear regression）可以追溯到19世纪初， 它在回归的各种标准工具中最简单而且最流行。 线性回归基于几个简单的假设： 首先，假设自变量和因变量之间的关系是线性的， 即可以表示为中元素的加权和，这里通常允许包含观测值的一些噪声； 其次，我们假设任何噪声都比较正常，如噪声遵循正态分布。

第3.1节 线性回归

3.1.1 理论基础

1. 数据

• 给定一个维输入：
  $\mathbf{X} = \left[x{1}, x{2}, \ldots, x_{n} \right]^{T}$
• 线性模型的权重参数和偏置：
  $\mathbf{W} = \left[w{1}, w{2}, \dots, w_{n} \right]^{T}, \enspace b$
• 输出则是输入的加权和：
  $ {y} = w{1}x{1} + w{2}x{2} + \dots, + w{n}x{n} + b $​

1500	3	300000
2000	4	400000
1200	2	250000

2. 损失函数

• 目标：根据输入数据去预测输出

  $$\hat{y} = \mathbf{w}^{T}{x} + b$$

• 衡量预估质量：用预测值 - 真实值

  $$\begin{aligned} \mathcal{l}^* &= y - \hat{y} \\ => \enspace \mathcal{l} &= \frac{1}{2}(y - \hat{y})^{2}\end{aligned}$$

• 合并：正对每一个样本都进行评估

  $$\mathcal{l}^{i}_{(\mathbf{w},b)} = \frac{1}{2}(\hat{y}^{(i)} - y^{i})^{2}$$

• 损失函数：对每个样本的衡量求均值

  $$\begin{aligned} \mathcal{L}_{(\mathbf{w},b)} &= \frac{1}{n} \sum\limits_{i=1}^{n} \mathcal{l}^{i} \\ &= \frac{1}{n} \sum\limits_{i=1}^{n} \frac{1}{2} (\mathbf{w}^{T}{x}^{(i)} + b - y^{i})^{2} \end{aligned}$$

3. 优化函数

• 目标：最小化损失

  $\mathbf{w}^*,{b}^* = \text{arg} \min_\limits{w,b}\mathcal{L}_{(\mathbf{X},\mathbf{y},\mathbf{w},b)}$​

• 简化：将设为1，则可以将偏置列入权重

  $\begin{aligned} \mathcal{L}_{(\mathbf{X},\mathbf{y},\mathbf{w},b)} &= \frac{1}{n} \sum\limits_{i=1}^{n} \frac{1}{2} (\mathbf{w}^{T}{x}^{(i)} + b - y^{i})^{2} \\ =>\mathcal{L}_{(\mathbf{X},\mathbf{y},\mathbf{w})} &= \frac{1}{2n} \Big\lvert\Big\lvert ({y} - \mathbf{w}{X}) \Big\rvert\Big\rvert^{2} \end{aligned}$​

• 显示解：凸函数，最优解满足梯度

  $$\begin{aligned} &\frac{\partial}{\partial{\mathbf{w}}} \mathcal{L}_{(\mathbf{X},\mathbf{y},\mathbf{w})} = 0 \\ \Leftrightarrow &\frac{1}{n} (y - \mathbf{w}{X})^{T}{X} = 0 \\ \Leftrightarrow &\mathbf{w}^* = ({X}^{T} {X})^{-1}{Xy} \end{aligned}$$

4. 优化方法——梯度下降

梯度下降通过不断沿着反梯度方向更新参数求解

• 梯度下降
• 小批量下降：超参数p
• 随机梯度下降

4.总结

• 线性回归是对n维输入的加权，外加偏差
• 使用平方损失来衡量预测值与真实值之间的差异
• 线性回归有显示解
• 线性回归可以看做是单层的神经网络
• 小批量随机梯度下降是深度学习默认的求解算法
• 小批量梯度下降两个重要的超参数是批量大小和学习率

3.1.2 从零实现线性回归

数据、模型、损失函数、优化器

%matplotlib inline
import random, torch
from d2l import torch as d2l

1.数据构建

def generate_data(w, b, num_examples):
    """生成 y = wX + b + 噪声"""
    X = torch.normal(2, 0.3, (num_examples, len(w)))
    y = torch.matmul(X, w) + b
    y += torch.normal(0, 0.2, y.shape)
    return X, y.reshape((-1, 1))


ture_w = torch.tensor([2, 0.6])
ture_b = 0.2
features, labels = generate_data(ture_w, ture_b, 8000)

features[:3, :], labels[:3]

(tensor([[2.3478, 2.4763],
         [1.7897, 2.3384],
         [1.6261, 2.0841]]),
 tensor([[6.7362],
         [5.0191],
         [4.6524]]))

d2l.set_figsize()
d2l.plt.scatter(features[:1000, 0].detach().numpy(),
                labels[:1000].detach().numpy(), 1)
d2l.plt.scatter(features[:1000, 1].detach().numpy(),
                labels[:1000].detach().numpy(), 1)

<matplotlib.collections.PathCollection at 0x7fe64fc797c0>

2. 数据读取

# 按照一个batch地获取
def get_data_iter(batch_size, features, labels):
    num_examples = len(features)
    idxs = list(range(num_examples))
    random.shuffle(idxs)
    for i in range(0, num_examples, batch_size):
        batch_idxs = torch.tensor(
            idxs[i: min(batch_size + i, num_examples)]
        )
        yield features[batch_idxs, :], labels[batch_idxs]

batch_size = 10
for X, y in get_data_iter(batch_size, features, labels):
    print("训练数据：", X, '\n', "训练目标：", y)
    break

训练数据： tensor([[2.4767, 1.9958],
        [2.2292, 2.3632],
        [2.1951, 2.1529],
        [2.0536, 1.8707],
        [1.6970, 1.8042],
        [2.0869, 1.4939],
        [2.0516, 2.2665],
        [1.8934, 2.2656],
        [1.7037, 2.1092],
        [1.8330, 1.8787]]) 
 训练目标： tensor([[6.4159],
        [6.0410],
        [5.9194],
        [5.4723],
        [4.5526],
        [5.0979],
        [5.7323],
        [5.0205],
        [5.0299],
        [4.8499]])

3. 构建模型

### 随机初始化模型参数
w = torch.normal(0, 1, size=(2, 1), requires_grad=True)
b = torch.ones(1, requires_grad=True)

### 定义模型
def liner_model(X, w, b):
    return torch.matmul(X, w) + b


liner_model(features, w, b).shape

torch.Size([8000, 1])

4. 定义损失函数

def ms_loss(y_hat, y):
    """定义均方损失"""
    return (y_hat - y.reshape(y_hat.shape)) ** 2 / 2

5. 定义优化器

def sgd(params, lr, batch_size):
    """小批量随机梯度下降"""
    with torch.no_grad():
        for param in params:
            param -= lr * param.grad / batch_size
            param.grad.zero_()

6. 模型训练

### 超参数
lr = 0.1
num_epochs = 5
net_model = liner_model
loss = ms_loss

### 训练
for epoch in range(num_epochs):
    for X, y in get_data_iter(batch_size, features, labels):
        l = loss(net_model(X, w, b), y)
        l.sum().backward()
        sgd([w, b], lr, batch_size)
    with torch.no_grad():
        train_l = loss(net_model(features, w, b), labels)
        print(f'epoch {epoch + 1 :2}, loss {train_l.mean() :.8f}')

epoch  1, loss 0.02702038
epoch  2, loss 0.02084376
epoch  3, loss 0.02002947
epoch  4, loss 0.02400172
epoch  5, loss 0.01999601

7.与真实参数做对比

print(f'w的误差: {ture_w - w.reshape(ture_w.shape)}')
print(f'b的误差: {(ture_b - b)}')

w的误差: tensor([0.0069, 0.0213], grad_fn=<SubBackward0>)
b的误差: tensor([-0.0697], grad_fn=<RsubBackward1>)

3.1.3 线性回归简单实现

import numpy as np
import torch
from torch.utils import data
from d2l import torch as d2l

ture_w, ture_b

(tensor([2.0000, 0.6000]), 0.2)

true_w = torch.tensor([2, 0.6], requires_grad=True)
true_b = torch.ones(1, requires_grad=True)

features, labels = generate_data(true_w, true_b, 1000)

def load_array(data_arrays, batch_size, is_train=True):
    dataset = data.TensorDataset(*data_arrays)
    return data.DataLoader(dataset, batch_size, shuffle=True)


batch_size = 10
data_iter = load_array((features, labels), batch_size)
next(iter(data_iter))

[tensor([[2.5015, 2.3537],
         [1.5924, 1.8605],
         [2.1028, 2.1819],
         [1.8190, 2.0113],
         [1.6774, 2.3362],
         [1.7908, 2.0683],
         [1.7124, 2.2071],
         [2.3189, 1.6855],
         [2.1480, 2.2916],
         [1.5104, 1.6872]]),
 tensor([[7.1117],
         [5.2422],
         [6.9261],
         [5.8172],
         [5.9285],
         [6.1434],
         [5.7847],
         [6.5754],
         [6.5749],
         [4.7918]], grad_fn=<StackBackward0>)]

from torch import nn

net = nn.Sequential(nn.Linear(2, 1))

net[0].weight.data.normal_(0, 0.1)
net[0].bias.data.fill_(0)

tensor([0.])

loss = nn.MSELoss()

# 实例化SGD
trainer = torch.optim.SGD(net.parameters(), lr=0.01)

# 训练
num_epochs = 5
for epoch in range(num_epochs):
    for X, y in data_iter:
        l = loss(net(X), y)
        trainer.zero_grad()
        l.backward(retain_graph=True)
        trainer.step()
    l = loss(net(features), labels)
    print(f'epoch {epoch + 1 :2}, loss {l:f}')

epoch  1, loss 0.096368
epoch  2, loss 0.077876
epoch  3, loss 0.067644
epoch  4, loss 0.059806
epoch  5, loss 0.054634

print(f'w的误差: {true_w - w.reshape(true_w.shape)}')
print(f'b的误差: {(true_b - b)}')

w的误差: tensor([0.0495, 0.0135], grad_fn=<SubBackward0>)
b的误差: tensor([0.7791], grad_fn=<SubBackward0>)