1, Linear neural network
(1) Linear regression
1. Linear model
The linear model is regarded as a singlelayer neural network.
2. Loss function
The loss function can quantify the difference between the actual value and the predicted value of the target.
3. Analytical solution
4. Optimization method: small batch gradient descent algorithm
For the case without analytical solution, the gradient descent reduces the error by continuously updating the parameters in the decreasing direction of the loss function. Calculate the derivative (gradient) of the loss function with respect to the model parameters. But in practice, the execution may be very slow: because we have to traverse the entire data set before each parameter update. Therefore, we usually take a small batch of samples at random every time we need to calculate the update. This variant is called small batch random gradient descent, and the batch size is b.
(2) Implementation of linear regression
1. Generate dataset
def synthetic_data(w, b, num_examples): #@save """generate y = Xw + b + Noise.""" X = torch.normal(0, 1, (num_examples, len(w))) y = torch.matmul(X, w) + b y += torch.normal(0, 0.01, y.shape) return X, y.reshape((1, 1)) true_w = torch.tensor([2, 3.4]) true_b = 4.2 features, labels = synthetic_data(true_w, true_b, 1000)
2. Read dataset
Disrupt samples in the data set and obtain data in small batches. We define a data_iter function, which receives batch size, characteristic matrix and label vector as input, and generates batch size_ Small batch size. Each small batch contains a set of features and labels.
def data_iter(batch_size, features, labels): num_examples = len(features) indices = list(range(num_examples)) # These samples are read randomly without a specific order random.shuffle(indices) for i in range(0, num_examples, batch_size): batch_indices = torch.tensor( indices[i: min(i + batch_size, num_examples)]) yield features[batch_indices], labels[batch_indices]
3. Initialize model parameters
The weight is initialized by sampling random numbers from the normal distribution with mean value of 0 and standard deviation of 0.01, and the offset is initialized to 0.
w = torch.normal(0, 0.01, size=(2,1), requires_grad=True) b = torch.zeros(1, requires_grad=True)
4. Define model
def linreg(X, w, b): #@save """Linear regression model.""" return torch.matmul(X, w) + b
5. Define loss function
Here we use the square loss function. In the implementation, we need to convert the shape of the real value y into and the predicted value y_hat has the same shape.
def squared_loss(y_hat, y): #@save """Mean square loss.""" return (y_hat  y.reshape(y_hat.shape)) ** 2 / 2
6. Define optimization algorithm
This function accepts the set of model parameters, learning rate and batch size as inputs. The size of each update step is determined by the learning rate lr. Because the loss we calculate is the sum of a batch of samples, we use batch_size to normalize the step size, so that the step size does not depend on our choice of batch size.
def sgd(params, lr, batch_size): #@save """Small batch random gradient descent.""" with torch.no_grad(): for param in params: param = lr * param.grad / batch_size param.grad.zero_()
7. Training
Number of iteration cycles num_epochs and learning rate lr are both super parameters. Setting super parameters is very difficult and needs to be adjusted through repeated experiments.
lr = 0.03 num_epochs = 3 net = linreg loss = squared_loss for epoch in range(num_epochs): for X, y in data_iter(batch_size, features, labels): l = loss(net(X, w, b), y) # `Small batch loss of X 'and' y ' # Because the ` L 'shape is (` batch_size`, 1), not a scalar` All elements in l ` are added together, # And calculate the gradient about [` w`, `b `] l.sum().backward() sgd([w, b], lr, batch_size) # Update the parameter with the gradient of the parameter with torch.no_grad(): train_l = loss(net(features, w, b), labels) print(f'epoch {epoch + 1}, loss {float(train_l.mean()):f}')
(3) Simple implementation of linear regression
A deep learning framework is used to concisely implement the linear regression model in (2).
1. Generate dataset
import numpy as np import torch from torch.utils import data from d2l import torch as d2l true_w = torch.tensor([2, 3.4]) true_b = 4.2 features, labels = d2l.synthetic_data(true_w, true_b, 1000)
2. Read dataset
We can call the existing API in the framework to read the data. We pass features and labels as API parameters, and specify batch when instantiating the data iterator object_ size. In addition, the Boolean value is_train indicates whether you want the data iterator object to scramble the data in each iteration cycle.
def load_array(data_arrays, batch_size, is_train=True): #@save """Construct a PyTorch Data iterator.""" dataset = data.TensorDataset(*data_arrays) return data.DataLoader(dataset, batch_size, shuffle=is_train) batch_size = 10 data_iter = load_array((features, labels), batch_size)
3. Define model
In PyTorch, the full connection layer is defined in the Linear class. It is worth noting that we pass two parameters to nn.Linear. The first specifies the input feature shape, i.e. 2, and the second specifies the output feature shape. The output feature shape is a single scalar, so it is 1.
# `nn ` is the abbreviation of neural network from torch import nn net = nn.Sequential(nn.Linear(2, 1))
4. Initialize model parameters
Just as we specify the input and output dimensions when constructing nn.Linear. Now let's access the parameters directly to set the initial value. We select the first layer in the network through net[0], and then use the weight.data and bias.data methods to access the parameters. Then use the replacement method normal_ And fill_ To override parameter values. Here, we specify that each weight parameter should be randomly sampled from a normal distribution with a mean of 0 and a standard deviation of 0.01, and the bias parameter will be initialized to zero.
net[0].weight.data.normal_(0, 0.01) net[0].bias.data.fill_(0)
5. Define loss function
The mselos class, also known as the square L2 norm, is used to calculate the mean square error. By default, it returns the average of all sample losses.
loss = nn.MSELoss()
6. Define optimization algorithm
The small batch random gradient descent algorithm is a standard tool for optimizing neural networks. PyTorch implements many variants of the algorithm in optim module. When we instantiate the SGD instance, we need to specify the optimized parameters (which can be obtained from our model through net.parameters()) and the super parameter dictionary required by the optimization algorithm. Small batch random gradient descent only needs to set lr value, which is set to 0.03 here.
trainer = torch.optim.SGD(net.parameters(), lr=0.03)
7. Training
In each iteration cycle, we will completely traverse the train_data and constantly obtain a small batch of input and corresponding labels. For each small batch, we will carry out the following steps:

The prediction is generated by calling net(X) and the loss l (forward propagation) is calculated.

The gradient is calculated by back propagation.

Update model parameters by calling the optimizer.

num_epochs = 3 for epoch in range(num_epochs): for X, y in data_iter: l = loss(net(X) ,y) trainer.zero_grad() l.backward() trainer.step() l = loss(net(features), labels) print(f'epoch {epoch + 1}, loss {l:f}')
(4) Softmax regression
Common loss function:
(1)L2 Loss (2)L1 Loss (3)Huber's Robust Loss
(5) Image classification dataset
%matplotlib inline import torch import torchvision from torch.utils import data from torchvision import transforms from d2l import torch as d2l d2l.use_svg_display()
1. Read dataset
The fashion MNIST dataset can be downloaded and read into memory through the builtin functions in the framework.
# The image data is transformed from PIL type to 32bit floatingpoint format through ToTensor instance # Divide by 255 so that the values of all pixels are between 0 and 1 trans = transforms.ToTensor() mnist_train = torchvision.datasets.FashionMNIST( root="../data", train=True, transform=trans, download=True) mnist_test = torchvision.datasets.FashionMNIST( root="../data", train=False, transform=trans, download=True)
Fashion MNIST contains 10 categories, each of which includes 6000 training data and 1000 test data. The following function is used to convert between a numeric label index and its text name.
def get_fashion_mnist_labels(labels): #@save """return FashionMNIST The text label of the dataset.""" text_labels = ['tshirt', 'trouser', 'pullover', 'dress', 'coat', 'sandal', 'shirt', 'sneaker', 'bag', 'ankle boot'] return [text_labels[int(i)] for i in labels]
2. Read small batch
Using the builtin data iterator, the data loader will read a small batch of data each time in each iteration_ size. We also randomly scrambled all samples in the training data iterator.
batch_size = 256 def get_dataloader_workers(): #@save """Four processes are used to read data.""" return 4 train_iter = data.DataLoader(mnist_train, batch_size, shuffle=True, num_workers=get_dataloader_workers())
Take a look at the time it takes to read the training data.
timer = d2l.Timer() for X, y in train_iter: continue f'{timer.stop():.2f} sec'
3. Consolidate all components
We define load_data_fashion_mnist function, which is used to obtain and read the fashion MNIST dataset. It returns the data iterators for the training set and the validation set. In addition, it accepts an optional parameter resize to resize the image to another shape.
def load_data_fashion_mnist(batch_size, resize=None): #@save """download FashionMNIST The dataset and then load it into memory.""" trans = [transforms.ToTensor()] if resize: trans.insert(0, transforms.Resize(resize)) trans = transforms.Compose(trans) mnist_train = torchvision.datasets.FashionMNIST( root="../data", train=True, transform=trans, download=True) mnist_test = torchvision.datasets.FashionMNIST( root="../data", train=False, transform=trans, download=True) return (data.DataLoader(mnist_train, batch_size, shuffle=True, num_workers=get_dataloader_workers()), data.DataLoader(mnist_test, batch_size, shuffle=False, num_workers=get_dataloader_workers()))
Test load by specifying the resize parameter_ data_ fashion_ Image resizing function of MNIST function.
train_iter, test_iter = load_data_fashion_mnist(32, resize=64) for X, y in train_iter: print(X.shape, X.dtype, y.shape, y.dtype) break
torch.Size([32, 1, 64, 64]) torch.float32 torch.Size([32]) torch.int64
(6) Implementation of Softmax regression
import torch from IPython import display from d2l import torch as d2l batch_size = 256 train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)
1. Initialize model parameters
Each sample in the original dataset is 28 × two thousand eight hundred and twentyeight × 28. In this section, we will flatten each image as a vector of length 784. In softmax regression, we have as much output as categories. Because our dataset has 10 categories, the network output dimension is 10. As with linear regression, we will initialize our weight W with a normal distribution and an offset of 0.
num_inputs = 784 num_outputs = 10 W = torch.normal(0, 0.01, size=(num_inputs, num_outputs), requires_grad=True) b = torch.zeros(num_outputs, requires_grad=True)
2. Define softmax operations
softmax consists of three steps: (1) exponentiation of each term (using exp); (2) Sum each row (each sample is a row in a small batch) to obtain the normalization constant of each sample; (3) Divide each row by its normalization constant to ensure that the sum of the results is 1.
def softmax(X): X_exp = torch.exp(X) partition = X_exp.sum(1, keepdim=True) return X_exp / partition # The broadcast mechanism is applied here
3. Define model
The following code defines how the input is mapped to the output through the network. Note that before passing the data to our model, we use the reshape function to flatten each original image into a vector.
def net(X): return softmax(torch.matmul(X.reshape((1, W.shape[0])), W) + b)
4. Define loss function
5. Classification accuracy
6. Training
7. Forecast
(7) Concise implementation of Softmax regression
2, Multilayer perceptron
(1) Multilayer perceptron
(2) Implementation of multilayer perceptron
1. Initialize model parameters
num_inputs, num_outputs, num_hiddens = 784, 10, 256 W1 = nn.Parameter(torch.randn( num_inputs, num_hiddens, requires_grad=True) * 0.01) b1 = nn.Parameter(torch.zeros(num_hiddens, requires_grad=True)) W2 = nn.Parameter(torch.randn( num_hiddens, num_outputs, requires_grad=True) * 0.01) b2 = nn.Parameter(torch.zeros(num_outputs, requires_grad=True)) params = [W1, b1, W2, b2]
2. Activation function
def relu(X): a = torch.zeros_like(X) return torch.max(X, a)
3. Model
We use reshape to convert each twodimensional image into a length of num_ Vector of inputs.
def net(X): X = X.reshape((1, num_inputs)) H = relu(X@W1 + b1) # Here "@" represents matrix multiplication return (H@W2 + b2)
4. Loss function
Directly use the builtin functions in the advanced API to calculate softmax and cross entropy loss.
loss = nn.CrossEntropyLoss()
5. Training
The training process of multilayer perceptron is exactly the same as that of softmax regression. You can call the train of d2l package directly_ CH3 function.
num_epochs, lr = 10, 0.1 updater = torch.optim.SGD(params, lr=lr) d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, updater)
(3) Simple implementation of multilayer perceptron
import torch from torch import nn from d2l import torch as d2l
Compared with the concise implementation of softmax regression, the only difference is that we have added two full connection layers (we only added one full connection layer before). The first layer is the hidden layer, which contains 256 hidden units and uses the ReLU activation function. The second layer is the output layer.
net = nn.Sequential(nn.Flatten(), nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10)) def init_weights(m): if type(m) == nn.Linear: nn.init.normal_(m.weight, std=0.01) net.apply(init_weights);
batch_size, lr, num_epochs = 256, 0.1, 10 loss = nn.CrossEntropyLoss() trainer = torch.optim.SGD(net.parameters(), lr=lr) train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size) d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, trainer)
(4) Model selection, under fitting and over fitting
Training error: the error calculated by the model on the training data set.
Generalization error: the error calculated by the model on the new data set.
Validation dataset: a dataset used to evaluate the quality of the model.
Test data set: a data set used only once.
kfold cross validation method:
Overfitting:
Under fit:
(5) Weight attenuation
(6) Dropout