7. Loss function loss

Loss usually refers to the difference between the predicted value and the real value of the output of the neural network, which is used to:
① Calculate the gap between the actual output and the target
② Provide some basis for us to update the output (back propagation)

For example: the specific definitions of each function are in the official pytorch document https://pytorch.org/docs/stable/nn.html#loss-functions
We define two variables inputs and targets to represent the output and target values, and use L1Loss function and mselos function to calculate the loss value respectively:

import torch
from torch.nn import L1Loss
from  torch import nn
inputs= torch.tensor([1,2,3],dtype=torch.float32)
targets = torch.tensor([1,2,5],dtype=torch.float32)

inputs = torch.reshape(inputs,(1,1,1,3))
targets = torch.reshape(targets,(1,1,1,3))

loss =L1Loss(reduction='sum')          #Use reduction to adjust the calculation method loss function
result = loss(inputs,targets)

loss_mse =nn.MSELoss()
result_mse = loss_mse(inputs,targets)
print(result)
print(result_mse)

Output results:

tensor(2.)
tensor(1.3333)

Use of cross entropy function:

x=torch.tensor([0.1,0.2,0.3])
y=torch.tensor([1])
x=torch.reshape(x,(1,3))
loss_cross = nn.CrossEntropyLoss()
result_cross = loss_cross(x,y)
print(result_cross)

Output:

tensor(1.1019)

8. Back propagation

Back propagation effect:
Calculate the total error, that is, quickly calculate the partial derivatives of all parameters to update the new weight.

This example uses the previously designed network:
Import corresponding modules, data sets and neural network definitions

import torch
import torchvision
from torch import nn
from torch.nn import Conv2d, MaxPool2d, Flatten, Linear, Sequential
from torch.utils.data import DataLoader
from torch.utils.tensorboard import SummaryWriter

dataset = torchvision.datasets.CIFAR10('./dataset',train=False,transform=torchvision.transforms.ToTensor(),download=True)
dataloader = DataLoader(dataset,batch_size=1)

class ChenYu(nn.Module):
    def __init__(self):
        super(ChenYu, self).__init__()
        self.model1 =Sequential(
        Conv2d(3,32,5,padding=2),
        MaxPool2d(2),
        Conv2d(32,32,5,padding=2),
        MaxPool2d(2),
        Conv2d(32,64,5,padding=2),
        MaxPool2d(2),
        Flatten(),
        Linear(1024,64),
        Linear(64,10)
        )

    def forward(self,x):
        x=self.model1(x)
        return x

The cross entropy is used for loss calculation, and the backward() function is used for back propagation:

loss= nn.CrossEntropyLoss()      
chenyu=ChenYu()
for data in dataloader:
    imgs, targets = data
    outputs= chenyu(imgs)
    result_loss = loss(outputs,targets)      #Cross entropy use
    result_loss.backward()         #Back propagation update data
    print(result_loss)

9. Optimizer

Optimizer function: it is used to update and calculate the network parameters affecting model training and model output to make them approximate or reach the optimal value, so as to minimize (or maximize) the loss function E. the most commonly used method is the gradient descent method.

Using the SGD optimizer, we need to set its input parameters and learning rate lr

Similarly, the above neural network is used for direct analysis without repeating here:
① Input the samples into the neural network to get the output
② Use the loss function to calculate the error between the output and the actual value
③ Using optim.zero_grad() clears the previous gradient to 0
④ Use the. backward() function for back propagation
⑤ optim.step() tunes each parameter

loss= nn.CrossEntropyLoss()
chenyu=ChenYu()
optim = torch.optim.SGD(chenyu.parameters(),lr=0.01)
for epoch in range(20):
    running_Loss = 0.0
    for data in dataloader:
        imgs, targets = data
        outputs= chenyu(imgs)
        result_loss = loss(outputs,targets)
        optim.zero_grad()         #Set the parameter to be adjusted to 0
        result_loss.backward()         #Back propagation
        optim.step()
        running_Loss = running_Loss+result_loss
    print(running_Loss)

Final result: we can see that with the continuous iteration of the network, the loss value is continuously decreasing:

tensor(18689.9395, grad_fn=<AddBackward0>)
tensor(16173.0996, grad_fn=<AddBackward0>)
tensor(15504.9697, grad_fn=<AddBackward0>)