A simple third-party CNN self coding matlab toolbox

The handwriting database is used. The database has been integrated into the toolbox. Just use it directly. Show a part of the database. The goal is to train the database to achieve the purpose of recognition:

Then, the database data is normalized and preprocessed. nnsetup establishes a network, in which many parameters will be initialized, and opts numepochs = 1; The personal feeling of this parameter is to repeat the test times for all data, and setting 1 is to experiment once. opts.batchsize = 100; This parameter is to send a large number of samples every 100 at random as a wave into the experiment. Then there's the training test.


function nn = nnsetup(architecture)
%NNSETUP Create forward neural network
% nn = nnsetup(architecture) Returns a neural network structure, architecture Is a structural parameter
% architecture It's a n x 1 Vector representing the number of neurons in each layer
%such as architecture=[784 100 10],Indicates that the input layer is 784 dimensional input, 100 hidden layers and 10 output layers
%Why is the input 784: because each handwriting size is 28*28 That is, the 784 dimension
%Why the hidden layer is 100: it can be set freely, can be modified freely, and needs to be designed
%Why is the output 10: the handwriting has 0-9 These 10 results, so it is 10
    nn.size   = architecture;
    nn.n      = numel(nn.size);
    nn.activation_function              = 'tanh_opt';   % Hidden layer activation function: 'sigm' (sigmoid) or 'tanh_opt' (default tanh).
    nn.learningRate                     = 2;            %  Learning rate: typically needs to be lower when using 'sigm' activation function and non-normalized inputs.
    nn.momentum                         = 0.5;          %  Momentum Weight momentum factor
    nn.scaling_learningRate             = 1;            %  Learning rate change factor (each epoch)
    nn.weightPenaltyL2                  = 0;            %  L2 regularization Regularization
    nn.nonSparsityPenalty               = 0;            %  Non sparse penalty
    nn.sparsityTarget                   = 0.05;         %  Sparse target value
    nn.inputZeroMaskedFraction          = 0;            %  Denoising effect of automatic coding
    nn.dropoutFraction                  = 0;            %  Dropout level (http://www.cs.toronto.edu/~hinton/absps/dropout.pdf)
    nn.testing                          = 0;            %  Internal variable. nntest sets this to one.
    nn.output                           = 'sigm';       %  Output activation output unit 'sigm' (=logistic), 'softmax' and 'linear'
    for i = 2 : nn.n   
        % weights and weight momentum
        nn.W{i - 1} = (rand(nn.size(i), nn.size(i - 1)+1) - 0.5) * 2 * 4 * sqrt(6 / (nn.size(i) + nn.size(i - 1)));
        nn.vW{i - 1} = zeros(size(nn.W{i - 1}));
        % average activations (for use with sparsity)
        nn.p{i}     = zeros(1, nn.size(i));   


This function is very simple to understand. It initializes the network. What does the network need to initialize, A lot of initialization is suitable for all networks (cnn,dbn, etc.). Now you only need to know the structure of the network and the parameters related to sparse coding representation: nn.nonSparsityPenalty, nn.sparsityTarget, which is why the sparse representation and the specific methods do not need to be controlled. In actual use, there are only a few parameter settings. Let's leave the rest to the program. Then Yes, just notice the activation function NN activation_ function. , Then the network weights are initialized randomly.


Let's talk about the whole function: [nn, L] = nntrain(nn, train_x, train_y, opts);

It can be seen that what nntrain needs is the designed network nn and training data train_x. Training corresponding target value train_y. And the additional parameter opts. Additional parameters include: number of repeated training, opts Numepochs, the size of each block of training data, opts Batch size, etc. The function comes out of the trained network nn, which is very important. The trained nn is the structure, which includes all the information you need, such as the weight coefficient of each layer of network, training error, etc. you can find it, and this trained nn is also used in nntest. See the introduction of the blog above for the specific implementation details of nntrain.


setup is about this process. Next, we'll get to train. Open \ NN \ nntrain m

We skip the code that verifies that the incoming data is correct and go straight to the key parts

For the denoising part, please refer to the paper: extracting and composing robot features with denoising autoencoders

m = size(train_x, 1);  
//m is the number of training samples  
//Note that we set opt when calling. Batch size is the size of batch gradient  
batchsize = opts.batchsize; numepochs = opts.numepochs;  
numbatches = m / batchsize;  //Calculate the number of batches. Divide the total number of training samples by the size of a batch to get the total number of batches  
assert(rem(numbatches, 1) == 0, 'numbatches must be a integer');//That is, the total data is grouped in the previous step, and the number of groups should be an integer
L = zeros(numepochs*numbatches,1);  
n = 1;  
//numepochs is the number of cycles  
for i = 1 : numepochs  
    tic;  // tic is used to save the current time, and then toc is used to record the program completion time.
    kk = randperm(m);  
    //The random perm (m) function is to randomly disrupt a number sequence and generate an array of 1 to M. For example, randperm(5), ans = 2 3 4 1 5
    for l = 1 : numbatches  
        batch_x = train_x(kk((l - 1) * batchsize + 1 : l * batchsize), :);  
        //Add noise to input (for use in denoising autoencoder)  
        //Add noise, which is the part that denoising autoencoder needs to use  
        //For this part, please refer to the paper "extracting and composing robot features with denoising autoencoders"  
        //The specific method is to adjust some data in the training sample to 0. inputZeroMaskedFraction represents the adjusted proportion  
        if(nn.inputZeroMaskedFraction ~= 0)  
            batch_x = batch_x*
        batch_y = train_y(kk((l - 1) * batchsize + 1 : l * batchsize), :);  
        //These three functions  
        //nnff is forward propagation, nnbp is backward propagation, and nnapplygrads is gradient descent  
        //We analyze the code of these functions below  
        nn = nnff(nn, batch_x, batch_y);  
        nn = nnbp(nn);  
        nn = nnapplygrads(nn);  
        L(n) = nn.L;  
        n = n + 1;  
    t = toc;  
    if ishandle(fhandle)  
        if opts.validation == 1  
            loss = nneval(nn, loss, train_x, train_y, val_x, val_y);  
            loss = nneval(nn, loss, train_x, train_y);  
        nnupdatefigures(nn, fhandle, loss, opts, i);  
    disp(['epoch ' num2str(i) '/' num2str(opts.numepochs) '. Took ' num2str(t) ' seconds' '. Mean squared error on training set is ' num2str(mean(L((n-numbatches):(n-1))))]);  
    nn.learningRate = nn.learningRate * nn.scaling_learningRate;  

The three functions nnff,nnbp and nnapplygrads are analyzed below


nnff is a feed forward pass. In fact, it is very simple, that is, the whole network can run forward once

Of course, there are calculations of dropout and sparsity

For details, see the paper "Improving Neural Networks with Dropout" and Autoencoders and Sparsity


function nn = nnff(nn, x, y)  
//NNFF performs a feedforward pass  
// nn = nnff(nn, x, y) returns an neural network structure with updated  
// layer activations, error and loss (nn.a, nn.e and nn.L)  
    n = nn.n;  
    m = size(x, 1);  
    x = [ones(m,1) x];  
    nn.a{1} = x;  
    //feedforward pass  
    for i = 2 : n-1  
        //Forward propagation calculation is performed according to different activation functions selected  
        //You can look back at the first parameter activation in nnsetup_ function  
        //sigm is the sigmoid function, tanh_opt is the function of tanh. The toolbox seems to have changed a little  
        //tanh_opt is 1.7159*tanh(2/3.*A)  
        switch nn.activation_function   
            case 'sigm'  
                // Calculate the unit's outputs (including the bias term)  
                nn.a{i} = sigm(nn.a{i - 1} * nn.W{i - 1}');  
            case 'tanh_opt'  
                nn.a{i} = tanh_opt(nn.a{i - 1} * nn.W{i - 1}');  
        //dropout fraction is a parameter that can be set in nnsetup  
        if(nn.dropoutFraction > 0)  
                nn.a{i} = nn.a{i}.*(1 - nn.dropoutFraction);  
                nn.dropOutMask{i} = (rand(size(nn.a{i}))>nn.dropoutFraction);  
                nn.a{i} = nn.a{i}.*nn.dropOutMask{i};  
        //Calculate sparsity. nonSparsityPenalty is the penalty coefficient for parameters that fail to reach sparsitytarget  
        //calculate running exponential activations for use with sparsity  
            nn.p{i} = 0.99 * nn.p{i} + 0.01 * mean(nn.a{i}, 1);  
        //Add the bias term  
        nn.a{i} = [ones(m,1) nn.a{i}];  
    switch nn.output   
        case 'sigm'  
            nn.a{n} = sigm(nn.a{n - 1} * nn.W{n - 1}');  
        case 'linear'  
            nn.a{n} = nn.a{n - 1} * nn.W{n - 1}';  
        case 'softmax'  
            nn.a{n} = nn.a{n - 1} * nn.W{n - 1}';//N here is the specific number in the loop n-1, representing the last output layer (not participating in the loop)  
            nn.a{n} = exp(bsxfun(@minus, nn.a{n}, max(nn.a{n},[],2))); 
//max this expression means to compare and take the maximum value of each row, and the result is a column vector.
            nn.a{n} = bsxfun(@rdivide, nn.a{n}, sum(nn.a{n}, 2));   
     // bsxfun stands for efficient operation, which refers to the binary operation of calculating elements one by one between two arrays, @ rdivide here refers to left division@ minus is subtraction
    //error and loss 

    //Calculation error  
    nn.e = y - nn.a{n};  
    //Calculate loss function  
    switch nn.output  
        case {'sigm', 'linear'}  
            nn.L = 1/2 * sum(sum(nn.e .^ 2)) / m;   
        case 'softmax'  
            nn.L = -sum(sum(y .* log(nn.a{n}))) / m;  


Code: \ NN \ nnbp m

nnbp is the process of back propagation, which is more regular than that in ufldl Neural Network Basically the same

It is also worth noting the dropout and sparsity parts

    pi = repmat(nn.p{i}, size(nn.a{i}, 1), 1);  
    sparsityError = [zeros(size(nn.a{i},1),1) nn.nonSparsityPenalty * (-nn.sparsityTarget ./ pi + (1 - nn.sparsityTarget) ./ (1 - pi))];  
// Backpropagate first derivatives  
if i+1==n % in this case in d{n} there is not the bias term to be removed               
    d{i} = (d{i + 1} * nn.W{i} + sparsityError) .* d_act; // Bishop (5.56)  
else // in this case in d{i} the bias term has to be removed  
    d{i} = (d{i + 1}(:,2:end) * nn.W{i} + sparsityError) .* d_act;  
    d{i} = d{i} .* [ones(size(d{i},1),1) nn.dropOutMask{i}];  

This is only the content of the implementation. d{i} in the code is the delta value of this layer, which is described in ufldl

dW{i} is basically the gradient of calculation, but some things need to be added and modified later

See the paper "Improving Neural Networks with Dropout" and Autoencoders and Sparsity Content of


Code file: \ NN \ nnapplygrads m

for i = 1 : (nn.n - 1)  
        dW = nn.dW{i} + nn.weightPenaltyL2 * nn.W{i};  
        dW = nn.dW{i};  
    dW = nn.learningRate * dW;  
        nn.vW{i} = nn.momentum*nn.vW{i} + dW;  
        dW = nn.vW{i};  
    nn.W{i} = nn.W{i} - dW;  

This content is simple, NN Weightpenaltyl2 is part of weight decay and a parameter that can be set during nnsetup

If yes, add weight Penalty to prevent over fitting, then adjust it according to the size of momentum, and finally change NN W {I} is enough


Ok, let's take another look at nntest, as follows:

function [ri, right] = nntest(nn, x, y)
    labels = nnpredict(nn, x);
    [~, expected] = max(y,[],2);
    right = find(labels == expected);    
    ri = numel(right) / size(x, 1);

Call nnpredict. The function needs test data X and label y. if there is y, the accuracy can be calculated. If there is no y, you can directly call labels = nnpredict(nn, x) to get the predicted label.


Code file: \ NN \ nnpredict m

[cpp]  view plain  copy

  1. function labels = nnpredict(nn, x)  
  2.     nn.testing = 1;  
  3.     nn = nnff(nn, x, zeros(size(x,1), nn.size(end)));  
  4.     nn.testing = 0;  
  6.     [~, i] = max(nn.a{end},[],2);  
  7.     labels = i;  
  8. end  

It is very simple to continue. predict is only nnff once to get the final output~~

max(nn.a{end},[],2); Is to return the maximum value of each row and the number of columns, so labels returns the label

(this test seems to be specially used to test classification problems. We know that nnff can get the final value.)

Keywords: MATLAB neural networks CNN

Added by moleculo on Thu, 16 Dec 2021 11:41:18 +0200