1.image classification
Data is divided into train_data, val_data and test_data by data-driven algorithm. Different results are debugged with different hyperparameters on train, evaluated on verification set, and then applied to test with the best performance hyperparameters on verification set.
image classifier,data_driven approach,
Example 1: knn nearest neighbor algorithm
The code is divided into five parts: loading data (cifar-10), processing data, training model, testing data, and cross-validation.
Focus on knn's algorithmic representation:
(1) Calculating distance of two-layer cycle:
def compute_distances_two_loops(self, X):
num_test = X.shape[0]
num_train = self.x_train.shape[0]
dists = np.zeros((num_test, num_train))
for i in xrange(num_test):
for j in xrange(num_train):
dists[i,j]=np.sqrt(np.sum((X[i,:]-self.x_train[j,:])**2))#Computing Euclidean Distance
return dists
(2) Calculating distance of one layer cycle:
def compute_distances_one_loop(self, X):
num_test = X.shape[0]
num_train = self.x_train.shape[0]
dists = np.zeros((num_test, num_train))
for i in xrange(num_test):
dists[i]=np.sqrt(np.sum(np.square(X[i,:]-self.x_train),axis=1))#Calculate Euclidean Distance with a Layer of Cycle
return dists
prints(dists)
(3) Matrix computing distance:
def compute_distances_no_loops(self, X):
num_test = X.shape[0]
num_train = self.x_train.shape[0]
dists = np.zeros((num_test, num_train))
x_2=np.sum(np.square(X),axis=1)#Calculate x squares
x_train_2=np.sum(np.square(self.x_train),axis=1)#Computing x_train square
x_xtrain=np.dot(X,self.x_train.T)#Compute x.*x_train
dists=np.sqrt(x_2.reshape(-1,1)-2*x_xtrain+x_train_2)#Calculating distance
return dists
(4) Prediction part (voting mechanism)
def predict_labels(self, dists, k=1):
num_test = dists.shape[0]
y_pred = np.zeros(num_test)
for i in xrange(num_test):
order_dists=np.argsort(dists[i,:],axis=0)#Sort by column
target_k=self.y_train[order_dists[: k]]#Record Category
y_pred[i]=np.argmax(np.bincount(target_k))
return y_pred
The python function in the voting mechanism:
1.np.argsort sorting, with axis=0 operation is sorted by column
2.np.argmax finds the index corresponding to the maximum value in the matrix
3.np.bincount count count count
(5)cross-validation
num_folds = 5
k_choices = [1, 3, 5, 8, 10, 12, 15, 20, 50, 100]
x_train_folds = []
y_train_folds = []
y_train = y_train.reshape(-1, 1)
x_train_folds = np.array_split(x_train, num_folds)
y_train_folds = np.array_split(y_train, num_folds)
k_to_accuracies={}
for k in k_choices:
k_to_accuracies.setdefault(k, []) # Setting up a dictionary
pass
classifier = KNearestNeighbor()
for i in range(num_folds):
x_train = np.vstack(x_train_folds[0:i] + x_train_folds[i + 1:]) # Set up the training set and remove the val where i is
y_train = np.vstack(y_train_folds[0:i] + y_train_folds[i + 1:]) # label
y_train = y_train[:, 0]
classifier.train(x_train, y_train)
for k in k_choices:
x_pred = x_train_folds[i] # x of prediction set
y_pred = classifier.predict(x_pred, k=k) # Predict y
num_correct = np.sum(y_pred == y_train_folds[i][:,0])
accuracy = float(num_correct) / len(y_pred) # Accuracy of calculation
k_to_accuracies[k] = k_to_accuracies[k] + [accuracy] # Put precision into dictionary
# Print out the computed accuracies
for k in sorted(k_to_accuracies):
for accuracy in k_to_accuracies[k]:
print('k = %d, accuracy = %f' % (k, accuracy))
Dictionary operation
Here we construct a precision dictionary, k_to_accuracies, corresponding to k-accuracy. The setdefault function is used here. Compared with get, this function can add a query value when there is no value to query in the dictionary.
Example 2 SVM Support Vector Machine
Code is divided into: data loading, data processing, SVM-classifier, super-parameter adjustment.
Focus on the svm algorithm.
with loops
def svm_loss_naive(W, X, y, reg):
dW = np.zeros(W.shape) # initialize the gradient as zero
num_classes = W.shape[1]
num_train = X.shape[0]
loss = 0.0
for i in xrange(num_train):
scores = X[i].dot(W)
correct_class_score = scores[y[i]]#Scores of correct categories
for j in xrange(num_classes):
if j == y[i]:
#Skip the correct category
continue
margin = scores[j] - correct_class_score + 1 # note delta = 1, score - correct category score + 1, reference formula
if margin > 0:
loss += margin
dW[:,j] += X[i].T
dW[:,y[i]] -= X[i].T
loss /= num_train
dW/=num_train
# Add regularization to the loss.
loss += 0.5*reg * np.sum(W * W)
dW +=reg*W
return loss, dW
no loops
def svm_loss_vectorized(W, X, y, reg): loss = 0.0 dW = np.zeros(W.shape) # initialize the gradient as zero num_classes=W.shape[1] num_train=X.shape[0] scores=X.dot(W) correct_class_scores=scores[np.arange(num_train),list(y)].reshape(-1,1) margins=scores-correct_class_scores+1 margins[margins<0]=0 margins[np.arange(num_train),y]=0 loss=np.sum(margins)/num_train loss+=0.5*reg*np.sum(W*W) margins[margins>0]=1 e_number=np.sum(margins,axis=1)#Quantity of eligible contributions margins[np.arange(num_train),y]-=e_number dW=np.dot(X.T,margins)/num_train dW+=reg*W return loss, dW
Some practical python functions, such as (axis=1, line-by-line operation) (reshape(-1,1), turn a row matrix into a column)
(In the matrix, y.shape[0] returns the total number of rows in y, basically the same as len(y), and y.shape[1] returns the total number of columns in y.)
Super-parameter debugging
learning_rates = [1e-7, 5e-5]
regularization_strengths = [2.5e4, 5e4]
results = {}
best_val = -1 # The highest validation accuracy that we have seen so far.
best_svm = None
from cs231n.classifiers import LinearSVM
for lr,reg in zip(learning_rates,regularization_strengths):
svm=LinearSVM()
svm.train(X_train,y_train,learning_rate=lr,reg=reg,num_iters=1000,verbose=False)
y_train_pred=svm.predict(X_train)
train_accuracy=np.mean(y_train==y_train_pred)
y_val_pred=svm.predict(X_val)
val_accuracy=np.mean(y_val==y_val_pred)
results[(lr,reg)]=(train_accuracy,val_accuracy)
if best_val<val_accuracy:
best_val=val_accuracy
best_svm=svm
pass
Here, some combinations of lr and reg are output by using zip(learning_rates,regularization_strengths) to facilitate superparametric debugging.
Example 3 softmax loss function
Code is divided into data loading, data processing, software max-classfier, hyperparameter debugging
Here, the difference between soft Max and svm is that the loss function is different, and the rest of the training process and other basic ideas are the same.
with loops
import numpy as np
from random import shuffle
from past.builtins import xrange
def softmax_loss_naive(W, X, y, reg):
# Initialize the loss and gradient to zero.
loss = 0.0
dW = np.zeros_like(W)
num_class=W.shape[1]
num_train=X.shape[0]
for i in range(num_train):
scores=X[i].dot(W)
shift_scores=scores-max(scores)
loss -= np.log( np.exp(shift_scores[y[i]]) / np.sum(np.exp(shift_scores)) )
for j in xrange(num_class):
softmax_output = np.exp(shift_scores[j]) / np.sum(np.exp(shift_scores))
if j == y[i]:
dW[:,j] += (-1 + softmax_output) * X[i,:]#According to Back Propagation Algorithms
else:
dW[:,j] += softmax_output * X[i,:]
loss /= num_train
loss += 0.5 * reg * np.sum(W * W)
dW /= num_train
dW += reg * W
pass
return loss, dW
no loops
def softmax_loss_vectorized(W, X, y, reg): # Initialize the loss and gradient to zero. loss = 0.0 dW = np.zeros_like(W) num_class=W.shape[1] num_train=X.shape[0] scores=X.dot(W) shift_scores=scores-np.max(scores,axis=1).reshape(-1,1)#Avoiding the x-th power spillover of e can easily become nan softmax_out=np.exp(shift_scores)/np.sum(np.exp(shift_scores),axis=1).reshape((-1,1)) loss = np.sum( -1 * np.log( softmax_out[range(num_train),y] ) ) loss /= num_train loss += 0.5 * reg * np.sum(W * W) dS = softmax_out.copy() dS[range(num_train), list(y)] += -1 dW = (X.T).dot(dS) dW = dW / num_train + reg * W return loss, dW
Here, when calculating dw, we use the chain criterion of back propagation algorithm. At first, we don't understand it very well. If we look back, we will understand it. The shift processing of scores is also to avoid the x-Power of e being too large, so that the function value is nan.