as13351_DL_HW1Q4 | Deepnote

OK, thus far we have been talking about linear models. All these can be viewed as a single-layer neural net. The next step is to move on to multi-layer nets. Training these is a bit more involved, and implementing from scratch requires time and effort. Instead, we just use well-established libraries. I prefer PyTorch, which is based on an earlier library called Torch (designed for training neural nets via backprop).

import numpy as np import torch import torchvision import tensorflow as tf from torch.autograd import Variable import matplotlib.pyplot as plt %matplotlib inline

Use CUDA if it is available, otherwise use CPU.

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

Torch handles data types a bit differently. Everything in torch is a tensor.

a = np.random.rand(2,3) b = torch.from_numpy(a) # Q4.1 Display the contents of a, b print('contents of a:\n', a) print('contents of b:\n', b)

The idea in Torch is that tensors allow for easy forward (function evaluations) and backward (gradient) passes.

A = torch.rand(2,2) b = torch.rand(2,1) x = torch.rand(2,1, requires_grad=True) y = torch.matmul(A,x) + b print(y) z = y.sum() print(z) z.backward() print(x.grad) print(x)

Notice how the backward pass computed the gradients using autograd. OK, enough background. Time to train some networks. Let us load the Fashion MNIST dataset, which is a database of grayscale images of clothing items.

trainingdata = torchvision.datasets.FashionMNIST('./FashionMNIST/',train=True,download=True,transform=torchvision.transforms.ToTensor()) testdata = torchvision.datasets.FashionMNIST('./FashionMNIST/',train=False,download=True,transform=torchvision.transforms.ToTensor())

Let us examine the size of the dataset.

The dataset contains images of size (28, 28). After flattening, the data points are 784.

# Q4.2 How many training and testing data points are there in the dataset? # What is the number of features in each data point? print('Training data points: ', len(trainingdata)) print('Test data points: ', len(testdata)) img = trainingdata.data[0] img = img.reshape(28, 28) img = img.flatten() print('Number of features in each data point: ', len(img))

Let us try to visualize some of the images. Since each data point is a tensor (not an array) we need to postprocess to use matplotlib.

randnums= np.random.randint(1,60000,15) # Q4.3 Assuming each sample is an image of size 28x28, show it in matplotlib. plt.figure(figsize=(10,10)) for i in range(15): image, label = trainingdata[randnums[i]] image = image.reshape(28, 28) plt.subplot(5,5,i+1) plt.xticks([]) plt.yticks([]) plt.imshow(image, cmap=plt.cm.binary) plt.xlabel(label) plt.show()

Let's try plotting several images. This is conveniently achieved in PyTorch using a data loader, which loads data in batches.

trainDataLoader = torch.utils.data.DataLoader(trainingdata, batch_size=64, shuffle=True) testDataLoader = torch.utils.data.DataLoader(testdata, batch_size=64, shuffle=False) trainDataIter = iter(trainDataLoader) images, labels = trainDataIter.next() print(images.size(), labels)

# Q4.4 Visualize the first 10 images of the first minibatch # returned by testDataLoader. testDataIter = iter(testDataLoader) test_images, test_labels = testDataIter.next() plt.figure(figsize=(10,10)) for i in range(10): image = test_images[i].reshape(28, 28) plt.subplot(5,5,i+1) plt.xticks([]) plt.yticks([]) plt.imshow(image, cmap=plt.cm.binary) plt.xlabel(test_labels[i]) plt.show()

Now we are ready to define our linear model. Here is some boilerplate PyTorch code that implements the forward model for a single layer network for logistic regression (similar to the one discussed in class notes).

class LinearReg(torch.nn.Module): def __init__(self): super(LinearReg, self).__init__() self.linear = torch.nn.Linear(28*28,10) def forward(self, x): x = x.view(-1,28*28) transformed_x = self.linear(x) return transformed_x net = LinearReg().cuda() Loss = torch.nn.CrossEntropyLoss() optimizer = torch.optim.SGD(net.parameters(), lr=0.01)

Cool! Now we have set everything up. Let's try to train the network.

This block trains the neural network for 20 epochs and prints the training loss for every 5 epochs.

train_loss_history = [] test_loss_history = [] # Q4.5 Write down a for-loop that trains this network for 20 epochs, # and print the train/test losses. # Save them in the variables above. If done correctly, you should be able to # execute the next code block. num_epochs = 20 for epoch in range(num_epochs): epoch_train_loss = 0 current_epoch_train_loss = 0 count = 0 for images, labels in trainDataLoader: # Transfering images and labels to GPU if available images, labels = images.to(device), labels.to(device) # Forward pass outputs = net(images) loss = Loss(outputs, labels) # Initializing the gradient as 0 so there is no mixing of gradient among the batches optimizer.zero_grad() #Backward propagation of error loss.backward() # Optimizing the parameters optimizer.step() epoch_train_loss += loss.data count += 1 current_epoch_train_loss = epoch_train_loss/count train_loss_history.append(current_epoch_train_loss) if count % 5 == 0: print("Epoch: {}, Train Loss: {}".format(epoch, current_epoch_train_loss))

This block runs on test data for 20 epochs and prints the loss for every 5 epochs.

for epoch in range(num_epochs): epoch_test_loss = 0 current_epoch_test_loss = 0 count = 0 for test_images, test_labels in testDataLoader: test_images, test_labels = test_images.to(device), test_labels.to(device) predictions = net(test_images) epoch_test_loss += Loss(predictions, test_labels) count += 1 current_epoch_test_loss = epoch_test_loss/count test_loss_history.append(current_epoch_test_loss) if count % 5 == 0: print("Epoch: {}, Train Loss: {}".format(epoch, current_epoch_test_loss))

plt.plot(range(20),train_loss_history,'-',linewidth=3,label='Train error') plt.plot(range(20),test_loss_history,'-',linewidth=3,label='Test error') plt.xlabel('epoch') plt.ylabel('loss') plt.grid(True) plt.legend()

Neat! Now let's evaluate our model accuracy on the entire dataset. The predicted class label for a given input image can computed by looking at the output of the neural network model and computing the index corresponding to the maximum activation. Something like

predicted_output = net(images) _, predicted_labels = torch.max(predicted_output,1)

predicted_output = net(images) print(torch.max(predicted_output, 1)) fit = Loss(predicted_output, labels) print(labels)

def evaluate(dataloader): # Q4.6 Implement a function here that evaluates training and testing accuracy. # Here, accuracy is measured by probability of successful classification. accuracy = 0 for images, labels in dataloader: images, labels = images.to(device), labels.to(device) predictions = net(images) predicted_labels = predictions.max(1).indices accuracy += (predicted_labels == labels).sum(dtype=torch.float32) return accuracy/len(dataloader.dataset) print('Train acc = %0.2f, test acc = %0.2f' % (evaluate(trainDataLoader), evaluate(testDataLoader)))

Recommended on Deepnote

Stock market analysis

10 ways to create NumPy arrays

Wide Residual Networks