[Tensorflow 2][Keras][Custom and Distributed Training with TensorFlow] Week3 - Programming Assignment: Horse or Human?

본 포스팅은 다음 과정을 정리 한 글입니다.

 

Custom and Distributed Training with TensorFlow

https://www.coursera.org/learn/custom-distributed-training-with-tensorflow?specialization=tensorflow-advanced-techniques 

Custom and Distributed Training with TensorFlow

지난 시간 리뷰

 

[Tensorflow 2][Keras][Custom and Distributed Training with TensorFlow] Week3 - AutoGraph and Creating Graphs for Complex Code

[Tensorflow 2][Keras][Custom and Distributed Training with TensorFlow] Week3 - AutoGraph and Creating Graphs for Complex Code

Horse or Human? In-graph training loop Assignment

This assignment lets you practice how to train a Keras model on the horses_or_humans dataset with the entire training process performed in graph mode. These steps include:

• loading batches
• calculating gradients
• updating parameters
• calculating validation accuracy
• repeating the loop until convergence

Setup

Import TensorFlow 2.0:

from __future__ import absolute_import, division, print_function, unicode_literals
import numpy as np

import tensorflow as tf
import tensorflow_datasets as tfds
import tensorflow_hub as hub
import matplotlib.pyplot as plt

Prepare the dataset

Load the horses to human dataset, splitting 80% for the training set and 20% for the test set.

splits, info = tfds.load('horses_or_humans', as_supervised=True, with_info=True, split=['train[:80%]', 'train[80%:]', 'test'], data_dir='./data')

(train_examples, validation_examples, test_examples) = splits

num_examples = info.splits['train'].num_examples
num_classes = info.features['label'].num_classes

BATCH_SIZE = 32
IMAGE_SIZE = 224

Pre-process an image (please complete this section)

You'll define a mapping function that resizes the image to a height of 224 by 224, and normalizes the pixels to the range of 0 to 1. Note that pixels range from 0 to 255.

• You'll use the following function: tf.image.resize and pass in the (height,width) as a tuple (or list).
• To normalize, divide by a floating value so that the pixel range changes from [0,255] to [0,1].

# Create a autograph pre-processing function to resize and normalize an image
### START CODE HERE ###
@tf.function
def map_fn(img, label):
    image_height = 224
    image_width = 224
### START CODE HERE ###
    # resize the image
    img = tf.image.resize(img, (image_height,image_width))
    # normalize the image
    img /= 255.0
### END CODE HERE
    return img, label

## TEST CODE:

test_image, test_label = list(train_examples)[0]

test_result = map_fn(test_image, test_label)

print(test_result[0].shape)
print(test_result[1].shape)

del test_image, test_label, test_result

Apply pre-processing to the datasets (please complete this section)

Apply the following steps to the training_examples:

• Apply the map_fn to the training_examples
• Shuffle the training data using .shuffle(buffer_size=) and set the buffer size to the number of examples.
• Group these into batches using .batch() and set the batch size given by the parameter.

Hint: You can look at how validation_examples and test_examples are pre-processed to get a sense of how to chain together multiple function calls.

# Prepare train dataset by using preprocessing with map_fn, shuffling and batching
def prepare_dataset(train_examples, validation_examples, test_examples, num_examples, map_fn, batch_size):
    ### START CODE HERE ###
    train_ds = train_examples.map(map_fn).shuffle(num_examples).batch(batch_size)
    ### END CODE HERE ###
    valid_ds = validation_examples.map(map_fn).batch(batch_size)
    test_ds = test_examples.map(map_fn).batch(batch_size)
    
    return train_ds, valid_ds, test_ds

train_ds, valid_ds, test_ds = prepare_dataset(train_examples, validation_examples, test_examples, num_examples, map_fn, BATCH_SIZE)

## TEST CODE:

test_train_ds = list(train_ds)
print(len(test_train_ds))
print(test_train_ds[0][0].shape)

del test_train_ds

Define the model

MODULE_HANDLE = 'data/resnet_50_feature_vector'
model = tf.keras.Sequential([
    hub.KerasLayer(MODULE_HANDLE, input_shape=(IMAGE_SIZE, IMAGE_SIZE, 3)),
    tf.keras.layers.Dense(num_classes, activation='softmax')
])
model.summary()

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
keras_layer (KerasLayer)     (None, 2048)              23561152  
_________________________________________________________________
dense (Dense)                (None, 2)                 4098      
=================================================================
Total params: 23,565,250
Trainable params: 4,098
Non-trainable params: 23,561,152
_________________________________________________________________

Define optimizer: (please complete these sections)

Define the Adam optimizer that is in the tf.keras.optimizers module.

def set_adam_optimizer():
    ### START CODE HERE ###
    # Define the adam optimizer
    optimizer = tf.keras.optimizers.Adam()
    ### END CODE HERE ###
    return optimizer

## TEST CODE:

test_optimizer = set_adam_optimizer()

print(type(test_optimizer))

del test_optimizer

<class 'tensorflow.python.keras.optimizer_v2.adam.Adam'>

Define the loss function (please complete this section)

Define the loss function as the sparse categorical cross entropy that's in the tf.keras.losses module. Use the same function for both training and validation.

def set_sparse_cat_crossentropy_loss():
    ### START CODE HERE ###
    # Define object oriented metric of Sparse categorical crossentropy for train and val loss
    train_loss = tf.keras.losses.SparseCategoricalCrossentropy()
    val_loss = tf.keras.losses.SparseCategoricalCrossentropy()
    ### END CODE HERE ###
    return train_loss, val_loss

## TEST CODE:

test_train_loss, test_val_loss = set_sparse_cat_crossentropy_loss()

print(type(test_train_loss))
print(type(test_val_loss))

del test_train_loss, test_val_loss

Define the acccuracy function (please complete this section)

Define the accuracy function as the spare categorical accuracy that's contained in the tf.keras.metrics module. Use the same function for both training and validation.

def set_sparse_cat_crossentropy_accuracy():
    ### START CODE HERE ###
    # Define object oriented metric of Sparse categorical accuracy for train and val accuracy
    train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy()
    val_accuracy = tf.keras.metrics.SparseCategoricalAccuracy()
    ### END CODE HERE ###
    return train_accuracy, val_accuracy

## TEST CODE:

test_train_accuracy, test_val_accuracy = set_sparse_cat_crossentropy_accuracy()

print(type(test_train_accuracy))
print(type(test_val_accuracy))

del test_train_accuracy, test_val_accuracy

Call the three functions that you defined to set the optimizer, loss and accuracy

optimizer = set_adam_optimizer()
train_loss, val_loss = set_sparse_cat_crossentropy_loss()
train_accuracy, val_accuracy = set_sparse_cat_crossentropy_accuracy()

Define the training loop (please complete this section)

In the training loop:

• Get the model predictions: use the model, passing in the input x
• Get the training loss: Call train_loss, passing in the true y and the predicted y.
• Calculate the gradient of the loss with respect to the model's variables: use tape.gradient and pass in the loss and the model's trainable_variables.
• Optimize the model variables using the gradients: call optimizer.apply_gradients and pass in a zip() of the two lists: the gradients and the model's trainable_variables.
• Calculate accuracy: Call train_accuracy, passing in the true y and the predicted y.

# this code uses the GPU if available, otherwise uses a CPU
device = '/gpu:0' if tf.config.list_physical_devices('GPU') else '/cpu:0'
EPOCHS = 2

# Custom training step
def train_one_step(model, optimizer, x, y, train_loss, train_accuracy):
    '''
    Trains on a batch of images for one step.
    
    Args:
        model (keras Model) -- image classifier
        optimizer (keras Optimizer) -- optimizer to use during training
        x (Tensor) -- training images
        y (Tensor) -- training labels
        train_loss (keras Loss) -- loss object for training
        train_accuracy (keras Metric) -- accuracy metric for training
    '''
    with tf.GradientTape() as tape:
    ### START CODE HERE ###
        # Run the model on input x to get predictions
        predictions = model(x)
        # Compute the training loss using `train_loss`, passing in the true y and the predicted y
        loss = train_loss(y,predictions)

    # Using the tape and loss, compute the gradients on model variables using tape.gradient
    grads = tape.gradient(loss, model.trainable_variables)
    
    # Zip the gradients and model variables, and then apply the result on the optimizer
    optimizer.apply_gradients(zip(grads, model.trainable_variables))

    # Call the train accuracy object on ground truth and predictions
    train_accuracy(y, predictions)
    ### END CODE HERE
    return loss

## TEST CODE:

def base_model():
    inputs = tf.keras.layers.Input(shape=(2))
    x = tf.keras.layers.Dense(64, activation='relu')(inputs)
    outputs = tf.keras.layers.Dense(1, activation='sigmoid')(x)
    model = tf.keras.Model(inputs=inputs, outputs=outputs)
    return model

test_model = base_model()

test_optimizer = set_adam_optimizer()
test_image = tf.ones((2,2))
test_label = tf.ones((1,))
test_train_loss, _ = set_sparse_cat_crossentropy_loss()
test_train_accuracy, _ = set_sparse_cat_crossentropy_accuracy()

test_result = train_one_step(test_model, test_optimizer, test_image, test_label, test_train_loss, test_train_accuracy)
print(test_result)

del test_result, test_model, test_optimizer, test_image, test_label, test_train_loss, test_train_accuracy

Define the 'train' function (please complete this section)

You'll first loop through the training batches to train the model. (Please complete these sections)

• The train function will use a for loop to iteratively call the train_one_step function that you just defined.
• You'll use tf.print to print the step number, loss, and train_accuracy.result() at each step. Remember to use tf.print when you plan to generate autograph code.

Next, you'll loop through the batches of the validation set to calculation the validation loss and validation accuracy. (This code is provided for you). At each iteration of the loop:

• Use the model to predict on x, where x is the input from the validation set.
• Use val_loss to calculate the validation loss between the true validation 'y' and predicted y.
• Use val_accuracy to calculate the accuracy of the predicted y compared to the true y.

Finally, you'll print the validation loss and accuracy using tf.print. (Please complete this section)

• print the final loss, which is the validation loss calculated by the last loop through the validation dataset.
• Also print the val_accuracy.result().

HINT If you submit your assignment and see this error for your stderr output:

Cannot convert 1e-07 to EagerTensor of dtype int64

Please check your calls to train_accuracy and val_accuracy to make sure that you pass in the true and predicted values in the correct order (check the documentation to verify the order of parameters).

# Decorate this function with tf.function to enable autograph on the training loop
@tf.function
def train(model, optimizer, epochs, device, train_ds, train_loss, train_accuracy, valid_ds, val_loss, val_accuracy):
    '''
    Performs the entire training loop. Prints the loss and accuracy per step and epoch.
    
    Args:
        model (keras Model) -- image classifier
        optimizer (keras Optimizer) -- optimizer to use during training
        epochs (int) -- number of epochs
        train_ds (tf Dataset) -- the train set containing image-label pairs
        train_loss (keras Loss) -- loss function for training
        train_accuracy (keras Metric) -- accuracy metric for training
        valid_ds (Tensor) -- the val set containing image-label pairs
        val_loss (keras Loss) -- loss object for validation
        val_accuracy (keras Metric) -- accuracy metric for validation
    '''
    step = 0
    loss = 0.0
    for epoch in range(epochs):
        for x, y in train_ds:
            # training step number increments at each iteration
            step += 1
            with tf.device(device_name=device):
                ### START CODE HERE ###
                # Run one training step by passing appropriate model parameters
                # required by the function and finally get the loss to report the results
                loss = train_one_step(model, optimizer, x, y, train_loss, train_accuracy)
                ### END CODE HERE ###
            # Use tf.print to report your results.
            # Print the training step number, loss and accuracy
            tf.print('Step', step, 
                   ': train loss', loss, 
                   '; train accuracy', train_accuracy.result())

        with tf.device(device_name=device):
            for x, y in valid_ds:
                # Call the model on the batches of inputs x and get the predictions
                y_pred = model(x)
                loss = val_loss(y, y_pred)
                val_accuracy(y, y_pred)
        
        # Print the validation loss and accuracy
        ### START CODE HERE ###
        tf.print('val loss', loss, '; val accuracy', val_accuracy.result())
        ### END CODE HERE ###

Run the train function to train your model! You should see the loss generally decreasing and the accuracy increasing.

Note: Please let the training finish before submitting and do not modify the next cell. It is required for grading. This will take around 5 minutes to run.

train(model, optimizer, EPOCHS, device, train_ds, train_loss, train_accuracy, valid_ds, val_loss, val_accuracy)

Step 1 : train loss 0.756068707 ; train accuracy 0.53125
Step 2 : train loss 0.510951698 ; train accuracy 0.65625
Step 3 : train loss 0.303860724 ; train accuracy 0.739583313
Step 4 : train loss 0.276302457 ; train accuracy 0.7890625
Step 5 : train loss 0.234697416 ; train accuracy 0.81875
Step 6 : train loss 0.167217016 ; train accuracy 0.84375
Step 7 : train loss 0.0925243944 ; train accuracy 0.866071403
Step 8 : train loss 0.0859617367 ; train accuracy 0.87890625
Step 9 : train loss 0.0440456383 ; train accuracy 0.892361104
Step 10 : train loss 0.0738788471 ; train accuracy 0.9
Step 11 : train loss 0.0402740166 ; train accuracy 0.909090936
Step 12 : train loss 0.0227937121 ; train accuracy 0.916666687
Step 13 : train loss 0.0204026215 ; train accuracy 0.923076928
Step 14 : train loss 0.0186813027 ; train accuracy 0.928571403
Step 15 : train loss 0.0119501781 ; train accuracy 0.933333337
Step 16 : train loss 0.0124373045 ; train accuracy 0.9375
Step 17 : train loss 0.00590448547 ; train accuracy 0.941176474
Step 18 : train loss 0.00546534359 ; train accuracy 0.944444418
Step 19 : train loss 0.0108059682 ; train accuracy 0.947368443
Step 20 : train loss 0.00294282753 ; train accuracy 0.95
Step 21 : train loss 0.00296216598 ; train accuracy 0.952380955
Step 22 : train loss 0.0023206952 ; train accuracy 0.954545438
Step 23 : train loss 0.0078078229 ; train accuracy 0.956521749
Step 24 : train loss 0.00188160746 ; train accuracy 0.958333313
Step 25 : train loss 0.00315087754 ; train accuracy 0.96
Step 26 : train loss 0.00336355204 ; train accuracy 0.961070538
val loss 0.00343822781 ; val accuracy 1
Step 27 : train loss 0.00155152602 ; train accuracy 0.962529302
Step 28 : train loss 0.00226262351 ; train accuracy 0.963882625
Step 29 : train loss 0.00430524489 ; train accuracy 0.965141594
Step 30 : train loss 0.00471592275 ; train accuracy 0.966315806
Step 31 : train loss 0.00463974569 ; train accuracy 0.967413425
Step 32 : train loss 0.00102221372 ; train accuracy 0.968441844
Step 33 : train loss 0.00278860074 ; train accuracy 0.96940726
Step 34 : train loss 0.00134344574 ; train accuracy 0.970315397
Step 35 : train loss 0.00547727 ; train accuracy 0.9711712
Step 36 : train loss 0.00128802785 ; train accuracy 0.971978962
Step 37 : train loss 0.00112945435 ; train accuracy 0.972742736
Step 38 : train loss 0.00118560134 ; train accuracy 0.973466
Step 39 : train loss 0.00248970464 ; train accuracy 0.97415185
Step 40 : train loss 0.00375873619 ; train accuracy 0.97480315
Step 41 : train loss 0.00164634362 ; train accuracy 0.975422442
Step 42 : train loss 0.000895871199 ; train accuracy 0.976012
Step 43 : train loss 0.00087104633 ; train accuracy 0.976573944
Step 44 : train loss 0.00175065256 ; train accuracy 0.977110147
Step 45 : train loss 0.00780364731 ; train accuracy 0.97762239
Step 46 : train loss 0.00339399581 ; train accuracy 0.978112161
Step 47 : train loss 0.000925442728 ; train accuracy 0.978581
Step 48 : train loss 0.0367161259 ; train accuracy 0.978374839
Step 49 : train loss 0.00101024506 ; train accuracy 0.978819
Step 50 : train loss 0.00213793316 ; train accuracy 0.979245305
Step 51 : train loss 0.00154432189 ; train accuracy 0.979654729
Step 52 : train loss 0.00122441642 ; train accuracy 0.979927
val loss 0.00184391928 ; val accuracy 1

Evaluation

You can now see how your model performs on test images. First, let's load the test dataset and generate predictions:

test_imgs = []
test_labels = []

predictions = []
with tf.device(device_name=device):
    for images, labels in test_ds:
        preds = model(images)
        preds = preds.numpy()
        predictions.extend(preds)

        test_imgs.extend(images.numpy())
        test_labels.extend(labels.numpy())

Let's define a utility function for plotting an image and its prediction.

# Utilities for plotting

class_names = ['horse', 'human']

def plot_image(i, predictions_array, true_label, img):
    predictions_array, true_label, img = predictions_array[i], true_label[i], img[i]
    plt.grid(False)
    plt.xticks([])
    plt.yticks([])

    img = np.squeeze(img)

    plt.imshow(img, cmap=plt.cm.binary)

    predicted_label = np.argmax(predictions_array)
    
    # green-colored annotations will mark correct predictions. red otherwise.
    if predicted_label == true_label:
        color = 'green'
    else:
        color = 'red'
    
    # print the true label first
    print(true_label)
  
    # show the image and overlay the prediction
    plt.xlabel("{} {:2.0f}% ({})".format(class_names[predicted_label],
                                100*np.max(predictions_array),
                                class_names[true_label]),
                                color=color)

Plot the result of a single image

Choose an index and display the model's prediction for that image.

# Visualize the outputs 

# you can modify the index value here from 0 to 255 to test different images
index = 8 
plt.figure(figsize=(6,3))
plt.subplot(1,2,1)
plot_image(index, predictions, test_labels, test_imgs)
plt.show()