Fix Tensorflow Object Detection Framework taking too much disk space while training

The model_lib_tf2.py file contains all the training and evaluation loops.

In the function:

def eager_train_step(detection_model,
                     features,
                     labels,
                     unpad_groundtruth_tensors,
                     optimizer,
                     learning_rate,
                     add_regularization_loss=True,
                     clip_gradients_value=None,
                     global_step=None,
                     num_replicas=1.0):

We can see that every training iteration it saves a few training dataset images.

tf.compat.v2.summary.image(
    name='train_input_images',
    step=global_step,
    data=features[fields.InputDataFields.image],
    max_outputs=5)

There are three problems with this:

1. It takes A LOT of space

2. It actually slows up training 

3. Images look saturated.

We can fix this easily by replacing the above snippet with this:

if global_step % 100 == 0:
    # --- get images and normalize them
    images_normalized = \
        (features[fields.InputDataFields.image] + 128.0) / 255.0
    tf.compat.v2.summary.image(
        name='train_input_images',
        step=global_step,
        data=images_normalized,
        max_outputs=5)