Sigsoftmax: Reanalysis of the Softmax Bottleneck

This paper was kind of overlooked last year (even thought it showed up on NIPS) but it caught my attention.

The paper suggests replacing the usual softmax output layer for classification (the paper makes this suggestions for NLP problems but the theory stands for all domains with large number of classes) because its very design produces an information bottleneck.

I have yet to see this applied to computer vision problems but I will implement it  in Tensorflow or Caffe and report back.

It's essence boils down to the following excerpts :



The paper then goes to suggest the properties of a replacement activation layer:



and the final outcome is this:



which satisfied the properties defined earlier

Visualizing frozen models in tensorflow and tensorboard

In tensorflow frozen inference graphs are normal graphs with their variables turned to constants and some training layers stripped away. Very often we get these binary files in protobuf (pb) and we want to check them out.

A fast way to do it is by using tensorboard and the tensorflow tool called import_pb_to_tensorboard.py

To use you need to clone the tensorflow repo locally.

First you convert the model into tensorboard event log

 python ~/tensorflow/tensorflow/python/tools/import_pb_to_tensorboard.py --model_dir PATH_TO_PB_FILE --log_dir PATH_LOG_DIR

afterwards you run tensorboard to visualize it

tensorboard --logdir=visualization:PATH_TO_LOG_DIR