Keras is a great way to try new ideas in deep learning. It is also a great way to learn the current state of the art because the syntax is so simple. As I was trying out new activation functions, I found that their documentation was a bit lacking for visualizations. Many of the activation functions lack an equation or visualization, so here is my shot at understanding them.

In [1]:
%matplotlib inline
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.layers import Dense, LeakyReLU, PReLU, ThresholdedReLU
import numpy as np

Using TensorFlow backend.
In [2]:
def CreateModelWithActivation(act):
    mdl = Sequential();
    if type(act) is str:
        mdl.add(Dense(1, input_shape=[1], activation=act, use_bias = False));
    else:
        mdl.add(Dense(1, input_shape=[1], use_bias = False));
        mdl.add(act())
    mdl.compile(loss='mse', optimizer='SGD')
    unitWeight = [np.array(1).reshape((1,1))];
    mdl.layers[0].set_weights(unitWeight);
    return mdl
In [3]:
def PlotActivationFunction(act):
    model = CreateModelWithActivation(act)
    x = np.linspace(-5,5,100)
    plt.plot(x,model.predict(x))
    plt.grid(True)
    plt.xlabel('Input')
    plt.ylabel('Output')

Activation functions for Keras, in order of their documentation found here: https://keras.io/layers/advanced-activations/

Note on softmax : softmax only makes sense in the use of multiple outputs, as each output is relative to the others (see first equation in https://en.wikipedia.org/wiki/Softmax_function). Because I am creating simple input-output plots, I am not going to attempt to visualize that :(

In [4]:
PlotActivationFunction('elu')
In [5]:
PlotActivationFunction('selu')
In [6]:
PlotActivationFunction('softplus')
In [7]:
PlotActivationFunction('softsign')
In [8]:
PlotActivationFunction('relu')
In [9]:
PlotActivationFunction('tanh')
In [10]:
PlotActivationFunction('sigmoid')
In [11]:
PlotActivationFunction('hard_sigmoid')

Advanced Activation Functions: https://keras.io/layers/advanced-activations/

In [12]:
PlotActivationFunction(PReLU)
In [13]:
PlotActivationFunction(LeakyReLU)
In [14]:
PlotActivationFunction(ThresholdedReLU)