In mathematics and machine learning, the softplus function is
The names softplus [1] [2] and SmoothReLU [3] are used in machine learning.
It is a smooth approximation (in fact, an analytic function) to the ramp function, which is known as the rectifier or ReLU in machine learning. For large negative it is , so just above 0, while for large positive it is , so just above .
The derivative of softplus is the logistic function:
The logistic sigmoid function is a smooth approximation of the derivative of the rectifier, the Heaviside step function.
The multivariable generalization of single-variable softplus is the LogSumExp with the first argument set to zero:
The LogSumExp function is
and its gradient is the softmax; the softmax with the first argument set to zero is the multivariable generalization of the logistic function. Both LogSumExp and softmax are used in machine learning.
The convex conjugate (specifically, the Legendre transform) of the softplus function is the negative binary entropy (with base e). This is because (following the definition of the Legendre transform: the derivatives are inverse functions) the derivative of softplus is the logistic function, whose inverse function is the logit, which is the derivative of negative binary entropy.
Softplus can be interpreted as logistic loss (as a positive number), so by duality, minimizing logistic loss corresponds to maximizing entropy. This justifies the principle of maximum entropy as loss minimization.
This function can be approximated as:
By making the change of variables , this is equivalent to
A sharpness parameter may be included:
Since the sigmoid h has a positive first derivative, its primitive, which we call softplus, is convex.
Rectifier and softplus activation functions. The second one is a smooth version of the first.
In mathematics and machine learning, the softplus function is
The names softplus [1] [2] and SmoothReLU [3] are used in machine learning.
It is a smooth approximation (in fact, an analytic function) to the ramp function, which is known as the rectifier or ReLU in machine learning. For large negative it is , so just above 0, while for large positive it is , so just above .
The derivative of softplus is the logistic function:
The logistic sigmoid function is a smooth approximation of the derivative of the rectifier, the Heaviside step function.
The multivariable generalization of single-variable softplus is the LogSumExp with the first argument set to zero:
The LogSumExp function is
and its gradient is the softmax; the softmax with the first argument set to zero is the multivariable generalization of the logistic function. Both LogSumExp and softmax are used in machine learning.
The convex conjugate (specifically, the Legendre transform) of the softplus function is the negative binary entropy (with base e). This is because (following the definition of the Legendre transform: the derivatives are inverse functions) the derivative of softplus is the logistic function, whose inverse function is the logit, which is the derivative of negative binary entropy.
Softplus can be interpreted as logistic loss (as a positive number), so by duality, minimizing logistic loss corresponds to maximizing entropy. This justifies the principle of maximum entropy as loss minimization.
This function can be approximated as:
By making the change of variables , this is equivalent to
A sharpness parameter may be included:
Since the sigmoid h has a positive first derivative, its primitive, which we call softplus, is convex.
Rectifier and softplus activation functions. The second one is a smooth version of the first.