How ReLU Allows Neural Networks to Approximate Steady Nonlinear Features? | by Thi-Lam-Thuy LE | Jan, 2024

Activation features play an integral function in Neural Networks (NNs) since they introduce non-linearity and permit the community to be taught extra complicated options and features than only a linear regression. Probably the most generally used activation features is Rectified Linear Unit (ReLU), which has been theoretically proven to allow NNs to approximate a variety of steady features, making them highly effective perform approximators.

On this put up, we research specifically the approximation of Steady NonLinear (CNL) features, the primary function of utilizing a NN over a easy linear regression mannequin. Extra exactly, we examine 2 sub-categories of CNL features: Steady PieceWise Linear (CPWL), and Steady Curve (CC) features. We are going to present how these two perform varieties might be represented utilizing a NN that consists of 1 hidden layer, given sufficient neurons with ReLU activation.

For illustrative functions, we contemplate solely single function inputs but the thought applies to a number of function inputs as effectively.

ReLU is a piecewise linear perform that consists of two linear items: one which cuts off unfavorable values the place the output is zero, and one that gives a steady linear mapping for non unfavorable values.

CPWL features are steady features with a number of linear parts. The slope is constant on every portion, than modifications abruptly at transition factors by including new linear features.

In a NN with one hidden layer utilizing ReLU activation and a linear output layer, the activations are aggregated to kind the CPWL goal perform. Every unit of the hidden layer is chargeable for a linear piece. At every unit, a brand new ReLU perform that corresponds to the altering of slope is added to provide the brand new slope (cf. Fig.2). Since this activation perform is at all times constructive, the weights of the output layer comparable to items that improve the slope shall be constructive, and conversely, the weights comparable to items that decreases the slope shall be unfavorable (cf. Fig.3). The brand new perform is added on the transition level however doesn’t contribute to the ensuing perform previous to (and typically after) that time because of the disabling vary of the ReLU activation perform.

Instance

To make it extra concrete, we contemplate an instance of a CPWL perform that consists of 4 linear segments outlined as under.

To characterize this goal perform, we are going to use a NN with 1 hidden layer of 4 items and a linear layer that outputs the weighted sum of the earlier layer’s activation outputs. Let’s decide the community’s parameters so that every unit within the hidden layer represents a phase of the goal. For the sake of this instance, the bias of the output layer (b2_0) is ready to 0.

The subsequent kind of steady nonlinear perform that we’ll research is CC perform. There may be not a correct definition for this sub-category, however an off-the-cuff method to outline CC features is steady nonlinear features that aren’t piecewise linear. A number of examples of CC features are: quadratic perform, exponential perform, sinus perform, and so forth.

A CC perform might be approximated by a sequence of infinitesimal linear items, which known as a piecewise linear approximation of the perform. The higher the variety of linear items and the smaller the scale of every phase, the higher the approximation is to the goal perform. Thus, the identical community structure as beforehand with a big sufficient variety of hidden items can yield good approximation for a curve perform.

Nevertheless, in actuality, the community is skilled to suit a given dataset the place the input-output mapping perform is unknown. An structure with too many neurons is vulnerable to overfitting, excessive variance, and requires extra time to coach. Due to this fact, an acceptable variety of hidden items should not be too small to correctly match the info, nor too massive to result in overfitting. Furthermore, with a restricted variety of neurons, approximation with low loss has extra transition factors in restricted area, reasonably than equidistant transition factors in an uniform sampling approach (as proven in Fig.10).

On this put up, we have now studied how ReLU activation perform permits a number of items to contribute to the ensuing perform with out interfering, thus permits steady nonlinear perform approximation. As well as, we have now mentioned concerning the selection of community structure and variety of hidden items with the intention to receive approximation consequence.

I hope that this put up is helpful on your Machine Studying studying course of!

Additional questions to consider:

1. How does the approximation skill change if the variety of hidden layers with ReLU activation improve?
2. How ReLU activations are used for a classification downside?

*Except in any other case famous, all photos are by the writer