Skip to main content

Add Fully-Connected Layer

Synopsis​

Adds a fully-connected layer to your neural net structure.

Description​

This operator has to be placed into the subprocess of the Deep Learning, Deep Learning (Tensor) or Autoencoder operator. It adds a fully-connected layer to the neural net structure.

A fully-connected layer consists of neurons. The number of neurons to use can be set using the number of neurons parameter. A neuron can be seen as a new attribute that takes into account information from all neurons of the previous layer. This new attribute is calculated by multiplying the current weight of a neuron with the input value and adding a bias value. Afterwards the so-called activation function is applied to check whether an input should be taken into account or not.

Most of the time the standard "ReLU" activation function provides the best results.

The weight and bias values are optimized by the network and setup with a starting value at the beginning of the training process. In general, the weight and bias values are set using a unified method for all layers. This method is chosen in the Deep Learning operator through the weight inititialization and bias value parameters. These values can be overwritten by selecting the overwrite networks weight initialization or overwrite networks bias initialization parameter.

Furthermore, it is possible to apply a regularization technique called Dropout to each layer. This option is recommended in order to avoid overtraining. Therefore, select the use dropout parameter and set a dropout ratio. With this option enabled neurons are ignored randomly (based on the chosen ratio).

The dropout rate can also be disabled for this layer and set using the "Add Dropout Layer".

Input​

layerArchitecture​

A network configuration setup with previous operators. Connect this port to the layerArchitecture output port of another add layer operator or to the layer port of the "Deep Learning" operator if this layer is the first one.

Output​

layerArchitecture​

The network with the configuration for this fully-connected layer added. Connect this port to the next input port of another layer or the layer port on the right side of the "Deep Learning" operator.

Parameters​

Neurons​

Provide the number of neurons used in this layer. A neuron can be seen as a new attribute that takes into account information from all neurons of the previous layer. This new attribute is calculated by multiplying the current weight of a neuron with the input value and adding a bias value. Afterwards the so-called activation function is applied to check whether an input should be taken into account or not.

In general the number of neurons shouldn't be to big. The more neurons are added, the more weight and bias values need to be optimized, resulting in a more complex model. A rule of thumb that might apply for your problem is to choose a number smaller than the number of input attributes minus the number of output neurons. E.g. for a data set with 100 attributes and 3 class values, the number of neurons present in the network should not exceed 100-3=97. Keep in mind this rule might not apply to your problem, but it might help you getting started.

Activation function​

Activation functions allow networks to create complex nonlinear decision boundaries. Mathematically speaking the chosen activation function is wrapped around the result of multiplying weights to input data and adding the bias. Hence activation functions ensure that a layers output is within a given range and a general decision whether to use the output or not can be made.

Because these none linear functions increase the computational load during training, choosing a simple function (with a monotonic derivative) is recommended for many situations.

Choosing the activation function for the last layer of a network is slightly different from previous layers. At this point the activation functions provides a conversion from the internal network state to the awaited output. For regression tasks "None (identity)" might be chosen, while for classification problems "Softmax" converts the results to probabilities for the given class values.

  • ReLU (Rectified Linear Unit): Rectified linear unit. Activation function is max(0, x). Monotonic derivative.
  • Sigmoid: Sigmoid or logistic function. None monotonic derivative. Sensitive to small changes in the present data. Results are in the range between 0 and 1.
  • Softmax: Softmax or normalized exponential function. Resulting values are in a range between 0 and 1, while adding up to one. Hence this function can be used to map values to probability like values.
  • TanH: TanH function, similar to the sigmoid function. None monotonic derivative with values in the range -1 and +1.
  • Cube: Cubic function. Output is the cubic of input values. https://cs.stanford.edu/people/danqi/papers/emnlp2014.pdf
  • ELU (Exponential Linear Unit): Same as ReLU for values above zero, but an exponential function below. Hence the derivative is only monotonic for values above zero.
  • GELU (Gaussian Error Linear Unit): Gaussian Error Linear Unit. Activation function is x * Phi(x), with Phi(x) as the standard Gaussian cumulative distribution function. Difference to ReLU: input is weighted based on its value instead of its sign. https://arxiv.org/abs/1606.08415 Sigmoid version of the implementation is used.
  • MISH: A self-regularized non-monotonic activation function. Activation function is x tanh (ln(1 + exp(x))). https://arxiv.org/abs/1908.08681 Sigmoid version of the implementation is used.
  • Leaky ReLU: Same as ReLU for values above zero, but with a linear function for values below. Monotonic derivative.
  • Rational TanH: Rational TanH approximation, element-wise function.
  • Randomized ReLU: Similar to ReLU but with a randomly chosen scaling factor for the linearity. Monotonic derivative.
  • Rectified TanH: Similar to ReLU, but with a TanH function for positive values instead of a linearity. None monotonic derivative.
  • Softplus: A logarithmic function with values ranging vom zero to infinity. Monotonic derivative.
  • Softsign: Similar to TanH with same range and monotonicity but less prone to changes.
  • SELU (Scaled ELU): Scaled exponential linear unit. Similar to ELU, but with a scaling factor. None monotonic derivative. https://arxiv.org/pdf/1706.02515.pdf
  • None (identity): Output equals input. This function can be used e.g. within the last layer of a network to obtain a regression result. Monotonic derivative.

Layer name​

Provide a name for the layer for ease of identification, when inspecting the model or re-using it.

Use dropout​

Enable dropout regularization for this layer.

Dropout rate​

Define a rate between 0 and 1 that is used to randomly remove a neuron of this layer during training. This is only applied during training and helps to reduce overtraining.

Overwrite networks weight initialization​

Enabling this parameter allows to choose a weight initialization method for this layer, that is different from the general networks setting.

Weight initialization​

A Deep Learning model is defined by so called weights. Weights are set within most layers and define the model. The process of finding the best weight values during training is an iterative process and requires start values. Weight values are multiplied to respective input data. At the first layer the input data is the data provided at the training port of the Deep Learning operator. For successive layers weights are multiplied to the output of the previous layer. Select one of the provided pre-defined methods to initialize all weights by the given strategy. Change this parameter, if during training the score is not decreasing or it takes a long time before the scores values goes down.

  • Identity: Use identity matrices.
  • Normal: Use a Gaussian distribution with a mean of zero and a standard deviation of 1 / sqrt(number of layer inputs).
  • Ones: Use ones. This is rarely a good idea.
  • ReLU: Use a Gaussian distribution with a mean of zero and a variance of 2 / (number of layer inputs).
  • ReLU Uniform: Use a Uniform distribution from -a to a, where a = sqrt(6/(number of layer inputs)).
  • Sigmoid Uniform: Use a Uniform distribution from -a to a, where a = sqrt(6 / (number of layer inputs + number of layer outputs)).
  • Uniform: Use a Uniform distribution from -a to a, where a = 1 / sqrt(number of layer inputs).
  • Xavier: Use a Gaussian distribution with a mean of zero and a variance of 2 / (number of layer inputs + number of layer outputs).
  • Xavier Uniform: Use a Uniform distribution from -a to a, where a = sqrt(6/(number of layer inputs + number of layer outputs)).
  • Zero: Initialize all weights with zero. This is rarely a good idea.

Overwrite networks bias initialization​

Enabling this parameter allows to choose a bias initialization value for this layer, that is differnt from the general networks setting.

Bias initialization​

As described for the weight initialization method parameter, a Deep Learning model needs starting values for the training process. While the weights are multiplied to input data, the bias values are added ontop of this product. When training a regression model, for a data set with a mean target value of 10, starting with a bias initialization value of 10 could enable a network to find a fitting bias value more quickly.