Skip to main content

Add LSTM Layer

Synopsis​

Adds a long short term memory layer to your neural net structure.

Description​

This operator has to be placed into the subprocess of the Deep Learning, Deep Learning (Tensor) or Autoencoder operator. It adds a long short term memory layer to the neural net structure.

A long short term memory layer uses the concept of various hidden state types to adjust the amount of information stored across so called states. This can be used e.g. when working on sequential data (like time-series or text), since the hidden states can store a given amount of information from previous states beyond the just handled one. This means, that for example a connection between the first word of a long text and the last word can be created even though the complete paragraph is quite long. To account for the importance of in-between states long short term memory layers use mechanisms to adjust the importance and amount of influence a hidden state has for the current calculation.

Input​

layerArchitecture​

A network configuration setup with previous operators. Connect this port to the layerArchitecture output port of another add layer operator or to the layer port of the "Deep Learning" operator if this layer is the first one.

Output​

layerArchitecture​

The network with the configuration for this fully-connected layer added. Connect this port to the next input port of another layer or the layer port on the right side of the "Deep Learning" operator.

Parameters​

Neurons​

Provide the number of neurons used in this layer. A neuron can be seen as a new attribute that takes into account information from all neurons of the previous layer. This new attribute is calculated by multiplying the current weight of a neuron with the input value and adding a bias value. Afterwards the so-called activation function is applied to check whether an input should be taken into account or not.

For long short term memory layers each neuron creates hidden states depending on the sequence length of the input data. Four types of gates are used to control the information flux between different states to e.g. account for the importance of a given state. This is mostly done automatically but can be influenced using the forget gate initialization bias.

Activation function​

Activation functions allow networks to create complex nonlinear decision boundaries. Mathematically speaking the chosen activation function is wrapped around the result of multiplying weights to input data and adding the bias. Hence activation functions ensure that a layers output is within a given range and a general decision whether to use the output or not can be made.

Because these none linear functions increase the computational load during training, choosing a simple function (with a monotonic derivative) is recommended for many situations.

Choosing the activation function for the last layer of a network is slightly different from previous layers. At this point the activation functions provides a conversion from the internal network state to the awaited output. For regression tasks "None (identity)" might be chosen, while for classification problems "Softmax" converts the results to probabilities for the given class values.

  • ReLU (Rectified Linear Unit): Rectified linear unit. Activation function is max(0, x). Monotonic derivative.
  • Sigmoid: Sigmoid or logistic function. None monotonic derivative. Sensitive to small changes in the present data. Results are in the range between 0 and 1.
  • Softmax: Softmax or normalized exponential function. Resulting values are in a range between 0 and 1, while adding up to one. Hence this function can be used to map values to probability like values.
  • TanH: TanH function, similar to the sigmoid function. None monotonic derivative with values in the range -1 and +1.
  • Cube: Cubic function. Output is the cubic of input values. https://cs.stanford.edu/people/danqi/papers/emnlp2014.pdf
  • ELU (Exponential Linear Unit): Same as ReLU for values above zero, but an exponential function below. Hence the derivative is only monotonic for values above zero.
  • GELU (Gaussian Error Linear Unit): Gaussian Error Linear Unit. Activation function is x * Phi(x), with Phi(x) as the standard Gaussian cumulative distribution function. Difference to ReLU: input is weighted based on its value instead of its sign. https://arxiv.org/abs/1606.08415 Sigmoid version of the implementation is used.
  • MISH: A self-regularized non-monotonic activation function. Activation function is x tanh (ln(1 + exp(x))). https://arxiv.org/abs/1908.08681 Sigmoid version of the implementation is used.
  • Leaky ReLU: Same as ReLU for values above zero, but with a linear function for values below. Monotonic derivative.
  • Rational TanH: Rational TanH approximation, element-wise function.
  • Randomized ReLU: Similar to ReLU but with a randomly chosen scaling factor for the linearity. Monotonic derivative.
  • Rectified TanH: Similar to ReLU, but with a TanH function for positive values instead of a linearity. None monotonic derivative.
  • Softplus: A logarithmic function with values ranging vom zero to infinity. Monotonic derivative.
  • Softsign: Similar to TanH with same range and monotonicity but less prone to changes.
  • SELU (Scaled ELU): Scaled exponential linear unit. Similar to ELU, but with a scaling factor. None monotonic derivative. https://arxiv.org/pdf/1706.02515.pdf
  • None (identity): Output equals input. This function can be used e.g. within the last layer of a network to obtain a regression result. Monotonic derivative.

Layer name​

Provide a name for the layer for ease of identification, when inspecting the sequentialModel or re-using it.

Forget gate initialization bias​

Provide a number, often between 1 and 5, to potentially ease the learning process for long-term dependencies. This values is used as a bias for the internal forget gate, influencing the behaviour of keeping information in a hidden state or not.