9.1 LSTM architecture and gating mechanisms

LSTMs are powerful recurrent neural networks designed to handle long-term dependencies in sequential data. Their unique architecture includes a cell state and three gates that control information flow, allowing the network to selectively remember or forget information over time.

The LSTM cell's key components are the input, forget, and output gates, which work together to manage the cell state. This structure enables LSTMs to learn and retain important information across long sequences, making them effective for tasks like language modeling and time series prediction.

LSTM Architecture

Components of LSTM cells

Top images from around the web for Components of LSTM cells

• 

  Machine learning de séries temporais – LSTM – Monolito Nimbus View original

  Is this image relevant?

• 

  Long Short-Term Memory View original

  Is this image relevant?

• 

  Long Short-Term Memory View original

  Is this image relevant?

• 

  Machine learning de séries temporais – LSTM – Monolito Nimbus View original

  Is this image relevant?

• 

  Long Short-Term Memory View original

  Is this image relevant?

1 of 3

Top images from around the web for Components of LSTM cells

• 

  Machine learning de séries temporais – LSTM – Monolito Nimbus View original

  Is this image relevant?

• 

  Long Short-Term Memory View original

  Is this image relevant?

• 

  Long Short-Term Memory View original

  Is this image relevant?

• 

  Machine learning de séries temporais – LSTM – Monolito Nimbus View original

  Is this image relevant?

• 

  Long Short-Term Memory View original

  Is this image relevant?

1 of 3

• LSTM cell structure consists of cell state and three gates acting as neural networks with sigmoid activation functions
• Input gate controls addition of new information to cell state using sigmoid and tanh layers
• Forget gate discards information from cell state using sigmoid layer outputting values between 0 and 1
• Output gate determines cell state information to output combining sigmoid layer with tanh operation

Gating mechanisms in LSTMs

• Gating mechanism uses element-wise multiplication to control information flow with gates outputting values between 0 and 1
• Input gate filters candidate values created by tanh layer determining cell state updates
• Forget gate multiplies previous cell state by its output allowing selective forgetting of irrelevant information
• Output gate filters cell state through tanh function controlling exposed parts to next time step

Role of cell state

• Cell state acts as LSTM memory flowing through entire chain with minimal linear interactions
• Allows relevant information to flow unchanged through many time steps mitigating vanishing gradient problem
• Enables selective updates by adding and removing information facilitating long-term dependency learning
• Provides direct path for gradients to flow backwards through time facilitating long-range temporal dependency learning

Implementation of LSTM cells

• LSTM cell equations:
  1. Forget gate: $f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)$
  2. Input gate: $i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i)$
  3. Candidate values: $\tilde{C}_t = \tanh(W_C \cdot [h_{t-1}, x_t] + b_C)$
  4. Cell state update: $C_t = f_t * C_{t-1} + i_t * \tilde{C}_t$
  5. Output gate: $o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o)$
  6. Hidden state: $h_t = o_t * \tanh(C_t)$
• Weight matrices $W_f$, $W_i$, $W_C$, and $W_o$ for forget, input, cell, and output gates with dimensions based on input and hidden state sizes
• Bias vectors $b_f$, $b_i$, $b_C$, and $b_o$ correspond to each gate
• Activation functions: sigmoid ($\sigma$) for gates (0 to 1 output) and tanh for cell state and candidate values (-1 to 1 output)
• Input concatenation combines previous hidden state $h_{t-1}$ with current input $x_t$
• Initialization uses Xavier/Glorot for weight matrices and zero or small positive values for bias vectors