Gated Recurrent Unit (GRU): Simplifying Sequential Data Learning

In deep learning, it was believed that the more layers we implemented, the result should have been that much more accurate. It changed when we noticed that after a certain point, the accuracy of the model decreased when more layers were introduced. We call this the vanishing gradient problem. The gated recurrent unit helped tackle this problem. Let me walk you through the basics of GRU so you can also use this method in your next project.

Over time, I learned how GRUs use gating methods to control the flow of information throughout the network, helping reduce the vanishing gradient problem. The latter commonly occurs in deep RNNs and hampers learning long-range dependencies. Gated recurrent units changed deep learning forever, making it a staple in our industry.

What is a gated recurrent unit?

Let me start by giving you a brief idea about GRU in this tutorial on GRU neural network. A gated recurrent unit (GRU) is a (RNN) recurrent neural network architecture that aims to solve the problems of RNNs, which include vanishing gradient problems and difficulty getting long-term dependency. Kyunghyun Cho et al in 2014 introduced GRUs.

GRUs are similar to Long Short-Term Memory (LSTM) networks in that they use gating mechanisms to control the flow of information inside the network. However, GRUs have a simpler architecture with fewer gating units, resulting in higher computational efficiency.

Components in gated recurrent unit architecture

Let's discuss the components used in the gated recurrent unit to better understand how it works. Here are the components of GRU.

Input

At each time step (t), the GRU receives an input vector containing the current data point in the series.

Previous hidden state

The GRU also accepts the previous hidden state as input, which provides information from the previous time step that helps detect dependencies over time.

Update gate

The update gate controls how much of the previous concealed state is kept and how much new information is added to the current state. It is calculated as Z(t)= σ(Wz . [h(t-1) . x(t)]. Here, the weight matrix for the update gate is 𝑊_𝑧. The sigmoid activation function is σ, and the concatenation of the previous hidden state and the current input is [ℎ 𝑡 − 1 , 𝑥 𝑡 ].

Reset gate

The reset gate controls how much of the previous concealed state should be forgotten. It is calculated as r_t = σ(W_r ⋅[h t−1,x t]). The weight matrix for the reset gate is denoted as 𝑊_𝑟.

Candidate memory content

The GRU creates potential hidden states based on the reset gate (𝑟𝑡). It's calculated as:

h(t) = tanh(Wh) ⋅[r × t ⊙h( t−1),x ( t)]. Here, 𝑊_ℎ​ is the weight matrix for the candidate memory content, tanh is the hyperbolic tangent activation function, ⊙ denotes element-wise multiplication, and [ 𝑟 𝑡 ⊙ ℎ(𝑡 − 1), 𝑥( 𝑡) ] [r t​⊙h( t−1​),x( t)​] represents the concatenation of the reset gate applied to the previous hidden state and the current input.

New hidden state

Finally, the new hidden state h(t) is calculated by mixing the preceding concealed state. h(t-1) and the candidate memory content using the update gate z t​: ℎ 𝑡 = ( 1 − 𝑧 𝑡 ) ⊙ ℎ 𝑡 − 1 + 𝑧 𝑡 ⊙ ℎ ~ 𝑡 h t​=(1−z t​)⌙h t−1​+z t​⊙ h(t).

How does the GRU in deep learning model work?

GRU is like other recurrent neural network architectures. It analyzes sequential data one element at a time, changing its hidden state using both the current and prior inputs. At each time step(t), the GRU generates a "candidate activation vector" by combining information from the input and the prior hidden state. This candidate vector is then used to update the concealed state in the following time step.

The Update and reset gates are used to calculate the candidate activation vector. The reset gate chooses how much of the prior concealed state it should forget. The update gate, on the other hand, decides how much of the candidate activation to insert in the current hidden state.

Let me explain the maths behind this whole spectacle.

• The reset gate r is calculated using the current input x along with the previous hidden state h(t-1). The update gate is calculated at the same time.

r(t) = σ(W(r) * [h(t-1), x(t)])

z(t) = σ(W(z) * [h(t-1), x(t)])

In the above expression, W(r) and W(z) are called weight matrices. They are evaluated during the training of the neural network.

• The candidate activation vector h(t)~ is generated using the current input x and a modified version of the prior hidden state, which is "reset" by the reset gate.

h(t)~ = tanh( W(h) * [r(t) * h(t-1), x(t)] )

Here, W(h) is another weight matrix.

• The candidate activation vector is combined with the previous hidden state, weighted by the update gate, to compute the new hidden state, h(t).

h(t) = (1 - z(t) ) * h(t-1) + z(t ) * h(t)~

The end result is a compact architecture that can selectively update its hidden state based on the input and prior hidden state. This eliminates the need for a separate memory cell state, which is used in LSTM (Long Short-Term Memory).

Input processing

At each time step 𝑡, the GRU gets an input vector 𝑥, which represents the current data point in the series.

Previous hidden gate

The function accepts the previous concealed state as input (ℎ, 𝑡, -1). It contains information from the preceding time step and aids in the identification of temporal dependencies.

Update gate

The GRU generates an update gate (z t) to determine how much of the prior hidden state [ℎ (𝑡 - 1)] should be kept and how much new information should be added.

Reset gate

The GRU computes a reset gate r(t). This reset gate determines how much of the previous step should be forgotten.

Candidate memory content

After this, the GRU uses the reset gates to create candidate memory content. This is the new potential hidden state. This is based on the current input and previous hidden state.

New hidden state

After all this, the GRU computes the new hidden state h(t-1) and the candidate memory content (h ~ t​) using the update gate.

Advantages and disadvantages of gated recurrent unit in deep learning

GRU networks have their own set of advantages and disadvantages. Let me discuss them one by one.

The advantages of the GRU neural network are as follows:

• Efficient training: GRUs are computationally more efficient than other recurrent architectures, such as LSTMs, because their gating mechanism is simpler, resulting in fewer parameters and calculations.
• Long-term dependencies: GRUs, like LSTMs, are designed to capture long-range dependencies in sequential data, making them useful for tasks that require context across longer sequences, such as natural language processing and time series analysis.
• Gradient flow: The gating mechanism in GRUs helps to mitigate the vanishing gradient problem, allowing for more stable and successful deep recurrent network training.
• Avoiding overfitting: GRUs are less prone than LSTMs in some cases to get overfitting due to their simpler architecture. This results in greater generalization on smaller datasets. As a result, a greater generalization is achieved in smaller datasets.
• Easy implementation: GRUs are easier to implement than more sophisticated designs such as LSTMs, making them a viable option for a variety of sequential data tasks.

Now, let me discuss some of the disadvantages of GRU (gated recurrent unit):

• Limited capacity: GRUs have a simpler structure than LSTM networks and use fewer gating mechanisms. This simplicity may limit their ability to record highly intricate connections in long sequences.
• Problem getting precise timing: GRUs may suffer with jobs that need accurate timing information because they lack independent methods of controlling input and output gates, which LSTMs possess.
• Less memory control: GRUs have a merged update and reset gate; therefore, they have less fine-grained control over how much information to update or discard from the previous time step.
• Variable performance: In certain applications and datasets, GRUs may not always outperform LSTMs or other more advanced designs, especially when dealing with highly subtle or detailed sequential patterns.
• Complexity trade-off: While GRUs are simpler than LSTMs, their simplicity can occasionally limit the model's ability to handle very complex temporal interactions, particularly in tasks with strong context dependencies.
  

Wrapping up

The gated recurrent unit has helped solve a lot of problems we faced in deep learning. It helped us especially mitigate the vanishing gradient problem. This tutorial has given you a good idea of how to use GRUs in your next deep-learning project.

Just like gated recurrent units, there are a lot of advanced concepts you need to master in deep learning. I would suggest checking out certified courses from reputed platforms. One such platform that comes to mind is upGrad. Their brilliant courses are in collaboration with some of the best universities around the world. Some of the best professors in the field curate their courses.

Frequently Asked Questions

1. What is ResNet used for?

ResNet, or Residual Network, is a deep neural network architecture that is specifically developed to solve the problem of vanishing gradients in very deep networks. This problem occurs when training deep networks with numerous layers because gradients can become exceedingly tiny during backpropagation, making it impossible for the model to learn efficiently.

2. How many layers are there in ResNet?

ResNet architectures can contain anywhere from ten to hundreds of layers. The original ResNet, introduced by Kaiming He et al. in 2015, contains versions with 18, 34, 50, 101, and 152 layers. Later variants and adaptations may include more layers, such as ResNet-1001 or ResNet-200.

3. Why ResNet is better than others?

ResNet's efficiency comes from its usage of skip connections, which allow for the training of extremely deep networks. This method helps to solve the vanishing gradient problem, making it easier to train deeper models.

4. What is the difference between VGG and ResNet?

The fundamental difference between VGG and ResNet is their architecture. VGG uses a deep but uniform architecture with modest 3x3 filters, whereas ResNet uses skip connections to ease the training of extremely deep networks, allowing it to overcome the depth restrictions experienced by networks such as VGG.

5. What are the advantages of ResNet?

ResNet's advantages include successful deep network training with skip connections, which leads to enhanced performance, better feature reuse, and scalability to hundreds of layers without losing performance.

6. Why ResNet is better than VGG?

ResNet is often seen to be better than VGG due to its capacity to train deeper networks more successfully, which is achieved using skip connections that mitigate the vanishing gradient problem. This leads to better performance and feature representation.

7. Which is faster VGG or ResNet?

VGG is often faster than ResNet for inference (generating predictions on fresh data) due to its simpler architecture with fewer layers and computations.

8. Which is best VGG or ResNet?

One is not better than the other. The job and dataset determine whether VGG or ResNet should be used. ResNet is often favored for deeper networks and jobs that require extracting complicated features, whereas VGG may be appropriate for simpler structures or smaller datasets.