Types of Optimizers in Deep Learning Every AI Engineer Should Know

Introduction

Deep learning is a great advancement over machine learning in terms of flexibility, higher accuracy, and a wide range of possibilities in industry applications. Whether it’s a chat application, grammar auto-correction, translation among different languages, fake news detection, or automatic story writing based on some initial wordings, Deep learning finds its usage in almost every sector. 

With this much usage, it becomes important that these algorithms run under minimum resources so we can reduce recurring costs and provide efficient results in less time. An optimizer is a method or algorithm to update the various parameters that can reduce the loss in much less effort. Let’s look at some popular Deep learning optimizers that deliver acceptable results.

A deep learning model comprises an input, output, activation function, loss function, hidden layers, etc. loss function, etc. All deep learning models generalize the data through an algorithm and make predictions on the unseen data.  You require an algorithm that maps the inputs’ examples to that of the outputs as well as an optimization algorithm. The particular optimization algorithm searches the parameters’ value, minimising the error while mapping the inputs to outputs. Such optimizers and optimization significantly influence the accuracy of the deep learning model. Moreover, they influence the model’s speed training. The question now arises is what an optimizer in deep learning is.

When training a deep learning model, you must adapt every epoch’s weight and minimize the loss function. An optimizer is an algorithm or function that adapts the neural network’s attributes, like learning rate and weights. Hence, it assists in improving the accuracy and reduces the total loss. But it is a daunting task to choose the appropriate weights for the model. This is because a deep learning model usually comprises millions of parameters.

You can use various optimizers to modify your learning rate and weights. But selecting the best optimizer in deep learning depends on your application.

Learn AI ML Courses from the World’s top Universities. Earn Masters, Executive PGP, or Advanced Certificate Programs to fast-track your career.

Before proceeding to different types of optimizers, let’s get familiar with some of their important terms.

Sample: It depicts a single row of a dataset.

Epoch: It denotes the number of times the algorithm operates on the entire training dataset.

Batch: It is the number of samples to be considered for updating the model parameters.

Cost Function/Loss Function: A cost function helps you calculate the cost, representing the difference between the actual value and the predicted value.

Learning rate: It offers a degree that denotes how much the model weights should be updated.

Weights/ Bias: They are learnable parameters that control the signal between two neurons in a deep learning model.

Gradient Descent (GD)

This is the most basic optimizer that directly uses the derivative of the loss function and learning rate to reduce the loss and achieve the minima. This approach is also adopted in backpropagation in neural networks where the updated parameters are shared between different layers depending upon when the minimum loss is achieved. It is easy to implement and interpret the results, but it has various issues. 

The weights are updated when the whole dataset gradient is calculated, which slows down the process. It also requires a large amount of memory to store this temporary data, making it a resource-hungry process. Though the idea behind this algorithm is well suited, it needs to be tweaked.

GD optimization algorithm uses calculus to change the values and obtain the local minimum consistently. You might be confused about what a gradient is. Suppose you are holding a ball present at the bowl’s top. But when you lose it, it follows the steepest direction and ultimately settles at the bowl’s bottom. With the help of a gradient, the ball follows the steepest direction to attain the local minimum at the bowl’s bottom.

GD algorithm is one of the easy-to-understand optimizers in deep learning because its formula s as below:

a_new = x –alpha *f’(a)

 The above equation calculates the gradient. The alpha shows the step size that denotes how far to move against each gradient with every iteration.

This optimizer in neural network works as follows:

• It begins with a few coefficients, observes their cost, and finds a cost value lower than what it is currently.
• It proceeds towards the lower weight and updates the coefficients’ value.
• The process iterates until the local minimum is attained. A local minimum is a point outside which it can’t proceed.

Check out our free NLP online course 

Stochastic Gradient Descent

This is a changed version of the GD method, where the model parameters are updated on every iteration. It means that after every training sample, the loss function is tested and the model is updated. These frequent updates result in converging to the minima in less time, but it comes at the cost of increased variance that can make the model overshoot the required position.

But an advantage of this technique is low memory requirement as compared to the previous one because now there is no need to store the previous values of the loss functions. 

A gradient descent optimizer may not be the best option for huge data. The stochastic gradient descent (SGD) optimizer tackles this problem. The term “stochastic” implies randomness upon which the algorithm is built upon. Rather than accepting the entire dataset for all iterations, a stochastic gradient allows you to choose the data’s batches randomly. It suggests that you only need to take a few samples from the dataset.

The process of this optimizer in neural network first selects the initial parameters ‘w’ and learning rate ‘n’. The data is then randomly shuffled at every iteration to obtain an approximate minimum.

Instead of the entire dataset, you only take the dataset’s batches for iteration. So, the path adopted by the algorithm consists of noise, unlike the gradient descent algorithm. Hence, SGD uses a higher number of iterations to attain the local minima. The total computation time increases because of an increase in the number of iterations. However, even after raising the number of iterations, the computation expense is still lesser than that of the GD optimizer. So, it concludes that an SGD optimizer in deep learning is preferable if the data is massive and the computational time is significant.

Mini-Batch Gradient Descent

Another variant of this GD approach is mini-batch, where the model parameters are updated in small batch sizes. It means that after every n batches, the model parameters will be updated and this ensures that the model is proceeding towards minima in fewer steps without getting derailed often. This results in less memory usage and low variance in the model.

Read: Machine Learning Project Ideas

Rather than taking the whole training data, Mini-Batch Gradient Descent only takes a subset of the dataset to calculate the loss function. So, fewer iterations are required. Therefore, the mini-batch gradient descent algorithm is comparatively faster than both batch gradient descent and stochastic gradient descent algorithms.

It is more robust and efficient than the former variants of gradient descent algorithms. It uses batching, so all the training data need not be processed on the memory. Hence, this optimizer in neural network makes the process more efficient.

Momentum Based Gradient Descent

Let’s revisit the method we are using to update the parameters. Based on the first-order derivative of the loss function, we are back-propagating the gradients. The frequency of updates can be after every iteration, a batch, or at the last, but we are not considering how many updates we have in the parameters. 

If this history element is included in the next updates, then it can speed the whole process and this is what momentum means in this optimizer. This history element is like how our mind memorizes things. If you are walking on a street and you cover a pretty large distance, then you will be sure that your destination is some distance ahead and you will increase your speed.

This element depends on the previous value, learning rate, and a new parameter called gamma, which controls this history update. The update rule will be something like w = w – v, where v is the history element.

Nesterov Accelerated Gradient (NAG)

The momentum-based GD gave a boost to the currently used optimizers by converging to the minima at the earliest, but it introduced a new problem. This method takes a lot of u-turns and oscillates in and out in the minima valley adding to the total time. The time taken is still way too less than normal GD, but this issue also needs a fix and this is done in NAG.  

The approach followed here was that the parameters update would be made with the history element first and then only the derivative is calculated which can move it in the forward or backward direction. This is called the look-ahead approach, and it makes more sense because if the curve reaches near to the minima, then the derivative can make it move slowly so that there are fewer oscillations and therefore saving more time.

Also Read: Deep Learning Techniques You Should Know

Adagrad

Till now we are only focusing on how the model parameters are affecting our training, but we haven’t talked about the hyper-parameters that are assigned constant value throughout the training. One such important hyper-parameter is learning rate and varying this can change the pace of training.

For a sparse feature input where most of the values are zero, we can afford a higher learning rate which will boost the dying gradient resulted from these sparse features. If we have dense data, then we can have slower learning. 

The solution for this is to have an adaptive learning rate that can change according to the input provided. Adagrad optimizer tries to offer this adaptiveness by decaying the learning rate in proportion to the updated history of the gradients.

It means that when there are larger updates, the history element is accumulated, and therefore it reduces the learning rate and vice versa. One disadvantage of this approach is that the learning rate decays aggressively and after some time it approaches zero. 

It uses unique learning rates for every iteration. The modification in learning rate depends on the variance in the parameters during the training. If the parameters change frequently, the learning rate, too, changes frequently. This change is advantageous because practical datasets include dense and sparse features. Hence, it is impartial to assign the same value of the learning rate for every feature.

One of the key benefits of using Adagrad optimizer in neural networks is that it does not need manual modification of the learning rate. It is more reliable than the gradient descent algorithms and their other variants. Moreover, it attains convergence at a faster speed.

RMSProp

It is an improvement to the Adagrad optimizer. This aims to reduce the aggressiveness of the learning rate by taking an exponential average of the gradients instead of the cumulative sum of squared gradients. Adaptive learning rate remains intact as now exponential average will punish larger learning rate in conditions when there are fewer updates and smaller rate in a higher number of updates. 

Must Read: Regularization in Deep Learning

RPPROP uses the gradient sign to change the step size discretely for each weight. In the RMSProp algorithm, two gradients are first compared based on the signs. If their signs are the same, you proceed in the right direction. So, the step size increases by a small amount. But if their signs are opposite, you need to reduce the step size. Subsequently, you limit the step size and proceed with the weight update.

This algorithm primarily accelerates the optimization process by reducing the number of function estimates to obtain the local minima. It stores the moving average of squared gradients for each weight and divides the gradient by the mean square’s square root. Hence, it is one of those optimizers in deep learning that precisely calculates the gradients.

Adam

Adaptive Moment Estimation combines the power of RMSProp (root-mean-square prop) and momentum-based GD. In Adam optimizers, the power of momentum GD to hold the history of updates and the adaptive learning rate provided by RMSProp makes Adam optimizer a powerful method. It also introduces two new hyper-parameters beta1 and beta2 which are usually kept around 0.9 and 0.99 but you can change them according to your use case.

Conclusion

In this article, we looked at 8 Deep learning optimizers in the order of ease of their usage and how one optimizer’s limitation is overcome by the next one, and so on. There are more modifications of one or the other optimizers mentioned here, but these are the fundamental ones that you should consider before going for complex solutions.

Picking a winner among these is highly subjective to the use case and the problem you are dealing with but one can surely rank Adam Optimizer on the top because of its combination with the momentum concept that changed how the model parameters should be updated and adapting the changing learning rate for different scenarios enabling efficient processing of any types of inputs. 

A general trend shows that for the same loss, these optimizers converge at different local minima. While adaptive learning optimizers converge at sharper minima, other types of techniques converge at flatter minima which is better for generalization. These techniques can only help to some extent because as the Deep neural networks are becoming bigger, more efficient methods are required to get good results.

If you are curious to master Machine learning and AI, boost your career with an our Master of Science in Machine Learning & AI with IIIT-B & Liverpool John Moores University.