Choosing the right optimizer to minimize the loss between the predictions and the ground truth is one of the crucial elements of designing neural networks.