Deep Learning Cheat Sheet

Neural Networks

Neural Networks are the fundamental models of Deep Learning, consisting of layers of nodes or neurons. The operation is divided into Forward Propagation for prediction and Backpropagation for adjusting weights and biases to minimize the Loss Function.

Activation Functions

Activation functions add non-linearity to the model, allowing Neural Networks to learn complex patterns.

• Sigmoid: Outputs a value between 0 and 1, commonly used in the Output Layer for Binary Classification.
• Tanh: Outputs a value between -1 and 1.
• ReLU (Rectified Linear Unit): Returns 0 if the value is less than zero, and returns the original value if it is greater than zero. Widely used in most Hidden Layers.
• Softmax: Used in the Output Layer for Multi-class Classification to convert values into probabilities.

Convolutional Neural Networks (CNN)

CNNs are designed for Computer Vision tasks and have the following main components:

• Convolutional Layers: Use Filters or Kernels to extract features from images.
• Pooling Layers: Reduce the size of data and parameters, often using Max Pooling.
• Fully Connected Layers: Take the extracted features to perform classification.

Recurrent Neural Networks (RNN)

RNNs are suitable for Sequential Data such as time series data or Natural Language Processing (NLP).

• Hidden State: Allows previous data to affect the processing of current data.
• Vanishing Gradient Problem: Basic RNNs often struggle to remember long-term information. Therefore, LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) were developed to solve this problem.