Convolutional Neural Networks

A convolutional Neural Network is a deep Neural Network most commonly used to analyze images. They are similar to a multilayer perceptron, but instead of the fully connected layers usually found in these networks, they aim to extract features through the use of local patterns in each layer whose complexity increases in each layer such that you have the distinctive features in the final layer.

Commonly used Layers

A convolutional Neural Network consists of an input, output and an arbitrary amount of hidden layers. Each hidden layer has an input of shape (number of inputs) x (height of input) x (width of input) x (input channels (e.g. rgb colors in a picture)) and a coresponding output shape. There are different types of hidden layers that are commonly used, usually to reduce the dimensionality of the data, but at the cost of increasing the amount of features.

Convolutional Layers

Each convolutional layer consists of a kernel of a small size (usually 3x3 or 5x5) which is applied at regular intervals, called stride, by multiplying the values from the kernel to the values at the target, thus effectly convolving the data into abstract features. The output is, depending on how application at the edges is handled, a new shape slightly smaller than the input. To apply the kernel at the edges there are two methods often used, using a padding that does not discard data treats points outside the input as zeros, called "same"-padding, and a padding that only applies the kernel where it is possible, thus potentially discarding data if stride and size of the kernel is large, called "valid"-padding.
The Application of multiple different kernels creates the ability to extract several different abstract features. A big advantage of this method compared to regular fully connected neural networks is that the amount of weights for each kernel is very small.

Pooling Layers

Pooling layers aim to reduce the dimensionality by combining the values of clusters of data point into singular data points. The area which is combined into a single data point is usually small, for example a 3x3 area. There are two popular pooling functions used today: max and average. Max pooling uses the maximum value of each local cluster, while average pooling calculates the average value. A pooling layer does not increase the amount of features but decreases the dimensionality of the data, while keeping significant values.

Activation Layer

An activation layer applies an activation function to each value of the input, thus keeping the same dimensionality of the input and the output. This provides another non-linearity in the network. Nowadays there are three different activation functions mainly used, the Rectified Linear Unit (or a variation thereof), the tangens hyperbolicus or a sigmoid function.

Batch normalization

To prevent the unrestricted growth of values in the network usually some form of normalization is applied. Batch normalization aims to normalize the input by re-scaling and re-centering the data of a feature, thus fixing the means and the variances.

Fully connected Layer

Usually at the end of a Convolutional Neural Network a few fully connected layers are applied after transforming the current data into a flat vector (sometimes also listed as a seperate flattening-layer). This is the same as in a normal multi layer perceptron. Sometimes, to avoid the network overfitting on trainingsdata, a so called dropout is applied. The dropout p describes the chance of a neuron being deactivated in a single training stage where only the present weights are updated. At testing the full Layer is used, although each neurons output is reduced by a factor of (1-p) to account for the difference in layer width. This reduces overfitting by not training the neurons on all training data.

Softmax activation function

To predict a single class out of multiple classes usually a softmax activation function is used. This provides a much clearer distinction and highlights usually one dominant output over the others.

Hyperparameters

A convolutional Neural Network has a lot of freedom choosing the parameters for individual layers. Due to the nature of the kernels and application of the pooling layers, there are the parameters of size of a kernel or pooling layer as well as the stride, which describes the number of datapoints the window moves after each application. There is also the issue of finding a good amount of filters in each layer, a large number can easily overcome the computational capabilites of the user, due to each filter producing an output of usually only a slightly smaller size as the input, while a too small amount of filters might not empower the network to capture all important features in this layer.