Neural network

A neural network is a data structure inspired by the biological brain. It consists of neurons linked by synapses. A neuron receives input impulses via synapses and performs some function upon those inpulses that determines the strength of the output impulse that should be sent to further neuron(s) via further synapse(s). The general idea during training is that the processing of an impulse by a neuron slightly changes the way that neuron will react to other impulses in the future. This allows the whole network to mould itself to the data it processes: to learn from experience just as the brain does.

The most important ways in which the  various types of neural networks differ are:

• Their topology. Some types of neural networks have strict rules about which neurons can accept external input and provide external output and maintain a concept of layers where neurons in each layer are only allowed to accept input from the preceding layer and to provide output to the subsequent layer. Others consist of a homogenous set of neurons with little or no restrictions on which pairs can communicate with one another.
   
• The nature of their input values, which can be binary or quantitative. Note that several binary neurons can be used to represent a categorical variable.
   
• The nature of their output values. Observing which of the last layer of neurons fire yields binary or categorical output, while capturing the propagation values from those neurons yields quantitative output;
   
• Their propagation function: the way in which input impulses received via a synapse are combined and processed to form a single input value to the activation function (see below). In a typical neural network, each synapse has a weight that is multiplied by the value of each impulse received along it, and the results of the multiplications for all input synapses are added together to generate the input value to the activation function. The weights are what most typically change as a neural network learns.
   
• Their activation function, which broadly speaking decides whether or not a neuron should fire based on its input. The simplest type of activation function is a binary threshold function that outputs 1 if the input was above a certain value and 0 otherwise. The binary threshold function is easy to understand but infeasible to work with mathematically because its differential is infinity at the threshold value. For this reason, other functions that approximate to the binary threshold function are normally used instead, including the logit function also used in logistic regression. All neurons in a network using such a function will always fire whatever input they receive, but that those that are not activated will fire so weakly as to have no practical effect on their successors.