There are two main categories of neural network structures:
1. Acyclic or feed forward networks and
2. Cyclic or recurrent networks.
The manner in which the neurons of a neural network are structured is intimately linked with the learning algorithm used to train the network. We may therefore speak of learning algorithms used in the design of neural networks as being structured. Thus, we focus our attention on network architectures (structures).
ADVERTISEMENTS:
Now, we shall identify fundamentally different classes of network architectures in order to know the preliminary difference between connectionist AI and symbolic AI, and how do neurons learn?
1. Feed Forward Networks:
a. Single-Layer Feed Forward Networks (Perceptions):
In a layered neural network the neurons are organised in the form of layers. In the simplest form of layered network we have an input layer of source nodes which projects onto an output layer of neurons (computation nodes), but not vice versa. In other words, this network is strictly a feed forward or acyclic type.
It is illustrated in Fig. 11.15, for the case of four nodes in both the input and output layers. Such a network is called a single layer network with the designation ‘single-layer’ referring to the output layer of computation nodes (neurons). We do not count the input layer of source nodes because no computation is performed there. Since each output unit is independent of the others-each weight affects only one of the outputs.
b. Multilayer Feed Forward Networks:
The second class of a feed forward neural network distinguishes itself by the presence of one or more hidden layers, whose computation nodes are correspondingly called hidden neurons or hidden units. The function of hidden neurons is to intervene between the external input and the network output in some useful manner. By adding one or more hidden layers, the network is enabled to extract higher-order statistics.
In a rather loose sense the network acquires a global perspective despite its local connectivity due to the extra set of synaptic connections and the extra dimension of neural interactions. The ability of hidden neurons to extract higher-order statistics is particularly valuable when the size of the input layer is large.
The source nodes in the input layer of the network supply respective elements of the activation pattern (input vector), which constitute the input signals applied to the neurons (computation nodes) in the second layer (i.e., the first hidden layer). The output signals of the second layer are used as inputs to the third layer, and so on for the rest of the network. Typically the neurons in each layer of the network have as their inputs the output signals of the preceding layer only.
ADVERTISEMENTS:
The set of output signals of the neurons in the output (final) layer of the network constitutes the overall response of the network to the activation pattern supplied by the source nodes in the input layer. The architectural graph in Fig. 11.16., illustrates the layout of a multilayer feed forward neural network for the case of a single hidden layer.
For brevity the network in Fig. 11.16., is referred to as a 10-4-2 network because it has 10 source nodes, 4 hidden neurons, and 2 output neurons. In general, a feed forward network with m source nodes, h1 neurons in the first hidden layer, h2 neurons in the second hidden layer, and q neurons in the output layer is referred to as an m – h1– h2– q network.
The neural network in Fig. 11.16., is said to be fully connected in the sense that every node in each layer of the network is connected to every other node in the adjacent forward layer. If, however, some of the communication links (synaptic connections) are missing from the network, we say that the network is partially connected.
Learning algorithms for multilayer networks are similar to that of the perceptron learning. One minor difference is that we may have several outputs so we have an output vector rather than a single value, and each example has an input vector.
ADVERTISEMENTS:
The major difference is that, whereas the error at the output layer is clear, the error at the hidden layers seems mysterious because the training data does not say what value type hidden nodes should have. It turns out that we can back-propagate the error from the output layer to the hidden layers. The back propagation process emerges directly from a derivation of the over all error gradient.
2. Recurrent Networks:
A recurrent neural network distinguishes itself from a feed forward neural network in that it has units linked in cycles and at least one feed back loop. For example, a recurrent network may consist of a single layer of neurons with each neuron feeding its output signal back to the inputs of all the other neurons, as illustrated in the architectural graph in Fig. 11.17. In the structure depicted in this figure there are no self-feedback loops in the network; self-feedback refers to a situation where the output of a neuron is feedback into its own input.
The recurrent network illustrated in Fig. 11.17, also has no hidden neurons. In Fig. 11.18, we illustrate another class of recurrent networks with hidden neurons. The feedback connections shown in Fig. 11.18, originate from the hidden neurons as well as from the output neurons.
The presence of feedback loops, whether in the recurrent structure of Fig. 11.17 or that of Fig. 11.18, has a profound impact on the learning capability of the network and on its performance. Moreover, the feedback loops involve the use of particular branches composed of unit-delay elements (denoted by z -1), which result in a nonlinear dynamical behaviour, assuming that the neural network contains nonlinear units.
Hopfield Networks:
The best understood class of recurrent networks. They use bidirectional connections with symmetric weight (wi,j = wj,i), all of the units are both input and output units, the activation function is the sign function and the activation levels can be + 1.
A Hopfield network functions as an associative memory after the networks train on a set of examples, a new stimulus will cause it to settle into an activation pattern corresponding to the example in the training set which resembles most closely to the new stimulus.
For example, if the training set consists of a set of photographs, in the new stimulus is a small piece of one of the photographs, then the network activation levels will reproduce the photograph from which the piece was taken.
The original photographs are not stored separately in the network; each weight is a partial encoding of all the photographs. One of the interesting theoretical results is that Hopfield networks can reliably store upto 0.138N training example, N being the number of units in the networks.