Hello and welcome! Today we will look at Radial Basis Functions, RBF. RBF networks approximate functions by stretching and compressing a house on bells and then signing them spatially shifted. Description of their function and their learning process.

Comparison with multilayer perceptrons. According to Podgeo and Gyrosi radial basis function networks, RBF networks are a paradigm of neural networks, which was developed considerably

later than that of perceptrons. Like perceptrons, the RBF networks are built in layers, but in this case they have exactly three layers, only one single layer of hidden layers. Like perceptrons, the networks have a feed-forward structure and their layers are completely

linked. Here, the input layer again does not participate in informational processing. The RBF networks are like MLPs universal function approximators. Despite all things

in common, what is the difference between RBF networks and perceptrons? The difference lies in the informational processing itself and in the computational rule within the neurons outside of the input layer. So, in a moment we will define as so far unknown type of

neurons. Components and structure of an RBF network. Initially, we want to discuss colloquially and then define some concepts considering RBF networks.

Output neurons. In an RBF network, the output neurons only contain the identity as activation function and one weighted sum as propagation function. Thus, they do little more than

adding all input values and returning the sum. Hiding neurons are also called RBF neurons as well as the layer in which they are located in referred to as RBF layer. As propagation function, each hiding neuron calculates a norm that represents the distance

between the input to the network and the so-called position of the neuron center. This is inserted into a radial activation function which calculates and outputs the activation of the neuron. Definition 1. Input neuron. An input neuron is an identity neuron. It

exactly forwards the information received. Thus, it represents the identity function. Definition 2. Center of an RBF neuron. The center of an RBF neuron, H, is the point

in the input space where the RBF neuron is located. In general, the closer the input vector is to the center vector of an RBF neuron, the higher is its activation.

Definition 3. RBF neuron. The so-called RBF neurons have a propagation function that determines the distance between the center of a neuron and the input vector. This distance represents the network input. Then the network input is sent through a radial basis function

which returns the activation or the output of the neuron. Definition 4. RBF output neuron. RBF output neurons are the weighted

sum as propagation function and the identity as activation function. Components and structure of an RBF network. Definition 5. RBF network. An RBF network has exactly three layers in the following order. The input layer

consisting of input neurons, the hidden layer also called RBF layer consisting of RBF neurons and the output layer consisting of RBF output neurons. Each layer is completely linked with

the following one. It is a feed-forward topology. The connections between input layer and RBF layer are underweighted. For example, they only transmit the input.

The connections between RBF layer and output layer are weighted. The original definition of an RBF network only referred to an output neuron, but in analogy to the perceptrons it is apparent that such a definition can be generalized.

A piece neuron is not used in RBF networks. The set of input neurons shall be represented by the set of hidden neurons by H and the set of output neurons by O. Therefore, the inner neurons are called radial basis neurons because from their

definition follows directly that all input vectors with the same distance from the center of a neuron also produce the same output value. Figure 2. Information processing of an RBF network. Now, the question is, what can be realized

by such a network and what is its purpose? Let us go over the RBF network from top to bottom. An RBF network receives the input by means of the underweighted connections.

Then the input vector is sent through a norm so that the result is a scalar. This scalar, which by the way can only be positive due to the norm, is perceived by a radial basis function, for example by a Gaussian bell, figure 3.

The output values of the different neurons of the RBF layer or of the different Gaussian bells are added within the third layer. Basically, in relation to the full input space,

Gaussian bells are added here. Suppose that we have a second, a third and a fourth RBF neuron and therefore four differently located centers. Each of those neurons now measures another distance from the input to its own center.

De facto provides different values, even if the Gauss bell is the same. Since these values are finally simply accumulated in the output layer, one can easily see that any surface can be shaped by dragging, compressing and removing Gaussian bells and subsequently accumulating them.

Here, the parameters for the superposition of the Gaussian bells are in the weights of the connections between the RBF layer and the output layer. Furthermore, the network architecture offers the possibility to freely define off-train

high-train width of the Gaussian bells, due to which the network becomes even more versatile. Informational processing in RBF neurons

RBF neurons process information by using norms and radius-basis functions. At first, let us take as an example a simple one-for-one RBF network. It is apparent that we will receive a one-dimensional output,

which can be represented as a function Fig. 4. Additionally, the network includes the centers of the four inner neurons, and therefore it has Gaussian bells, which are finally added within the output neuron.

The network also possesses four values which influence the width of the Gaussian bells. On the contrary, the height of the Gaussian bells is influenced by the subsequent weights, since the individual output values of the bells are multiplied by those weights.

You can see a two-dimensional example in the Fig. 5. Informational processing in RBF neurons Since we use a norm to calculate the distance between the input vector and the center of a neuron H, we have different choices.

Often, the Euclidean norm is chosen to calculate the distance. Remember, the input vector was referred to as X, here the index I runs through the input neurons and thereby through the input vector components and

the neuron center components. As we can see, the Euclidean distance generates the score differences of all vector components, adds them and extracts the root of the sum. In two-dimensional space, this corresponds to the Pythagorean theorem.

From the definition of a normal directly follows that the distance can only be positive. Strictly speaking, we hence only use the positive part of the activation function. By the way, activation functions other than the Gaussian bell are possible.

Normally, functions that are monotonically decreasing over the interval from 0 to infinity are chosen. Now that we know the distance between the input vector and the center of the RBF neuron, this distance has to be passed through the activation function.

Here we use, as already mentioned, a Gaussian bell. It is obvious that both the center and the width can be seen as part of the activation function, and hence the activation function should not be referred to as FAG simultaneously.

Combinations of equation system and gradient strategies are useful for training. Analogous to the MLP, we perform a gradient descent to find the suitable weights by means of the already well-known delta rule. Here backpropagation is unnecessary,

since we only have to train one single weight layer, which requires less computing time. It is very popular to divide the training into two phases by analytically computing a set of weights and then refining it by training with the delta rule.

There is still the question whether to learn offline or online. Here the answer is similar. To the answer for the multilayer perceptron, initially one often trains online faster movement across the error surface. Then, after having approximated the solution,

the errors are once again accumulated and for a more precise approximation one trains offline in a short learning phase. However, similar to the MLPs,

you can be successful by using many methods. As already indicated, in an RBF network not only the weights between the hidden and the output layer can be optimized. It is not always trivial to determinize centers and weights of RBF neurons. It is obvious that the approximation

occurrence of RBF networks can be increased by adapting the weights and positions of the Gaussian bells in the input space to the problem that needs to be approximated. There are several methods to deal with the centers and the weights of the Gaussian bells.

Fixed selection. The centers and weights can be selected in a fixed manner and regardless of training samples, this is what we have assumed until now. Conditional fixed selection. Again, centers and weights are selected fixedly,

but we have previous knowledge about the functions to be approximated and comply with it. Adaptive to the learning process, this is definitely the most elegant variant, but certainly the most challenging one too. Fixed selection.

In any case, the goal is to cover the input space as evenly as possible. Here, weights of two-thirds of the distance between the centers can be selected so that the Gaussian bells overlap by approx. One-third. Figure 6.

The closer the bells are, the more time consuming the full thing becomes. This may seem to be very annoying, but in the field of function approximation we cannot avoid even coverage. Here, it is useless if the function to be approximated is precisely

represented at some positions, but at other positions the return value is only zero. However, the height input dimension requires a great many RBF neurons, which increase

the computation effort approximately with the dimension and is responsible for the fact that 6-10 dimensional problems in RBF networks are already called high dimensional. An MLP, for example, does not cause any problems here.

Conditional fixed selection. Suppose that our training samples are not evenly distributed across the input space. It then seems obvious to arrange the centers and sigma of the RBF neurons by means of the pattern distribution.

So, the training patterns can be analyzed by statistical techniques such as cluster analysis, and so it can be determined whether there are statistical factors according to which we should distribute the centers and sigmas, figure 7.

A more trivial alternative would be to set centers on positions randomly, selected from the set of patterns. So, this method would allow for every training pattern to be directly in the center of an neuron, figure 8. This is not yet very elegant,

but a good solution when time is an issue. Generally, for this method, the bits are fixedly selected. If we have reason to believe that the set of training samples is clustered, we can use clustering methods to determinize them.

There are different methods to determinize clusters in an arbitrarily dimensional set of points. One neural clustering method are the so-called ROLFs, self-organizing maps.

Growing RBF networks automatically adjust the neuron density. In growing RBF networks, the number of RBF neurons is not constant. A certain number of neurons as well as their centers and widths are previously selected,

by means of clustering method, and then extended or reduced. In the following text, only simple mechanisms are sketched. Neurons are added to places with large error values. After generating this initial configuration, the vector of the weights is analytically calculated.

Then all specific errors concerning the set of the training samples are calculated and the maximum specific error is those. The extension of the network is simple,

we replace this maximum error with a new RBF neuron. Of course, we have to exercise care in doing this. If the sigma are small, the neurons width only influence each other if the distance between them is sharp.

But if the sigma are large, the already existing neurons are considerably influenced by the new neuron because of the overlapping of the Gaussian bales. So it is obvious that we will adjust the already existing RBF neurons when adding the new neuron.

To put it simply, this adjustment is made by moving the center of the other neurons away from the new neuron and reducing their width a bit. Then the current output vector of the network is compared to the teaching input and the weight

vector is improved by means of training. Subsequently, a new neuron can be inserted if necessary. This method is particularly suited for function approximations, limiting the number of neurons. Here, it is mandatory to see that the network will not grow

at infinitum, which can happen very fast. Thus, it is very useful to previously define a maximum number of neurons. Less imported neurons are deleted, which leads to the question whether it is possible to continue learning

when this limit is reached. The answer is, this would not stop learning, we only have to look for the most unimportant neuron and delete it. A neuron is, for example, unimportant for the network if there is another neuron that has a similar function.

If open occurs, the two Gaussian bales exactly overlap and add such a position, for instance, one single neuron with a higher Gaussian bale would be appropriate. These RBF neurons are multilayer perceptrons we have already become a sequent with

and extensively discussed two network paradigms for similar problems. Therefore, we want to compare these two paradigms and look at their advantages and disadvantages. Comparing RBF networks and multilayer perceptrons

We will compare multilayer perceptrons and RBF networks with respect to different aspects. Input dimension. We must be careful with RBF networks in high dimensional functional spaces since the network could very quick require huge memory storage and computation

effort. Here a multilayer perceptron would cause less problems because its number of neurons does not grow exponentially with the input dimension. Center selection. However, selection of centers for RBF networks is despite the introduced approaches still a major problem.

Please use any previous knowledge you have when applying them. Such problems do not occur with the MLP. Output dimension. The advantage of RBF networks

is that the training is not much influenced when the output dimension of the network is high. For an MLP a learning procedure such as backpropagation thereby will be very time consuming.

Extrapolation. Advantage SPS disadvantage of RBF networks is the lack of extrapolation capability. An RBF network returns the result zero far away from the centers of the RBF layer. On the one hand it does not extrapolate, unlikely the MLP it cannot be used for

extrapolation, whereby we could never know if the extrapolated values of the MLP are reasonable, but experience shows that MLPs are suitable for that measure.

On the other hand, unlike the MLP, the network is capable to use this zero to tell us I don't know, which could be an advantage. Lesson tolerance. For the output of an MLP it is not so important if a weight or an neuron is missing, it will only worsen a little in

an RBF network, then large parts of the output remain particularly uninfluenced, but one part of the output is heavily affected because a Gaussian bell is directly missing.

Rose. We can choose between a strong local error for a lesion and a weak but global error. Spread. Here, the MLP is advantaged, since RBF networks are used considerably less often,

which is not always understood by professionals, at least as far as low-dimensional input spaces are concerned. Thank you for your attention, see you at the next lectures.