|
Zoo database An example of a multivariate data type classification problem using Neuroph by Nevena Jovanovic, Faculty of Organisation Sciences, University of Belgrade an experiment for Intelligent Systems course
Introduction In this example we will be testing Neuroph 2.4 with Zoo database, which can be found : here. Several architectures will be tried out, and it will be determined which ones represent a good solution to the problem, and which ones do not. First here are some useful information about our Zoo database:
Introducing the problem In this project I will work on the classification of animals, according to some of their characteristics. Trying to train through a network of networks learn to recognize our exhibits based on the output.. Class# -- Set of animals:
This variable, named type, represents the output variable. Except the output variable, there were 17 input variables for each animal species. Information of input variables:
Each variable is type of Boolean, except variable animal name which is nominal variable and variable legs is a numeric variable (set of values: {0, 2, 4, 6, 8}). Handling non-numeric data, such as Boolean = {true, false}, is more difficult. However, nominal-valued variables can be represented numerically. Value true will be replaced with value 1, and value false will be replaced with value 0. We wil not use variable animal name in experiment, because this variable is unique for each case. In this example we will be using 70%,85% and 90% of data for training the network and 30%,15% and 10% of data for testing it. To be able to use this data set in Neuoroph, it is necessary to normalize the data, but in this dataset we used the standard formula for normalization, but will vary for each input or output assign one neuron. So in the case of inputs related to the number of legs of the animals we will have 6 different values, so we will make one column and 7 columns, which will be a combination of 1 and 0. So it will be the last column refers to the exhibit, which is seventh In this way, our dataset now have only 0 and 1 values. Before you start reading our experiment we suggest to first get more familiar with Neuroph Studio and Multi Layer Perceptron.You can do that by clicking on the links below: Network designHere you can see the structure of our network with its inputs,outputs and hidden neurons in the middle layer. 
Training attempt 1
Network Type: Multi Layer Perceptron
Training Parameters:
Training Results: For this training, we used Sigmoid transfer function. As you can see, the neural network took 33 iterations to train. Total Net Error is acceptable 0.0095
Total Net Error graph look like this:
Practical Testing:
The final part of testing this network is testing it with several input values. To do that, we will select 4 random input values from our data set. Those are:
The network guessed correct in all five instances. After this test, we can conclude that this solution does not need to be rejected. It can be used to give good results in most cases. In our next experiment we will be using the same network,but some of the parametres will be diferent and we will see how the result is going to change. Training attempt 2
Network Type: Multi Layer Perceptron
Training Parameters:
Training Results: For this training, we used Sigmoid transfer function.As you can see, the neural network took 15 iterations to train. Total Net Error is acceptable 0.009
Total Net Error graph look like this:
Practical Testing:
The only thing left is to put the random inputs stated above into the neural network. The result of the test are shown in the table. The network guessed right in all five cases.
In this table we see exhibited in appropriate random number insanci. These are the value obtained in testing. Based on the results we can conclude that the deviations were smaller than in the previous case. In the next two attempts we will be making a new neural network.The main difference will be the number of hidden neurons in the structure of our network and other parametres will also be changed. Training attempt 3
Network Type: Multi Layer Perceptron
For this training, we used Sigmoid transfer function.As you can see, the neural network took 7 iterations to train. Total Net Error is acceptable 0.008169
Total Net Error graph look like this:
The final part of testing this network is testing it with several input values. To do that, we will select 5 random input values from our data set. Those are:
Based on this test, we conclude that the results are worse compared to previous tests and that the error is greater than the earlier ones. In our next experiment we will be using the same network,but some of the parametres will be diferent and we will see how the result is going to change. Training attempt 4
Network Type: Multi Layer Perceptron
For this training, we used Sigmoid transfer function.As you can see, the neural network took 24 iterations to train. Total Net Error is acceptable 0.0099
Total Net Error graph look like this:
The only thing left is to put the random inputs stated above into the neural network. The result of the test are shown in the table. The network guessed right in all four cases.
Training attempt 5This time we will be making some more significant changes in the structure of our network.Now we will try to train a network with 5 neurons in its hidden layer.
Network Type: Multi Layer Perceptron
We use here 85% of data set for traning.
For this training, we used Sigmoid transfer function.Training was completed in 826 iterations, but the error is large. So this is a not good selection of combinations of network and training set.
Total Net Error graph look like this:So the conclusion of this experiment is that the choice of the number of hidden neurons is crucial to the effectiveness of a neural network. One of the "rules" for determining the correct number of neurons to use in the hidden layers is that the number of hidden neurons should be between the size of the input layer and the size of the output layer. Formula that we used looks like this:((number of inputs + number of outputs)/2)+1.In that case we made a good network that showed great results.Then we made a network with less neurons in its hidden layer and the results were not as good as before. So,in the next example we are going to see how will the network react with a greater number of hidden neurons. Training attempt 6
Network Type: Multi Layer Perceptron We use here 85% of data set for traning, and 15 % data set for testing.
For this training, we used Sigmoid transfer function.As you can see, the neural network took 23 iterations to train. Total Net Error is acceptable 0.0099
Total Net Error graph look like this:
Practical Testing:
The final part of testing this network is testing it with several input values. To do that, we will select 4 random input values from our data set. Those are:
We can conclude that a decrease in the number of hidden neurons reduces the total error in the testing and to appear in the test outputs deviate less than anticipated. As you can see,this number of hidden neurons with appropriate combination of parametres also gave a good results except the third instances, where the error is 0.0000455. Training attempt 7Now we will see how the same network is going to work with a diferent set of parametres.
Network Type: Multi Layer Perceptron
For this training, we used Sigmoid transfer function.As you can see, the neural network took 463 iterations to train. Total Net Error is acceptable 0.0099.
Total Net Error graph look like this:
Practical Testing:
The only thing left is to put the random inputs stated above into the neural network. The result of the test are shown in the table. The network guessed right in all four cases. In this case we can see that the deviations from the defined high value on the basis of error in the test. And based on the total error in the test.
Although in this example we used considerably different set of parametres the network gave a good results in the test. Some statistics Four different solutions tested in this experiment have shown that the choice of the number of hidden neurons is very important for the effectiveness of a neural network. We have concluded that one layer of hidden neurons is enough in this case. Also, the experiment showed that the success of a neural network is very sensitive to parameters chosen in the training process. The learning rate must not be too high, and the maximum error must not be too low. Below is a table that summarizes this experiment. The best solution for the problem is marked in the table.
Advanced Training TechniquesWhen the training is complete, you will want to check the network performance. A learning neural network is expected to extract rules from a finite set of examples. It is often the case that the neural network memorizes the training data well, but fails to generate correct output for some of the new test data. Therefore, it is desirable to come up with some form of regularization. One form of regularization is to split the training set into a new training set and a validation set. After each step through the new training set, the neural network is evaluated on the validation set. The network with the best performance on the validation set is then used for actual testing. Your new training set consisted of the say it for example 80% - 90% of the original training set, and the remaining 10% - 20% would be classified in the validation set. Then you have to compute the validation error rate periodically during training and stop training when the validation error rate starts to go up. However, validation error is not a good estimate of the generalization error, if your initial set consists of a relatively small number of instances. Our initial set, we named it TS1, consists only of 101 instances (animal species). In this case 10% or 20%, of the original training set, consisted of the 10 or 20 instances. This is the insufficient number of instances to perform validation. In this case instead validation we will use a generalization as a form of regularization. In the following examples we will check the generalization error, such as from the example to the example we will increase the number of instances in the training set, which we use for training, and we will decrease the number of instances in the sets that we used for testing. Training attempt 8In this case we'll create a new training set that will cover 10% sample, a mine that does not have to be located in previously trained dataset.That part of data you can get from this link:TS1.
First we train the network to 90%, and then test the new test of 10%. On this basis, we will see how fast you learn, if the test data that are not in the training set.We will train on another network that we created earlier, and in doing so we will use the parameters: learning rate 0.21, momentum 0.7 and the maximum error 0.01Of course, previously it was necessary to isolate the total dataset of 10 instances we will place at TS1. The test results are:
Total Net Error graph look like this: Here we have error 0.002 which is ok, and The individual errors are also negligible. Also we can conclude that a decrease in the number of hidden neurons, mainly reduces toltalna error. Dynamic BackpropagationThese are the results of a Dynamic Backpropagation algoritam used on the best example in our experiment.
Training Results:
Total Net Error graph look like this:
Practical Testing:
Graph
Conclusion During this experiment, we created three different architectures, one basic training set and six training sets derived from the basic training set. We normalize the original data set using a linear scaling method. Through six basic steps we explained in detail the creation, training and testing neural networks. If the network architecture using a small number of hidden neurons training will become excessively and the network may over fit no matter what are the values of training parameters. We point out major differences between the Perceptron and MultiLayerPerceptron, as network types. Through the various tests we have demonstrated the sensitivity of neural networks to high and low values of learning parameters. We have shown that the best solution to the problem of classification of animal species, in seven different groups, is architecture with one hidden layer and six hidden neurons. Finally, we explain the importance of generalization and we pointed to the importance of validation as an important form of regularization. In the table below can been seen the overall results of this experiment. Best solution is indicated in green color. DOWNLOAD See also:
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
