Hayes-Roth Data Set

An example of a multivariate data type classification problem using Neuroph

by Sava Necin, Faculty of Organisation Sciences, University of Belgrade

an experiment for Intelligent Systems course

Introduction

In this example we will be testing Neuroph 2.4 with Hayes-Roth Data Set , which can be found : here. Several architectures will be tried out, and it will be determined which ones represent a good solution to the problem, and which ones do not.

First here are some useful information about our Hayes-Roth Data Set:
Data Set Characteristics: Multivariate
Number of Instances: 160
Attribute Characteristics: Categorical
Number of Attributes: 5
Associated Tasks: Classification

Introducing the problem

This database contains 5 numeric-valued attributes. Only a subset of 3 are used during testing (the latter 3)l.
Some instances could be placed in either category 0 or 1. I've followed the authors' suggestion, placing them in each category with equal probability. I've replaced the actual values of the attributes (i.e., hobby has values chess, sports and stamps) with numeric values. I think this is how the authors' did this when testing the categorization models described in the paper. I find this unfair. While the subjects were able to bring background knowledge to bear on the attribute values and their relationships, the algorithms were provided with no such knowledge. I'm uncertain whether the 2 distractor attributes (name and hobby) are presented to the authors' algorithms during testing. However, it is clear that only the age, educational status, and marital status attributes are given during the human subjects' transfer tests.

Attribute Information:

-- 1. name: distinct for each instance and represented numerically
-- 2. hobby: nominal values ranging between 1 and 3
-- 3. age: nominal values ranging between 1 and 4
-- 4. educational level: nominal values ranging between 1 and 4
-- 5. marital status: nominal values ranging between 1 and 4
-- 6. class: nominal value between 1 and 3

For this experiment to work we had to transform our data set in binary format (0, 1).We replaced each attribute value with suitable binary combination.

In this example we will be using 80% of data for training the network and 20% of data for testing it.

Before you start reading our experiment we suggest to first get more familiar with Neuroph Studio and Multi Layer Perceptron.You can do that by clicking on the links below:

Neuroph Studio Geting started

Multi Layer Perceptron

Here you can see the structure of our network with its inputs,outputs and hidden neurons in the middle layer.

Network Type: Multi Layer Perceptron
Training Algorithm: Backpropagation with Momentum
Number of inputs: 16
Number of outputs: 3
Hidden neurons: 10

Training Parameters:
Learning Rate: 0.2
Momentum: 0.7
Max. Error: 0.01

Training Results:
For this training, we used Sigmoid transfer function. As you can see, the neural network took 5783 iterations to train. Total Net Error is acceptable 0.00998

Total Net Error graph look like this:

Practical Testing:

The final part of testing this network is testing it with several input values. To do that, we will select 5 random input values from our data set. The output neural network produced for this input is, respectively, in the last coloumn.

Inputs

Output

number

Name

Hobby

Age

Educational level

Marital status

Nominal outputs 1,2,3

Obtained outputs 1,2,3

The network guessed correct in all five instances. After this test, we can conclude that this solution does not need to be rejected. It can be used to give good results in most cases.

In our next experiment we will be using the same network,but some of the parametres will be diferent and we will see how the result is going to change.

Network Type: Multi Layer Perceptron
Training Algorithm: Backpropagation with Momentum
Number of inputs: 16
Number of outputs: 3
Hidden neurons: 10

Training Parameters:
Learning Rate: 0.3
Momentum: 0.6
Max. Error: 0.01

Training Results:

For this training, we used Sigmoid transfer function. Total Net Error graph look like this:


As you can see, the neural network took 12 iterations to train. Total Net Error is acceptable 0.00029

Practical Testing:

The only thing left is to put the random inputs stated above into the neural network. The result of the test are shown in the table. The network guessed right in all five cases.

Inputs

Output

number

Name

Hobby

Age

Educational level

Marital status

Nominal outputs 1,2,3

Obtained outputs 1,2,3

As we can see from this table,network guessed allmost every instance in this test without any error,so we can say that the second combination of parametres is even better than the first one.

In the next two attempts we will be making a new neural network.The main difference will be the number of hidden neurons in the structure of our network and other parametres will also be changed.

Network Type: Multi Layer Perceptron
Training Algorithm: Backpropagation with Momentum
Number of inputs: 16
Number of outputs: 3
Hidden neurons: 12

Training Parameters:
Learning Rate: 0.3
Momentum: 0.6
Max. Error: 0.01

Training Results:

For this training, we used Sigmoid transfer function. This time we used second network with 80% data for training and 20% data for test. In this case we used 12 hidden neurons instead of 10. Total Net Error graph look like this:



As you can see, the neural network took 3970 iterations to train. Total Net Error is acceptable 0.0096

Practical Testing:

The final part of testing this network is testing it with several input values. To do that, we will select 5 random input values from our data set. The output neural network produced for this input is, respectively:

Inputs

Output

number

Name

Hobby

Age

Educational level

Marital status

Nominal outputs 1,2,3

Obtained outputs 1,2,3

The network guessed correct in all five instances. After this test, we can conclude that this solution does not need to be rejected. It can be used to give good results in most cases.

In our next experiment we will be using the same network,but some of the parametres will be diferent and we will see how the result is going to change.

Network Type: Multi Layer Perceptron
Training Algorithm: Backpropagation with Momentum
Number of inputs: 16
Number of outputs: 3
Hidden neurons: 12

Training Parameters:
Learning Rate: 0.2
Momentum: 0.5
Max. Error: 0.01

Training Results:
For this training, we used Sigmoid transfer function.
Total Net Error graph look like this:

As you can see, the neural network took 10 iterations to train. Total Net Error is good: 0.00052

Practical Testing:

The only thing left is to put the random inputs stated above into the neural network. The result of the test are shown in the table. The network guessed right in all five cases.

Network Type: Multi Layer Perceptron
Training Algorithm: Backpropagation with Momentum
Number of inputs: 16
Number of outputs: 3
Hidden neurons: 8

Training Parameters:
Learning Rate: 0.2
Momentum: 0.7
Max. Error: 0.01

Training Results:
For this training, we used Sigmoid transfer function

Total Net Error graph look like this:

So the conclusion of this experiment is that the choice of the number of hidden neurons is crucial to the effectiveness of a neural network.

One of the "rules" for determining the correct number of neurons to use in the hidden layers is that the number of hidden neurons should be between the size of the input layer and the size of the output layer.Formula that we used looks like this:((number of inputs + number of outputs)/2)+1.In that case we made a good network.Then we made a network with less neurons in its hidden layer and the results are better than in the first place.So,in the next example we are going to see how will the network react with a fewer number of hidden neurons.

Network Type: Multi Layer Perceptron
Training Algorithm: Backpropagation with Momentum
Number of inputs: 16
Number of outputs: 3
Hidden neurons: 8

Training Parameters:
Learning Rate: 0.3
Momentum: 0.8
Max. Error: 0.01

Training Results:
For this training, we used Sigmoid transfer function. As you can see, the neural network took 23 iterations to train. Total Net Error is acceptable 0.0099

Total Net Error graph look like this:

Practical Testing:

The final part of testing this network is testing it with several input values. To do that, we will select 5 random input values from our data set. Those are:

Inputs

Output

number

Name

Hobby

Age

Educational level

Marital status

Nominal outputs 1,2,3

Obtained outputs 1,2,3

The output neural network produced for this input is, respectively:

As you can see,this number of hidden neurons with appropriate combination of parametres also gave a good results and guessed all five instances.

Now we will see how the same network is going to work with a diferent set of parametres.

Network Type: Multi Layer Perceptron
Training Algorithm: Backpropagation with Momentum
Number of inputs: 16
Number of outputs: 3
Hidden neurons: 8

Training Parameters:
Learning Rate: 0.5
Momentum: 0.3
Max. Error: 0.01

Training Results:

Total Net Error graph look like this:

For this training, we used Sigmoid transfer function.


As you can see, the neural network took 2662 iterations to train. Total Net Error is acceptable 0.0099

Practical Testing:

Conclusion

Four different solutions tested in this experiment have shown that the choice of the number of hidden neurons is very important for the effectiveness of a neural network. We have concluded that one layer of hidden neurons is enough in this case. Also, the experiment showed that the success of a neural network is very sensitive to parameters chosen in the training process. The learning rate must not be too high, and the maximum error must not be too low.

Below is a table that summarizes this experiment. The best solution for the problem is marked in the table.

DOWNLOAD

• Original training set

• Normalised training set

• Training set page at UCI




• Neuroph Studio 2.7 project

• Neural network training in Java source (NetBeans project with Neuroph 2.7)

See also:
Multi Layer Perceptron Tutorial