Page 1.CD – CSC-20023

Page 1.CD – CSC-20023 Computational Intelligence 1 Semester One Reassessment CSC-20023 Neural Networks Reassessment – Spring 2015 Introduction This assignment is based upon the use of the Multi-Layer Perceptron (MLP) neural network architecture. In Part 1 you will be required to use MLP neural networks for a pattern classification task using digitised character data. In Part 2 you will be required to develop and test auto-associative MLP networks. This assignment will require that you: ? learn how to use the Java Neural Network Simulator (JNNS); ? learn to create and operate simulations of MLPs; ? learn to train, test and evaluate the performance of MLP neural networks with JNNS; ? briefly sumarise your results in a report. Part 1: Pattern Classification of Digitised Numerals (60%) For this part of the assignment you will train and test several multi-layer perceptron (MLP) networks to act as pattern classifiers. Each network will try to identify whether a particular input pattern is a digitised numeral in the range 0-9. Several important files that you will need are stored in the numerals.zip zip-file accompanying this assignment specification on the KLE. You should make a copy of this zip-file and unzip it in your own area on the School’s teaching network. Amongst the files that you have copied is one called numerals_trg.pat. You should inspect the contents of this JNNS pattern file, but do not change anything. This file contains a set of ten training patterns for the digitised numerals 0 to 9. Each input pattern for a numeral is represented by 63 binary values: where a zero indicates a pixel is off and a one indicates that a pixel is on. For convenience each input pattern in the pattern file is arranged as a 7x9 tiling of input-pixel values e.g.: convenience each input pattern in the pattern file is arranged as a 7x9 tiling of input-pixel values eg: . . . but you should realise that, despite this cosmetic tiling appearance, each input pattern actually remains as a 63 unit vector! Page 2 CD – CSC-20023 Along with each input pattern there is a corresponding target pattern in the numerals_trg.pat file. Each target pattern is a 10 unit vector (i.e. units 1 to 10) with a single unit set to one and all other units set to zero – to indicate the position in the alphabet (i.e. the category) that an input pattern should be classified into. The numerals_trg.pat file will be the training data you should use for this part of the assignment. Using this set of training data you are required to train a number of MLP neural networks to try and perform the numeral classification task. The MLPs used for this task must all have 63 input units and must also have 10 output units – these constraints are dictated by the training data we have just reviewed in the numerals_trg.pat file. The distinguishing characteristic of each of your MLPs will be the number of hidden units in their middle layer (see below). Amongst the other files you have copied from the numerals directory is numerals_test.pat. This file contains some corrupted, or noisy, input patterns which can be used to test the generalising power of your trained MLPs. You should read the JNNS-comments which accompany the input patterns in this file (but take care not to alter any of the data!) to see what the correct classification for each corrupted/noisy input ought to be. You will be restricted to using only three-layer MLPs for this part of the assignment. So the only feature of the network topology that you can vary is the number of units in the hidden layer. As a minimum you should attempt to train at least one MLP with each of the following network topologies: 63_2_10 i.e. 2 hidden units 63_4_10 i.e. 4 hidden units 63_6_10 i.e. 6 hidden units 63_8_10 i.e. 8 hidden units 63_10_10 i.e. 10 hidden units [When you create each of the above MLP networks using JNNS you are strongly recommended to specify the layout of the input-layer’s nodes as a 7 by 9 tiling. This is so that when you have the network visible in the JNNS display window, the input-layer activations take on the appearance of the digitised numeral as they are shown in the diagrams above. In order to do this, when you enter the input-layer specification in the (Tools|Create|Layers) Create Layers dialog you should specify that the 63 input units are arranged as follows: Width is 7; Height is 9; Top left position is 1 1 1. Your specifications for the hidden- and output-layers should be supplied in the usual way: but take care to specify an appropriate Top left position for each of these succeeding layers.] Page 3 CD – CSC-20023 The decision function that you should use to interpret the MLP’s output layer activations during training is: unit is off if 0.0 <= activation <= 0.2 unit is on if 0.8 <= activation <= 1.0 What to hand-in for Part 1 For all of the MLPs that you train you should save the trained network in a file and record the following information in a table in your assignment report: ? Network topology and file name ? Number of training epochs to correctly classify the training data (or the number of epochs at which you stopped training if not correctly classifying all training patterns: stating why you chose to stop at that point) ? Number of training patterns correctly classified (if less than all ten) ? Number of testing patterns (from the numerals_test.pat file) correctly classified Plot a graph to show the relationship between the number of training epochs (y-axis) and the MLP hidden layer size (x-axis) using the data from your table. Finally, in no more than 500 words: summarise the results of your testing and you must clearly state what decision function you applied in order to interpret output-unit activations during your testing with the numerals_test.pat file. Part 2: Auto-Association of Digitised Numerals (40%) The MLP neural network architecture can also be used to build auto-associative memory networks. Such networks are trained to reproduce copies of their input patterns at their output layer. Once trained, these networks constitute a form of neural network memory of their training dataset and can often be used to clean-up/restore noisy/damaged copies of the data they were originally trained to remember. The aim of this part of the assignment is to train the smallest auto-associator MLP that will correctly clean-up nosiy/damaged versions of the digitised numerals. To train neural networks to act as auto-associators for the digitised numerals we will require slightly modified network topologies and training datasets. Since the MLPs will be trained to regurgitate approximate copies of their input patterns at their output layers it is important that both input and output layers have exactly the same number of units. For our digitised numerals this means that the input and output layers will both need to consist of 63 units. (Once again it would be best to specify that the input and output units should be displayed in JNNS as a 7x9 tiling.) The size of the hidden layer of each MLP network is a network-topology parameter that we can choose freely: however, we usually seek the network with the smallest hidden layer that can Page 4 CD – CSC-20023 successfully accomplish a given task. Clearly, the smallest possible hidden layer consists of 1 hiddenunit (i.e. a 63_1_63 network), then 2 hidden-units (i.e. a 63_2_63 network) and so on. If the hidden layer is too small the MLP will not be able to accomplish its memorisation task to a good level of accuracy. The pattern files that hold the auto-associative training and testing data are also different: numerals_aatrg.pat numerals_aatest.pat In addition to the various JNNS pattern files that you have unzipped from the numerals.zip file there should also be an executable Java program called DFuncAA. Inside the auto-associative training dataset the target data has been edited so that it is exactly the same as the input data. For the testing dataset the noisy/damaged versions of the digitised numerals are presented to the input layer, but you also need perfect copies of the desired target-pattern for each numeral (this is just to verify our off-line checking of the network’s outputs, see below). When the test dataset is used to test a trained network it is possible to laboriously check the activation of each output unit manually and apply a decision function to each one. Though with 63 output units and several networks to train this is not recommended! Instead we can use another feature of JNNS and a special Java program to produce output that is easier to interpret. This can be done as follows: 1. train an auto-associative MLP with the auto-associative training dataset; 2. load the auto-associative testing dataset and select it for use; 3. on the JNNS Save Data dialog (File | Save data) select the Result files *.res filetype from the combo-box that appears in the dialog window. Then specify a file-name (with a .res extension) of your own choosing, that the test-results should be stored in and generate the file by clicking on Save; a. in the Saving details dialog box that now appears make sure that you check the boxes to ensure that both input patterns and output patterns are placed into the results file, and that the Create (rather than Append) radio-button is selected; b. then click OK 4. now open the Windows Command Prompt window and navigate to your JNNS working directory, and ensure that your JNNS results file and the DFuncAA program are both present, then you can execute the program DFuncAA as follows: java DFuncAA <resfilename> where <resfilename> is the name of the JNNS results file you have just created. The DFuncAA program applies a decision function as follows: unit is off if 0.0 <= activation < 0.5 unit is on if 0.5 <= activation <= 1.0 Page 5 CD – CSC-20023 and then displays the resulting patterns in form that is easier to read than manually scrutinising the data on the JNNS display screen. What to hand-in for Part 2 Summarise the results you have obtained with auto-associative networks that you have trained and tested in a table. In the table clearly indicate which network you consider to be the best auto-associator that you have trained. The filename of each trained auto-associative MLP should be clearly recorded in your table and all of those network files should be part of your electronic submission for the assignment (see below). Marking criteria Part 1 and Part 2 are marked on the comprehensiveness of your search for good neural network implementations of the numeral classification and auto-association tasks. Your final assignment report should be between 500 to 1000 words long and contain a short description of the work you have carried out to solve both of the problems described above. You should ensure that any tables and graphs that you produce in your report are clearly labeled and annotated. The file-names of all of the fully trained neural networks should also be clearly recorded. For Part 1 remember you must clearly state what decision function(s) you used to interpret the output from your networks during your testing. For Part 2 you should describe how well the best auto-associative network you obtained performed and explain your rationale for deciding that it was the best. Some objective reflection, on the common pitfalls and problems that arose with the properties of the networks you developed during the assignment, should be present in a good report. Submission Details: The deadline for this assignment is 12noon on Thursday 11/6/2015. As part of your submission for this assignment you must write a word-processed assignment-report and store it in a pdf file called assignment.pdf. Along with all of the JNNS network and pattern files needed to run your neural networks, this report should be incorporated into a single zip-file using the compression software tools in the School’s Labs. (Please do not use any other exotic/freeware compression tools that you might have access to, as this could impair the examiners when they are assessing your work!) Also, take care to ensure that none of the zipped files/folders are password protected – as this will prevent the marker from being able to assess your submission. The compressed file should then be submitted via the CSC-20023 KLE pages: go to the Assignments section of the module in KLE and attach the compressed file according to the instructions found there for submitting a single zipped-file. It is important that you follow these instructions precisely. (Remember to use the correct file name for your assignment report!)