Introduction

Figure 1: Screen shot of the character recognition demo.

Starting earlier this week (around 10/24/2004), I finally got around to creating one for the first time. Doing so has really helped my understanding of how these things work and what one really can do with them.

There has been so much hype about the subject of neural nets (NNs). There have also been many attempts to explain how they work that I've found difficult to grasp. The NN concept was the brainchild of mathematicians, and it's no surprise then that people introduce the subject from the perspective of the mathematics. Honestly, though, the nomenclature of the mathematicians' explanations as I've read them are dizzying for me, despite my college education.

I obviously am no expert in NNs, but I was surprised to see how much I was able to do with just a little code. Like most NN researchers, I chose to simulate an NN in software rather than create one in hardware. It occurred to me that my rudimentary experiment might be a good starting point for other programmers curious about the topic. Also, I want to explain NNs from my own perspective as a programmer, in the hopes that my outsider's context might be helpful for programmers and nonprogrammers alike.

The Neuron

Many essays on how neural networks work are available online. I've found all of the ones I've skimmed difficult to follow; again, because I'm not a mathematician. Let me skip the proofs, then, and summarize the exact algorithm I used in my own neurons for the "thinking" done by a single neuron:

    Public Sub Think()
        Dim Sum As Single, D As Dendrite
        For Each D In Dendrites
            D.Value = D.FromNeuron.AxonValue
            Sum += D.Value * D.Weight
        Next
        AxonValue = -2 / (1 + Math.Exp(2 * Sum)) + 1
    End Sub

To be sure, I've left out some extra lines of code that are to deal with odd cases that are not essential for this discussion.

The goal of a neuron of this sort is to fire when it recognizes a known pattern of inputs. Let's say for example that the inputs come from a black and white image 10 x 10 pixels in size. That would mean we have 100 input values. For each pixel that is white, we'll say the input on its corresponding dendrite has a value of -1. Conversely, for each black pixel, we have a value of 1. Let's say our goal is to get this neuron to fire when it sees a letter "A" in the image, but not when it sees any other pattern. Figure 3 below illustrates this:

"Firing" occurs when the output is above some threshold. We could choose 0 as the threshold, but in my program, I found 0.5 is a good threshold. Remember; our only output (axon) for this neuron is always going to have a value between -1 and 1.

The key to getting our neuron to recognize an "A" in the picture is to set the weights on each dendrite so that each of the input-times-weight values will be positive. So if one dendrite (for a pixel) is expecting to match white (-1), its weight should be -1. Then, when it sees white, it will contribute -1 * -1 = 1 to the sum of all inputs. Conversely, when a dendrite is expecting to match black (1), its weight should be 1. Then, its contribution when it sees black will be 1 * 1 = 1. So if the input image is exactly like the archetype our single neuron has of the letter "A" built into its dendrite weights, the sum will be exactly 100, the highest possible sum for 100 inputs. Using our "sigma" function, this reduces to an output of 1, the highest possible output value, and a sure indication that the image is of an A.

Now lets say one of the input pixels was inverted from black to white or vice-versa. That pixel's dendrite would contribute a -1 to the sum, yielding 98 instead of 100. The resulting output would be just a little under 1. In fact, each additional "wrong" pixel color will reduce the sum and hence the output. If 50% of the pixels were "wrong" - not matching what the dendrites say they are expecting as input - the sum would be exactly 0 and hence the output would be 0. Conversely, if the entire image of the "A" were inverted, 100% of them would be wrong, the sum would be -100, and the output would be -1. Since 0 and -1 are obviously below our threshold of 0.5, we would say this neuron does not "fire" in these cases.

In our example above, we might assume that some programmer hard-coded the "knowledge" - the weights for each dendrite - into the neuron at design time. What makes NNs interesting is that they can be given examples of patterns and hence learn to recognize them thereafter, without any designer explicitly calculating what the weights should be. Most explanations of NNs speak of learning as a collective function of a network of neurons, but I want to introduce the idea of learning in a single neuron instead. I find it's easier to grasp that way.

Let's take our previous example of a single neuron with 100 dendrites taking input from the pixels of a 10 x 10 black (1) and white (-1) image. Let's say we start with weights of zero for all dendrites to start with. The goal of training will be to get those dendrite weights to morph their way to being able to match the desired pattern.

Knowing what we do about this case, we could just find the difference between the result of each weighted input and the desired outcome and just set the weights in one step. The problem with this, though, will become apparent when we start considering matching or not matching different patterns. That is, we will gradually adjust weights in successive steps instead of all at once because we don't want to wipe out previously acquired knowledge. It's important to stress that each time we morph the dendrite weights of a neuron in order to advance its ability to recognize one pattern, we degrade its acquired knowledge of other patterns it should also recognize.

In the sample program I wrote, each neuron can have its own set of what I call "training cases". Each training case identifies a desired output value given a set of input values. In the case of the character recognition example, I have a set of images of letters from "A" to "F". As with our current example, I have one neuron for each letter I want to be able recognize. I use these images to build training cases such that each letter-recognizing neuron gets one positive-output case for its own pattern and one negative-output case for each of the other letters' patterns. For example, the neuron I want to match the letter "B" in an input image has one test case for the letter "B" that says the output should be as close as possible to 1 if the neuron sees a "B" in its input. It also has test cases for "A", "C", "D", "E", and "F" that says the output for this same neuron should be as close as possible to -1 if it sees any of these.

Returning to our "A" neuron, let's consider what training means. Each neuron goes through its own individual training cycle based on its set of training cases. We take each training case. For each, we ask the neuron to "think"; to generate an output based on the input values that training case holds. We then find the difference between the desired output for that training case and the actual output resulting from thinking. We use that difference as an adjustment rate for each of the dendrites' weights. We only adjust a small percentage of the distance - 1% in my program - to the desired outcome.

We repeat the training cycle several times. With each cycle, the ability of our "A" matching neuron to properly match an "A" image and to not match a "B", "C", etc. image gets better and better. Because with each step, we only morph the dendrite values a small part of the way towards the "ideal" weights, we guarantee that each training case is factored in. The resulting set of weights will be a compromise that equally balances the interests of all the training cases.

Following is the actual code for a neuron to execute one training cycle:

    Public Sub AutoTrain()
        Dim T As TrainingCase, Td As TrainingCaseDendrite
        Dim ErrorTerm As Single
        Const LearningRate As Single = 0.01

        'Learn from each training case in this one cycle
        For Each T In Me.TrainingCases

            'Preset my dendrite's source axons' values to reflect
            For Each Td In T.Dendrites
                Td.ForDendrite.FromNeuron.AxonValue = Td.Value
            Next

            'Given the training inputs, generate an output
            Think()

            'Morph toward the desired output
            ErrorTerm = T.AxonValue - Me.AxonValue
            For Each Td In T.Dendrites
                Td.ForDendrite.Weight += Td.Value * ErrorTerm * LearningRate
            Next
        Next
    End Sub

I must at this point remind the reader that I have not done much experimentation beyond this point. I've technically created a neural network program that supports stacks of layers of neurons and even clusters of stacks that can all interact, but I haven't really gone beyond the point of having a single set of neurons in one "layer". I have a knowledge of these concepts, but again, please take what I say with a grain of salt.

Figure 4 below illustrates a classic view of how an NN should be arranged:

A layer in this context, if it's not obvious by the diagram, is just a set of neurons that all share the same inputs. That is, every neuron in one layer has dendrites that extend to all the axons of neurons in a prior layer. Technically, an NN may have any number of "layers" of neurons. But it seems in practice, there are usually only two or three.

The first layer, referred to generally as the "input" layer, is actually just there to make programming easier. Its neurons don't actually have any dendrites. Instead, the input of some input matrix - say, an image of a letter - gets pumped straight into the axon values for each neuron. The neurons themselves are just placeholders so the next layer can tap into these input values in the same way each subsequent layer does.

The last layer is referred to as the "output" layer. Layers between the input and output layers are generally referred to as "hidden" layers. These names sound mysterious, so let me try to ground the thinking behind this. In my sample program (at least, for the character recognition demo), I have two layers. The first layer is just for dumping input values into the network. The second is the output layer, and its neurons are tasked with identifying characters from an input image. Each neuron is for matching a specific character: "A", "B", "C", and so on.

If two layers is good enough for recognizing a pattern, then you might well ask what the point of adding another layer is. The point, generally, is to add layers of abstraction. Let's take the character recognition example and say that the neurons that recognize characters are now in the hidden layer and we add a new output layer.

Perhaps our goal is to determine when a character recognized represents an even or odd character, in an ordinal sense. That is, for "A", "C", and "E", we want to report that we see an odd character and for "B", "D", and "F", we'll report that we see an even character. To do so, we will add two neurons to the output layer - one for even and one for odd - and provide them with training cases, just like we did for the hidden layer to recognize characters. The training cases for the "even" neuron would indicate that the neuron should fire when the previous layer fires to indicate one of the even characters and should suppress any urge to fire when it indicates one of the odd characters.

When researchers speak of learning in neural networks, they often refer to the concept of "back-propagation". Technically, the concept of giving each neuron its own training case is not back-propagation. In back-propagation, one would give training cases where the desired inputs would be for the input layer and the desired outputs would be for the output layer. All the layers would then go through a self-organizing process to figure out how best to achieve the goals of the training cases.

Let me voice my own criticisms of such bold claims. First, one could argue that a sigmoid neuron of the sort I created is not a bad model for how an individual neuron in the human brain functions; I'd be willing to buy that. But I have no doubt that neurons in the brain are not lined up in layers like I illustrated in figure 4 above, with each layer serving a different functional purpose and each neuron only talking to neurons in adjacent layers. Evolution is just not that good of a planner. With each new phase of experimentation, evolution makes subtle adjustments that serve some short-term good of a given individual. They are always "band aid" solutions. The result is a very function but hideously complicated tangle of neurons.

Second, neurons in the brain are not all the same. Some have short axons and dendrites that only span perhaps millimeters and thus propagate local information, while others can have axons that run many inches or perhaps even several feet. The neurons in your spinal column are structurally and functionally different from the ones in your brain. The ones in your brain vary by where they are, and they all certainly have differing functions. Most artificial neural nets, by contrast, are composed of just one type of neuron, and they are all in a very homogeneous kind of arrangement.

Third, the ideas of training cases and back-propagation seem intuitive enough as a learning paradigm for NNs, but I've never seen any indication that such mechanisms exist in the human brain. Neurons don't take time off every now and then to undergo training and then get switched back into a performance mode. They seem to learn and perform as they go.

A person should take away from this exercise an understanding that a human brain could not start off "tabula rasa", having no knowledge from the beginning. A mass of billions of neurons with no initial weights would do nothing and have no chance of learning anything. Such an amorphous brain would require a long period of orientation before it could even begin to do even the most basic things like regulate breathing and bowel functions, let alone get a person crawling and interpreting what it sees. I suspect our genes are responsible not only for telling cellular machinery how to specialize a stem cell into a neuron and where certain specialized types of neurons should develop and how far and where their axons should reach out, but perhaps even provide initial biases for synaptic connections and their effects on how a neuron responds to input. New research is showing that the 90% of DNA we previously thought was junk because it doesn't code for the proteins that make up the body may actually be largely responsible for differentiating humans from other species. Many of them do code RNAs that we are starting to understand are responsible for various activities. Could controlling initial neural wiring be one of them? Of course, there's no denying that our brains do learn in ways that are not directly defined by our genes. Once the development process is set in motion and the first neurons emerge in an embryo, interconnections and reinforcements are surely beginning to take shape based on variables in the environment. Such is the essence of learning.

So I think I can fairly say that, no matter how many layers deep and how many neurons wide one makes a neural network, no such network will ever be able to function like a human brain or even become vaguely sentient. That said, I think it's fair to also say that NNs can play a role in AI research. Perhaps an intelligent being can be borne out of an assemblage of just NNs, but it can't just be one big stack. It would have to be lots of individual stacks- what I call a "cluster" of stacks - interacting. Most of the parts of the cluster would have to be pre-trained so they can perform largely unchanging, autonomous functions. They would also have to be able to function in such a way that learning and acting can happen in parallel, or at least in alternating sequences. More realistically, NNs can be used as just one kind of machine in a heterogeneous platform of technologies to provide practical value. Perhaps one part of an AI might decide it wishes to learn to recognize certain patterns in its vision - perhaps different faces or machine parts - and allocate a new neural net to do the job and control when and what it learns, for example.

Figure 5 below shows what you'll see when you first open the Neural Net Demos program:

Figure 5: The main menu.

Figure 6: The "Demo 1" window.

All of the stuff on the window does something useful, but let's focus our attention first on the top inch - everything above the first full-width text area with "Layer: ..." written in it.

A neural network is just a processing machine, and as such has inputs and outputs. In this case, the main inputs are the two check boxes on the far left. The main outputs are the four check boxes around the middle of the form that are immediately to the right of the inputs. The four check boxes immediately below those four are used for training the net, but for now, we'll ignore those.

Figure 7 illustrates all four of the possible combinations of inputs we can use in our experiment.

Figure 7: Input combinations.

I'll refer to these cases as "00", "01", "10", and "11" from now on. I'll use the same sort of annotation for the output and training check boxes; e.g., refer to an output of "0001", "1010", etc.

At this point, no training has taken place and all the weights are set to 0. Try the various combinations of input (e.g., "01", "11") and you'll see the output is always "0000". Now try the automatic training. This comes from the XML file and is designed to encourage the neural network to learn the following response pattern:

This is why I also name this the "multiplexer" demo, since the resulting network is supposed to simulate a basic multiplexing output element. With the input set to "00", Click the "Train" button repeatedly until you see the output change to "1000". This should take about 39 clicks. (Hint: you can click the button once and then hold down the Enter key to auto-repeat pushing the button.) Now try each of the four input combinations and you'll now see the outputs mirror the truth table above.

Now let's try doing our own custom training by hand. Start by clicking the "Reload Brain" button. Then be sure to check the "Use ---->" check box to indicate that we'll use our own training values instead of the predefined ones in the XML file. Now set the training set (the four check boxes below the output check boxes) to "0101". Then set the input to "01". Then start clicking the "Train" button until you see the output matching the training set.

Start trying the different input combinations and you should notice something interesting. The outputs do different things depending on the inputs. The only particular input we trained for is "01", and it works as we trained it to, but the others just emerged automatically from the rest of the system. "10" as an input yields "1010", for example. Now let's say we want to train "10" to yield "1001" instead. Go ahead and enter that input and the desired output of "1001" into the training set. Keep hitting "Train" until you see the desired output.

At this point, something seemingly discouraging has happened. Your "10" input combination should yield "1001" as you wanted, but now your "01" input, depending on how many training cycles you went through, will probably output "0110", definitely not what you trained it for. Why is this? Because training on one pattern is inherently destructive of other learned patterns. This is why training is done in incremental steps instead of all in one quick step. The goal is to get training to occur on multiple patterns so that the dendrite weights that represent the knowledge will be compromise values. How do we do this?

Now let's try doing our own custom training by hand. Start by clicking the "Reload Brain" button. Then be sure to check the "Use ---->" check box to indicate that we'll use our own training values instead of the predefined ones in the XML file. Now set the training set (the four check boxes below the output check boxes) to "0001". Then set the input to "00". Then start clicking the "Train" button until you see the output matching the training set. Continue on with each of the following input and training set combinations:

So we basically reversed the output pattern of our multiplexer. If you're curious, feel free to choose a different pattern, but make sure you assign exactly one output check box for each of the four combinations (unless you don't care about one or more of the input patterns).

Incidentally, if you play around for a while, you'll notice some limitations. For instance, you can assign one of the outputs to multiple input patterns, but you can't do so for inputs that have polar opposite inputs. That is, you can't get a neuron to recognize one input pattern and its antithesis.

If you're curious to know what the other features are, let me describe them briefly. The first text box, which begins "Layer:" represents each of the neuron layers' axon value outputs. The two -1 or 1 values in the first line represents the two input neurons. Watch that first line change as you change the input check boxes. Because these neurons are sigmoid (quasi-linear) and not digital, I thought it would be worthwhile to see the actual values and not just the check boxes. So the second row represents the output layer's axon values.

If you're curious to know what the other features are, let me describe them briefly. The first text area, which begins "Layer:" represents each of the neuron layers' axon value outputs. The two -1 or 1 values in the first line represents the two input neurons. Watch that first line change as you change the input check boxes. Because these neurons are sigmoid (quasi-linear) and not digital, I thought it would be worthwhile to see the actual values and not just the check boxes. So the second row represents the output layer's axon values.

Figure 8: The "Demo 8" window.

The goal of this window is to demonstrate basic character recognition. The left side in an input you can draw on and the six check boxes labeled "A" through "F" are the outputs.

This window is basically a modified version of the Demo 1 window. This time, though, the inputs are represented by a paint-style grid. You can click and drag on this grid to paint. Starting from the cell you click on, the opposite color is selected from what's already there and that color is applied to all cells you drag over until you let go and click somewhere else. You can also use the "Clear" button to white out all the inputs.

The "A" through "F" outputs, as you might imagine, indicate the best matches for the input pattern. It's not a given that any will be checked for a given input, nor that only one will be.

If you want to see what the pattern for each character is, right-click on the label for that letter and you'll see the pattern to the right. Figure 8 above shows the "A" pattern after right-clicking on the "A" check box.

One way to kick off the process is to show the pattern for one of the letters and to draw some semblance of that letter in the input grid. Keep in mind that it's not really the shape that counts, but the overlap of the black pixels you draw with the black pixels in the model image. To start with, none of the outputs will be checked, because we have to undergo training. Click on the "Train" button repeatedly (again, you can click once and hold down the "Enter" key to rapidly repeat this button) until you see that letter's check box checked.

One thing that's different about this example is that I didn't hand-craft an XML file with all the training cases. Instead, I created GIF images with each of the letters and wrote code to import them as training cases. In fact, each of the A - F neurons in the output layer have not just one training case to positively identify their own letters, but in fact have one case for each of the other letters to avoid accidentally recognizing them. For example, letter "B" has a training case to tend toward an output of 1 (true) when it recognizes a "B" in the input, but will tend toward an output of -1 (false) when it recognizes "A", "C", "D", "E", or "F" (or "blank", which is one extra test case they all share). The result is 6 * 6 = 36 total training cases. I should point out that when I only trained on the 6 positive cases, I got way too many false positives. It was critical to also train on the negatives to avoid having every output neuron firing as soon as a few pixels get into the input grid.

Following are some example input patterns and the resulting guesses. Most are pretty good.

Figure 9: Sample cases of input and pretty good guesses by the neural network.

And now for some carefully chosen stinkers that should help illustrate the limitations of this solution.

Figure 10: Sample cases of input and pretty bad guesses by the neural network.

I encourage you to experiment for yourself with different inputs and see what the output looks like. Be sure to use the "cheat sheet" to see what the characters are expected to look like as you go. It will help explain why you get the results you get.

Figure 11: Files in the project.

Figure 11 shows a list of the code modules that are in this project to give a hint of the project's modest complexity.

I wrote with expandability and reusability of the design in mind. Although the demos in the code only show two-layer neural nets, in fact the structures support any number of clusters of arbitrarily deep stacks of layers of neurons, all in one brain. You are not even restricted to pure layering, either. You can connect the dendrites of one layer to dendrites anywhere else in the brain. This makes cross-connecting different stacks together possible and even circular loops within any stack possible. You could even create a stack with just one layer and lots of loop-backs within that layer if you want to get creative.

One thing to note, though, is that I originally started with a design that was meant to be cascading, so I buffered axon outputs throughout the brain in one iteration of the cycle, then propagated those buffered values forward for use in the next cycle. That made it act like one would expect if all the neurons were acting purely in parallel. One problem was that that meant that for a stack of N layers deep, it would take N thinking cycles to get a result from input to output. Another problem was that the logic for dealing with all sorts of exception cases was getting very complicated. So I got rid of axon output buffering. The downside comes when you start connecting neurons in non-layering configurations. Take neurons A and B in the same layer and B has a dendrite (input) connected to the axon (output) of A. If A is processed before B in one thought, then B's inputs will include the output of A in this same cycle. If B is processed first, on the other hand, then B's inputs will not include the output of A from this cycle. How would you know? You'd have to study the code carefully to recognize how processing propagates through the entire brain. Neural network theory, though, should suggest that the network will self-organize itself in its own way to solve the problems given it. That's my cop-out and I'm sticking to it.