Introduction to neural networks and neural network programming using the Encog Artificial Intelligence Framework.

Introduction.

This article provides a basic introduction to neural networks and neural network programming using the Encog Artificial Intelligence Framework. 

Encog is an AI framework that is available for both Java and Microsoft .NET. 

In this article, I will show you how to create and train a very basic neural network with Encog. 

I will likely follow this up with more complex neural network examples in the future. However, for now, this article shows you how to get started with Encog.

Download.

Download source code - 306 KB

Background.

Encog is an advanced Artificial Intelligence Framework. Using Encog, you can create advanced neural network applications. Though Encog supports other aspects of Artificial Intelligence programming, this article will focus on neural network programming. Neural Network programming is the primary focus of Encog, as of version 2.3. Encog is released under the Lesser GNU Public License (LGPL). Encog can be downloaded from the following URL: http://www.heatonresearch.com/encog/.

This is a simple introductory example that shows how to create a neural network that recognizes the XOR operator. The XOR operator is essentially the "Hello World" of the neural network world. It is often used to demonstrate a new neural network.

Before I show you how to create a neural network in Encog, it is important to understand how a neural network works. Nearly all neural networks contain layers. A layer is a group of neurons that behave similarly. There are many different layer types used by the different types of neural networks that are supported by Encog. However, there are two very important layers that nearly every neural network will have. These are the input and output layers. The input layer is how you feed data to the neural network. The output layer is how you get the response back from the neural network.

The input and output from the neural network are both Java double values, ordinary floating-point numbers. Both the input and output layers will have some number of neurons. This determines how many floating-point numbers the layer will deal with. The input and output layers will typically have a different number of neurons. Deciding the structure of the input and output layer is how you define the "problem" that you are trying to solve to the neural network.

Let's see how we could create a neural network to perform as an XOR operator. The XOR operator requires that the two inputs be different, for the output to be true. The XOR operator's truth table can be expressed as follows:

Collapse | Copy Code

0 XOR 0 = 0
1 XOR 0 = 1
0 XOR 1 = 1
1 XOR 1 = 0

 

I would like to now create a neural network that can perform this operator. Such a network will have two input neurons and one output neuron. It will accept the two operands and return the result.

Using the Code.

We will now take a look at how the code for this example is constructed. This example was created with Encog v2.3, which is the current version of Encog at the time of this writing. The Encog 2.3 JAR file is provided with the download for this example. However, you may wish to grab the latest Encog JAR from the URL provided earlier in this article.

Neural networks must be trained before they are of any use. To train this neural network, me must provide training data. The training data is the truth table for the XOR operator. The XOR has the following inputs:

Collapse | Copy Code

public static double XOR_INPUT[][] = { 
  { 0.0, 0.0 }, 
  { 1.0, 0.0 },
  { 0.0, 1.0 }, 
  { 1.0, 1.0 } };

These are all of the possible inputs to the XOR operator. Likewise, the expected outputs are also stored as an array.

Collapse | Copy Code

public static double XOR_IDEAL[][] = { 
  { 0.0 }, 
  { 1.0 }, 
  { 1.0 }, 
  { 0.0 } };

These two arrays will be combined to create the training data for the XOR operator. The following line of code combines these two arrays to create training data:

Collapse | Copy Code

NeuralDataSet trainingSet = new BasicNeuralDataSet(XOR_INPUT, XOR_IDEAL);

We must now create a neural network. The following lines of code do this:

Collapse | Copy Code

BasicNetwork network = new BasicNetwork();

network.addLayer(new BasicLayer(new ActivationSigmoid(), true,2));
network.addLayer(new BasicLayer(new ActivationSigmoid(), true,4));
network.addLayer(new BasicLayer(new ActivationSigmoid(), true,1));
network.getStructure().finalizeStructure();
network.reset();

You can see the three layers being created. The three addLayer methods above create the three layers. The first layer is the input layer, it has two neurons. The final layer created is the output layer, it has a single neuron. The layer created in the middle is called the hidden layer. The hidden layer helps the rest of the neural network learn. More hidden neurons must be added to allow the neural network to learn more complex patterns. Four hidden neurons is more than enough for the XOR operator. Picking the number of hidden neurons is usually a process of trial and error.

The call to the reset method above randomizes the weights between all of the neurons in the network. These random weights will not allow the neural network to function as an XOR operator. However, the training process, that we will see shortly, will refine these weights and encourage the neural network to produce output similar to the XOR operator.

The ActivationSigmoid class specifies the type of activation that is to be used. The sigmoid function is a good choice here because we use no negative values on the XOR operator. If the network needed to recognize negative, the hyperbolic tangent activation function would be more appropriate. Encog supports a number of different activation functions, all of which have their unique uses.

A training object must be created to train the neural network. Encog supports a number of different training methods. Some are used for specific neural network types. Others perform better on different types of data. The best 'general purpose' trainer for this type of neural network is called resilient propagation (RPROP). The following code creates an RPROP trainer:

Collapse | Copy Code

final Train train = new ResilientPropagation(network, trainingSet);

Once the trainer has been created, we should loop through a series of iterations. Each iteration will take the neural network closer to being able to function as an XOR operator.

Collapse | Copy Code

int epoch = 1;

do {

  train.iteration();
  System.out.println("Epoch #" + epoch + 
                     " Error:" + train.getError());
  epoch++;

} while(train.getError() > 0.01);

At each iteration, or epoch, we check the error. We continue until the error is less than 1%. The error defines how far the output of the neural network is from the ideal output defined earlier.

Once the neural network has been trained satisfactorily, we will make use of it. We will feed each of the inputs into the neural network and observe the output. The following code does this:

Collapse | Copy Code

System.out.println("Neural Network Results:");

for(NeuralDataPair pair: trainingSet ) {

  final NeuralData output = network.compute(pair.getInput());
  System.out.println(pair.getInput().getData(0) + 
         "," + pair.getInput().getData(1)  + 
         ", actual=" + output.getData(0) + 
         ",ideal=" + pair.getIdeal().getData(0));

}

We loop through each of the training data items and present the output from the neural network as well as the actual output. The output from the program is shown here:

Collapse | Copy Code

Epoch #1 Error:0.5494587158070631
Epoch #2 Error:0.5494587158070631
Epoch #3 Error:0.5142230676643447
Epoch #4 Error:0.5005001549387983
Epoch #5 Error:0.5004690225170658
Epoch #6 Error:0.5004317073796488
...
Epoch #40 Error:0.026928155758252966
Epoch #41 Error:0.014605778809936418
Epoch #42 Error:0.007180166468750018
Neural Network Results:
0.0,0.0, actual=0.003348914814615583,ideal=0.0
1.0,0.0, actual=0.995118794356202,ideal=1.0
0.0,1.0, actual=0.9999200785659956,ideal=1.0
1.0,1.0, actual=8.741041885168623E-4,ideal=0.0

As you can see, the neural network loops through a number of iterations before the error percent drops below 1%. After 42 iterations, the network is trained. When you run the application, it may take more or less, as the neural network starts with random weights.

Once it is done, you see the actual output from the neural network. The output from the neural network does not exactly match the ideal output. This is expected. However, the value 0.99 is very close to 1.0.

Points of Interest.

This article demonstrated a very simple neural network application. It is massive overkill to use a neural network to create an XOR operator. There are many more complex examples provided with Encog. I will likely write further articles here demonstrating some of the more advanced features of Encog.

One of the interesting features of a neural network is that it is created completely differently from a traditional program. For a traditional program, you would think about how to implement the XOR operator and create all of the necessary programming logic to do it. For a neural network, you just provide input examples and the expected outputs. It is up to the neural network to learn how to provide these expected outputs. Usually, you really have no idea how it actually learns to provide its output. This is especially true with large neural networks that may have hundreds of neurons.

Custom Search

 

If you liked this article, subscribe to the feed by clicking the image below to keep informed about new contents of the blog: