在 Go 中构建一个简单的 神经网络
I have written a lot of computer programs in my career, most of the time to solve various problems or perform some tasks (or sometimes just for fun). For most part, other than bugs, as long as I tell the computer what to do very clearly (in whichever the programming language I use) it will obediently follow my instructions.
This is because computer programs are really good at executing algorithms -- instructions that follow defined steps and patterns that are precise and often repetitious. And in most cases they work well for us for tasks like number crunching or repetitious boring work.
What computer programs are not so good at doing though, are tasks that are not so well defined, and doesn’t follow precise patterns.
So how can we use computers to do such tasks? Just think about how you do this task. You probably learned about birds when you're young, and you've been told certain animals are birds and certain animals are not, mostly through seeing them either in real life or through picture books. When you get it wrong, you'll be told and you remember that. Over time you have a mental model of what's a bird and what's not. Every time you see something parts of a bird (clawed feet, feathered wings, sharp beak) you don't even need to see the whole animal anymore, you'll automatically identify it correctly by comparing it with your mental model.
So how do we do this with computer programs? Basically we do the same thing. We try to create a model that we can use to compare inputs with, through a trial and error process. And since computer programs are all mathematics, you can guess that it's going to be a mathematical model that we're going to be talking about.
Let's take a simple example create a black box that accepts an input and tries to predict the output.
We feed it with an input and get the output from this predictor. Since we know what the actual output should be, we can tell how different the predicted output is from the actual output. This difference between the actual and the predicted output becomes the error.
Of course, if the predictor is static and can't be changed, it's all pretty much moot. When we feed our predictor with an input, an output is produced with an error and that's the end of the story. Not very useful.
To make our predictor more useful let's give it a configurable parameter that we can use to influence the output. Since it only predicts correctly if there is no error, we want to change the parameter such that the error shrinks as we keep on feeding the predictor with data. The aim is to get a predictor that predicts the correct output most of the time, without actually needing to give clear instructions to the predictor.
In other words, this is very much like a numbers guessing game.
Let’s see this in a more practical way. Let’s say we have a predictor with the simple mathematical formula o = i x c where o is the output, i is the input and c is configurable parameter.
We are also given a confirmed valid output with a given input, that is, we know if i is 10, o is 26. How do we find c using the predictor?
First, we need take a random prediction, let's say c is 2. Let's put in the input 10, and crank up the predictor. The output o is 20. Since the error e = t - o where t is the truth (or target), this means e = 26 - 20 = 6. Our error e is 6 and we want to achieve 0, so let's try again.
Let's make c to be 3. The output is then 30 and is e is now -4. Oops, we overshot! Let's go back a bit and make c to be 2.5. That makes o to be 25, and e to be 1. Finally we try c to be 2.6 and we get the error e to be 0!
Once we know what c is, we can use the predictor to predict the output for other inputs. Let's say the input i is now 20, then we can predict o to be 52.
As you can see, this method tries to find answers iteratively and improving itself as it goes along, until we get the best fit. This in essence is what [machine learning[(A Beginner’s Guide to AI/ML – Machine Learning for Humans – Medium) is. The computer program tries to find answers iteratively and ‘learns’ through its mistakes until it achieves a model that can produce the best answer. Once it has the correct model, we can use the model to correctly guess the answers. This is very similar to what we humans do (by learning from past mistakes and correcting ourselves) but how exactly do we do it?
Let’s step out a bit. We talked a bit about how a machine can possible learn using mathematical functions. How humans do the same thing (as research over the years have shown) is using something called a [_neuron_](Understanding Neurons’ Role in the Nervous System).
A neuron, or a nerve cell, is a cell that receives information, processes them and transmits through electrical and chemical signals. Our brain and spinal cord (part of what is called our central nervous system) consists of neurons.
A neuron consists of a cell body, dendrites and an axon and can connected to each other to form neural networks. In a neural network, a neuron's axon is connected to the next neuron's dendrites and synaptic signals are transmitted from a neuron through its axon, and received by the next neuron through its dendrites. The connections between the axon and the dendrites is the synapse.
The incoming signals through the dendrite are strengthened or weakened based on how often the synaptic connections are being used and these strengthened or weakened signals are pooled together in the cell body.
If the pooled signals that are received are strong enough, it will trigger a new signal that is sent down the line through the axon to other neurons.
As you can see, how a neuron work is somewhat analogous to our predictor earlier. It has a bunch of inputs through its dendrites which it processes and an output through its axon. Instead of a configurable parameter, each input is paired with the strength (or weight) of the synaptic connection.
With this information, let’s get back into our predictor and make some changes.
We start off with building an artificial neuron that emulates the actual biological neuron. This artificial neuron is our upgraded predictor.
Instead of a single input we have a bunch of inputs, each of the modified with a weight (in place of a configurable parameter). These modified inputs are summed up and passed through a triggering or [activation function](Understanding Activation Functions in Neural Networks) which determines if an output should be sent.
So why an activation function (beyond the fact that a biological neuron behaves the same way)? There are a few good reasons but one the most important is that activation functions introduce non-linearity into the network. A neural network without activation functions (or a linear activation function) is basically just a [linear regression](Introduction to Linear Regression) model and is not able to do more complicated tasks such as language translations and image classifications. You will see later how non-linear activation functions enable back propagation.
For now, we will assume the use of a common activation function, the [sigmoid function](Sigmoid function).
The interesting thing to note about this function is that the output is always within the range between 0 and 1 but never reaches either.
Just like we had neurons forming neural networks, we can also connect our artificial neurons to form artificial neural networks.
It seems quite a bit more complicated now!
However we’re simply stacking the neurons up in different layers. All the inputs come in through the input layer, which sends its output to the hidden layer, which in turn sends its outputs to the final output layer. While the output from each node is the same (there is only 1 output) but the connections to the neurons in the next layer are weighted differently. For example, the inputs to the first node in the hidden layer would be (w11 x i1) + (w21 x i2).
Calculating the final outputs in this network can be a bit tedious if we have to do it one at a time, especially if we have a lot of neurons. Fortunately there’s an easier way. If we represent the inputs as well as the weights as matrices, we can use the matrix operations to make the calculations easier. In fact we don’t need to do individual neuron input summation and output activation anymore, we simply do it layer by layer.
This will help a lot later in the code, as you will see.
We used the [matrix dot product](How to Multiply Matrices) to handle the multiplication and summation of the input and weights, but for the activation function we will need to apply the sigmoid function to each of the matrix elements. We’ll have to do the same for every hidden layer, and also the output layer.
You might realise at this point in time, our neural network is (conceptually) simply a larger version of the neuron, and therefore is very much like our predictor earlier. And just like our predictor we want to train our neural network to learn from its mistakes by passing it input with known output. Then using the difference (error) between the known and actual outputs, we change the weights to minimise the error.
However you’d probably realise that the neural network is quite a bit more complicated than our predictor. First, we have multiple neurons arranged in layers. As a result, while we know the final target output, we don’t know the intermediate target outputs of the different layers in between. Secondly, we while our predictor is linear, our neurons are passed through a non-linear activation function so the output is not linear. So how do we change the weights of the different connections?
We know from our predictor earlier, we are looking to minimise the final output error Ek by changing the various output weights that connect between the hidden layer to the output layer, wjk.
That’s very well and good but how do we minimise a value of a function by changing its input variable?
Let’s look at this from a different perspective. We know the final output error Ek is:
However just subtracting ok from tk isn’t a great idea because it will often result in negative numbers. If we are trying to find the final output error of the network, we’re actually adding up all the errors, so if some of them are negative numbers it will result in the wrong final output error. A common solution is to use the squared error, which the name suggests is:
At the same time we know:
So we know (roughly speaking) if we map Ek with wjk we will get a range of values (blue line) plotted on a chart (actually this is a multi- dimensional chart but in order to keep our collective sanity, I’m going to use a 2 dimensional chart):
As you can see, to reach the minimal value of Ek we follow the gradient downwards or the negative gradient. In other words, we try to find the negative gradient, change the weight accordingly, then find negative gradient again, until we reach the minimal point of Ek. This algorithm is called [_gradient descent_](An Introduction to Gradient Descent and Linear Regression).
You might remember from secondary school calculus, in order to find the gradient of a point in a function we u
暂无开放 Issues,或尚未同步最近议题。