Neurons and weights

In this chapter, we will learn what a weight is. We will begin with the brain, because the idea was borrowed from it. We will see how a child learns what a car is and why the face of an old friend fades. Then we will watch one artificial neuron make a decision with little more than multiplication and addition.

The brain in four numbers

A neuron is a cell in the brain that receives small signals from other neurons and, if the signals together are strong enough, sends a signal of its own. The place where one neuron passes its signal to another is called a synapse. In this book we will call it a connection.

These are estimates, and honest sources give ranges. Nobody has counted the connections of a whole human brain. We will use 200 trillion as a round number and remember that the truth may be half or five times as much.

neurons                                  86,000,000,000
connections, our round number       200,000,000,000,000
connections for each neuron         200,000,000,000,000 / 86,000,000,000 = about 2,300

The important point is the last line, which is derived from two estimates. There are a few thousand times more connections than neurons. What the brain has learned is held mostly in its connections, as far as science can tell, and much less in the cells themselves.

How a child learns "car"

Learning changes the strength of connections. A connection that is used becomes stronger. The signal then passes more easily the next time.

A small child sees a car for the first time. Light falls on the eye, and signals run through many neurons: some react to the round shape of the wheels, some to the noise, some to the movement. Someone says "car". A set of connections between all of these has been used together, once, and each has become a little stronger.

The child sees a car the next day, and the day after, in the street, in a book, as a toy. Each time the same paths are walked. After a few thousand times the paths are bare earth. The child sees two wheels and a piece of a door behind a wall and says "car" at once, without any effort.

   what the eye and ear report           connections                 what is recognised

   round wheels     -----------\
   engine noise     ------------+=====  strong, used daily  =====>   "a car"
   moves on a road  -----------/

   a face, 12 years old  -------\
   a voice              ---------+-----  weak, not used for 20 years ----->   "... do I know you?"
   a school yard        --------/

The friend you have not seen for twenty years

A connection that is not used becomes weaker, and a weak connection does not always carry the signal.

You sat next to a friend at school for six years. You saw that face hundreds of times, so the paths were well worn. Then you moved away. Twenty years later someone stops you in the street. Something feels familiar. Sometimes the name comes. Sometimes it does not, and it arrives an hour later, when you are thinking of something else.

The memory was not deleted. The path is still in the field, but it has grown over. Whether the signal gets through depends on small things: the light, the voice, whether you just thought about school.

The interactive below turns this story into two small formulas. They are made up for this book, so that we have something to move. Before you touch it, guess: how strong is the memory of the school friend, compared with the memory of a car?

The first formula builds strength by repetition: the number of meetings, divided by that number plus 50. It rises quickly at first and never quite reaches 1. In the second formula, the sign ^ means "to the power of": 0.95 ^ 20 is 0.95 multiplied by itself 20 times. Every year without a meeting leaves 95% of the strength.

a car                 seen 5,000 times, last seen today
  built              5,000 / (5,000 + 50)       = 0.990
  kept               0.95 ^ 0                   = 1.000
  strength now       0.990 x 1.000              = 0.990     you know it at once

a school friend       seen 300 times, last seen 20 years ago
  built              300 / (300 + 50)           = 0.857
  kept               0.95 ^ 20                  = 0.358
  strength now       0.857 x 0.358              = 0.307     it comes back, sometimes

The artificial neuron

An artificial neuron is a small calculation: multiply each input by a weight, add everything up, and pass the sum on. The weight is the strength of one connection, written as a number. It is one dial on the wall of Chapter 1.

Let us build one neuron whose job is to answer a single question: is this thing a car? It gets four inputs. Each input is 1 if the thing has the property and 0 if it does not. We chose the weights by hand, for illustration.

A positive weight says "this speaks for it". A negative weight says "this speaks against it". A weight near zero says "this does not matter". One more number is added at the end, the bias. It says how hard the neuron is to convince. Ours is -3.0, so a thing needs more than 3 points to count as a car.

a car: wheels 1, engine 1, fur 0, carries people 1

  has wheels        +2.0 x 1   =  +2.0
  has an engine     +3.0 x 1   =  +3.0
  has fur           -4.0 x 0   =   0.0
  carries people    +1.5 x 1   =  +1.5
  bias                         =  -3.0
  sum                          =  +3.5

The sum can be any number. To read it as "how sure", the neuron squashes it into the range from 0 to 1. A large positive sum becomes nearly 1, a large negative sum becomes nearly 0, and a sum of 0 becomes exactly 0.5. The formula for this squashing is 1 / (1 + e^(-sum)), where e is a fixed number, about 2.718. Nobody works this out by hand. A calculator or the interactive below does it. For a sum of +3.5 it gives 0.971. We read that as "97% sure it is a car". Our neuron says "car" when the result is above 0.5, which is the same as a sum above 0.

Our program ran the neuron on five things.

  weights  has wheels +2.0   has an engine +3.0   has fur -4.0   carries people +1.5   bias -3.0
  a car               (+2.0 x 1) + (+3.0 x 1) + (-4.0 x 0) + (+1.5 x 1) + (-3.0) = +3.5   squashed to 0.971   car
  a bicycle           (+2.0 x 1) + (+3.0 x 0) + (-4.0 x 0) + (+1.5 x 1) + (-3.0) = +0.5   squashed to 0.622   car
  a dog               (+2.0 x 0) + (+3.0 x 0) + (-4.0 x 1) + (+1.5 x 0) + (-3.0) = -7.0   squashed to 0.001   not a car
  a bus               (+2.0 x 1) + (+3.0 x 1) + (-4.0 x 0) + (+1.5 x 1) + (-3.0) = +3.5   squashed to 0.971   car
  a shopping trolley  (+2.0 x 1) + (+3.0 x 0) + (-4.0 x 0) + (+1.5 x 0) + (-3.0) = -1.0   squashed to 0.269   not a car

Two of the five answers are wrong. The neuron says that a bicycle is a car, with 0.622, and that a bus is a car, with 0.971. Let us take them one at a time.

The bicycle is a mistake of the weights. It can be repaired. Choose the bicycle below, then raise the weight of the engine and lower the bias until the bicycle falls below 0.5 and the car stays above it. A lower bias pulls every thing down. A higher engine weight lifts only the things that have an engine.

You have just done by hand what training does. Nobody tells the neuron a rule such as "a car needs an engine". Someone shows it a bicycle, says "wrong", and the weights are moved a little in the direction that makes the answer better.

The bus is a different kind of mistake, and no setting of the weights can repair it. For this neuron, a bus and a car are the same thing: both have wheels, an engine and seats, so both send the inputs 1, 1, 0, 1. A neuron can only tell apart what its inputs tell apart. To separate the bus from the car it would need a new input, such as "has more than nine seats". We will meet the inputs again in the next chapter, under the name X.

From one neuron to a model

A neural network is a very large number of artificial neurons, arranged in layers, where the outputs of one layer are the inputs of the next.

One neuron can answer one simple question. A layer of many neurons answers many simple questions at once. The next layer uses those answers as its inputs and asks questions about them, and so on. The questions of the first layers are simple. The questions of the later layers are built from the answers of the earlier ones.

You may hear that a large language model is made of many neural networks. It is more exact to say that it is one network with tens or hundreds of layers. GPT-2 small has 12. Every layer holds millions of weights.

the brain, connections     200,000,000,000,000
GPT-2 small, weights               124,439,808
ratio       200,000,000,000,000 / 124,439,808   = about 1,600,000

The model we open in this book has about 1.6 million times fewer weights than a brain has connections. The largest models of today are far bigger than GPT-2, and Chapter 13 puts them side by side with the brain.

pip install numpy
python 02_neuron_and_training.py

(+2.0 x 0) + (+3.0 x 0) + (-4.0 x 1) + (+1.5 x 1) + (-3.0) = -5.5
squashed: 1 / (1 + e^(5.5)) = 0.004      not a car