The core idea
A neural network is a computer program built from many very simple parts, called units or neurons, connected in layers. Each unit does almost nothing on its own: it takes some numbers in, combines them, and passes a number on. Useful behavior comes from having many of these units, arranged in layers and tuned together.
The name comes from a loose analogy with the brain, where cells pass signals to one another. The analogy is only a starting point. Artificial networks are mathematical structures running on computers, and they differ from biological brains in many important ways.
What a single unit does
Imagine a unit deciding whether a photo might contain a cat. It receives several numbers as inputs. Each input has a weight, which is a number saying how much that input matters. The unit multiplies each input by its weight, adds the results together, and then applies a simple rule, called an activation function, that decides how strongly to pass the signal on.
A hypothetical example with made-up values: suppose a unit receives inputs 2 and 3 with weights 0.5 and 1. It computes 2 × 0.5 + 3 × 1 = 1 + 3 = 4. If the activation rule says “pass on the value if it is positive, otherwise pass on zero,” the output is 4. Real units do the same sort of arithmetic, just with many more inputs.
Layers
Units are arranged in layers.
- Input layer. Receives the raw data, such as the brightness of each pixel in an image, or numbers representing words.
- Hidden layers. One or more layers in the middle that transform the information step by step.
- Output layer. Produces the result, such as a probability for each possible category.
Networks with many hidden layers are called deep, which is where the term deep learning comes from. A common way to picture what happens is that early layers pick up simple features, such as edges in an image, and later layers combine them into more complex features, such as shapes and then objects. This picture is helpful but simplified; the features that networks actually learn are not always easy for people to name.
Where the learning happens
The weights are the network’s memory. At the start of training they are set to small, essentially random values, so the network’s outputs are nonsense. Training then follows a loop.
- Feed in an example, such as an image with a known label.
- Let the network compute its output, called the forward pass.
- Compare the output with the correct answer and calculate the error.
- Work out how much each weight contributed to the error, using a method known as backpropagation.
- Adjust each weight slightly in the direction that reduces the error.
- Repeat across a large number of examples, many times.
Over many rounds, the weights settle into values that give good answers. The size of each adjustment is controlled by a setting called the learning rate: too large and training becomes unstable, too small and it takes a very long time.
This is a specific case of machine learning, where a system learns from examples instead of following rules written by a person.
Why compute and data matter
Large networks may have millions or billions of weights, and each training step involves a great deal of arithmetic. The operations are mostly the kind that can be done in parallel, which is part of why graphics processors became so important for this work. Training also needs a lot of examples. The range and quality of that material shapes what the network can do, a point covered under training data.
What neural networks are good at
They tend to do well on problems involving messy, high-dimensional data where writing rules by hand is impractical: recognizing images, converting speech to text, translating languages, and predicting the next word in a sentence. The language models behind many chatbots are neural networks of a particular design, trained on enormous amounts of text.
Limits and cautions
They are hard to interpret. Billions of numbers do not explain themselves. This is why explainable AI exists as a research area, aiming to make decisions easier to inspect.
They can be fooled. Small, carefully chosen changes to an input, sometimes invisible to people, can make a network give a wrong answer.
They reflect their data. If the training examples are biased or incomplete, the network is likely to be too, which feeds into concerns about algorithmic bias.
They can be confidently wrong. A network will produce an output even for input unlike anything it has seen.
They need a lot of resources. Training large networks uses substantial computing power and energy, though smaller ones can run on modest hardware.
Are they like brains?
Only loosely. Real neurons are far more complex, they communicate through timing and chemistry, and brains learn from far fewer examples and continue to learn throughout life. Artificial networks are better thought of as flexible mathematical functions that can be fitted to data. Some researchers study designs more closely inspired by biology, an idea seen in neuromorphic computing, but standard networks do not work that way.
A summary you can remember
A neural network is many simple calculations stacked in layers. Its behavior is determined by the weights on the connections between units. Training is the process of adjusting those weights, bit by bit, so the network’s answers match the examples it is given. That simple recipe, scaled up, powers much of modern AI, and it also explains many of its weaknesses.