What is a neural network made of?

A neural network is built from many small, simple units wired together. Each unit is called an artificial neuron (or a node), and the units are joined by connections. Every connection carries a number called a weight, which sets how strongly one neuron’s signal affects the next.^[1]

A single neuron does something modest. It takes the outputs of the neurons feeding into it, multiplies each by the weight on that connection, and adds the results together. It adds one more number, called a bias. Then it passes that total through a function (a nonlinear activation function) to produce its own output.^[1] In practice that function is often something as simple as ReLU, which outputs zero for any negative input and passes positive input straight through unchanged — a rule cheap enough to apply billions of times per forward pass, which matters once a network has billions of neurons doing it. The activation function matters more than it sounds: without it, stacking layers would gain you nothing, because a chain of plain linear steps is still just one linear step. Google’s own crash course states the math plainly: “linear calculations performed on the output of linear calculations are also linear, which means this model cannot learn nonlinearities.”^[2] A hundred purely linear layers would still only be able to draw a straight line through the data; the nonlinear step is what lets the network bend that line into a curve.

The neurons are organized into layers. The first layer takes in the raw data. The last layer gives the answer. Any layers in between are called hidden layers, and they let the network recombine the data into more useful forms before reaching a result.^{[1, 2]} What makes a network learn anything is the set of weights, and how they get set is the subject of the next page.