How does a neural network learn?

When a neural network is new, its weights are essentially random, so its answers are too. Learning is the process of fixing that. Put plainly, “training/learning involves adjusting the weights of the network to improve the accuracy of the result.”^[1] Nothing about the network’s wiring changes; only the numbers on the connections do.

The loop is short and repeats many times. The network takes an example, runs it through to produce a guess, and that guess is compared to the known right answer to get an error. The bigger the error, the more the weights need to move. The usual method for figuring out which weights to move, and in which direction, is called backpropagation, which “spreads the error (adjusts the weights) from the output nodes across the network to the input nodes.”^[1] Underneath that spreading is calculus: backpropagation “calculates the gradient (the derivative) of the loss function… with respect to the weights,”^[1] which is what tells the network not just how wrong it was, but which direction to move each individual weight to be less wrong.

How big a step to take on each nudge is its own setting, the learning rate, which “defines the size of the corrective steps that the model takes to adjust for errors in each observation.”^[1] Push it too high and training goes faster but lands on a worse final answer; keep it too low and training crawls, even if the eventual result is better.^[1]

Each pass nudges the weights a little, so any single example barely shifts them. The learning comes from doing this across a great many examples until the errors get small. A network does not memorize a rule someone wrote for it. It settles, gradually, into a set of weights that happens to give good answers.