How the closest-example model decides
Measure how far apart two images are, find the closest stored examples, and let them vote, first by hand and then with the library.
Work here, beside the explanation
Before you train a model on real digits, work out by hand how the model you are about to use makes a decision. It is called the closest-example model, and most people call it k nearest neighbours, or KNN for short. You used it as a block in Level 3 to predict late deliveries. Here you rebuild it yourself with the Python you already know, on images so small you can check every number, and then confirm that the library gives exactly the same answer.
Each numbered stage below shows a complete program. Try this stage copies it into the editor beside the article, including every line it needs from earlier stages, so it works even after you reload the page. Read the program first, predict what it will print, then press Run. Loading a stage replaces what is in the editor; Undo brings your own version back.
The idea in one paragraph
Imagine someone hands you an unfamiliar fruit and asks what it is. You could compare it with fruits you already know, pick the three that look most like it, and go with whatever most of those three are. That is the whole method. The model keeps every training example it was given. To label a new input, it finds the stored examples that are closest to it and lets them vote. The k in k nearest neighbours is simply how many of the closest examples get a vote.
To do this with images we need one precise thing: a number that says how far apart two images are. The next few stages build that number.
Build with me · 1
1. Store a few labelled examples
Each image here has just four pixels, arranged as a two-by-two square and written as a list in reading order: top left, top right, bottom left, bottom right. Each number is a brightness from 0 (dark) to 1 (bright), the same scale the digit images use after dividing by 16. Images A and B are bright down their left side, so they are labelled left. C and D are bright down their right side, labelled right. E is a blurry example whose right side is only slightly brighter; a person labelled it right. new_image has no label: it is the one we want the model to decide about. The zip over three lists hands out one name, one image, and one label per turn.
names = ["A", "B", "C", "D", "E"]training_images = [ [0.9, 0.1, 0.8, 0.0], [1.0, 0.0, 0.6, 0.2], [0.2, 0.9, 0.1, 1.0], [0.0, 0.7, 0.3, 0.9], [0.5, 0.6, 0.4, 0.5],]training_labels = ["left", "left", "right", "right", "right"]new_image = [0.6, 0.4, 0.5, 0.3]for name, image, label in zip(names, training_images, training_labels): print(name, image, label)print("New image:", new_image)Run it. For each stored image, check that the label matches where the bright pixels are.
What to look for
Five labelled images print, followed by the new image [0.6, 0.4, 0.5, 0.3].
Make it yours
Look at the new image and decide which label you would give it before the model does. Keep that guess for the end of the lesson.
What close means for two images
Two images are close when their pixels are similar in the same places. A bright pixel in the new image matched by a bright pixel in the same spot of a stored image makes them close. A bright pixel matched by a dark one pushes them apart. So the comparison works pixel by pixel, and only pixels in the same position are compared.
Build with me · 2
2. Compare two images pixel by pixel
To compare the new image with image A, take the two images one pixel at a time, in the same order, and subtract. zip lines up the first pixel of each image, then the second, and so on. The new image is 0.6 in its top-left pixel and A is 0.9, so that difference is 0.6 minus 0.9, which is -0.3. The last line rounds each difference to two decimal places for display, because computers store decimals like 0.3 with tiny rounding errors such as 0.30000000000000004.
names = ["A", "B", "C", "D", "E"]training_images = [ [0.9, 0.1, 0.8, 0.0], [1.0, 0.0, 0.6, 0.2], [0.2, 0.9, 0.1, 1.0], [0.0, 0.7, 0.3, 0.9], [0.5, 0.6, 0.4, 0.5],]training_labels = ["left", "left", "right", "right", "right"]new_image = [0.6, 0.4, 0.5, 0.3]differences = []for new_pixel, stored_pixel in zip(new_image, training_images[0]): differences.append(new_pixel - stored_pixel)print("Differences from A:", [round(difference, 2) for difference in differences])Work out the four differences on paper first: 0.6 - 0.9, 0.4 - 0.1, 0.5 - 0.8, and 0.3 - 0.0. Then run.
What to look for
Differences from A: [-0.3, 0.3, -0.3, 0.3].
Make it yours
Change training_images[0] to training_images[4] to compare with E instead, and predict whether its differences will be larger or smaller.
Build with me · 3
3. Turn the differences into one distance
Four separate differences are hard to compare between images, so we combine them into one number called the distance. The recipe has three steps. First, square each difference, so a negative difference counts the same as a positive one and larger differences count much more than small ones. Second, add the squares. Third, take the square root, which brings the total back to the same scale as the pixels. For A: each difference is 0.3 or -0.3, each square is 0.09, four of them add up to 0.36, and the square root of 0.36 is 0.6. This is the same distance a ruler would measure between two points on a map, extended to more than two directions. The function wraps the recipe so it can be reused for any two images of the same size.
names = ["A", "B", "C", "D", "E"]training_images = [ [0.9, 0.1, 0.8, 0.0], [1.0, 0.0, 0.6, 0.2], [0.2, 0.9, 0.1, 1.0], [0.0, 0.7, 0.3, 0.9], [0.5, 0.6, 0.4, 0.5],]training_labels = ["left", "left", "right", "right", "right"]new_image = [0.6, 0.4, 0.5, 0.3]from math import sqrt def distance(first, second): total = 0 for a, b in zip(first, second): total += (a - b) ** 2 return sqrt(total) print("Distance to A:", round(distance(new_image, training_images[0]), 3))Check the arithmetic for A by hand, then run.
What to look for
Distance to A: 0.6.
Make it yours
Call distance(new_image, new_image) and explain why the answer must be 0.
Build with me · 4
4. Measure the distance to every stored example
Now apply the same function to every stored image. Nothing about the labels is used yet: distance only compares pixels. A smaller distance means a more similar image. Reading down the list, E is the closest by some way, B and A come next and are almost tied, and C and D are far away.
names = ["A", "B", "C", "D", "E"]training_images = [ [0.9, 0.1, 0.8, 0.0], [1.0, 0.0, 0.6, 0.2], [0.2, 0.9, 0.1, 1.0], [0.0, 0.7, 0.3, 0.9], [0.5, 0.6, 0.4, 0.5],]training_labels = ["left", "left", "right", "right", "right"]new_image = [0.6, 0.4, 0.5, 0.3]from math import sqrt def distance(first, second): total = 0 for a, b in zip(first, second): total += (a - b) ** 2 return sqrt(total) for name, image, label in zip(names, training_images, training_labels): print(name, label, "distance", round(distance(new_image, image), 3))Before running, guess which image is closest to the new one.
What to look for
The distances are A 0.6, B 0.583, C 1.03, D 0.922, and E 0.316.
Make it yours
Change one pixel of the new image to make it look more like C, and watch which distances shrink and which grow.
Build with me · 5
5. Sort from closest to farthest
To find the closest examples, store three facts about each stored image together in one tuple: its distance from the new image, its label, and its name. pairs.sort() puts the tuples in order. Python sorts tuples by their first value, so the list ends up ordered from the smallest distance to the largest. The first tuple is now the single closest example. If only one neighbour gets a vote, which is k = 1, the prediction is simply that example's label: pairs[0] is the first tuple and [1] picks its label. The loop names each distance gap so it does not replace the distance function.
names = ["A", "B", "C", "D", "E"]training_images = [ [0.9, 0.1, 0.8, 0.0], [1.0, 0.0, 0.6, 0.2], [0.2, 0.9, 0.1, 1.0], [0.0, 0.7, 0.3, 0.9], [0.5, 0.6, 0.4, 0.5],]training_labels = ["left", "left", "right", "right", "right"]new_image = [0.6, 0.4, 0.5, 0.3]from math import sqrt def distance(first, second): total = 0 for a, b in zip(first, second): total += (a - b) ** 2 return sqrt(total) pairs = []for name, image, label in zip(names, training_images, training_labels): pairs.append((distance(new_image, image), label, name))pairs.sort()for gap, label, name in pairs: print(name, label, round(gap, 3))print("k=1 prediction:", pairs[0][1])Run and read the order from top to bottom.
What to look for
The order is E, B, A, D, C, and the k=1 prediction is right, because E is labelled right.
Make it yours
Compare this with the guess you made in stage 1. Does a single neighbour agree with you?
Why ask more than one neighbour
E is the single closest image, but it is a blurry example that a person could easily have labelled the other way. Trusting one neighbour means trusting its label completely, mistakes included. Asking three neighbours lets a clear majority outvote one odd example. Using an odd number such as 3 also avoids a tie when there are two labels. There is no perfect k: very small values follow single odd examples, and very large values drown out the local detail. Later lessons compare several values of k on measured evidence.
Build with me · 6
6. Let the three closest vote
With k = 3, the three closest examples vote. pairs[:k] is the first three tuples: E (right), B (left), and A (left). The dictionary counts the votes for each label, exactly as you counted labels in the previous lesson. max(votes, key=...) looks at each label in the dictionary and picks the one with the most votes. Left wins two votes to one. So one neighbour says right and three neighbours say left: the value of k changed the answer, even though no example and no pixel changed.
names = ["A", "B", "C", "D", "E"]training_images = [ [0.9, 0.1, 0.8, 0.0], [1.0, 0.0, 0.6, 0.2], [0.2, 0.9, 0.1, 1.0], [0.0, 0.7, 0.3, 0.9], [0.5, 0.6, 0.4, 0.5],]training_labels = ["left", "left", "right", "right", "right"]new_image = [0.6, 0.4, 0.5, 0.3]from math import sqrt def distance(first, second): total = 0 for a, b in zip(first, second): total += (a - b) ** 2 return sqrt(total) pairs = []for name, image, label in zip(names, training_images, training_labels): pairs.append((distance(new_image, image), label, name))pairs.sort() k = 3votes = {}for gap, label, name in pairs[:k]: votes[label] = votes.get(label, 0) + 1print("Votes from the 3 closest:", votes)prediction = max(votes, key=lambda label: votes[label])print("k=3 prediction:", prediction)Predict the vote counts before running.
What to look for
Votes from the 3 closest: right 1, left 2. The k=3 prediction is left.
Make it yours
Set k to 5 and predict the vote before running. With all five examples voting, the label with more examples overall wins, which shows why k should stay much smaller than the number of stored examples.
The library version
You now know every step the model takes. The library does the same steps, faster and on much larger data. Checking that it gives the same answer as your hand calculation is how you know you understand it.
Build with me · 7
7. Ask the library to do the same
scikit-learn, the machine learning library you imported digits from, has the whole method built in. KNeighborsClassifier(n_neighbors=3) creates an empty model and sets k to 3; this is like the make-a-model block in Level 3. fit hands it the labelled examples. For this model, fitting simply stores them, because there is nothing to tune: the work happens later, when a new image arrives. predict does what you did by hand in stages 3 to 6 and returns the winning label. It expects a batch of images and returns one answer per image, which is why the new image is wrapped in an extra pair of square brackets and why [0] picks the first, and only, answer. kneighbors shows the working: the distances to the chosen neighbours and their positions in the training list.
names = ["A", "B", "C", "D", "E"]training_images = [ [0.9, 0.1, 0.8, 0.0], [1.0, 0.0, 0.6, 0.2], [0.2, 0.9, 0.1, 1.0], [0.0, 0.7, 0.3, 0.9], [0.5, 0.6, 0.4, 0.5],]training_labels = ["left", "left", "right", "right", "right"]new_image = [0.6, 0.4, 0.5, 0.3]from sklearn.neighbors import KNeighborsClassifier model = KNeighborsClassifier(n_neighbors=3)model.fit(training_images, training_labels)print("Library prediction:", model.predict([new_image])[0])found_distances, positions = model.kneighbors([new_image])print("Library distances:", [round(float(gap), 3) for gap in found_distances[0]])print("Neighbour names:", [names[position] for position in positions[0]])Run and compare every number with your hand calculation.
What to look for
The library predicts left, reports distances 0.316, 0.583, and 0.6, and names the neighbours E, B, and A.
Make it yours
Change n_neighbors=3 to n_neighbors=1 and confirm the library now agrees with your k=1 answer.
From four pixels to sixty-four
A digit image in this course is eight by eight pixels, which is 64 brightness values in a row once it is flattened, as in the arrays lesson. The distance recipe does not care how many pixels there are, as long as both images have the same number in the same order.
Build with me · 8
8. The same idea on real digits
The first lines load the digits and split them into three parts, which the next lesson explains line by line; for now, X_train and y_train are the stored training images and their labels, and X_dev holds images the model has not stored. Each digit image has 64 pixels instead of 4, so each distance adds up 64 squared differences instead of 4. Nothing else changes. The model stores 1,077 training digits, and for one new development image it finds the three stored digits at the smallest distance and lets them vote. plt.subplots(1, 4) makes one row of four small picture areas, and the loop draws the new image followed by its three neighbours, each titled with its label.
from sklearn.datasets import load_digitsimport numpy as np digits = load_digits()X = digits.data.astype("float64") / 16.0y = digits.targetfrom sklearn.model_selection import train_test_split # Reserve the final test before trying model settings.X_pool, X_final, y_pool, y_final = train_test_split( X, y, test_size=0.2, random_state=42, stratify=y)X_train, X_dev, y_train, y_dev = train_test_split( X_pool, y_pool, test_size=0.25, random_state=42, stratify=y_pool)from sklearn.neighbors import KNeighborsClassifierimport matplotlib.pyplot as plt model = KNeighborsClassifier(n_neighbors=3)model.fit(X_train, y_train)index = 0query = X_dev[index:index + 1]found_distances, positions = model.kneighbors(query)print("True label of the new image:", y_dev[index])for gap, position in zip(found_distances[0], positions[0]): print("Stored image", position, "label", y_train[position], "distance", round(float(gap), 2))print("Prediction:", model.predict(query)[0])pictures = [query[0]] + [X_train[position] for position in positions[0]]titles = ["New image"] + ["Label " + str(y_train[position]) for position in positions[0]]figure, axes = plt.subplots(1, 4, figsize=(8, 2.4))for axis, picture, title in zip(axes, pictures, titles): axis.imshow(picture.reshape(8, 8), cmap="gray", vmin=0, vmax=1) axis.set_title(title) axis.axis("off")plt.tight_layout()plt.show()Run and look at the four pictures side by side. Do the neighbours look like the new image to you?
What to look for
The new image has true label 9. Its three closest stored images are all labelled 9, so the prediction is 9, and the picture shows why.
Make it yours
Set index = 34 and run again. The closest stored image is a 4, which is correct, but the next two are 7s, so the vote says 7: the same effect as E, B, and A in your tiny example, this time producing a mistake. Look at the pictures and decide whether you can see why those 7s were close.
What this model is good at, and where it struggles
The closest-example model is easy to explain: every prediction can be traced to the stored examples that voted for it, and you can look at them. It needs no training time beyond storing the examples, and it often works well when images that look alike really do share a label.
Its weaknesses come from the same simplicity. It compares pixels in fixed positions, so a digit drawn a little to the left, or with a thicker pen, can look far away from stored examples of the same digit. Every prediction compares the new image with every stored one, which gets slow when there are millions of examples. And it has no way to say "none of these": a scribble still gets the label of whatever happens to be closest.
In the next lesson you give this model every development image, count how often it is right, and compare it with a model that ignores the pixels entirely.
What runs in this page
Everything here, including training, runs inside your browser. The first Run of a visit loads Python and its libraries, which can take a little while; wait for the loading message to finish before deciding something is wrong. Training speed depends on your device. Closing the page stops an unfinished run, so download any file you want to keep.
The complete reference is folded away below. Compare it with your work after trying the steps; changing a personal choice such as a name, a colour, or a display threshold can produce a different valid program.
Full reference solution
This is the final complete program built in the walkthrough. All its setup is included. Personal choices may differ in your own version; model scores are measured when you run, not promises about a future dataset.
from sklearn.datasets import load_digitsimport numpy as np digits = load_digits()X = digits.data.astype("float64") / 16.0y = digits.targetfrom sklearn.model_selection import train_test_split # Reserve the final test before trying model settings.X_pool, X_final, y_pool, y_final = train_test_split( X, y, test_size=0.2, random_state=42, stratify=y)X_train, X_dev, y_train, y_dev = train_test_split( X_pool, y_pool, test_size=0.25, random_state=42, stratify=y_pool)from sklearn.neighbors import KNeighborsClassifierimport matplotlib.pyplot as plt model = KNeighborsClassifier(n_neighbors=3)model.fit(X_train, y_train)index = 0query = X_dev[index:index + 1]found_distances, positions = model.kneighbors(query)print("True label of the new image:", y_dev[index])for gap, position in zip(found_distances[0], positions[0]): print("Stored image", position, "label", y_train[position], "distance", round(float(gap), 2))print("Prediction:", model.predict(query)[0])pictures = [query[0]] + [X_train[position] for position in positions[0]]titles = ["New image"] + ["Label " + str(y_train[position]) for position in positions[0]]figure, axes = plt.subplots(1, 4, figsize=(8, 2.4))for axis, picture, title in zip(axes, pictures, titles): axis.imshow(picture.reshape(8, 8), cmap="gray", vmin=0, vmax=1) axis.set_title(title) axis.axis("off")plt.tight_layout()plt.show()Compare this with your version. Different names and personal choices are fine when the program follows the same logic.
Keep your progress
Sign in and every reading, quiz, and exercise you finish is saved.