Your drawing, the network, and the closest-example model
Give the same drawing to your trained network and to the closest-example model, then change the drawing on purpose to find what each one is sensitive to.
Work here, beside the explanation
In Module 2 you gave the closest-example model a digit you drew yourself. Now you have trained a neural network too, so give the same drawing to both models and compare how they answer. Then change the drawing on purpose, with thinner strokes and a small shift, to find out what each model is sensitive to. The drawing preparation is the same as in Module 2: eight rows of eight symbols, turned into 64 brightness values from 0 to 1.
Each numbered stage below shows a complete program. Try this stage copies it into the editor beside the article, including every line it needs from earlier stages, so it works even after you reload the page. Read the program first, predict what it will print, then press Run. Loading a stage replaces what is in the editor; Undo brings your own version back.
Build with me · 1
1. Train both models on the same images
The program trains the network from the previous lessons (model) and the closest-example model (nearest) on the same 1,077 training images. Giving the second model its own name keeps the two apart. Both development accuracies are printed as a reminder of how the two models compare on the packaged digits, before either sees your drawing.
from sklearn.datasets import load_digitsimport numpy as np digits = load_digits()X = digits.data.astype("float64") / 16.0y = digits.targetfrom sklearn.model_selection import train_test_split # Reserve the final test before trying model settings.X_pool, X_final, y_pool, y_final = train_test_split( X, y, test_size=0.2, random_state=42, stratify=y)X_train, X_dev, y_train, y_dev = train_test_split( X_pool, y_pool, test_size=0.25, random_state=42, stratify=y_pool)from sklearn.neural_network import MLPClassifier model = MLPClassifier( hidden_layer_sizes=(32,), activation="relu", solver="adam", learning_rate_init=0.003, batch_size=64, max_iter=100, random_state=42, tol=0.0001,)model.fit(X_train, y_train)from sklearn.neighbors import KNeighborsClassifier nearest = KNeighborsClassifier(n_neighbors=3)nearest.fit(X_train, y_train)print("Network development accuracy:", round(model.score(X_dev, y_dev), 3))print("Closest-example development accuracy:", round(nearest.score(X_dev, y_dev), 3))Run and wait for both models to train. You may see the ConvergenceWarning from earlier; it is only a note.
What to look for
In our run the network scores about 0.978 and the closest-example model about 0.981 on development.
Make it yours
Write down both numbers so you can compare them with how the models do on your drawings.
Build with me · 2
2. Prepare your drawing
These are the drawing lines from Module 2. Each string is one row: a dot is dark, a plus sign is half bright, a hash is bright. The checks stop the program if a row has the wrong length or an unknown symbol. The dictionary and the list comprehension turn the symbols into an eight-by-eight grid of numbers, and reshape(1, 64) flattens it into a batch of one image, which is what both models expect. The starter drawing is a seven with strokes two pixels thick, like the sevens in the dataset.
from sklearn.datasets import load_digitsimport numpy as np digits = load_digits()X = digits.data.astype("float64") / 16.0y = digits.targetfrom sklearn.model_selection import train_test_split # Reserve the final test before trying model settings.X_pool, X_final, y_pool, y_final = train_test_split( X, y, test_size=0.2, random_state=42, stratify=y)X_train, X_dev, y_train, y_dev = train_test_split( X_pool, y_pool, test_size=0.25, random_state=42, stratify=y_pool)from sklearn.neural_network import MLPClassifier model = MLPClassifier( hidden_layer_sizes=(32,), activation="relu", solver="adam", learning_rate_init=0.003, batch_size=64, max_iter=100, random_state=42, tol=0.0001,)model.fit(X_train, y_train)from sklearn.neighbors import KNeighborsClassifier nearest = KNeighborsClassifier(n_neighbors=3)nearest.fit(X_train, y_train) rows = [ "..####..", "..####..", "....##..", "...##...", "...##...", "..##....", "..##....", "........",]if len(rows) != 8 or any(len(row) != 8 for row in rows): raise ValueError("Use exactly eight rows of eight characters.")if any(character not in ".+#" for row in rows for character in row): raise ValueError("Use only . for dark, + for half-bright, and # for bright.")brightness = {".": 0.0, "+": 0.5, "#": 1.0}image = np.array([[brightness[character] for character in row] for row in rows])if image.max() == 0: raise ValueError("The drawing is blank. Add a digit before predicting.")one_input = image.reshape(1, 64)import matplotlib.pyplot as plt plt.figure(figsize=(3, 3))plt.imshow(image, cmap="gray", vmin=0, vmax=1)plt.title("My prepared drawing")plt.axis("off")plt.tight_layout()plt.show()Run and look at the picture. Edit the rows into a digit of your own if you like, and write down which digit you meant before going further.
What to look for
An eight-by-eight picture of your drawing appears.
Make it yours
If you change the drawing, keep every row exactly eight characters long and keep an empty border where you can.
Build with me · 3
3. Ask the network
predict_proba returns the network's ten chances for this image, the softmax output from the neural network lesson. np.argsort(chances) lists the positions from the smallest chance to the largest, and [::-1] reverses that list so the largest comes first. The loop prints the top three digits with their chances. A chance near 1 means the network strongly prefers one digit; it does not prove that digit is right.
from sklearn.datasets import load_digitsimport numpy as np digits = load_digits()X = digits.data.astype("float64") / 16.0y = digits.targetfrom sklearn.model_selection import train_test_split # Reserve the final test before trying model settings.X_pool, X_final, y_pool, y_final = train_test_split( X, y, test_size=0.2, random_state=42, stratify=y)X_train, X_dev, y_train, y_dev = train_test_split( X_pool, y_pool, test_size=0.25, random_state=42, stratify=y_pool)from sklearn.neural_network import MLPClassifier model = MLPClassifier( hidden_layer_sizes=(32,), activation="relu", solver="adam", learning_rate_init=0.003, batch_size=64, max_iter=100, random_state=42, tol=0.0001,)model.fit(X_train, y_train)from sklearn.neighbors import KNeighborsClassifier nearest = KNeighborsClassifier(n_neighbors=3)nearest.fit(X_train, y_train) rows = [ "..####..", "..####..", "....##..", "...##...", "...##...", "..##....", "..##....", "........",]if len(rows) != 8 or any(len(row) != 8 for row in rows): raise ValueError("Use exactly eight rows of eight characters.")if any(character not in ".+#" for row in rows for character in row): raise ValueError("Use only . for dark, + for half-bright, and # for bright.")brightness = {".": 0.0, "+": 0.5, "#": 1.0}image = np.array([[brightness[character] for character in row] for row in rows])if image.max() == 0: raise ValueError("The drawing is blank. Add a digit before predicting.")one_input = image.reshape(1, 64)chances = model.predict_proba(one_input)[0]order = np.argsort(chances)[::-1]for position in order[:3]: print("Digit", int(model.classes_[position]), "chance", round(float(chances[position]), 3))Run and compare the top digit with the digit you meant.
What to look for
For the starter seven, the network gives 7 a chance of about 0.978.
Make it yours
Make one stroke of your drawing fainter with + symbols and watch how the chances shift.
Build with me · 4
4. Ask the closest-example model and see its neighbours
The closest-example model can show its reasons: kneighbors returns the three stored training images nearest to your drawing, exactly as in the lesson where you worked it out by hand. The picture shows your drawing next to those three neighbours with their labels. The network cannot show its reasons this directly, because its answer comes from 2,410 learned numbers rather than from a few stored examples.
from sklearn.datasets import load_digitsimport numpy as np digits = load_digits()X = digits.data.astype("float64") / 16.0y = digits.targetfrom sklearn.model_selection import train_test_split # Reserve the final test before trying model settings.X_pool, X_final, y_pool, y_final = train_test_split( X, y, test_size=0.2, random_state=42, stratify=y)X_train, X_dev, y_train, y_dev = train_test_split( X_pool, y_pool, test_size=0.25, random_state=42, stratify=y_pool)from sklearn.neural_network import MLPClassifier model = MLPClassifier( hidden_layer_sizes=(32,), activation="relu", solver="adam", learning_rate_init=0.003, batch_size=64, max_iter=100, random_state=42, tol=0.0001,)model.fit(X_train, y_train)from sklearn.neighbors import KNeighborsClassifier nearest = KNeighborsClassifier(n_neighbors=3)nearest.fit(X_train, y_train) rows = [ "..####..", "..####..", "....##..", "...##...", "...##...", "..##....", "..##....", "........",]if len(rows) != 8 or any(len(row) != 8 for row in rows): raise ValueError("Use exactly eight rows of eight characters.")if any(character not in ".+#" for row in rows for character in row): raise ValueError("Use only . for dark, + for half-bright, and # for bright.")brightness = {".": 0.0, "+": 0.5, "#": 1.0}image = np.array([[brightness[character] for character in row] for row in rows])if image.max() == 0: raise ValueError("The drawing is blank. Add a digit before predicting.")one_input = image.reshape(1, 64)import matplotlib.pyplot as plt found_distances, positions = nearest.kneighbors(one_input)print("Closest-example vote:", nearest.predict(one_input)[0])print("Neighbour labels:", y_train[positions[0]])pictures = [image] + [X_train[position].reshape(8, 8) for position in positions[0]]titles = ["My drawing"] + ["Label " + str(y_train[position]) for position in positions[0]]figure, axes = plt.subplots(1, 4, figsize=(8, 2.4))for axis, picture, title in zip(axes, pictures, titles): axis.imshow(picture, cmap="gray", vmin=0, vmax=1) axis.set_title(title) axis.axis("off")plt.tight_layout()plt.show()Run and look at the neighbours. Do they look like your drawing?
What to look for
For the starter seven, all three neighbours are sevens, so the vote is 7.
Make it yours
Compare the neighbours' stroke thickness with your drawing's.
Build with me · 5
5. Change the drawing on purpose
Now test what the models are sensitive to. The program makes three versions of a seven and stores them in a dictionary, each under a name. The first is your prepared drawing. The second, thin_rows, is a seven drawn with strokes only one pixel wide. prepare is a small function that does the symbols-to-numbers step for any drawing. The third moves your drawing one column to the right: np.roll(image, 1, axis=1) shifts every pixel one place along each row, and the last column wraps round to the first, which is harmless here because that column is empty. The loop asks both models about every version.
from sklearn.datasets import load_digitsimport numpy as np digits = load_digits()X = digits.data.astype("float64") / 16.0y = digits.targetfrom sklearn.model_selection import train_test_split # Reserve the final test before trying model settings.X_pool, X_final, y_pool, y_final = train_test_split( X, y, test_size=0.2, random_state=42, stratify=y)X_train, X_dev, y_train, y_dev = train_test_split( X_pool, y_pool, test_size=0.25, random_state=42, stratify=y_pool)from sklearn.neural_network import MLPClassifier model = MLPClassifier( hidden_layer_sizes=(32,), activation="relu", solver="adam", learning_rate_init=0.003, batch_size=64, max_iter=100, random_state=42, tol=0.0001,)model.fit(X_train, y_train)from sklearn.neighbors import KNeighborsClassifier nearest = KNeighborsClassifier(n_neighbors=3)nearest.fit(X_train, y_train) rows = [ "..####..", "..####..", "....##..", "...##...", "...##...", "..##....", "..##....", "........",]if len(rows) != 8 or any(len(row) != 8 for row in rows): raise ValueError("Use exactly eight rows of eight characters.")if any(character not in ".+#" for row in rows for character in row): raise ValueError("Use only . for dark, + for half-bright, and # for bright.")brightness = {".": 0.0, "+": 0.5, "#": 1.0}image = np.array([[brightness[character] for character in row] for row in rows])if image.max() == 0: raise ValueError("The drawing is blank. Add a digit before predicting.")one_input = image.reshape(1, 64) thin_rows = [ "..####..", ".....#..", "....#...", "...#....", "..#.....", "..#.....", "..#.....", "........",] def prepare(drawing_rows): grid = np.array([[brightness[character] for character in row] for row in drawing_rows]) return grid.reshape(1, 64) variants = { "thick seven": one_input, "thin seven": prepare(thin_rows), "thick seven, moved right": np.roll(image, 1, axis=1).reshape(1, 64),}for name, variant in variants.items(): chances = model.predict_proba(variant)[0] network_answer = int(model.classes_[np.argmax(chances)]) print(name, "| network:", network_answer, "chance", round(float(chances.max()), 3), "| closest example:", nearest.predict(variant)[0])Predict each model's answer for the thin seven before running.
What to look for
In our run both models read the thick seven and the moved seven as 7, but both read the thin seven as 8, the network with a chance of about 0.78.
Make it yours
Look back at the dataset images in the arrays lesson. Their strokes are thick and slightly blurred, so a one-pixel line looks unlike any stored seven. Add your own variant, such as a seven with a crossbar, and see how each model answers.
Build with me · 6
6. Keep a record of what you tried
A record lets you come back to a surprising result instead of relying on memory. The dictionary stores the digit you meant, both drawings, and each model's answer for every version, and json writes it to a file. record["results"][name] = {...} puts a small dictionary inside the bigger one, one entry per version. Set intended_digit to the digit you actually drew before running. A handful of your own drawings cannot tell you a model's accuracy; they are probes that show you what kind of input it finds hard.
from sklearn.datasets import load_digitsimport numpy as np digits = load_digits()X = digits.data.astype("float64") / 16.0y = digits.targetfrom sklearn.model_selection import train_test_split # Reserve the final test before trying model settings.X_pool, X_final, y_pool, y_final = train_test_split( X, y, test_size=0.2, random_state=42, stratify=y)X_train, X_dev, y_train, y_dev = train_test_split( X_pool, y_pool, test_size=0.25, random_state=42, stratify=y_pool)from sklearn.neural_network import MLPClassifier model = MLPClassifier( hidden_layer_sizes=(32,), activation="relu", solver="adam", learning_rate_init=0.003, batch_size=64, max_iter=100, random_state=42, tol=0.0001,)model.fit(X_train, y_train)from sklearn.neighbors import KNeighborsClassifier nearest = KNeighborsClassifier(n_neighbors=3)nearest.fit(X_train, y_train) rows = [ "..####..", "..####..", "....##..", "...##...", "...##...", "..##....", "..##....", "........",]if len(rows) != 8 or any(len(row) != 8 for row in rows): raise ValueError("Use exactly eight rows of eight characters.")if any(character not in ".+#" for row in rows for character in row): raise ValueError("Use only . for dark, + for half-bright, and # for bright.")brightness = {".": 0.0, "+": 0.5, "#": 1.0}image = np.array([[brightness[character] for character in row] for row in rows])if image.max() == 0: raise ValueError("The drawing is blank. Add a digit before predicting.")one_input = image.reshape(1, 64) thin_rows = [ "..####..", ".....#..", "....#...", "...#....", "..#.....", "..#.....", "..#.....", "........",] def prepare(drawing_rows): grid = np.array([[brightness[character] for character in row] for row in drawing_rows]) return grid.reshape(1, 64) variants = { "thick seven": one_input, "thin seven": prepare(thin_rows), "thick seven, moved right": np.roll(image, 1, axis=1).reshape(1, 64),}import json intended_digit = 7record = {"intended_digit": intended_digit, "rows": rows, "thin_rows": thin_rows, "results": {}}for name, variant in variants.items(): chances = model.predict_proba(variant)[0] record["results"][name] = { "network": int(model.classes_[np.argmax(chances)]), "network_chance": round(float(chances.max()), 3), "closest_example": int(nearest.predict(variant)[0]), }print(json.dumps(record, indent=2))with open("my-digit-comparison.json", "w") as handle: json.dump(record, handle, indent=2)print("Saved my-digit-comparison.json")Set intended_digit, run, and download my-digit-comparison.json if you want to keep it.
What to look for
The printed record lists the three versions with both models' answers, and the file is offered for download.
Make it yours
Draw two or three more digits in different styles and record every result, including the ones the models get wrong.
What the comparison shows
Both models learned from neat, thick digits, so they share a blind spot: a thin, sharp drawing can look more like a different digit than like its own. This is called a distribution shift, when the inputs a model receives differ from the examples it learned from. Neither model knows it is out of its depth; each still gives an answer, and the network even gives a fairly confident one.
The fix is not a cleverer model. Either prepare new inputs to look like the training data (thicker, centred, on the same brightness scale), or add training examples that look like the inputs you expect.
A photograph or a large drawing would need more preparation before it fits this model: turn it grey, make the digit bright on a dark background, crop to the digit, shrink it to eight by eight, and scale brightness to 0 to 1. Each of those steps changes the numbers the model sees, so inspect the final grid before trusting the answer.
What runs in this page
Everything here, including training, runs inside your browser. The first Run of a visit loads Python and its libraries, which can take a little while; wait for the loading message to finish before deciding something is wrong. Training speed depends on your device. Closing the page stops an unfinished run, so download any file you want to keep.
This course trains on the 1,797 small digit images that come with scikit-learn: eight by eight pixels, brightness 0 to 16. Many tutorials elsewhere use MNIST, a larger collection of 70,000 digit images that are 28 by 28 pixels with brightness 0 to 255, and a library called Keras. The ideas you learn here carry over, but the image sizes and scales differ, so a model trained on one cannot read the other's images.
The complete reference is folded away below. Compare it with your work after trying the steps; changing a personal choice such as a name, a colour, or a display threshold can produce a different valid program.
Full reference solution
This is the final complete program built in the walkthrough. All its setup is included. Personal choices may differ in your own version; model scores are measured when you run, not promises about a future dataset.
from sklearn.datasets import load_digitsimport numpy as np digits = load_digits()X = digits.data.astype("float64") / 16.0y = digits.targetfrom sklearn.model_selection import train_test_split # Reserve the final test before trying model settings.X_pool, X_final, y_pool, y_final = train_test_split( X, y, test_size=0.2, random_state=42, stratify=y)X_train, X_dev, y_train, y_dev = train_test_split( X_pool, y_pool, test_size=0.25, random_state=42, stratify=y_pool)from sklearn.neural_network import MLPClassifier model = MLPClassifier( hidden_layer_sizes=(32,), activation="relu", solver="adam", learning_rate_init=0.003, batch_size=64, max_iter=100, random_state=42, tol=0.0001,)model.fit(X_train, y_train)from sklearn.neighbors import KNeighborsClassifier nearest = KNeighborsClassifier(n_neighbors=3)nearest.fit(X_train, y_train) rows = [ "..####..", "..####..", "....##..", "...##...", "...##...", "..##....", "..##....", "........",]if len(rows) != 8 or any(len(row) != 8 for row in rows): raise ValueError("Use exactly eight rows of eight characters.")if any(character not in ".+#" for row in rows for character in row): raise ValueError("Use only . for dark, + for half-bright, and # for bright.")brightness = {".": 0.0, "+": 0.5, "#": 1.0}image = np.array([[brightness[character] for character in row] for row in rows])if image.max() == 0: raise ValueError("The drawing is blank. Add a digit before predicting.")one_input = image.reshape(1, 64) thin_rows = [ "..####..", ".....#..", "....#...", "...#....", "..#.....", "..#.....", "..#.....", "........",] def prepare(drawing_rows): grid = np.array([[brightness[character] for character in row] for row in drawing_rows]) return grid.reshape(1, 64) variants = { "thick seven": one_input, "thin seven": prepare(thin_rows), "thick seven, moved right": np.roll(image, 1, axis=1).reshape(1, 64),}import json intended_digit = 7record = {"intended_digit": intended_digit, "rows": rows, "thin_rows": thin_rows, "results": {}}for name, variant in variants.items(): chances = model.predict_proba(variant)[0] record["results"][name] = { "network": int(model.classes_[np.argmax(chances)]), "network_chance": round(float(chances.max()), 3), "closest_example": int(nearest.predict(variant)[0]), }print(json.dumps(record, indent=2))with open("my-digit-comparison.json", "w") as handle: json.dump(record, handle, indent=2)print("Saved my-digit-comparison.json")Compare this with your version. Different names and personal choices are fine when the program follows the same logic.
Keep your progress
Sign in and every reading, quiz, and exercise you finish is saved.