Why you hold data back

The question your test should answer

You want to know whether the model can classify mugs and glasses it has not seen, not whether it recognises the same desk scene again. So first decide what job the model is for. A model for one fixed camera on one desk has a different job from a model for anyone's phone in any kitchen.

A useful test uses photos that match that job and were not used in training. Testing on training photos usually makes the model look better than it really is, because training has already adjusted it to those exact photos and their labels.

Here is a comparison. A student who practises the same twenty arithmetic questions over and over can answer those twenty. Whether they can answer new questions of the same kind is a different question, and only new questions can answer it. Models do not learn the way people do, but the point about the test holds.

Three jobs, used the same way every time

You gave your photos these jobs when you set up the lab. Here is what each job is for, and what you are allowed to change because of it:

Photos kept out of training on purpose are often called held-out photos. Many tools call the development set a validation set.

You may check the development set as often as you like; that is what makes it useful for comparing versions. But each time its results lead you to change something, it helps shape the model, so it can no longer act as an untouched test.

That is why the final test stays closed until you have picked a version. If you look at its mistakes and then change the model, the final test has become one more development set. Say so honestly, and collect fresh photos if you need a fresh final score.

Split by what should be new

Split physical objects before you take photos. All photos of one mug belong to one job. So do photos taken in one quick burst, and any copies or edited versions of a photo. Otherwise the test contains near-copies of training photos and rewards the model for recognising them.

With enough objects, use several per class for training, at least one different object per class for development, and different objects again for the final test. More objects make a stronger test. No split percentage makes a tiny collection reliable.

When information from the test photos sneaks into training, even through a near-copy or a duplicate, it is called data leakage. Leakage makes a score look better than the model deserves. Never copy photos between jobs to improve a score.

1One mug, one quick burst of photos2  photo 1  Training3  photo 2  Training4  photo 3  Training5Do not move photo 3 to Final test just because it is a separate file.

Recognising overfitting

Overfitting means the model has learned details that are special to its training photos and do not carry over to new ones. The warning sign is a big gap: the model fits its training photos very well (a very low training loss) but does clearly worse on development photos. A model that does well on new examples it never trained on is said to generalise.

A gap does not prove overfitting on its own. Different conditions between the sets, wrong labels, or a very small development set can also cause one. Look at the photos it got wrong before deciding.

If the model does badly on both (a training loss that stays high, and poor development results), look elsewhere: labels that disagree, photos that do not show the object clearly, or too little training. Scores that hover near 50% on ordinary photos point the same way: the model has not found a clear difference, and labels applied with two different rules are a common reason. More rounds are not an automatic fix. If moving the same object from one background to another changes the prediction, you may have found a background shortcut. Test that idea by photographing both classes on both backgrounds.

A confident wrong answer shows that confidence is not a guarantee. On its own, it does not tell you which shortcut the model learned.

1Split record (example)2  Training:     mugs A, B, C and glasses A, B, C (desk and blue mat)3  Development:  mug D and glass D (kitchen)4  Final test:   mug E and glass E (another room), kept closed5  Chooses improvements:   Development6  Untouched until the end: Final test

11. Name the two classes and write a one-sentence rule for each.22. Reserve objects or sessions for Development and Final test before collecting.33. In Photos, choose Training and a label, then add varied photos of both classes.44. Add reserved photos under Development and Final test, never under Training.55. In Notes, record label rules, objects, conditions, the split, and a run name.66. In Train, check both counts, keep 20 rounds for the first run, and press Train model.77. Download project and Download model, with matching run names.88. In Test & export, predict Development and read every ID, actual label, prediction, and score.99. Work out accuracy yourself, read the confusion matrix, and compare with the Always- baseline.1010. Write one hypothesis. Change only the Training photos (or one stated setting), retrain, and predict the same Development collection.1111. Choose a version using Development results. Open its saved model if needed.1212. Predict Final test once, after choosing, and report its fraction, conditions, and limits.1313. For photos with no known answer, use Unlabelled targets, predict, and Download predictions.csv.1414. Keep project, model, notes, and predictions file together under one run name. None of them replaces another.