How many images does an AI model need? Planning training data properly

AI and Machine Learning

“How many images do we need?” is the first question in almost every AI project. The honest answer is: it depends. That sounds like an excuse, but it is the start of a plan. Because which images you collect matters more than how many.

Why there is no fixed number

A learner driver who practises for thirty hours in an empty car park is not ready for the city centre. An AI model is no different: it only learns what it has seen. How many images it needs depends mainly on five things:

• The task: classification assesses the whole image, detection finds objects with bounding boxes, segmentation assigns every pixel. The finer the task, the more work goes into labelling each image.
• The distance between classes: a missing component is easier to learn than a fine scratch on a textured surface.
• The variation: variants, suppliers, shifts, dirt and part position – everything that changes in operation has to appear in the data.
• The starting point: a model pre-trained on large public datasets needs far fewer images of your own than one that starts from scratch.
• The requirement: every further improvement costs more data than the one before. A Google Research study found that performance on vision tasks grows only logarithmically with the amount of training data. So every doubling brings less than the last.

Plan in cells, not totals

“5,000 images” sounds like a lot. If 4,800 of them show the same variant from the early shift, they are of little use. So plan in a table: variants and conditions in the rows, classes in the columns, a minimum number per cell. A short script checks whether the collected data meets the plan:

import pandas as pd

# One row per labelled image: file, variant, shift, label
data = pd.read_csv("images.csv")
table = pd.crosstab([data["variant"], data["shift"]], data["label"])

# Make combinations that are missing entirely visible
rows = pd.MultiIndex.from_product(
    [data["variant"].unique(), data["shift"].unique()],
    names=["variant", "shift"])
table = table.reindex(rows, fill_value=0)

MINIMUM = 50  # your target per cell, derived from the task and the learning curve
cells = table.stack()
for (variant, shift, label), count in cells[cells < MINIMUM].items():
    print(f"Too few: {variant} / {shift} / {label}: {count} images")

The value 50 is only a placeholder. Your minimum follows from the task and the learning curve. The “Too few” lines are your shopping list for the next round of image capture. Combinations that are missing entirely are the most dangerous. That is why the script fills them with zero instead of silently leaving them out.

The learning curve shows whether more data helps

You do not have to guess whether collecting more is worth it. Train with a quarter, half and all of the available images and measure each time on the same, unchanged test set. If the curve is still rising clearly, more images help. If it is flat, more images of the same kind will hardly change anything. Then you need different images, better labels or a different method.

Good labels beat many images

A model cannot be better than its labels. A 2021 study shows how common errors are: in the test sets of ten widely used datasets, at least 3.3 percent of the labels were wrong on average, and at least 6 percent in the ImageNet validation set. And that is in carefully curated public data. Three measures help:

• A labelling guide with image examples, above all for borderline cases: when does a scratch count as a defect?
• A sample is labelled twice, independently by two people. Where they disagree, the quality assurance experts decide.
• Corrections go into a new, versioned release of the dataset.

Augmentation does not replace real images

Rotating, mirroring, brightening and darkening: such artificial variations help if these fluctuations also occur on the line. They do not create a new defect type or the surface of a new supplier. Synthetic defect images are only useful, too, if the model is finally tested on real images.

Typical mistakes

• Setting a total as the target instead of a minimum per cell.
• Collecting images where it is convenient: on one line, in one shift, from one supplier.
• Counting burst shots of the same part as many images. They add hardly any new information.
• Reassembling the test set for every run. Then the results can no longer be compared.
• Labelling without a guide, with everyone applying their own standard.
• Generating variations that never occur in reality, such as parts upside down.
• Not versioning the data. Later nobody knows which images the model in operation learned from.

Sources: Northcutt, Athalye, Mueller, “Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks” (NeurIPS 2021); Sun, Shrivastava, Singh, Gupta, “Revisiting Unreasonable Effectiveness of Data in Deep Learning Era” (ICCV 2017).