AI on the device instead of in the cloud: image processing on embedded hardware

AI and Machine Learning

A device needs to evaluate images by itself – in the field, in the lab or on the machine. The cloud would be convenient, but response time, data protection and missing network connections often argue against it. Then image processing has to run on limited hardware, and that is exactly where many approaches fail that looked good on the developer PC.

Why on the device?

• Response time: decisions within the machine cycle cannot afford a detour through the network.
• Availability: the device keeps working without an internet connection.
• Data protection: images of people or patient samples never leave the device.
• Running costs: no fees per image and no dependence on a cloud provider.

The cloud remains useful for training, analyses across many devices and updates. A split that often works well: learn in the data centre, decide on the device.

Cloud or device?

The hardware determines the method

Embedded platforms range from microcontrollers and ARM processors to modules with their own GPU or AI accelerator. Each class has different limits in memory, computing power and energy. That is why the target hardware belongs at the start of the project, not at the end. A network that runs smoothly on a graphics card can be many times too slow on a small ARM processor.

Making models lean

• Choose smaller network architectures designed for mobile devices.
• Scale down input images or evaluate only the relevant region of the image.
• Quantise: convert weights from 32-bit floating point to 8-bit integers. According to the documentation of Google's LiteRT (formerly TensorFlow Lite), this makes a model about four times smaller and noticeably faster on the CPU. The loss of accuracy is usually small but has to be verified with real images.
• Use a runtime that matches the hardware, such as ONNX Runtime, LiteRT or the chip vendor's tools.
• Put classic preprocessing in front so that the network only has to assess the difficult cases.

Measure runtime – on the real device

Always measure inference time on the target hardware, and not only as an average. Outliers count for the cycle time too. A simple measurement script with ONNX Runtime:

import statistics
import time

import numpy as np
import onnxruntime as ort

session = ort.InferenceSession("model_int8.onnx", providers=["CPUExecutionProvider"])
model_input = session.get_inputs()[0]
shape = [d if isinstance(d, int) else 1 for d in model_input.shape]
data = np.random.rand(*shape).astype(np.float32)  # input assumed to be float32

for _ in range(10):  # warm-up
    session.run(None, {model_input.name: data})

times_ms = []
for _ in range(200):
    start = time.perf_counter()
    session.run(None, {model_input.name: data})
    times_ms.append((time.perf_counter() - start) * 1000)

p95 = statistics.quantiles(times_ms, n=20)[-1]
print(f"Median {statistics.median(times_ms):.1f} ms, 95th percentile: {p95:.1f} ms")

Image capture, preprocessing and passing on the result come on top. Only the total shows whether the cycle time holds.

From practice: finding faces without compute-heavy AI

A manufacturer of thermal imaging cameras wanted to measure people's body temperature. To do so, the camera first had to find where faces were located in the image – not who the people were. The software ran on an embedded system with very limited computing power, and lamps and other heat sources interfered with detection. We examined algorithms and neural networks for speed and detection quality. The result was our own lean methods that reliably distinguish faces from other heat sources and run in real time on the embedded hardware. The lesson: intelligence on the device does not automatically mean a neural network on the device.

Common mistakes

• Developing a model on the PC and moving it to the target hardware only at the end.
• Measuring only the inference, not the whole chain from image capture to result.
• Quantising without re-checking accuracy with real images afterwards.
• Overlooking memory usage and heat build-up in continuous operation.
• Providing no way to update models in the field.
• Using neural networks where a lean classic method would be faster and more robust.

From our projects: face detection on thermal imaging cameras with embedded hardware, real-time drivers for thermography cameras. Technical sources: LiteRT – post-training quantization, ONNX Runtime – quantization.