Edge AI Deployment, Part 1: Model Compression and On-Device Inference

Why Edge AI Now

Cloud AI is powerful, but its limits are clear. In environments where network latency exceeds 100ms, workloads that need real-time decisions—autonomous driving, industrial robots, smart cameras—cannot wait on a round trip to the cloud. Edge AI solves this by running inference on the device itself.

Cloud vs. Edge: Decision Criteria

When you should choose the edge:

• Latency SLA < 50ms
• Offline operation is required
• GDPR/HIPAA regulations apply
• High data volume → cloud upload costs are excessive

Edge Device Types and How to Choose

MCU (Microcontroller Unit)

• Examples: STM32, Arduino Nano 33 BLE
• RAM: 256KB–1MB, best for: TinyML (< 100KB models)
• Use cases: keyword spotting, abnormal vibration detection

Edge GPU / NPU

• Examples: NVIDIA Jetson Orin, Google Coral
• TOPS: 10–275, best for: MobileNet, EfficientDet
• Use cases: object detection, face authentication

Industrial Edge Servers

• Examples: Intel Core Ultra + OpenVINO
• Use cases: factory vision inspection, predictive maintenance

Comparing Major Deployment Frameworks

TensorFlow Lite

tflite_convert \
  --saved_model_dir=./model \
  --output_file=model.tflite \
  --optimizations=DEFAULT

• Pros: Largest ecosystem, first-class Android/iOS support
• Cons: TensorFlow models only

ONNX Runtime

import onnxruntime as ort
sess = ort.InferenceSession("model.onnx")
outputs = sess.run(None, {"input": input_data})

• Pros: Framework-agnostic (PyTorch/TF/sklearn all work)
• Cons: Mobile optimization is weaker than TFLite

OpenVINO (Intel)

mo --input_model model.onnx --output_dir ./ir_model

• Pros: Maximizes hardware acceleration on Intel CPU/GPU/VPU

Pre-Deployment Checklist

1. Power budget: < 100mW per inference
2. Peak memory: Runtime max RAM at or below 70% of the device limit
3. Thermal management: Stay under 80°C during continuous inference
4. Model versioning: Design an OTA update channel
5. Fallback strategy: Prepare a rule-based alternative if inference fails

Part 2 covers practical methods to cut model size by 75% with quantization and pruning while keeping accuracy loss under 1%.

Three Levers for Shrinking Models (Preview)

We go deep in Part 2, but edge deployment succeeds or fails on model lightweighting, so here are the essentials.

In most cases, INT8 quantization alone shrinks the model 4× with less than 1% accuracy loss.

Security Is Non-Negotiable on the Edge

Devices are physically exposed, so the model itself is a theft target.

• Encrypt model files and use Secure Boot
• Validate inference inputs (defend against adversarial examples)
• Verify signatures on the OTA update channel (block forged firmware)

Edge MLOps — Deployment Is Not the End

Models scattered across thousands of devices drift in version. You need central monitoring of model version, accuracy, and drift, plus staged (canary) rollouts and automatic rollback channels if operations are going to stay sustainable.

Checking Whether Compression Hurt Accuracy

A commonly missed edge problem is shipping a model that got faster but quietly lost accuracy.

Causes

• Quantization calibration data differs from real field input (lighting, camera, sensor noise), so value ranges are misestimated
• Preprocessing mismatch: training used RGB with 0–1 normalization, but device code feeds BGR 0–255
• Some ops are precision-sensitive, so full INT8 conversion drops accuracy sharply for specific classes

Verification: compare original and compressed models on the same validation set

import numpy as np, onnxruntime as ort
fp32 = ort.InferenceSession("model_fp32.onnx")
int8 = ort.InferenceSession("model_int8.onnx")
name = fp32.get_inputs()[0].name
agree = 0
for x, y in val_loader:                    # include samples collected in the field
    p1 = fp32.run(None, {name: x})[0].argmax(1)
    p2 = int8.run(None, {name: x})[0].argmax(1)
    agree += (p1 == p2).sum()
print("top-1 agreement:", agree / len(val_loader.dataset))

Compare per-class accuracy as well as overall agreement to see whether only some classes degrade.

Fix

• Include field samples in calibration data and use per-channel quantization where available.
• Keep layers that lose the most accuracy in FP16/FP32 (mixed precision).
• Share one preprocessing function between training and device code, or bake it into the model graph.

Prevention

• Gate CI on "maximum allowed accuracy drop vs. original" and don't produce a deployable artifact when it's exceeded.

FAQ

Q. Do I have to pick only edge or only cloud? No. In practice, hybrids are common — a cascade where the edge does first-pass inference (low latency) and only ambiguous cases go to the cloud for precise analysis. That balances cost and accuracy.

Q. Which device should I start with? For a PoC, start with developer-friendly boards like Jetson or Coral, measure inference performance and power, then decide at production based on unit cost and supply.