A reproducible experimental study of YOLO11n inference under reduced numerical precision, evaluating the trade-offs between object-detection accuracy, model size, latency, and CPU throughput.
The project compares:
- FP32 — full-precision baseline
- FP16 — reduced floating-point precision
- INT8 — post-training integer quantization
Accuracy is evaluated on COCO2017 validation data, while inference performance is benchmarked using ONNX Runtime on CPU.
Reducing numerical precision is a common model-optimization technique for deploying deep-learning models under computational and storage constraints. However, lower precision does not necessarily provide the same benefits across hardware and runtimes, and quantization can introduce a measurable accuracy trade-off.
This project investigates that trade-off experimentally using the lightweight YOLO11n object-detection model.
The experiment keeps the underlying architecture, input resolution, and evaluation dataset consistent while changing the numerical representation:
YOLO11n
│
┌─────────┼─────────┐
▼ ▼ ▼
FP32 FP16 INT8
│ │ │
└─────────┼─────────┘
▼
Accuracy + Performance
Comparison
The goal is not to identify a universally "best" precision, but to determine how each representation behaves under the tested deployment conditions.
| Precision | mAP50 | mAP50-95 | Mean Latency | Throughput | Model Size |
|---|---|---|---|---|---|
| FP32 | 0.5456 | 0.3887 | 17.26 ms | 57.92 FPS | 10.24 MB |
| FP16 | 0.5454 | 0.3883 | 17.70 ms | 56.48 FPS | 5.17 MB |
| INT8 | 0.4855 | 0.3363 | 16.53 ms | 60.48 FPS | 2.96 MB |
FP16
- Reduces model size by approximately 49.5%
- mAP50-95 decreases by only 0.09% relative to FP32
- Does not provide a CPU inference speedup in the tested ONNX Runtime configuration
- Mean latency changes from 17.26 ms to 17.70 ms
INT8
- Reduces model size by approximately 71.1%
- Provides the highest measured throughput at 60.48 FPS
- Reduces mean latency from 17.26 ms to 16.53 ms
- Produces a significant accuracy trade-off
- mAP50-95 decreases by 13.46% relative to FP32
The results demonstrate that numerical precision is a deployment trade-off rather than a simple progression toward "better" performance.
| Property | Value |
|---|---|
| Model | YOLO11n |
| Parameters | 2,616,248 |
| Computational cost | 6.5 GFLOPs |
| Input resolution | 640 × 640 |
| Batch size | 1 |
- Dataset: COCO2017
- Split:
val2017 - Images: 5,000
- Instances: 36,335
- Input size: 640 × 640
- Precision configurations: FP32, FP16, INT8
The same validation split is used for all three configurations.
Performance is measured using:
- ONNX Runtime
CPUExecutionProvider- Batch size: 1
- Input resolution: 640 × 640
- 20 warm-up runs
- 100 measured inference runs
- Fixed set of 100 COCO2017 validation images
Preprocessing is performed outside the timed inference loop, so the reported latency represents model inference time rather than complete end-to-end application latency.
| Precision | mAP50 | mAP50-95 | Precision | Recall |
|---|---|---|---|---|
| FP32 | 0.5456 | 0.3887 | 0.6516 | 0.5043 |
| FP16 | 0.5454 | 0.3883 | 0.6515 | 0.5044 |
| INT8 | 0.4855 | 0.3363 | 0.6134 | 0.4519 |
| Precision | mAP50 Degradation | mAP50-95 Degradation |
|---|---|---|
| FP32 | 0.00% | 0.00% |
| FP16 | 0.04% | 0.09% |
| INT8 | 11.01% | 13.46% |
FP16 is effectively lossless in terms of the measured detection metrics.
INT8, on the other hand, introduces a substantial reduction in detection quality under the current post-training quantization configuration.
| Precision | Mean | Median | P95 | FPS |
|---|---|---|---|---|
| FP32 | 17.26 ms | 16.98 ms | 19.19 ms | 57.92 |
| FP16 | 17.70 ms | 17.52 ms | 18.91 ms | 56.48 |
| INT8 | 16.53 ms | 16.33 ms | 18.18 ms | 60.48 |
INT8 is the fastest configuration in this benchmark, while FP16 is slightly slower than FP32.
This highlights an important deployment consideration: reducing numerical precision does not automatically result in faster inference. Performance depends on the hardware and runtime's ability to efficiently execute the chosen representation.
| Precision | Model Size | Reduction vs FP32 |
|---|---|---|
| FP32 | 10.24 MB | — |
| FP16 | 5.17 MB | 49.5% |
| INT8 | 2.96 MB | 71.1% |
INT8 provides the greatest model-size reduction, while FP16 provides a substantial reduction without the significant accuracy loss observed with INT8.
The experiment also records process-level resident memory:
| Precision | Recorded Process RSS |
|---|---|
| FP32 | 729.14 MB |
| FP16 | 969.50 MB |
| INT8 | 978.36 MB |
These values represent process-level memory usage, not the size of the model weights.
They include Python, ONNX Runtime, loaded libraries, runtime allocations, and other process-level state. Consequently, they should not be interpreted as a direct measurement of model memory requirements.
The experiment follows a consistent precision-comparison pipeline:
1. YOLO11n FP32 baseline
│
├── Accuracy evaluation
│
└── Performance benchmark
│
▼
2. FP16 conversion
│
├── Accuracy evaluation
│
└── Performance benchmark
│
▼
3. Post-training INT8 quantization
│
├── Calibration
├── Accuracy evaluation
└── Performance benchmark
│
▼
4. Final comparison
│
├── Accuracy
├── Latency
├── Throughput
├── Model size
└── Resource usage
All experiments use structured JSON/CSV outputs to make the comparison reproducible.
yolo-quant/
├── benchmarks/
│ ├── benchmark.py
│ └── benchmark_onnx.py
│
├── configs/
│ └── experiment.yaml
│
├── models/
│ ├── yolo11n.pt
│ ├── yolo11n_fp32.onnx
│ ├── yolo11n_fp16.onnx
│ └── yolo11n_int8.onnx
│
├── results/
│ ├── benchmark_images.txt
│ ├── comparison/
│ │ ├── accuracy_comparison.csv
│ │ ├── accuracy_comparison.json
│ │ └── performance_comparison.json
│ ├── final/
│ │ ├── final_comparison.csv
│ │ └── final_comparison.json
│ ├── fp32/
│ │ ├── metrics.json
│ │ ├── performance.json
│ │ └── performance_onnx.json
│ ├── fp16/
│ │ ├── metrics.json
│ │ ├── performance.json
│ │ └── performance_onnx.json
│ └── int8/
│ ├── metrics.json
│ └── performance_onnx.json
│
├── src/
│ ├── baseline.py
│ ├── baseline_fp32.py
│ ├── baseline_fp16.py
│ ├── baseline_int8.py
│ ├── int8_export.py
│ ├── metrics.py
│ ├── compare_accuracy.py
│ └── generate_final_report.py
│
├── environment.yaml
└── README.md
Generated datasets, experiment runs, Python cache files, and operating-system metadata are excluded from version control.
The benchmark uses a fixed manifest containing 100 COCO2017 validation images:
results/benchmark_images.txt
The same image set is used for FP32, FP16, and INT8 performance measurements.
Experiment results are stored in machine-readable formats:
- JSON
- CSV
This allows the final comparison to be regenerated without manually transcribing benchmark results.
The COCO2017 dataset itself is not included in this repository due to its size and licensing/distribution considerations. Users reproducing the experiment should download the dataset separately and place it in the expected project directory.
The project uses a Conda environment with Python dependencies installed through pip.
Key dependencies include:
- Python 3.11
- PyTorch
- Ultralytics
- ONNX
- ONNX Runtime
- NumPy
- OpenCV
- Pandas
- Matplotlib
- PyYAML
- tqdm
- psutil
The exact environment specification is provided in:
environment.yaml
The INT8 model uses post-training quantization (PTQ). Quantization-aware training was not performed.
The observed INT8 accuracy loss therefore does not necessarily represent the best achievable INT8 accuracy for YOLO11n.
Performance results are specific to the tested ONNX Runtime CPU configuration. Results may differ substantially on:
- NVIDIA GPUs
- Apple Silicon GPU/MPS
- TensorRT
- NPUs
- other CPU architectures
- other inference runtimes
The reported latency measures model inference only. Preprocessing, postprocessing, image loading, and application-level overhead are outside the timed inference loop.
The primary follow-up experiment is to evaluate quantization-aware training (QAT).
A future comparison would therefore include:
FP32
│
├── FP16
│
├── PTQ INT8
│
└── QAT INT8
The objective would be to determine whether QAT can recover some of the accuracy lost during post-training INT8 quantization while retaining the model-size and inference-efficiency advantages of INT8.
Additional future work could include benchmarking the models on GPU/NPU hardware and measuring complete end-to-end application latency.
This project experimentally evaluates the effect of numerical precision on YOLO11n deployment.
The results show that FP16 is an effective low-risk optimization: it reduces model size by approximately 50% while maintaining essentially unchanged detection accuracy.
INT8 provides substantially greater compression and the highest measured CPU throughput, but the current post-training quantization configuration introduces a significant detection-quality penalty.
The experiment therefore demonstrates that model optimization should be evaluated in the context of the target deployment environment rather than assuming that lower precision automatically provides better performance.
For the tested configuration:
- FP32 → strongest accuracy baseline
- FP16 → near-lossless accuracy with substantial compression
- INT8 → maximum compression and best CPU throughput, with a significant accuracy trade-off
The next step is to investigate whether quantization-aware training can reduce the INT8 accuracy loss while preserving its efficiency benefits.