Skip to content

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

YOLO11n Precision Benchmark: FP32 vs FP16 vs INT8

A reproducible experimental study of YOLO11n inference under reduced numerical precision, evaluating the trade-offs between object-detection accuracy, model size, latency, and CPU throughput.

The project compares:

  • FP32 — full-precision baseline
  • FP16 — reduced floating-point precision
  • INT8 — post-training integer quantization

Accuracy is evaluated on COCO2017 validation data, while inference performance is benchmarked using ONNX Runtime on CPU.


Overview

Reducing numerical precision is a common model-optimization technique for deploying deep-learning models under computational and storage constraints. However, lower precision does not necessarily provide the same benefits across hardware and runtimes, and quantization can introduce a measurable accuracy trade-off.

This project investigates that trade-off experimentally using the lightweight YOLO11n object-detection model.

The experiment keeps the underlying architecture, input resolution, and evaluation dataset consistent while changing the numerical representation:

                    YOLO11n
                       │
             ┌─────────┼─────────┐
             ▼         ▼         ▼
           FP32      FP16      INT8
             │         │         │
             └─────────┼─────────┘
                       ▼
             Accuracy + Performance
                    Comparison

The goal is not to identify a universally "best" precision, but to determine how each representation behaves under the tested deployment conditions.


Key Results

Precision mAP50 mAP50-95 Mean Latency Throughput Model Size
FP32 0.5456 0.3887 17.26 ms 57.92 FPS 10.24 MB
FP16 0.5454 0.3883 17.70 ms 56.48 FPS 5.17 MB
INT8 0.4855 0.3363 16.53 ms 60.48 FPS 2.96 MB

Main findings

FP16

  • Reduces model size by approximately 49.5%
  • mAP50-95 decreases by only 0.09% relative to FP32
  • Does not provide a CPU inference speedup in the tested ONNX Runtime configuration
  • Mean latency changes from 17.26 ms to 17.70 ms

INT8

  • Reduces model size by approximately 71.1%
  • Provides the highest measured throughput at 60.48 FPS
  • Reduces mean latency from 17.26 ms to 16.53 ms
  • Produces a significant accuracy trade-off
  • mAP50-95 decreases by 13.46% relative to FP32

The results demonstrate that numerical precision is a deployment trade-off rather than a simple progression toward "better" performance.


Experimental Setup

Model

Property Value
Model YOLO11n
Parameters 2,616,248
Computational cost 6.5 GFLOPs
Input resolution 640 × 640
Batch size 1

Accuracy Evaluation

  • Dataset: COCO2017
  • Split: val2017
  • Images: 5,000
  • Instances: 36,335
  • Input size: 640 × 640
  • Precision configurations: FP32, FP16, INT8

The same validation split is used for all three configurations.

Performance Benchmark

Performance is measured using:

  • ONNX Runtime
  • CPUExecutionProvider
  • Batch size: 1
  • Input resolution: 640 × 640
  • 20 warm-up runs
  • 100 measured inference runs
  • Fixed set of 100 COCO2017 validation images

Preprocessing is performed outside the timed inference loop, so the reported latency represents model inference time rather than complete end-to-end application latency.


Accuracy Results

Precision mAP50 mAP50-95 Precision Recall
FP32 0.5456 0.3887 0.6516 0.5043
FP16 0.5454 0.3883 0.6515 0.5044
INT8 0.4855 0.3363 0.6134 0.4519

Accuracy degradation relative to FP32

Precision mAP50 Degradation mAP50-95 Degradation
FP32 0.00% 0.00%
FP16 0.04% 0.09%
INT8 11.01% 13.46%

FP16 is effectively lossless in terms of the measured detection metrics.

INT8, on the other hand, introduces a substantial reduction in detection quality under the current post-training quantization configuration.


Performance Results

Precision Mean Median P95 FPS
FP32 17.26 ms 16.98 ms 19.19 ms 57.92
FP16 17.70 ms 17.52 ms 18.91 ms 56.48
INT8 16.53 ms 16.33 ms 18.18 ms 60.48

INT8 is the fastest configuration in this benchmark, while FP16 is slightly slower than FP32.

This highlights an important deployment consideration: reducing numerical precision does not automatically result in faster inference. Performance depends on the hardware and runtime's ability to efficiently execute the chosen representation.


Model Compression

Precision Model Size Reduction vs FP32
FP32 10.24 MB —
FP16 5.17 MB 49.5%
INT8 2.96 MB 71.1%

INT8 provides the greatest model-size reduction, while FP16 provides a substantial reduction without the significant accuracy loss observed with INT8.


Memory

The experiment also records process-level resident memory:

Precision Recorded Process RSS
FP32 729.14 MB
FP16 969.50 MB
INT8 978.36 MB

These values represent process-level memory usage, not the size of the model weights.

They include Python, ONNX Runtime, loaded libraries, runtime allocations, and other process-level state. Consequently, they should not be interpreted as a direct measurement of model memory requirements.


Methodology

The experiment follows a consistent precision-comparison pipeline:

1. YOLO11n FP32 baseline
        │
        ├── Accuracy evaluation
        │
        └── Performance benchmark
                │
                ▼
2. FP16 conversion
        │
        ├── Accuracy evaluation
        │
        └── Performance benchmark
                │
                ▼
3. Post-training INT8 quantization
        │
        ├── Calibration
        ├── Accuracy evaluation
        └── Performance benchmark
                │
                ▼
4. Final comparison
        │
        ├── Accuracy
        ├── Latency
        ├── Throughput
        ├── Model size
        └── Resource usage

All experiments use structured JSON/CSV outputs to make the comparison reproducible.


Project Structure

yolo-quant/
├── benchmarks/
│   ├── benchmark.py
│   └── benchmark_onnx.py
│
├── configs/
│   └── experiment.yaml
│
├── models/
│   ├── yolo11n.pt
│   ├── yolo11n_fp32.onnx
│   ├── yolo11n_fp16.onnx
│   └── yolo11n_int8.onnx
│
├── results/
│   ├── benchmark_images.txt
│   ├── comparison/
│   │   ├── accuracy_comparison.csv
│   │   ├── accuracy_comparison.json
│   │   └── performance_comparison.json
│   ├── final/
│   │   ├── final_comparison.csv
│   │   └── final_comparison.json
│   ├── fp32/
│   │   ├── metrics.json
│   │   ├── performance.json
│   │   └── performance_onnx.json
│   ├── fp16/
│   │   ├── metrics.json
│   │   ├── performance.json
│   │   └── performance_onnx.json
│   └── int8/
│       ├── metrics.json
│       └── performance_onnx.json
│
├── src/
│   ├── baseline.py
│   ├── baseline_fp32.py
│   ├── baseline_fp16.py
│   ├── baseline_int8.py
│   ├── int8_export.py
│   ├── metrics.py
│   ├── compare_accuracy.py
│   └── generate_final_report.py
│
├── environment.yaml
└── README.md

Generated datasets, experiment runs, Python cache files, and operating-system metadata are excluded from version control.


Reproducibility

The benchmark uses a fixed manifest containing 100 COCO2017 validation images:

results/benchmark_images.txt

The same image set is used for FP32, FP16, and INT8 performance measurements.

Experiment results are stored in machine-readable formats:

  • JSON
  • CSV

This allows the final comparison to be regenerated without manually transcribing benchmark results.

The COCO2017 dataset itself is not included in this repository due to its size and licensing/distribution considerations. Users reproducing the experiment should download the dataset separately and place it in the expected project directory.


Environment

The project uses a Conda environment with Python dependencies installed through pip.

Key dependencies include:

  • Python 3.11
  • PyTorch
  • Ultralytics
  • ONNX
  • ONNX Runtime
  • NumPy
  • OpenCV
  • Pandas
  • Matplotlib
  • PyYAML
  • tqdm
  • psutil

The exact environment specification is provided in:

environment.yaml

Limitations

Post-training quantization

The INT8 model uses post-training quantization (PTQ). Quantization-aware training was not performed.

The observed INT8 accuracy loss therefore does not necessarily represent the best achievable INT8 accuracy for YOLO11n.

Hardware and runtime dependence

Performance results are specific to the tested ONNX Runtime CPU configuration. Results may differ substantially on:

  • NVIDIA GPUs
  • Apple Silicon GPU/MPS
  • TensorRT
  • NPUs
  • other CPU architectures
  • other inference runtimes

Benchmark scope

The reported latency measures model inference only. Preprocessing, postprocessing, image loading, and application-level overhead are outside the timed inference loop.


Future Work

The primary follow-up experiment is to evaluate quantization-aware training (QAT).

A future comparison would therefore include:

FP32
  │
  ├── FP16
  │
  ├── PTQ INT8
  │
  └── QAT INT8

The objective would be to determine whether QAT can recover some of the accuracy lost during post-training INT8 quantization while retaining the model-size and inference-efficiency advantages of INT8.

Additional future work could include benchmarking the models on GPU/NPU hardware and measuring complete end-to-end application latency.


Conclusion

This project experimentally evaluates the effect of numerical precision on YOLO11n deployment.

The results show that FP16 is an effective low-risk optimization: it reduces model size by approximately 50% while maintaining essentially unchanged detection accuracy.

INT8 provides substantially greater compression and the highest measured CPU throughput, but the current post-training quantization configuration introduces a significant detection-quality penalty.

The experiment therefore demonstrates that model optimization should be evaluated in the context of the target deployment environment rather than assuming that lower precision automatically provides better performance.

For the tested configuration:

  • FP32 → strongest accuracy baseline
  • FP16 → near-lossless accuracy with substantial compression
  • INT8 → maximum compression and best CPU throughput, with a significant accuracy trade-off

The next step is to investigate whether quantization-aware training can reduce the INT8 accuracy loss while preserving its efficiency benefits.

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages