Skip to content

About

Measuring and ranking domain shift between 11 source datasets and 2 aerial targets (DOTA, DIOR) for few-shot object detection. Frozen ResNet-50 features, Wasserstein and MK-MMD distances, and a few-shot separability score (1 to 500 shots, LogReg/MLP/discriminator), with t-SNE/Isomap projections.

Topics

Resources

Stars

2 stars

Watchers

0 watching

Forks

Latest commit

Β 

History

19 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

πŸ›°οΈ Domain Shift Study

Quantifying Feature-Space Divergence in Few-Shot Object Detection Across Aerial Image Datasets

Python PyTorch scikit-learn HuggingFace Status License

Engineering project (3rd year) β€” Image Processing & Computer Vision, 2026

Cyprien Vial Β· Setthika Sun Β· Bilal Delais

Overview β€’ Method β€’ Datasets β€’ Metrics β€’ Results β€’ Usage β€’ Structure


πŸ“Œ Overview

Domain shift is the statistical gap between the distribution a model is trained on (source) and the one it is deployed on (target). This project builds a full pipeline to measure and rank domain shift across aerial imagery datasets, combining classical distribution-distance metrics with a few-shot separability score: the faster a lightweight classifier learns to tell two domains apart from just a handful of examples, the larger the shift between them.

πŸ’‘ Core idea β€” Train a binary classifier on 1 β†’ 500 labeled examples per domain, over many random episodes. A domain pair that becomes separable within a handful of shots is far apart in feature space; a pair that stays confused even at hundreds of shots is nearly indistinguishable.

The study benchmarks 11 heterogeneous source domains (aerial, generic, agriculture, fashion, underwater, synthetic...) against 2 aerial target domains (DOTA and DIOR, with an artificial background-blur degradation), using:

Approach Tools
🧭 Distribution-distance metrics Wasserstein distance, Multi-Kernel MMD
🎯 Few-shot separability Logistic Regression Β· MLP Β· Discriminator (1 β†’ 500 shots, 100 episodes)
πŸ—ΊοΈ Manifold visualization t-SNE, Isomap

All experiments, raw results, and analysis live in a single notebook: Technical_Study.ipynb.

Cross-target consistency between DOTA and DIOR
Fig. Domain-shift ranking is nearly identical whether measured against DOTA or DIOR (r β‰₯ 0.96 across all metrics).


πŸ”¬ Scientific Context

Aerial imagery is a particularly hard case for domain shift, driven by:

  • πŸ“‘ differing acquisition conditions (altitude, sensor, resolution)
  • 🌍 differing geographic context (urban vs. rural, country)
  • 🏷️ inconsistent annotation protocols and category taxonomies
  • 🌫️ image quality degradation (blur, compression, lighting)

Why few-shot?

Instead of the usual "train on thousands of samples" setup, the guiding question is:

How quickly can a classifier learn to distinguish two domains when given only a handful of labeled examples per class?

  • Low separability, even with many shots β†’ small domain shift β†’ features transfer well
  • High separability, even with very few shots β†’ large domain shift β†’ distributions are fundamentally different

Sweeping n_shots from 1 to 500 traces a full learning curve, from which principled, comparable shift scores are derived.


βš™οΈ Methodology

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ 1 Β· DATA LOADING & FILTERING                          β”‚
β”‚     Streamed from the HuggingFace Hub Β· blur/size     β”‚
β”‚     filtering (Laplacian variance, min bbox size)     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ 2 Β· FEATURE EXTRACTION                                β”‚
β”‚     Frozen ResNet-50 (ImageNet1K_V2) β†’ 2048-d vectors β”‚
β”‚     saved as float16 .npy files                       β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ 3 Β· DISTRIBUTION DISTANCE METRICS                     β”‚
β”‚     Wasserstein-1 (PCA-50) Β· Multi-Kernel MMD         β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ 4 Β· FEW-SHOT DOMAIN DISCRIMINABILITY                  β”‚
β”‚     LogReg Β· MLP Β· Discriminator                      β”‚
β”‚     n_shots ∈ {0,1,5,10,25,50,100,250,500} Γ— 100 eps  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ 5 Β· ANALYSIS & VISUALIZATION                          β”‚
β”‚     DSI score Β· AUC Β· PAD Β· cross-target consistency  β”‚
β”‚     t-SNE / Isomap projections                        β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
πŸ§ͺ Implementation notes worth knowing
Aspect Detail
Feature extractor Frozen ResNet-50, IMAGENET1K_V2 weights, final FC replaced with Identity()
Preprocessing Resize 224Γ—224, ImageNet mean/std normalization
Quality gate Reject if min(w, h) < 30px, or Laplacian-variance blur score < 20
Class balancing Downsample both sets to `min(
Few-shot protocol Sample n_shots/class for train, rest for test; StandardScaler fit on train only; averaged over multiple episodes
Discriminator by design Deliberately under-powered (128β†’32, dropout 0.3) vs. the standard MLP (256β†’64, dropout 0.2) β€” an over-confident classifier would make every domain pair look maximally separable, destroying the metric's discriminative range
Reproducibility Global seed = 1, 100 episodes/regime, resumable embedding extraction (skips already-processed datasets)

πŸ—‚οΈ Datasets

All data is streamed directly from the HuggingFace Hub (HichTala/datasets) β€” no full download required.

🎯 Target domains

Dataset Description
dota-bg_blurry DOTA, background artificially blurred, blurry images removed
dior-bg_blurry DIOR, background artificially blurred, blurry images removed

πŸ“š Source domains (11)

Key Description Domain
coco Common Objects in Context Generic
dota Aerial imagery (Google Earth) Aerial
dior Optical remote sensing Aerial
dior_bg DIOR backgrounds Aerial
deepfruits Fruit detection in orchards Agriculture
xview Overhead satellite imagery Aerial
oktoberfest Crowd & food scenes Event
fashionpedia Fashion article segmentation Fashion
cadot Overhead vehicle detection Aerial
artaxor Arthropod imagery Synthetic/nature
uodd Underwater object detection Underwater

The source pool is deliberately heterogeneous β€” spanning aerial, ground-level, synthetic, and underwater domains β€” to stress-test the shift metrics across a wide range of feature-space distances.


πŸ“ Metrics

Metric Definition Reads as
DSI-10 (AUC@10-shot βˆ’ 0.5) Γ— 200, clipped to [0, 100] 0 = same domain, 100 = maximal shift
AUC Learning Normalized area under the AUC-vs-shots curve Overall learning efficiency
Convergence Rate perf@10-shot / max_perf Relative convergence speed
Stability 1 βˆ’ 2 Γ— mean_std, clipped to [0, 100] Inverse of inter-episode variance
AUC-500 Final AUC (at 500 shots) Γ— 100 Asymptotic separability
PAD Proxy A-distance, 2 Γ— (1 βˆ’ 2Ξ΅) Classic domain-adaptation shift measure
Wasserstein Mean W-1 distance over 50 PCA components Location/shape gap between distributions
MK-MMD Multi-Kernel MMD, 30 log-spaced RBF bandwidths Gap in reproducing-kernel Hilbert space

Per-pair Γ— per-model metrics are aggregated into a consensus score (mean over the 3 classifiers) and joined with the geometric distances for the final report.


πŸ† Key Results

Domain-shift hierarchy (consensus DSI-10)

Tier Sources DSI-10
πŸ”΄ Extreme shift oktoberfest, fashionpedia, uodd, artaxor, deepfruits > 85
🟠 Strong shift coco, cadot, dior, xview, dota 55 – 85
🟒 Moderate / weak shift dior_bg, dior-bg_blurry < 50

Highlights

  • βœ… Near-perfect cross-target consistency β€” DOTA vs. DIOR correlate at r = 0.96 (DSI-10), r = 0.97 (Wasserstein), r = 1.00 (MK-MMD): the source ranking barely depends on which target is chosen.
  • βœ… Model hierarchy β€” LogReg (β‰ˆ83–84%) > MLP (β‰ˆ73%) > Discriminator (β‰ˆ65%) mean DSI-10, suggesting domain shift is largely linearly separable in ResNet-50 feature space.
  • ⚠️ Residual asymmetries β€” dota and dior_bg show a notable convergence gap between the two targets, indicating partial distribution overlap with one of them.
  • 🎯 Practical takeaway β€” oktoberfest / fashionpedia maximize shift (best stress-test sources); xview / coco sit closest to the aerial targets; 10 shots are enough to detect shift with LogReg (50 shots for MLP/Discriminator).

All generated figures (learning curves, metric heatmaps, cross-target comparisons, per-epoch DSI/PAD curves) live under domain_shift_analysis/.


πŸ—ƒοΈ Repository Structure

Domain_Shift_Study/
β”œβ”€β”€ Technical_Study.ipynb              # Main notebook β€” full pipeline + analysis
β”œβ”€β”€ domain_shift_analysis/             # Generated results & figures
β”‚   β”œβ”€β”€ metrics_detailed.csv           # Per-pair Γ— per-model metrics
β”‚   β”œβ”€β”€ metrics_consensus.csv          # Consolidated metrics (3-model average)
β”‚   β”œβ”€β”€ few_shot_learning_curves.png   # Aggregated learning curves
β”‚   β”œβ”€β”€ cross_target_comparison.png    # DOTA vs. DIOR consistency
β”‚   β”œβ”€β”€ dior-bg_blurry/                # Full report for the DIOR target
β”‚   β”‚   β”œβ”€β”€ summary.json / summary.txt
β”‚   β”‚   β”œβ”€β”€ metrics_consensus.csv
β”‚   β”‚   └── *.png                      # DSI, heatmap, per-epoch curves, PAD...
β”‚   └── dota-bg_blurry/                # Full report for the DOTA target
β”‚       β”œβ”€β”€ summary.json / summary.txt
β”‚       β”œβ”€β”€ metrics_consensus.csv
β”‚       └── *.png
└── README.md

πŸš€ Installation & Usage

Developed and tested on Python 3.9+. A GPU is recommended for feature extraction (the backbone also runs on CPU, just slower).

git clone https://github.com/cyyyp100/Domain_Shift_Study.git
cd Domain_Shift_Study

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate

pip install torch torchvision datasets scikit-learn scipy opencv-python \
            pandas numpy matplotlib seaborn tqdm jupyter

ℹ️ No requirements.txt is currently committed β€” the list above is reconstructed from the notebook's imports. Pinning exact versions with pip freeze > requirements.txt is recommended before external reuse.

Then open the notebook and run the cells in order:

jupyter lab Technical_Study.ipynb
Section Content
Β§1–3 Context, imports, global configuration (paths, seed, shot grid)
Β§4–13 Feature extraction β€” streaming + ResNet-50 inference across all domains (can be long)
Β§14–19 Distribution distances + few-shot evaluation (LogReg / MLP / Discriminator)
Β§20–22 Full analysis report, figures, summary and conclusions

Intermediate outputs (embeddings, incremental CSVs) make the pipeline resumable: an interruption doesn't require restarting extraction from scratch β€” already-processed datasets are automatically skipped. Final reports are written to domain_shift_analysis/, one subfolder per target domain.


πŸ”­ Limitations & Future Work

  • Relies on a single backbone (ImageNet ResNet-50); generalization to other architectures (ViT, self-supervised backbones) remains untested.
  • Shift metrics operate at the whole-image level, without accounting for the spatial structure relevant to object detection (object scale, localization).
  • The asymmetries observed between dota-bg_blurry and dior-bg_blurry for dota and dior_bg deserve dedicated investigation.
  • A natural extension would connect these shift scores to the actual performance of a few-shot object detector, to validate their predictive power beyond raw feature separability.

πŸ“„ Citation

@misc{domainshiftstudy2026,
  title  = {Domain Shift Evaluation in Few-Shot Object Detection: A Systematic
            Analysis of Feature Space Divergence Across Aerial Image Datasets},
  author = {Vial, Cyprien and Sun, Setthika and Delais, Bilal},
  year   = {2026},
  note   = {3rd-year engineering project, Image Processing \& Computer Vision},
  howpublished = {\url{https://github.com/cyyyp100/Domain_Shift_Study}}
}

πŸ“ License

No license is currently declared in this repository. Since this is an academic project, adding an explicit license (e.g. MIT, or CC BY-NC 4.0 for non-commercial use) is recommended before any external distribution or reuse.


Generated from Technical_Study.ipynb and the results in domain_shift_analysis/.

About

Measuring and ranking domain shift between 11 source datasets and 2 aerial targets (DOTA, DIOR) for few-shot object detection. Frozen ResNet-50 features, Wasserstein and MK-MMD distances, and a few-shot separability score (1 to 500 shots, LogReg/MLP/discriminator), with t-SNE/Isomap projections.

Topics

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages