time-cp

time-cp is a benchmarking and uncertainty quantification toolkit for Time Series Foundation Models (TSFMs) supporting my thesis Conformalized Zero-Shot Time Series Forecasting. It provides a unified inference interface across leading foundation models (Chronos-2, TiRex, FlowState, TimesFM), benchmark dataset utilities for GiftEval, TIME, and FEV, and a decoupled two-stage evaluation pipeline to rigorously benchmark Conformal Prediction (CP) algorithms against native foundation model quantile predictions.

📖 Full Documentation: For the in-depth API reference, complete method formulations, nonconformity score mathematics, CLI flag details, and dataset utilities, see src/timecp/README.md.

• Quickstart: Experiment Reproduction
• Project Structure
• Toolkit Overview & Architecture

---

Quickstart: Experiment Reproduction

Note: All operations have been tested on a Windows 10 machine with an NVIDIA RTX 3080 GPU and 16GB RAM.

Prerequisites

uv is used for project management. If you haven't installed it already, you should! You can follow the instructions here.

The virtual environment and required dependencies are automatically resolved and installed by uv upon the first command execution.

---

1. Quick Reproduction (Chronos-2 on FEV-bench mini)

To run a fast reproduction baseline using Chronos-2 on the fev-bench_mini benchmark across representative marginal CP methods (AgACI, DtACI, SplitCP, WeightedCP) and nonconformity scores (distributional, cdf_tail, cqr, abs) at target miscoverage $\alpha = 0.2$ ($80%$ nominal coverage):

Step 1: Batch Forecast Inference

Pre-computes foundation model predictions and caches ground-truth windows:

uv run python scripts/forecast.py \
    --model chronos2 \
    --tasks experiments/fev-bench_mini.yaml \
    --cal-windows 50

Step 2: Conformal Prediction Evaluation

Calibrates CP methods on the calibration windows and evaluates them on test windows:

uv run python scripts/cp_eval.py \
    --forecasters chronos2 \
    --base-datasets fev-bench_mini \
    --alpha 0.2 \
    --methods AgACI DtACI SplitCP WeightedCP \
    --score-type distributional cdf_tail cqr abs \
    --min-cal-windows 1 \                       # Don't skip tasks even if 1 calibration window is available
    --cal-windows 50                            # Clip calibration windows to 50 if a task has more available

Step 3: Parse Results and Build Interactive Dashboard

Aggregates task CSVs and builds a standalone interactive dashboard:

uv run python scripts/parse_summaries.py
uv run python scripts/generate_dashboard.py

Outputs:

• Per-task results: results/chronos2/fev-bench_mini/<task>/
• Global model summary: results/chronos2/fev-bench_mini/
• Interactive dashboard: results/dashboard.html

---

2. Full Paper Benchmark Reproduction

To reproduce the multi-model, multi-method paper benchmark across all four foundation models (Chronos-2, TiRex, FlowState, TimesFM) and all nonconformity score functions:

# 1. Run inference across all foundation models on the benchmark
uv run python scripts/forecast.py \
    --model chronos2 tirex flowstate timesfm \
    --tasks experiments/fev-bench_mini.yaml \
    --cal-windows 50

# 2. Run post-hoc conformal calibration across all methods and scores
uv run python scripts/cp_eval.py \
    --forecasters chronos2 tirex flowstate timesfm \
    --base-datasets fev-bench_mini \
    --alpha 0.2 \
    --methods ACI AgACI DtACI PID AcMCP TrailingWindow WeightedCP SplitCP \
    --score-type abs cqr squared signed iqr_scaled mad_scaled scaled_cqr distributional cdf_tail log diff \
    --min-cal-windows 1 \
    --cal-windows 50

# 3. Parse results and generate dashboard
uv run python scripts/parse_summaries.py
uv run python scripts/generate_dashboard.py

Note: For the full, non-mini benchmark, substitute --tasks experiments/fev-bench.yaml.

---

3. Paper Figures, Tables, and Appendix Generation

Once results/cp_calibration_data.json has been generated by scripts/parse_summaries.py, generate all analytical metrics, figures, and LaTeX tables from the paper:

# High-level coverage analysis, failure cases, and domain breakdowns
uv run python scripts/paper/generate_results.py

# Main paper figures (coverage comparison, ACI vs AgACI) and main LaTeX tables
uv run python scripts/paper/generate_plots_and_tables.py

# Comprehensive appendix tables across all models and score types
uv run python scripts/paper/generate_appendix.py

Generated artifacts:

• Figure: paper/coverage_comparison.png
• Figure: paper/appendix_all_results.png
• LaTeX tables: printed directly to terminal and saved into paper/appendix_tables.tex

---

Project Structure

experiments/                   # Experiment YAML configs (FEV, TIME, GiftEval)
paper/                         # Thesis manuscript source
results/                       # Generated predictions, evaluation CSVs, and dashboard
src/
└── timecp/                    # Core library package
    ├── data/                  # Dataset loaders & benchmark adapters (GiftEval, TIME, FEV)
    ├── methods/               # Conformal prediction algorithms (marginal & joint)
    └── models/                # TSFM wrappers (Chronos, TiRex, FlowState, TimesFM)
scripts/                       # Reproduction and evaluation CLI entry points
├── forecast.py                # Stage 1: Batch inference & prediction caching
├── cp_eval.py                 # Stage 2: Conformal prediction calibration & evaluation
├── parse_summaries.py         # Parses all summary CSVs into a single dashboard CSV
├── generate_dashboard.py      # Generates interactive dashboard from summary CSVs
└── paper/                     # Manuscript artifact generators (results, plots, appendix)
tests/                         # Unit and integration test suite

---

Toolkit Overview & Architecture

Two-Stage Decoupled Pipeline

Foundation models are computationally expensive to run on long time-series benchmarks. timecp introduces a two-stage architecture that separates foundation model inference from conformal prediction:

1. Stage 1 (Inference & Ground-Truth Caching): scripts/forecast.py queries the foundation models once across all benchmark windows, serializing point predictions, native quantiles, and true observed values to disk (Apache Arrow format).
2. Stage 2 (Post-Hoc Conformal Calibration): scripts/cp_eval.py iterates through hundreds of combinations of CP algorithms, nonconformity scores, and significance levels $\alpha$ in seconds without ever touching the GPU foundation models.

---

Unified Forecaster API

The Forecaster abstraction provides a consistent interface across model families:

from timecp.models import Forecaster
import numpy as np

# Instantiate model via factory pattern
forecaster = Forecaster('chronos2', device='cuda')

# Single context series or batched list of series
past = np.random.randn(512)
point, quantiles = forecaster.predict(past, horizon=24, output_format='numpy')
# point:     (24,)
# quantiles: (24, Q)

Supported model keys:

• chronos2 (amazon/chronos-2)
• tirex (NX-AI/TiRex)
• flowstate (ibm-research/flowstate)
• timesfm (google/timesfm-2.5-200m-pytorch)

---

Conformal Prediction Suite

timecp implements a comprehensive taxonomy of CP techniques tailored for time-series foundation models:

• Marginal Methods (Per-Horizon Tracking):
  • Static: SplitCP, TrailingWindow, WeightedCP
  • Online Adaptive: ACI, AgACI, DtACI, QuantileIntegrator (PID controller), AcMCP
  • Quantile Regression: CQR, AdaptiveCQR
• Joint Methods (Simultaneous Multi-Step Trajectory Coverage):
  • JointCFRNN, CopulaCPTS, NormMaxCP, CAFHT
• Nonconformity Scores:
  • Residual scores: abs, squared, signed, log, diff
  • Scaled & localized scores: iqr_scaled, mad_scaled, scaled_cqr
  • Distribution-native scores: distributional (pinball loss inversion), cdf_tail (probability-band quantile inversion)
• Asymmetric Prediction Intervals:
  • Independent lower/upper quantile tracking ($q_{lo}, q_{up}$) targeting $\alpha/2$ miscoverage per tail, critical for skewed forecast errors.

---

Minimal Programmatic Usage

import numpy as np
from timecp.methods import ACI, SplitCP

# Generate calibration and test residuals
rng = np.random.default_rng(42)
cal_scores = np.abs(rng.normal(0, 1, size=200))
test_scores = np.abs(rng.normal(0, 1, size=100))

# 1. Static Split Conformal Prediction
split_cp = SplitCP(alpha=0.1).fit(cal_scores)
threshold = split_cp.predict_quantile()
low, high = split_cp.predict_interval(y_hat=15.0)

# 2. Adaptive Conformal Inference (online updating)
aci = ACI(alpha=0.1, gamma=0.005).fit(cal_scores)
for observed_score in test_scores:
    low, high = aci.predict_interval(y_hat=15.0)
    aci.update(observed_score)  # Adapts alpha_t online

For higher-level multi-horizon batch evaluation across datasets, use CPEvaluator as detailed in src/timecp/README.md.