Looking for the training loss monitor? W-Twin has moved to github.com/Kretski/WTwin
Detect progressive neural network training degradation before it shows up in your loss curves.
ScalePredict compares your training loss against a scaling-law baseline at every step. When the trajectory drifts from what is expected, W-Twin raises an alert — before classical threshold and CUSUM detectors notice anything.
from scalepredict.monitor import WTwinMonitor
monitor = WTwinMonitor()
# Drop into any training loop — one line per step
for step, loss in enumerate(your_training_losses, 1):
state = monitor.update(step, loss)
if state.alert:
print(f"⚠ Degradation detected at step {step} (W={state.W:.2f})")
# → rollback checkpoint, send alert, or stop runOr from the command line — no code needed:
pip install git+https://github.com/Kretski/ScalePredict.git
scalepredict monitor training_log.csv
scalepredict demoCurrent training monitors are reactive: they catch NaN losses, gradient explosions, and hardware failures after the damage is done. Progressive degradation — slowly increasing label noise, gradual weight corruption, subtle data pipeline drift — accumulates undetected until significant GPU budget is wasted.
W-Twin fits a power-law baseline to early training steps, then at every step computes:
W(t) = Q(t) · (D(t) − α)
D(t) = (L_obs(t) − L_pred(t)) / σ_local(t) ← how far off the expected curve
Q(t) = exp(−MSE_fit / τ) ← how much to trust the baseline
α = 2.0 ← detection threshold (z-score)
An alert fires when W(t) > 0 for 5 consecutive steps. No tuning required for basic use.
From the paper — real nano-GPT training runs:
| Experiment | Runs | W-Twin | Threshold | CUSUM |
|---|---|---|---|---|
| Progressive drift | 9 | 9/9 (100%) | 0/9 (0%) | 0/9 (0%) |
| Mean detection delay | — | 223 ± 11 steps | — | — |
| False alarm rate | 30 clean runs | 0/30 (0%) | 0/30 | 0/30 |
| Abrupt spike | 2 | 2/2 (+5 steps) | 0/2 | 2/2 (+1 step, faster) |
W-Twin is the only method that detects progressive drift. For sudden spikes, CUSUM is faster — both are complementary.
Scope: Results are from controlled nano-GPT experiments with injected failures. External validation on independent architectures and real training logs is ongoing.
pip install git+https://github.com/Kretski/ScalePredict.gitDependencies: numpy, scipy only. No framework lock-in.
from scalepredict.monitor import WTwinMonitor
monitor = WTwinMonitor(
warmup_steps=100, # skip LR warmup phase
alpha=2.0, # detection sensitivity
n_consec=5, # steps above threshold before alert
)
for step, loss in training_loop():
state = monitor.update(step, loss)
if state.alert:
print(f"Step {step}: W={state.W:.3f} — possible degradation")
print(f"First alert: {monitor.first_alert_step()}")from transformers import TrainerCallback
from scalepredict.monitor import WTwinMonitor
class WTwinCallback(TrainerCallback):
def __init__(self):
self.monitor = WTwinMonitor()
def on_log(self, args, state, control, logs=None, **kwargs):
if logs and "loss" in logs:
st = self.monitor.update(state.global_step, logs["loss"])
if st.alert:
print(f"⚠ W-Twin alert at step {state.global_step}")
trainer = Trainer(..., callbacks=[WTwinCallback()])scalepredict monitor training_log.csv
scalepredict monitor wandb_export.csv --loss-col train/loss --step-col _step
scalepredict monitor training_log.csv --output wtwin_scores.csv
scalepredict demo# Detect alert AND get failure classification + recommendation
scalepredict suggest training_log.csv
# With full JSON output
scalepredict suggest training_log.csv --jsonOutput example:
⚠ W-Twin Alert at step 2390
Failure type : gradual_drift (confidence: 51%)
Suggestion : consider_lr_reduction
Action : manual_review
Reasoning : D(t) shows a sustained positive slope — training is
gradually deviating from the expected trajectory.
[EXPERIMENTAL — validate before acting]
suggestis advisory only — it does not modify any training state.
WTwinMonitor(
warmup_steps=50, # steps to skip (LR warmup)
alpha=2.0, # fixed z-score threshold
n_consec=5, # consecutive alerts required
mad_window=50, # window for local noise estimate
tau=1e-3, # baseline confidence decay
)| Method | Returns | Description |
|---|---|---|
update(step, loss) |
WTwinState |
Process one step |
first_alert_step() |
int | None |
Step of first alert |
history |
list[WTwinState] |
Full history |
reset() |
— | Reset state |
WTwinState fields: step, l_obs, l_pred, D, Q, T, W, alert
from scalepredict.monitor import WTwinMonitor, suggest
monitor = WTwinMonitor()
for step, loss in training_loop():
monitor.update(step, loss)
s = suggest(monitor)
print(s.failure_type) # "gradual_drift" | "abrupt_spike" | "uncertain"
print(s.suggestion) # "consider_lr_reduction" | "consider_rollback" | "manual_review"
print(s.confidence) # float in [0.5, 0.95]
print(s.as_dict()) # full JSON-serializable outputExperimental: classifier uses heuristics from synthetic experiments only. Not validated on labeled real failures. All suggestions require human review.
from scalepredict.monitor.baseline import BaseBaseline
from scalepredict.monitor import WTwinMonitor
class MyBaseline(BaseBaseline):
def fit(self, steps, losses): ...
def predict(self, t): ...
@property
def fit_mse(self): return 0.001
@property
def is_fitted(self): return True
monitor = WTwinMonitor(baseline=MyBaseline())git clone https://github.com/Kretski/ScalePredict.git
cd ScalePredict
pip install -e ".[train]"
# Clean run
python examples/train_real.py --mode none --steps 3000 --model-size small --seed 42
# Progressive drift (key result)
python examples/train_real.py --mode progressive_label \
--failure-step 2000 --steps 3000 --model-size small \
--seed 42 --ramp-steps 1000 --max-noise-prob 0.5
# Abrupt failure
python examples/train_real.py --mode weight_corrupt \
--failure-step 2000 --steps 3000 --model-size small --seed 42- Validated on nano-GPT (842K parameters) with synthetic byte-level text
- Power-law baseline assumes monotonically decreasing loss
- Failures are injected synthetically
suggest()classifier not validated on labeled real failures- External validation on independent architectures pending
Full details in Section 8 of the paper.
If you run ScalePredict on your own training logs — whether it works or not — please open an issue. External validation is the next priority.
@software{kretski2026wtwin,
author = {Kretski, Dimitar},
title = {W-Twin: Forecast-Based Detection of Progressive
Neural Network Training Degradation},
year = {2026},
doi = {10.5281/zenodo.21842461},
url = {https://zenodo.org/records/21842461},
publisher = {Zenodo}
}If you ran ScalePredict on a real training log — whether it detected something or missed it — please open an issue. One data point from a real run is worth more than 10 synthetic experiments.
MIT — see LICENSE.
Author: Dimitar Kretski, Center for Hydro- and Aerodynamics, Varna, Bulgaria ORCID: 0000-0001-5108-2243