sequential-testing

Runs ~20 options strategies against live market data in shadow mode, records every hypothetical fill under worst/base/optimistic assumptions, and grades each with anytime-valid e-processes. Places no orders.

Which model is really behind your API relay or agent IDE? Behavioral fingerprinting + anytime-valid sequential tests (FPR<=1%). LLMs can't be random - measured on 9 frontier models.

Sequential Hypothesis Testing with e-Values and p-Values

Sequential testing for efficient and reliable comparison of stochastic algorithms.

Fixed Sequence Multiple Testing Procedures

Bayesian A/B Testing for Proportions

Walds Sequential Probability Ratio Testing (SPRT) statistical evaluator for LLM and AI agent comparison in Ruby.

Sequential A/B Testing Framework in Python

Measure your agent harness, find where it wastes the model, and prove the fix worked. Harness-agnostic, agent-agnostic, zero dependencies. Reference implementation of HTP-1.

A/B testing and causal inference scored against known answers: simulations where I set the effect, and a randomised benchmark the observational methods have to recover.

Bayesian multi-armed bandits for continuous prompt experimentation: Thompson sampling routes traffic to the best prompt variant and a stopping rule promotes a winner without a fixed-N A/B test. Zero dependencies, TypeScript-first.

A/B testing engine: frequentist + Bayesian + CUPED variance reduction + peeking-safe sequential testing

A/B test analysis that is correct by default, informative by design, and composable by construction. 88 classes, 60+ paper references.

Ships ML models in stages - shadow, then 1/5/25/50% of traffic - and rolls a bad one back automatically. The guardrails stay valid under constant checking: 0.6% false rollback with two identical models, where a repeatedly-read A/B test acts wrongly 36.7% of the time.

Automated canary control plane for LLM configs: shadow testing, sequential inference (SPRT/CUSUM), drift monitoring

Causal bandit orchestration platform for real-time adaptive experimentation, sequential testing, and interference-aware decisioning.

Four free calculators for A/B tests and marketing experiments (sample size, minimum detectable effect, peeking, geo holdout) and an interactive CUPED explainer. Each result comes with one plain sentence on what it means and what would invalidate it. Every formula is checked against scipy, statsmodels and numpy on every push.

Ad campaign performance evaluation using AB Testing

A/B testing toolkit: power analysis, SRM, CUPED, BH-FDR, and a Monte Carlo calibration harness that checks whether naive peeking rules actually control the false-positive rate they claim to.

[Alpha / WIP] Agent-first CLI for Bayesian and sequential A/B/n experiment decisions with JSON, CSV, XLSX, and text inputs.