Skip to content

feat(experiments): add pluggable experiment lifecycle callback system - #423

Open
sriram687 wants to merge 2 commits into
ServiceNow:mainfrom
sriram687:feat/experiment-callbacks-system
Open

sriram687 wants to merge 2 commits into
ServiceNow:mainfrom
sriram687:feat/experiment-callbacks-system

Conversation

@sriram687

Copy link
Copy Markdown
Contributor

Experiment Lifecycle Callbacks

Closes #417

Summary

ExpArgs.run() now broadcasts structured events at five hook points, so researchers can attach monitoring, safety guardrails, and dashboard integrations without modifying any core loop code.

Motivation

As described in #417, the existing loop is a closed, synchronous black box. In practice this causes three problems:

Problem Impact
No live visibility Long multi-hour benchmark runs can't be monitored
No runtime safety guardrails Nothing stops runaway LLM token usage
No clean integration path Streaming to W&B, MLflow, Streamlit, etc. requires monkey-patching

Backward Compatibility

No existing behaviour changes. The callbacks field defaults to [], so CallbackList iterates over an empty list — zero overhead. All existing tests pass unchanged.

Verification

python -m black . --check                          →  95 files would be left unchanged.
pytest test_callbacks.py test_callbacks_cloud.py   →  60 passed in 2.38s

Usage Example

from browsergym.experiments import ExpArgs, EnvArgs
from browsergym.experiments.callbacks import ConsoleProgressCallback, BudgetLimitCallback

exp = ExpArgs(
agent_args=...,
env_args=EnvArgs(task_name="webarena.42", max_steps=30),
callbacks=[
ConsoleProgressCallback(),
BudgetLimitCallback(max_tokens=50_000),
],
)
exp.run()

PROOF OF WORK

Screenshot 2026-11-05 151525 Screenshot 2026-11-05 151512

…tLimitCallback)

- Add callbacks.py: AbstractCallback base, CallbackList dispatcher, ConsoleProgressCallback, BudgetLimitCallback
- Wire CallbackList into ExpArgs.run() at 5 lifecycle points (on_exp_start, on_step_start, on_action_selected, on_step_end, on_exp_end)
- Add callbacks field to ExpArgs dataclass (default=[]) -- 100% backward-compatible
- Export public API from experiments/__init__.py
- Add 24 unit tests in tests/experiments/test_callbacks.py (all passing)
Copilot AI balanced review requested due to automatic review settings November 5, 2026 09:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@sriram687

Copy link
Copy Markdown
Contributor Author

@Surya-5555 kindly review the integrations.......!!!

@sriram687
sriram687 force-pushed the feat/experiment-callbacks-system branch 2 times, most recently from a0d208b to 5899276 Compare November 9, 2026 05:35

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Proposal]: Add experiment lifecycle callbacks to ExpArgs in loop.py

2 participants