Reduce model-call amplification with helper tiers and workflow budgets - #93
Open
rica-v3 wants to merge 9 commits into
Open
Reduce model-call amplification with helper tiers and workflow budgets#93rica-v3 wants to merge 9 commits into
rica-v3 wants to merge 9 commits into
Conversation
Add local model cost and performance telemetry
* lifecycle.py: batch planner checkpoint reviews * operations_guide.md: document planner review batching
* prompt_context.py: compact request event history * configuration_guide.md: document prompt context compaction
* execution_policy.py: bound measurable benchmark calls * runner.py: add full-sprint A/B benchmark pipeline * performance_benchmarking.md: document sprint cost measurement * cli.py: restore benchmark JSON output * worker.py: harden live sprint measurement * performance_benchmarking.md: document measurement safeguards * codex_runner.py: isolate benchmark provider environment
There was a problem hiding this comment.
Pull request overview
Adds cost controls for model-backed workflows through helper tiers, bounded retries, prompt compaction, telemetry, and benchmarking.
Changes:
- Adds configurable internal-agent tiers and workflow budgets.
- Introduces bounded prompt context and model-call telemetry.
- Adds hardened A/B benchmarking and supporting tests/documentation.
Reviewed changes
Copilot reviewed 52 out of 53 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
workflows/sprints/lifecycle.py |
Extracts confirmation and checkpoint helpers. |
workflows/roles/research.py |
Compacts research prompt context. |
workflows/roles/__init__.py |
Adds workflow-budget policy defaults. |
workflows/repository_ops.py |
Hardens benchmark Git execution. |
workflows/orchestration/team_service.py |
Integrates tiers, telemetry, and budgets. |
workflows/orchestration/engine.py |
Enforces reopen and review limits. |
tests/test_workflow_state.py |
Tests persisted workflow budgets. |
tests/test_workflow_engine.py |
Tests reopen routing and blocking. |
tests/test_sprint_lifecycle.py |
Tests requirement checkpoints. |
tests/test_prompt_context.py |
Tests prompt projection behavior. |
tests/test_orchestration_sprint_execution.py |
Tests batched planner reviews. |
tests/test_orchestration_delegation.py |
Tests helper tiers and policy copying. |
tests/test_goal_store.py |
Updates CLI test dependencies. |
tests/test_execution_policy.py |
Tests benchmark execution safety. |
tests/test_config.py |
Tests new configuration and CLI behavior. |
tests/orchestration_test_utils.py |
Updates workflow fixtures. |
templates/scaffold/team_runtime.yaml |
Scaffolds tiers, context, and telemetry. |
templates/scaffold/orchestrator/.agents/skills/agent_utilization/SKILL.md |
Documents workflow limits. |
templates/prompts/orchestrator.md |
Documents helper configuration. |
shared/prompt_context.py |
Implements event-history projection. |
shared/paths.py |
Adds telemetry storage paths. |
shared/models.py |
Adds telemetry and prompt models. |
shared/config.py |
Loads and updates new configuration. |
shared/__init__.py |
Re-exports telemetry models. |
runtime/research_runtime.py |
Adds telemetry and research policy controls. |
runtime/internal/intent_parser.py |
Adds parser telemetry and policy. |
runtime/internal/goal_sourcing.py |
Adds sourcer telemetry and policy. |
runtime/execution_policy.py |
Defines bounded benchmark execution. |
runtime/benchmark_launcher.py |
Gates provider process startup. |
runtime/base_runtime.py |
Integrates telemetry and prompt projection. |
models.py |
Updates compatibility exports. |
docs/telemetry.md |
Documents telemetry operation. |
docs/specification.md |
Specifies prompt projection. |
docs/README.md |
Indexes telemetry documentation. |
docs/performance_benchmarking.md |
Documents live A/B benchmarks. |
docs/operations_guide.md |
Updates runtime operations guidance. |
docs/implementation.md |
Records prompt-context ownership. |
docs/configuration_guide.md |
Documents new configuration controls. |
docs/call_amplification_controls.md |
Explains rollout and rollback. |
cli.py |
Exposes metrics, configuration, and benchmark commands. |
benchmarking/reporting.py |
Produces benchmark reports. |
benchmarking/models.py |
Defines benchmark contracts. |
benchmarking/__init__.py |
Exports benchmark APIs. |
adapters/cli/commands.py |
Implements new CLI commands. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+227
to
+229
| raw_rate_cards = value.get("rate_cards") or {} | ||
| if not isinstance(raw_rate_cards, dict): | ||
| raise ValueError("team_runtime.yaml telemetry.rate_cards must be a mapping.") |
Comment on lines
+209
to
+216
| def _normalize_non_negative_rate(value: Any, *, field_name: str) -> float: | ||
| try: | ||
| normalized = float(value) | ||
| except (TypeError, ValueError) as exc: | ||
| raise ValueError(f"{field_name} must be a non-negative finite number.") from exc | ||
| if not math.isfinite(normalized) or normalized < 0: | ||
| raise ValueError(f"{field_name} must be a non-negative finite number.") | ||
| return normalized |
Comment on lines
358
to
360
| "MessageEnvelope", | ||
| "PromptContextRuntimeConfig", | ||
| "ReplyRoute", |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dependency
This PR is stacked on and depends on #79 (
1778636). It intentionally targetsmain, as required. Once #79 merges, GitHub will narrow this PR to commitsdf39733and808b5b4.Summary
internal_agent_defaultsfor parser, sourcer, and version-controller runtimesgpt-5.4-minihelper tiers with low/medium reasoning appropriate to each narrow workloadconfig internal setand expose effective helper tiers in status outputCost and quality guardrails
The review entry ceiling is structurally reduced by 17 calls, or 85%, but this PR does not claim realized production savings. Comparable live telemetry and QA outcomes are still required before checking off the impact-quantification subtask in #2.
Public role model defaults are unchanged. Existing configs without
internal_agent_defaultsinherit orchestrator settings. Existing in-flight workflows retain their persisted budgets.Validation
PYTHONPATH=.. python -m unittest discover -s tests: 874 tests passed, 1 skippedpython -m compileall -q .: passedgit diff --check: passedRefs #2