[Feature]: Team Improvement Metrics (DORA-inspired)

[jeromevdl]

Summary

Add a project-level Team Improvement Metrics view, inspired by DORA and value-stream measurement, so teams can see whether their use of Collaborative AI-DLC is improving delivery speed, quality, and cost over time.

The first version should use only data the platform already captures automatically. It should not claim to provide genuine DORA metrics yet: those require production deployment and operational data that Collaborative AI-DLC does not currently own.

The objective is not to score developers or compare teams. It is to help a team answer:

• Are we delivering intents faster?
• Where do we spend time waiting?
• Are we succeeding with less rework?
• Are quality signals improving?
• Is the cost per successful outcome improving?
• Which workflows, scopes, or stages create bottlenecks?

Motivation

We encourage customers to measure software delivery performance and improve from their own baseline, but the platform does not currently give them a consolidated measurement surface.

Collaborative AI-DLC already records enough structured events to calculate useful metrics automatically:

• intent lifecycle and status;
• stage execution, attempts, and duration;
• active time versus time waiting for humans;
• questions, gates, and answers;
• retries, rewinds, recompositions, and feedback batches;
• parallel unit execution and integration;
• sensor and reviewer outcomes;
• PR/MR creation;
• token usage and cost;
• graph coverage and traceability findings.

These measurements can provide value now, before the platform covers CI/CD and production operations.

Metrics available from current platform data

All metrics must be calculated from persisted platform events. No periodic manual data entry should be required.

Flow and speed

• Intent throughput: successful intents completed per week/month.
• Intent lead time: time from intent creation to completion.
• Intent cycle time: time from first execution activity to completion.
• Time by phase and stage: distribution of time spent in inception, construction, review, and individual stages.
• Active execution time: time agents are actively working.
• Human waiting time: time the workflow is parked on questions or gates.
• Flow efficiency: active execution time divided by total cycle time.
• Gate response time: time between opening a gate/question and receiving an answer.
• Work in progress: number of concurrent non-terminal intents over time.
• Parallel construction efficiency: elapsed construction time compared with the work performed across unit lanes.
• PR/MR creation lead time: time from intent start to final PR/MR creation.

Rework and reliability

• Intent outcome rate: succeeded, failed, cancelled, and still-blocked intents.
• Stage first-pass success rate: stages succeeding on their first attempt.
• Intent first-pass success rate: intents succeeding without retry, rewind, or recompose.
• Retry rate: number and proportion of stages requiring another attempt.
• Rewind rate: number and proportion of intents restarted from an earlier stage.
• Recomposition rate: intents whose remaining workflow was reshaped after execution started.
• Review-feedback rework: feedback batches and revision cycles per intent or unit.
• Failure concentration: stages, phases, workflows, or scopes responsible for the most failures and retries.
• Sensor quality trend: PASS, FAIL, and INCONCLUSIVE rates, including blocking findings.
• Reviewer revision trend: number of reviewer-driven correction cycles.

These are not production reliability metrics. They measure the reliability of the AI-DLC delivery process itself.

Cost and efficiency

• Cost per intent.
• Cost per successful intent.
• Cost of failed or cancelled intents.
• Cost of rework: usage attributed to retries, rewinds, and revision attempts.
• Token and context usage by phase, stage, workflow, scope, CLI, and model.
• Waiting versus compute efficiency.
• Comparison of cost and outcome between models or workflow configurations.

The primary economic measure should be cost per successful outcome, not the lowest token cost in isolation.

Quality and traceability

• Sensor/reviewer pass rate.
• Blocking finding rate.
• Must-have requirement coverage findings already produced by the graph audit.
• Graph structure and derivation health.
• Artifact and traceability completeness where the current graph has sufficient data.
• Percentage of intents completing without unresolved blocking quality findings.

Full traceability from FR/NFR and acceptance criteria to code and tests should be added when those relationships are reliably available in the graph.

Product experience

Add a project-level Team Improvement or Delivery Performance page containing:

• a selectable time range;
• current value, previous-period value, and trend;
• median and percentile distributions where relevant, rather than averages alone;
• breakdown by workflow, scope, phase, stage, and model;
• a drill-down from every aggregate to the contributing intents and events;
• an explanation of each metric and its formula;
• data-availability and sample-size indicators;
• highlighted bottlenecks, such as:
  • most cycle time spent waiting for a specific gate;
  • stages with the highest retry rate;
  • workflows with high rework;
  • rising cost per successful intent.

The default view should compare a team with its own previous periods. It should not rank individual developers or create cross-team league tables.

DORA metrics as a future extension

The current DORA model contains five software-delivery metrics:

• Change lead time
• Deployment frequency
• Failed deployment recovery time
• Change fail rate
• Deployment rework rate

Reference: DORA's software delivery performance metrics.

Collaborative AI-DLC cannot calculate these accurately today because it does not yet observe the complete lifecycle from committed code through production deployment, service failure, restoration, rollback, and rework.

These metrics should be added later when the platform can ingest reliable production lifecycle data. Intent completion, PR creation, or stage failure must not be presented as substitutes for DORA metrics.

Other future metrics enabled by the full lifecycle

Once source-control, CI/CD, test, deployment, and operations data are available, extend the same measurement surface with:

• PR/MR review and merge time;
• build success and CI duration;
• flaky-test rate;
• integration and E2E test outcomes;
• FR/NFR/acceptance-criteria-to-code-and-test traceability;
• deployment success and rollback rate;
• escaped defects;
• incident rate and recovery time;
• SLO and runtime regression signals;
• environment and infrastructure cost;
• the five genuine DORA metrics.

These should be follow-up capabilities rather than dependencies for the first useful release.

Measurement principles

• Metrics are calculated automatically from durable events.
• Missing data is shown as unavailable, never as zero.
• Definitions and formulas are visible.
• Metrics show sample size and time range.
• Users can drill down to the underlying intents and events.
• Trends and distributions are preferred over isolated totals.
• The product measures team/system improvement, not individual productivity.
• Metrics should help identify bottlenecks, not become targets that encourage gaming.
• Historical values remain reproducible when calculation definitions evolve.

Goals

• Give teams an automatic baseline for continuous improvement.
• Make speed, quality, rework, waiting time, and cost visible together.
• Deliver useful metrics using data already present in the platform.
• Identify workflow and human bottlenecks.
• Measure whether changes to models, workflows, scopes, and autonomy improve outcomes.
• Prepare a clean path toward DORA metrics when full lifecycle data becomes available.

Non-goals

• Claiming that intent-level measurements are DORA metrics.
• Requiring deployment support for the first release.
• Manual entry of recurring metric values.
• Ranking developers or using the metrics as individual performance measures.
• Defining universal targets that apply to every team.
• Supporting every CI/CD or operations provider in this issue.

Acceptance criteria

• A project-level metrics view is available with a configurable time range.
• The first release uses only automatically captured platform data.
• Flow, rework/reliability, quality, and cost metrics are clearly grouped.
• Every metric has a visible definition and formula.
• Values include sample size and comparison with the previous period.
• Users can drill down to the contributing intents and events.
• Missing or insufficient data is shown explicitly.
• Results can be filtered by workflow, scope, phase, stage, and model where applicable.
• Calculations are deterministic and covered by tests.
• No intent-level proxy is labelled as a DORA metric.
• No metric is used to rank individual developers.

Open questions

1. Which subset of the currently measurable metrics should appear in the first dashboard?
2. What should be the default comparison period: previous equal period, rolling average, or both?
3. Should metrics be scoped only to a project initially, or also aggregated across projects?
4. How should deleted intents be represented in historical aggregates?
5. Which retry, rewind, recompose, and feedback actions should count as rework, and should they be displayed separately?
6. What minimum sample size should be required before showing a trend?
7. Should teams be able to select a small number of improvement goals without turning them into individual performance targets?