Change Throughput Diagnostic

Maturity: assess — worth exploring; promising but not yet proven across enough contexts.

When to use: Deliberately invoke in Agent mode against a git repository to produce a point-in-time read on Lead Time For Changes (LTFC) and Deployment Frequency (DF), plus the contributor mix, batch size, and an experimental change-failure proxy. Re-run periodically (e.g. monthly) to track delivery throughput and set or check a per-active-contributor benchmark.

The agent needs read access only; the bundled scripts run entirely against the local git clone — no API key, no network, no writes to the target repo are required unless you ask it to save the report.

If the user asks to "run this here" (or equivalent), still run the scope gate below. You may proceed immediately only when window, target repo/branch, scope, and deploy trigger are already explicit in the request. Otherwise ask concise questions first and wait for answers.

Bundled scripts: ../scripts/README.md — a small, tested Python engine (metrics.py + test_metrics.py) and a git-only collector (collect.py). Stdlib only; no pip install. See ../scripts/README.md.

What this diagnostic covers

Every run produces evidence across three headline views:

View	What it answers
Throughput	LTFC (first commit → merge to trunk) and IF/DF (merges to trunk per week / work-day), split into meaningful vs micro changes
Contributor mix	Who is genuinely active vs occasional/part-time; how concentrated delivery is (bus-factor)
Benchmark check	Actual vs a target of 1 good change per N active contributors per work-day (default N=2)
Batch size	How much change rides in each landing — net commits, churn and batch age (§6)
Change-failure proxy	How often a landed change draws a follow-up fix — experimental, confidence-rated (§7)

Why this is trustworthy (and git-only)

• LTFC uses git author dates — immune to rebase (rebase rewrites committer dates only). Verify author≈committer dates to confirm no history rewriting distorts the measure.
• DF counts merge commits to the trunk. This is valid only if a merge to the trunk deploys. Confirm the deploy trigger for the target project before calling merge frequency "deployment frequency" — if deploys are scheduled or manual, DF is gated by that pipeline, not by merges.
• Cycle time (PR created_on → merge) is intentionally out of scope here: it requires the Bitbucket/GitHub API and a user-supplied token, and prior analysis showed PRs open a median ~4 min after the first commit, so cycle time ≈ LTFC. The optional API path is documented in scripts/collect.py if review-latency metrics are ever needed.

---

Scope gate — confirm before running (mandatory)

Do not assume scope. Before running any history commands, confirm two things with the user. If either is not explicit from the request or workspace, ask and wait for an explicit answer.

1. Time window. Which period should the diagnostic cover? Offer concrete options:
2. Last 30 days · last 90 days · a named month · an explicit --since YYYY-MM-DD [--until YYYY-MM-DD]
3. If the user wants a trend, run two adjacent windows (e.g. prior 30d and current 30d) so concentration/throughput shifts are visible.
4. Target project / repo and trunk branch. Which repository is being measured, and what is its deploy trunk? Default trunk is main. In a monorepo, confirm whether the read is for the whole repo or scoped to one domain/folder. If the workspace clearly is the target repo with a main branch, say so and proceed.

Also confirm the deploy trigger (see above) so the DF number is labelled correctly.

When interactive option tools are available, use them first. Otherwise ask in plain chat and wait for an explicit choice.

Execution preference for local machines:

• Confirm the shell. Windows (PowerShell or Git Bash), WSL, macOS, and Linux are all in scope. Match path style to the shell in use. Do not hand a Git Bash user a backslash path or vice versa.
• Pull latest before running. Before collecting any git history, confirm the local clone is up to date: git fetch origin && git status. If the branch shows [behind N], run git pull first. Stale clones silently under-count recent PRs and contributors.
• Prefer the workspace Python interpreter / active venv (.venv\Scripts\python.exe on Windows, or the py -3 launcher) rather than plain python.
• If scripts are not copied into the target repo, run them by absolute path from this prompt folder.
• Store scan output in the portfolio folder the user confirms, not in the target repo, unless they ask otherwise. A common layout is docs/management/compliance/scans/<repo_id>/ for the report and .../raw/ for JSON. Use repo_id as the join key to the portfolio registry — do not duplicate remote_url or other repo metadata in the scan doc.
• Locating the target repo at runtime: repos.yml is the stable source of truth for repo metadata (remote_url, web_url, default_branch). local_clone in that file is a convenience hint only — each team clones to a different path and it cannot be relied upon. Resolve the actual path by: (1) checking the open VS Code workspace folders, (2) deriving from remote_url if the repo is already cloned, or (3) asking the user to confirm the local clone path before proceeding. If a repo cannot be found, ask the user for the clone path and whether to use SSH or HTTPS. Do not guess a machine-local path from a registry hint.

Merge style is auto-detected (--merge-style auto) along two independent axes, because Azure DevOps does both:

Axis	Values	Consequence
PR wrapper	Bitbucket Merged in <branch> (pull request #N) · ADO Merged PR N: <title>	where the human-readable title lives
Landing	two-parent merge · single-parent squash	squash ⇒ LTFC unmeasurable, reported as such rather than as 0h

Override with --merge-style bitbucket|ado|ado-squash if detection looks wrong.

Measure the branch changes actually land on. Do not assume main. Check the portfolio registry when one exists, and verify against git ls-remote --symref origin HEAD rather than the local clone's checked-out branch. Where a repo uses a release train (feature → develop → master), master gives deployment frequency and the integration branch gives integration frequency. Report both and label each honestly — do not merge them into one number.

Always walk --first-parent. Counting all reachable commits inflates PR counts and miscounts every commit inside a merged branch as a direct push. A previous run of this diagnostic overstated both figures for a release-train repository for exactly this reason.

---

Prompt

Produce a Change Throughput Diagnostic for this repository, focused on Lead Time For Changes (LTFC) and Deployment Frequency (DF). Work entirely from the local git clone.

First, settle scope (see Scope gate). Confirm the time window, the target repo/branch (and whether it is whole-repo or scoped to one bundle), and the deploy trigger. If any of these is unclear from my request or the open workspace, ask me and wait. When interactive option tools are available, use them first. If scope is explicit, restate it in one line, then proceed.

1. Run the bundled engine

Use the scripts in this prompt folder by absolute path (do not copy into target repo). Ask which shell is in use and match it.

# PowerShell
$s = "<path-to-this-skill>\scripts"

# validation: run the unit tests first (all green before trusting output)
py -3 -m unittest discover -s $s -p 'test_*.py'

# collect metrics (stdout mode — no file output in target repo)
py -3 $s\collect.py --repo "$env:USERPROFILE\source\repos\<target>" --branch <trunk> `
  --since <YYYY-MM-DD> --until <YYYY-MM-DD>

# raw JSON to a temp location, then move into the confirmed scans folder
py -3 $s\collect.py --repo "<clone>" --branch <trunk> --since <d> --until <d> `
  --json $env:TEMP\throughput-report.json

# Git Bash / WSL / macOS / Linux
s=<path-to-this-skill>/scripts

python -m unittest discover -s "$s" -p 'test_*.py'
python "$s/collect.py" --repo <clone> --branch <trunk> --since <d> --until <d>
python "$s/collect.py" --repo <clone> --branch <trunk> --since <d> --until <d> \
  --json /tmp/throughput-report.json

On Windows prefer .venv\Scripts\python.exe or py -3 over a bare python, which can resolve to the Microsoft Store stub. --path is a git pathspec and always uses forward slashes; --repo uses whatever form the shell expects.

After running, capture the output and move any raw JSON into the confirmed scans folder (docs/management/compliance/scans/<repo_id>/raw/ when that layout is in use).

Report the engine output: PRs / meaningful / micro counts, LTFC (median, p75, max), DF (all and meaningful, per week and per work-day), the per-contributor table, the genuinely-active set, and the benchmark line.

2. Validate the measurement (cross-checks)

Before trusting the numbers, confirm the method holds for this repo: - No squash/rebase distortion: check that merges to trunk are true two-parent merge commits and that commit author dates == committer dates (a non-zero gap signals rebasing, which would compress LTFC). State the result. - Merge == deploy: restate the confirmed deploy trigger; if merges do not deploy, relabel DF as "integration frequency" and note what actually gates deployment. - Micro vs meaningful: a "micro" change is files<=1 and churn<=5 and commits<=1 (trivial one-liner). Note how many changes this excludes and whether loosening the threshold would move the headline.

3. Contributor mix and concentration

From the per-contributor table: - Classify each contributor as consistent core, rotating/regular, or occasional (genuinely-active default = >=3 PRs OR >=5 active days in the window). - Report top-N concentration (e.g. share of PRs held by the top 3) and flag bus-factor risk where a domain or the bulk of throughput sits with one or two people. - If you ran two windows, note who is stable across both vs who appeared/dropped off — the honest "genuinely active" count is those present in both windows. - Alias note: metrics.py carries a project-specific AUTHOR_ALIASES map (people who commit under both a name and a username/email). Extend it for the target project so one person is not counted as several; report the alias-merged headcount.

4. Area / domain breakdown (monorepo only)

If the repo is a monorepo, tabulate which top-level domains the core contributors actually touched in the window, and how many of the repo's total domains saw no core activity. This scopes the benchmark honestly: a per-active-contributor rate describes the actively-developed slice, not the dormant remainder. Pair throughput with a domain- ownership view so single-owner domains stay visible.

5. Benchmark check

State the benchmark — 1 good (meaningful) change per 2 genuinely-active contributors per work-day (tunable via --contributors-per-change semantics in benchmark()), the implied target for the window, the actual, and attainment %. Recommend the active- contributor denominator to anchor on (prefer the count stable across adjacent windows), and explicitly scope the benchmark to the active product lines rather than the whole monorepo.

6. Batch size — how much rides in each landing

Frequency alone hides lumpy delivery. Run scripts/batch_size.py and report net commits per landing on the trunk — not PRs, not branches, since one PR can carry a hundred commits.

Pass each clone explicitly — paths are never baked into the script:

# PowerShell
py -3 scripts\batch_size.py --since <d> --until <d> `
  --repo "plumbing prod=$env:USERPROFILE\source\repos\plumbing:master" `
  --repo "plumbing dev=$env:USERPROFILE\source\repos\plumbing:origin/develop" `
  --repo "$env:USERPROFILE\source\repos\pipe-dream"

# Git Bash / WSL / macOS / Linux
python scripts/batch_size.py --since <d> --until <d> \
  --repo "plumbing prod=<clone>:master" --repo "plumbing dev=<clone>:origin/develop" \
  --repo <clone>

Quote each target so the shell does not split it. Windows drive letters are handled: C:\src\repo reads as a path, C:\src\repo:master splits on the last colon. Omit :BRANCH to resolve the remote's default branch. Point two labels at one clone to profile a release train's production and integration branches separately.

Report per repo: landings, total net commits, median / p75 / max commits per landing, median files and churn, and batch age (oldest authored commit in the batch → landing). Then compare across the repos in scope in a single table — the comparison is where the signal is.

Interpretation rules:

• Check the landing strategy before calling 1.0 an artefact. Squash always reports 1 commit per landing; two-parent merges report the true count. collect.py already detects this. Verify per repo rather than attaching a blanket caveat to the whole table — a repo that can show large batches and does not is evidence, not noise.
• Churn and batch age are strategy-independent — use them as the cross-check.
• High frequency with large batches is still lumpy delivery. Report cadence (median gap between landings) and batch size as separate findings — a repo can deploy every day and still ship three weeks of aged work each time.
• Where batch size is an outlier, connect it to the failure proxy (§7): large, aged, co-mingled batches make root-cause isolation harder. State it as a correlation to test with the team, not a proven cause.
• Never present a release train as a justification. It explains the shape of delivery; the batch size is still the finding.

If asked how a team could improve the measurement, the strategy matrix is:

Strategy	Landing boundary	Net commits	Original author dates
Squash	✅	❌ always 1	❌ destroyed (author date = squash time)
Rebase + fast-forward	❌ none	⚠️ each commit is its own landing	✅ preserved
Rebase + merge commit (ADO semi-linear)	✅	✅	✅ preserved

Recommend semi-linear merge. Rebase alone rewrites only the committer date, so authoring time survives — but a strict fast-forward rebase removes the landing boundary and makes batch size unmeasurable. The merge commit is what makes a release legible.

7. Change-failure proxy (fix-follows-change)

Status: experimental. This subsection is deliberately kept inside the change throughput diagnostic because the same collected history answers it. Treat the numbers as a directional signal to be confirmed with the team, not as a measured change failure rate. It may graduate to a standalone scan type once enough runs exist to calibrate it.

Ask: after a change lands, how often does a follow-up change arrive that looks like it is repairing that change? There is no incident feed behind this — only commit text and file overlap — so confidence must be stated on every number.

Classification (implemented in scripts/failures.py, all unit-tested):

Confidence	Signal	Counted in the headline rate?
HIGH	conventional prefix — fix:, hotfix:, revert:, Revert "…"	yes
MEDIUM	strong keyword in the change title — fix / bug / hotfix / patch / rollback / defect / regression / broken	yes
LOW	ambiguous word only ("correct", "failure", "issue"), fix wording found only in the body, or bare file overlap with no wording	no — goes to a confirmation queue for the user to accept or reject

Report:

• Fix rate = fixes / changes landed in the window.
• Attributed-fix rate (the CFR proxy) = in-window changes that later drew a fix, over non-fix changes. Only this one is the failure-rate analogue.
• Median time-to-fix for attributed pairs.
• Fix hotspots — files repeatedly implicated. These are the practical follow-up.
• Confirmation queue — list the LOW-confidence candidates explicitly and ask the user to confirm or reject them. Do not silently fold them into the headline.

Read the result honestly:

• Where the linkage is only file overlap, say so and mark it low confidence. Look for corroborating evidence in the commit message or a code comment before promoting it.
• A repo whose titles are bare Jira IDs will score ~0% with a long confirmation queue. That is a commit-message hygiene finding, not evidence of defect-free delivery. State it that way.
• Fixes attributed to changes older than the window are counted as fixes but cannot implicate an in-window change. Report attributed vs unattributed separately.

8. Summary

Open with an At a glance block: window, branch, LTFC median/p75, DF (meaningful)/week, genuinely-active count, benchmark attainment. Then the sections above. Use tables for all quantitative data and cite actual numbers, not generalities. Flag any process observations (e.g. near-zero review latency straight to prod for large changes).

Output format: a single markdown document in the portfolio scans folder the user confirms. A common path is docs/management/compliance/scans/<repo_id>/change-throughput-YYYY-MM.md, compared against any prior report there under Progress since last run. Store raw JSON in that repo's raw/ folder. Do not save interim artifacts in the target repo unless the user asks.

---

Tips for reviewers

• Scope first, always. The window and target project/branch change the numbers materially. Never run history commands before the scope gate is satisfied.
• Two windows beat one. A single window hides whether the active-contributor count is stable. Run the prior period too before locking a benchmark denominator.
• Trust the source, not the headline. LTFC from git author dates and DF from merge counts are rewrite-immune; cycle time is not measurable without an API token and adds little here.
• Tune the definitions in metrics.py, not in prose: micro thresholds, active-contributor thresholds (min_prs, min_active_days, require_both), and the benchmark rate (contributors_per_change) are all parameters with unit tests.
• Keep aliases current. Update AUTHOR_ALIASES per project so headcount is honest. Repos where people commit as both a work address and a personal address will otherwise split one person across several rows.
• The failure proxy measures commit hygiene as much as quality. A 0% fix rate next to a 17-item confirmation queue means the titles carry no intent, not that nothing broke.
• Optional API path: to add cycle time / review latency, see the documented Bitbucket path in scripts/collect.py — it requires a personal API token (scopes read:repository:bitbucket, read:pullrequest:bitbucket) from https://id.atlassian.com/manage-profile/security/api-tokens.