Skip to content

Commit 2849a1b

Browse files
jasonleenaylorclaude
jasonleenaylor
and
claude
committed
Add native C++ coverage via OpenCppCoverage (Codecov flag: native)
dotnet-coverage is CLR-only, so the two native Unit++ executables (testGenericLib.exe, TestViews.exe) contributed nothing to the coverage uploaded from CI. OpenCppCoverage fills that gap: it is free (GPL-3, run-only in CI), debugger-based (needs only the PDBs already produced by a Debug build), exports Cobertura natively, and never rewrites binaries in the shared Output/<Configuration> folder - the same constraint that disqualified coverlet for the managed side. Validated locally against a Debug build (OpenCppCoverage 0.9.9.0 x64): - Fidelity: identical Unit++ summaries under coverage (testGenericLib [31-0-0], TestViews [301-0-0]), exit code 0, no assert dialogs or hangs, including testGenericLib's deliberate test-induced native assert - Exit codes: OpenCppCoverage propagates the child's non-zero exit code (verified by forcing a failing run); the wrapper additionally parses the Unit++ summary line, matching Invoke-CppTest.ps1 policy - Overhead: testGenericLib ~0.4s -> ~1.3s, TestViews ~5s -> ~20s (~4x, debugger-based collection) - Output: ~51% native line coverage across Views.dll, DebugProcs.dll and the statically linked Generic sources; ReportGenerator 5.3.11 parses the native cobertura files alone and merged with the managed one Changes: - Add Build/Agent/Run-NativeCoverage.ps1: opt-in post-build re-run of both exes under OpenCppCoverage with the same assert-dialog suppression and DistFiles environment as the regular native test run. Writes Output/<Configuration>/TestResults/ native.<exe>.cobertura.xml plus its own <exe>.exe.coverage.log, leaving the canonical <exe>.exe.log files untouched. Cobertura paths are rewritten repo-relative so Codecov matches files on any checkout path without environment-specific fixes - Consolidate duplicated test-run plumbing into FwBuildHelpers.psm1: Set-TestAssertDialogEnvironment (was copied in test.ps1 and Invoke-CppTest.ps1), Disable-CrashDialog (SetErrorMode P/Invoke), and Get-UnitppSummary (Unit++ summary-line parsing), consumed by test.ps1, Invoke-CppTest.ps1, and the new coverage script - CI: install OpenCppCoverage 0.9.9.0 (pinned) via choco, collect native coverage after the test run, and upload under flags: native / name: FieldWorks-native - deliberately separate from the managed upload so the two pipelines stay independently diagnosable - Extract the Codecov retry/verify step sequence into a local composite action (.github/actions/codecov-upload) used by both the managed and native uploads, so the retry policy and pinned action SHA live in one place - codecov.yml: ignore native Unit++ test sources (Src/Generic/Test, Src/views/Test), matching the existing managed test-source ignore - Update fieldworks-test-coverage skill notes for the native path Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 8dd2468 commit 2849a1b

8 files changed

Lines changed: 397 additions & 86 deletions

File tree

‎.claude/skills/fieldworks-test-coverage/SKILL.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,4 +40,5 @@ You will receive:
4040
- See `verify-test` for the broader build/test verification flow this slots into.
4141
- See `execute-implement`'s self-check step, which references this skill for new logic.
4242
- `test.ps1 -Coverage` restores the local dotnet tool manifest automatically (`dotnet-coverage` collects the raw `coverage.cobertura.xml`; ReportGenerator renders the summary/HTML). On PRs, Codecov reports the same data as a patch-coverage check.
43+
- `test.ps1 -Coverage` covers managed code only (`dotnet-coverage` is CLR-based). For native C++ changes (`Src/Generic`, `Src/views`), run `.\Build\Agent\Run-NativeCoverage.ps1 -Configuration Debug` after a `-BuildTests` build: it re-runs `testGenericLib.exe`/`TestViews.exe` under OpenCppCoverage (install via `choco install opencppcoverage`; debugger-based, needs only PDBs, never rewrites binaries in the shared Output folder) and writes `Output/<Configuration>/TestResults/native.<exe>.cobertura.xml`, which ReportGenerator accepts alongside the managed file. Overhead is ~4x wall time (TestViews ~5s bare, ~20s covered). CI uploads these under the Codecov flag `native`, separate from `managed`.
4344
</notes>

‎.github/actions/codecov-upload/action.yml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
name: Codecov upload with retry
2+
description: >-
3+
Upload coverage files to Codecov, retrying the whole invocation once after a
4+
backoff. The Codecov CLI retries individual HTTP requests, but
5+
whole-invocation failures (503 storms, commit-creation errors) slip past
6+
fail_ci_if_error, so the final step fails the job when neither attempt
7+
succeeded.
8+
inputs:
9+
token:
10+
description: Codecov upload token
11+
required: true
12+
files:
13+
description: Comma-separated coverage files to upload
14+
required: true
15+
flags:
16+
description: Codecov flag for this upload
17+
required: true
18+
name:
19+
description: Codecov upload name
20+
required: true
21+
22+
runs:
23+
using: composite
24+
steps:
25+
- name: Upload to Codecov (attempt 1)
26+
id: attempt1
27+
continue-on-error: true
28+
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
29+
with:
30+
token: ${{ inputs.token }}
31+
fail_ci_if_error: true
32+
files: ${{ inputs.files }}
33+
flags: ${{ inputs.flags }}
34+
name: ${{ inputs.name }}
35+
36+
- name: Wait before Codecov retry
37+
if: ${{ steps.attempt1.outcome == 'failure' }}
38+
shell: powershell
39+
run: Start-Sleep -Seconds 30
40+
41+
- name: Upload to Codecov (attempt 2)
42+
id: attempt2
43+
if: ${{ steps.attempt1.outcome == 'failure' }}
44+
continue-on-error: true
45+
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
46+
with:
47+
token: ${{ inputs.token }}
48+
fail_ci_if_error: true
49+
files: ${{ inputs.files }}
50+
flags: ${{ inputs.flags }}
51+
name: ${{ inputs.name }}
52+
53+
- name: Verify Codecov upload succeeded
54+
shell: powershell
55+
run: |
56+
$outcomes = @('${{ steps.attempt1.outcome }}', '${{ steps.attempt2.outcome }}')
57+
if ($outcomes -contains 'success') { Write-Host 'Coverage uploaded to Codecov.'; exit 0 }
58+
Write-Error 'Codecov upload failed after 2 attempts.'
59+
exit 1

‎.github/workflows/CI.yml‎

Lines changed: 19 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -38,46 +38,37 @@ jobs:
3838
.\test.ps1 -Configuration Debug -NoBuild -Coverage -TestFilter 'TestCategory!=LongRunning&TestCategory!=ByHand&TestCategory!=SmokeTest&TestCategory!=DesktopRequired'
3939
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
4040
41-
# The Codecov CLI retries individual HTTP requests, but whole-invocation failures (503 storms,
42-
# commit-creation errors) slip past it. Add one step-level retry after a backoff; the gate step
43-
# below fails CI if neither attempt succeeds.
44-
- name: Upload coverage to Codecov (attempt 1)
45-
id: codecov1
41+
- name: Upload coverage to Codecov
4642
if: ${{ !cancelled() }}
47-
continue-on-error: true
48-
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
43+
uses: ./.github/actions/codecov-upload
4944
with:
5045
token: ${{ secrets.CODECOV_TOKEN }}
51-
fail_ci_if_error: true
5246
files: Output/Debug/TestResults/CoverageReport/Cobertura.xml
5347
flags: managed
5448
name: FieldWorks
5549

56-
- name: Wait before Codecov retry
57-
if: ${{ !cancelled() && steps.codecov1.outcome == 'failure' }}
50+
- name: Install OpenCppCoverage
51+
if: ${{ !cancelled() }}
5852
shell: powershell
59-
run: Start-Sleep -Seconds 30
60-
61-
- name: Upload coverage to Codecov (attempt 2)
62-
id: codecov2
63-
if: ${{ !cancelled() && steps.codecov1.outcome == 'failure' }}
64-
continue-on-error: true
65-
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
66-
with:
67-
token: ${{ secrets.CODECOV_TOKEN }}
68-
fail_ci_if_error: true
69-
files: Output/Debug/TestResults/CoverageReport/Cobertura.xml
70-
flags: managed
71-
name: FieldWorks
53+
run: |
54+
choco install opencppcoverage --version=0.9.9.0 -y --no-progress
55+
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
7256
73-
- name: Verify Codecov upload succeeded
57+
- name: Collect native coverage
7458
if: ${{ !cancelled() }}
7559
shell: powershell
7660
run: |
77-
$outcomes = @('${{ steps.codecov1.outcome }}', '${{ steps.codecov2.outcome }}')
78-
if ($outcomes -contains 'success') { Write-Host 'Coverage uploaded to Codecov.'; exit 0 }
79-
Write-Error 'Codecov upload failed after 2 attempts.'
80-
exit 1
61+
.\Build\Agent\Run-NativeCoverage.ps1 -Configuration Debug
62+
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
63+
64+
- name: Upload native coverage to Codecov
65+
if: ${{ !cancelled() }}
66+
uses: ./.github/actions/codecov-upload
67+
with:
68+
token: ${{ secrets.CODECOV_TOKEN }}
69+
files: Output/Debug/TestResults/native.testGenericLib.cobertura.xml,Output/Debug/TestResults/native.TestViews.cobertura.xml
70+
flags: native
71+
name: FieldWorks-native
8172

8273
- name: Summarize native test results
8374
if: ${{ always() }}

‎Build/Agent/FwBuildHelpers.psm1‎

Lines changed: 73 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -601,6 +601,75 @@ function Test-ViewsNativeArtifactsStale {
601601
return $latestSource -gt $oldestArtifact
602602
}
603603

604+
function Set-TestAssertDialogEnvironment {
605+
<#!
606+
.SYNOPSIS
607+
Sets the DebugProcs.dll assertion environment variables for a test run.
608+
.DESCRIPTION
609+
By default puts asserts into unattended mode (converted to exceptions, no MessageBox)
610+
so an assert cannot block a headless run. Pass -AllowDialogs $true to get interactive
611+
Abort/Retry/Ignore dialogs for local debugger attachment.
612+
#>
613+
param(
614+
[bool]$AllowDialogs
615+
)
616+
617+
if ($AllowDialogs) {
618+
$env:AssertUiEnabled = 'true'
619+
$env:AssertExceptionEnabled = 'false'
620+
$env:FW_TEST_MODE = '0'
621+
$env:FW_TEST_ALLOW_ASSERT_DIALOGS = '1'
622+
return
623+
}
624+
625+
$env:AssertUiEnabled = 'false'
626+
$env:AssertExceptionEnabled = 'true'
627+
# Unconditional test-mode override: bypasses the registry AssertMessageBox key in DebugProcs.dll.
628+
$env:FW_TEST_MODE = '1'
629+
}
630+
631+
function Disable-CrashDialog {
632+
<#!
633+
.SYNOPSIS
634+
Suppresses Windows Error Reporting and crash/critical-error dialogs for this process
635+
and its children.
636+
#>
637+
# SEM_FAILCRITICALERRORS (0x1) | SEM_NOGPFAULTERRORBOX (0x2) | SEM_NOOPENFILEERRORBOX (0x8000)
638+
if (-not ('FwBuildHelpers.NativeMethods' -as [type])) {
639+
Add-Type -MemberDefinition '[DllImport("kernel32.dll")] public static extern uint SetErrorMode(uint uMode);' `
640+
-Name 'NativeMethods' -Namespace 'FwBuildHelpers'
641+
}
642+
[void][FwBuildHelpers.NativeMethods]::SetErrorMode(0x8003)
643+
}
644+
645+
function Get-UnitppSummary {
646+
<#!
647+
.SYNOPSIS
648+
Parses the last Unit++ "Tests [Ok-Fail-Error]: [n-n-n]" summary line from test output.
649+
.DESCRIPTION
650+
Returns an object with Ok/Fail/Error counts and the matched Text, or $null when the
651+
output contains no summary line (crash before completion, empty log).
652+
#>
653+
param(
654+
[string[]]$LogContent
655+
)
656+
657+
$summaryLine = $LogContent |
658+
Select-String -Pattern 'Tests \[Ok-Fail-Error\]: \[(\d+)-(\d+)-(\d+)\]' |
659+
Select-Object -Last 1
660+
if (-not $summaryLine) {
661+
return $null
662+
}
663+
664+
$match = [regex]::Match($summaryLine.Line, 'Tests \[Ok-Fail-Error\]: \[(\d+)-(\d+)-(\d+)\]')
665+
return [pscustomobject]@{
666+
Ok = [int]$match.Groups[1].Value
667+
Fail = [int]$match.Groups[2].Value
668+
Error = [int]$match.Groups[3].Value
669+
Text = $match.Value
670+
}
671+
}
672+
604673
# =============================================================================
605674
# Module Exports
606675
# =============================================================================
@@ -623,5 +692,8 @@ Export-ModuleMember -Function @(
623692
'Invoke-WithFileLockRetry',
624693
'Get-NewestWriteTimeUtc',
625694
'Get-OldestWriteTimeUtc',
626-
'Test-ViewsNativeArtifactsStale'
695+
'Test-ViewsNativeArtifactsStale',
696+
'Set-TestAssertDialogEnvironment',
697+
'Disable-CrashDialog',
698+
'Get-UnitppSummary'
627699
)

0 commit comments

Comments
 (0)