Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/CI.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ starting point. Two details matter:
| `--format` | Use |
|---|---|
| `console` (default) | Human-readable output on stdout, for build logs |
| `json` | Structured results for scripts and other tools |
| `json` | Structured results for scripts and other tools (versioned contract, see [RESULT_CONTRACT.md](architecture/RESULT_CONTRACT.md)) |
| `sarif` | SARIF 2.1.0 for code-scanning platforms |
| `html` | A standalone report page for humans |

Expand Down
3 changes: 2 additions & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,15 @@ Evergreen scope documents and living reference docs, updated in place and never
| [`CI.md`](CI.md) | CI integration: exit codes, `--fail-on`, output formats, and copy-paste recipes for GitHub Actions, Azure Pipelines, GitLab CI, Jenkins and Bitbucket Pipelines. |
| [`architecture/CORE_RULES.md`](architecture/CORE_RULES.md) | Product scope: the six policy pillars, non-goals, enforcement levels and waivers. |
| [`architecture/IMPLEMENTATION_STATUS.md`](architecture/IMPLEMENTATION_STATUS.md) | Build history, decisions and gotchas. Read before making non-trivial changes. |
| [`architecture/RESULT_CONTRACT.md`](architecture/RESULT_CONTRACT.md) | The versioned JSON result contract (v1) for `validate --format json`: shape, field meanings, versioning policy. |

## Roadmap (outstanding)

Documents describing work that is at least partly not yet built.

| Doc | What's left |
|---|---|
| [`roadmap/HIGH_LEVEL_ROADMAP.md`](roadmap/HIGH_LEVEL_ROADMAP.md) | Phase 2 (binaries, JSON/SARIF, exit codes) is mostly done. Still open: rule distribution ([#82](https://github.com/james-d12/CodeGuard/issues/82)), the result contract and CI governance (#66–#78), and server/portal/registry ([#83](https://github.com/james-d12/CodeGuard/issues/83)). |
| [`roadmap/HIGH_LEVEL_ROADMAP.md`](roadmap/HIGH_LEVEL_ROADMAP.md) | Phase 2 (binaries, JSON/SARIF, exit codes) is mostly done. Still open: rule distribution ([#82](https://github.com/james-d12/CodeGuard/issues/82)), CI governance (#67–#78), and server/portal/registry ([#83](https://github.com/james-d12/CodeGuard/issues/83)). |
| [`roadmap/HIGH_LEVEL_AI_ASSISTING.md`](roadmap/HIGH_LEVEL_AI_ASSISTING.md) | Phases 0–3 and 5 are done. Still open: the MCP server ([#62](https://github.com/james-d12/CodeGuard/issues/62)) and Phase 6 ([#86](https://github.com/james-d12/CodeGuard/issues/86)). |
| [`roadmap/SUPPORTING_TOOLS.md`](roadmap/SUPPORTING_TOOLS.md) | validate/explain/test/lint shipped as `codeguard rules …`. Still open: `inspect` (#59) and `generate` (#60, #61). |
| [`roadmap/REPORTING_API_PROMPT.md`](roadmap/REPORTING_API_PROMPT.md) | A prompt for the Reporting API MVP design. Nothing is built yet ([#83](https://github.com/james-d12/CodeGuard/issues/83)). |
Expand Down
5 changes: 5 additions & 0 deletions docs/USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,11 @@ cd codeguard
files. It recomputes `metadata.source.fingerprint` and `versionFingerprint` values in place, and
leaves other content untouched. Rules with a broken source link are not fixed automatically.

`validate --format json` writes the versioned result contract (`schemaVersion` `1.0`). It is a
breaking change from the earlier output, which serialised the internal result model. See
[`architecture/RESULT_CONTRACT.md`](architecture/RESULT_CONTRACT.md) for the shape, status values and
the JSON Schema location.

`--rule` on `validate` and `rules test` takes an ID. Repeat the flag to select several rules, for
example `--rule A --rule B`. Space-separated values are not supported.

Expand Down
31 changes: 30 additions & 1 deletion docs/architecture/IMPLEMENTATION_STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -652,7 +652,8 @@ on top of.
- Every `Violation` carries `RuleVersion` (`CodeGuard.Core.Results.ValidationResult.cs`), stamped
from `RuleDefinition.Version` at both construction sites in `RuleEvaluator`
(`EvaluateAnalyzerRule`/`CreateViolation`). This is the concrete consumer `metadata.source`'s
rejected `status` field lacked: `JsonViolationReporter` gets it for free (record serialization),
rejected `status` field lacked: `JsonViolationReporter` carries it in the v1 contract (see the
"Versioned JSON result contract" section below),
and `SarifViolationReporter` stamps it onto each rule's `ReportingDescriptor` via the SARIF SDK's
`properties` bag (`SetProperty("version", ...)`) - **not** a native `ReportingDescriptor.Version`
property, which does not exist in `Sarif.Sdk` 5.7.0 despite an early assumption otherwise (verified
Expand Down Expand Up @@ -716,6 +717,34 @@ rule by hand now uses `rules discover` (the engine's vocabulary) plus `rule.sche
path the AI skill already follows - same precedent as the earlier `list-standards` removal (see
"Things NOT done" below).

### Post-v1 addition: versioned JSON result contract (#66)

`--format json` no longer serialises `CodeGuard.Core.Results.ValidationResult` directly. It writes
the explicit `ValidationReportV1` DTO (`CodeGuard.Reporting/Contracts/`), built by
`ValidationReportMapper.ToV1`, which adds `schemaVersion`, `toolVersion`, `durationMs`, `status` and
a `metrics` block. The JSON Schema is embedded from
`CodeGuard.Reporting/Schemas/validation-report.v1.schema.json`, and tests validate real reporter
output against it. This is a breaking change to the JSON output. Full contract: `docs/architecture/RESULT_CONTRACT.md`.

Decisions and gotchas:

- The DTO maps enums through explicit `switch` expressions, not `ToString()`, so renaming an
internal enum member cannot change the wire format. Status values stay `passed`, `failed` and
`partiallyEvaluated`, as before.
- `toolVersion` cannot come from `typeof(...).Assembly` inside `CodeGuard.Reporting`, because MinVer
is referenced only by `CodeGuard.Cli`. `ValidateCommand` passes its own assembly's version into
`JsonViolationReporter`. The reporter's default falls back to its own assembly.
- `durationMs` is measured by `ValidateCommand` from the start of the action until the report is
written. It is passed through the reporter constructor, not `IViolationReporter.WriteAsync`, so the
interface stays unchanged for the console, SARIF and HTML reporters.
- The schema is embedded as a resource in `CodeGuard.Reporting` with an explicit `LogicalName`. It
is not in `examples/rules/`, so `scripts/verify-nupkg-contents.sh` needs no change. It only checks for
`rules/` paths and YAML files.
- `line` and `column` can be `0` (for example `(lineSpan?.StartLinePosition.Line ?? -1) + 1` in
`RoslynTypeExtractor`), so the schema allows a minimum of `0`.
- `JsonSchema.Net` is referenced by `CodeGuard.Core.Tests` only, to validate output. Each schema is
parsed once in a static `Lazy`, because `JsonSchema.FromText` throws on a duplicate `$id`.

## The 11 starter rules

All under `rules/`, all illustrative (`Contoso.*` namespace, `illustrative: true`), matching the
Expand Down
126 changes: 126 additions & 0 deletions docs/architecture/RESULT_CONTRACT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# Validation result contract (v1)

`codeguard validate --format json` writes a versioned report. It is the public contract that CI
integrations, baselines, the MCP server and the reporting API consume. It is deliberately **not** a
serialisation of the internal `CodeGuard.Core.Results.ValidationResult`: the internal model can
change without changing this output.

| Artifact | Location |
|---|---|
| Output DTOs | `src/CodeGuard.Reporting/Contracts/ValidationReportV1.cs` |
| Mapper (`ValidationResult` to DTO) | `src/CodeGuard.Reporting/Contracts/ValidationReportMapper.cs` |
| JSON Schema (draft 2020-12, embedded resource) | `src/CodeGuard.Reporting/Schemas/validation-report.v1.schema.json` |
| Tests | `tests/CodeGuard.Core.Tests/Reporting/ValidationReport*Tests.cs` |

The schema `$id` is `https://codeguard.internal/schemas/validation-report.v1.schema.json`. Every
object has `additionalProperties: false`, so a consumer can rely on the set of keys.

## Shape

```json
{
"schemaVersion": "1.0",
"toolVersion": "0.3.1",
"status": "failed",
"evaluatedAtUtc": "2026-10-13T14:42:10.7202958+00:00",
"durationMs": 6705,
"metrics": {
"rulesEvaluated": 11,
"rulesPassed": 7,
"rulesFailed": 4,
"rulesErrored": 0,
"violationCount": 4,
"violationsBySeverity": { "info": 0, "warning": 0, "error": 3, "critical": 1 },
"evaluationErrorCount": 0,
"analysisWarningCount": 0
},
"violations": [
{
"ruleId": "DDD-ENTITY-001",
"ruleVersion": 1,
"severity": "error",
"message": "'Contoso.Domain.Entities.LegacyThing' must inherit from 'Contoso.Domain.Entity<*>'.",
"file": "Contoso.Domain/Entities/LegacyThing.cs",
"line": 5,
"column": 14,
"symbol": "Contoso.Domain.Entities.LegacyThing",
"project": "Contoso.Domain",
"remediation": "Inherit from Contoso.Domain.Entity<TId>.",
"documentationReferences": []
}
],
"evaluationErrors": [],
"analysisWarnings": []
}
```

Optional fields (`file`, `line`, `column`, `symbol`, `project`, `remediation`, and the optional
fields of `evaluationErrors` and `analysisWarnings`) are omitted when unknown, not written as `null`.

## Top-level fields

| Field | Type | Meaning |
|---|---|---|
| `schemaVersion` | `"1.0"` | Version of this contract. A breaking change to the shape needs a new schema version. |
| `toolVersion` | string | Version of the CLI that produced the report. Taken from the assembly informational version with any `+commit` suffix removed. |
| `status` | enum | See below. |
| `evaluatedAtUtc` | date-time | When rule evaluation completed, ISO 8601 with offset. |
| `durationMs` | integer | Wall-clock milliseconds from the start of the `validate` command until the report is written. |
| `metrics` | object | Counts; see below. |
| `violations` | array | Violations that survived `--severity-threshold`. |
| `evaluationErrors` | array | Rules that threw during evaluation. |
| `analysisWarnings` | array | Signals that the analysis model is incomplete, e.g. a project that MSBuild could not fully load. |

### `status`

| Value | Meaning |
|---|---|
| `passed` | No violations reported and no evaluation errors. |
| `failed` | At least one violation reported and no evaluation errors. |
| `partiallyEvaluated` | At least one rule threw during evaluation, so results are incomplete. |

`status` describes the report, not the process exit code. It is computed after
`--severity-threshold`, so a report with only warnings is `failed` even when `--fail-on error`
makes the command exit 0. Exit `1` is returned when a violation at or above `--fail-on` is
reported, or when any evaluation error occurred.

### `durationMs`

An integer number of milliseconds, measured by the CLI. It covers rule loading, model building and
evaluation, and ends just before the report is written. A library caller that does not measure
time gets `0`.

### `metrics`

| Field | Meaning |
|---|---|
| `rulesEvaluated` | Enabled rules that were run (after `--rule` filtering). |
| `rulesPassed` / `rulesFailed` / `rulesErrored` | Per-rule outcome counts. |
| `violationCount` | Length of `violations`. |
| `violationsBySeverity` | Counts for `info`, `warning`, `error` and `critical`. All four keys are always present. |
| `evaluationErrorCount` | Length of `evaluationErrors`. |
| `analysisWarningCount` | Length of `analysisWarnings`. |

## Changes from the pre-v1 output

The earlier output was the internal `ValidationResult` serialised directly. Consumers of
`--format json` must update for v1:

- `rulesEvaluated`, `rulesPassed`, `rulesFailed` and `rulesErrored` moved under `metrics`.
- `schemaVersion`, `toolVersion`, `durationMs` and `metrics` are new.
- `status` keeps its values (`passed`, `failed`, `partiallyEvaluated`).
- Violation, evaluation error and analysis warning fields keep their names.

## Versioning policy

- Adding an optional field, or a new value to a documented enum, keeps `schemaVersion` `1.0`.
Consumers must ignore unknown enum values. Note that the schema's `additionalProperties: false`
means a new field also changes the schema, so the schema is updated in the same change.
- Removing or renaming a field, changing a type, or changing the meaning of a value requires a new
`schemaVersion` and a new `validation-report.vN.schema.json`.

## Not yet in the contract

Planned by the roadmap but not part of v1: `rulesVersion`, `repository`, `commit`, `branch`, a
`timestamp` separate from `evaluatedAtUtc`, and per-run size metrics such as projects and types
analysed. These need the run context to be passed into the reporter.
11 changes: 8 additions & 3 deletions src/CodeGuard.Cli/Commands/ValidateCommand.cs
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
using CodeGuard.Core.Results;
using CodeGuard.Reporting;
using CodeGuard.Reporting.Console;
using CodeGuard.Reporting.Contracts;
using CodeGuard.Reporting.Html;
using CodeGuard.Reporting.Json;
using CodeGuard.Reporting.Sarif;
Expand All @@ -18,7 +19,7 @@

public static class ValidateCommand
{
public static Command Build()

Check warning on line 22 in src/CodeGuard.Cli/Commands/ValidateCommand.cs

View workflow job for this annotation

GitHub Actions / Build And Test

Refactor this method to reduce its Cognitive Complexity from 29 to the 15 allowed.

Check warning on line 22 in src/CodeGuard.Cli/Commands/ValidateCommand.cs

View workflow job for this annotation

GitHub Actions / Build And Test

Refactor this method to reduce its Cognitive Complexity from 29 to the 15 allowed.
{
var pathOption = CommonOptions.CreatePathOption();
var configOption = CommonOptions.CreateConfigOption();
Expand Down Expand Up @@ -95,6 +96,7 @@

command.SetAction(async (parseResult, cancellationToken) =>
{
var totalStopwatch = Stopwatch.StartNew();
using var loggerFactory = CliLoggerFactory.Create(CliLoggerFactory.ParseVerbosity(parseResult.GetValue(verbosityOption)!));
var logger = loggerFactory.CreateLogger(typeof(ValidateCommand));

Expand Down Expand Up @@ -206,7 +208,8 @@
consoleOutputRedirected: Console.IsOutputRedirected,
noColorEnvVar: Environment.GetEnvironmentVariable("NO_COLOR"));

var reporter = CreateReporter(parseResult.GetValue(formatOption)!, useColor);
var reporter = CreateReporter(
parseResult.GetValue(formatOption)!, useColor, totalStopwatch.Elapsed);

logger.LogDebug("Writing {Format} report to {Destination}", parseResult.GetValue(formatOption), outputPath ?? "stdout");

Expand Down Expand Up @@ -256,13 +259,15 @@
RulesPassed = result.RulesEvaluated - rulesFailed - result.RulesErrored,
Status = result.EvaluationErrors.Count > 0
? ValidationStatus.PartiallyEvaluated
: filteredViolations.Count == 0 ? ValidationStatus.Passed : ValidationStatus.Failed

Check warning on line 262 in src/CodeGuard.Cli/Commands/ValidateCommand.cs

View workflow job for this annotation

GitHub Actions / Build And Test

Extract this nested ternary operation into an independent statement.

Check warning on line 262 in src/CodeGuard.Cli/Commands/ValidateCommand.cs

View workflow job for this annotation

GitHub Actions / Build And Test

Extract this nested ternary operation into an independent statement.
};
}

private static IViolationReporter CreateReporter(string format, bool useColor) => format switch
// durationMs in the JSON report is measured up to this point, i.e. just before the report is written.
private static IViolationReporter CreateReporter(string format, bool useColor, TimeSpan duration) => format switch
{
"json" => new JsonViolationReporter(),
"json" => new JsonViolationReporter(
ValidationReportMapper.ResolveToolVersion(typeof(ValidateCommand).Assembly), duration),
"sarif" => new SarifViolationReporter(),
"html" => new HtmlViolationReporter(),
_ => new ConsoleViolationReporter(useColor)
Expand Down
4 changes: 4 additions & 0 deletions src/CodeGuard.Reporting/CodeGuard.Reporting.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,8 @@
<PackageReference Include="Sarif.Sdk" Version="5.7.0" />
</ItemGroup>

<ItemGroup>
<EmbeddedResource Include="Schemas\validation-report.v1.schema.json" LogicalName="CodeGuard.Reporting.Schemas.validation-report.v1.schema.json" />
</ItemGroup>

</Project>
96 changes: 96 additions & 0 deletions src/CodeGuard.Reporting/Contracts/ValidationReportMapper.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
using System.Reflection;
using CodeGuard.Core.Results;
using CodeGuard.RuleModel.Rules;

namespace CodeGuard.Reporting.Contracts;

/// <summary>Maps the internal <see cref="ValidationResult"/> to the public <see cref="ValidationReportV1"/> contract.</summary>
public static class ValidationReportMapper
{
public const string SchemaVersion = "1.0";

public static ValidationReportV1 ToV1(ValidationResult result, TimeSpan duration, string toolVersion)
{
var violations = result.Violations.Select(MapViolation).ToList();

return new ValidationReportV1(
SchemaVersion: SchemaVersion,
ToolVersion: toolVersion,
Status: MapStatus(result.Status),
EvaluatedAtUtc: result.EvaluatedAtUtc,
DurationMs: (long)duration.TotalMilliseconds,
Metrics: new ValidationReportMetricsV1(
RulesEvaluated: result.RulesEvaluated,
RulesPassed: result.RulesPassed,
RulesFailed: result.RulesFailed,
RulesErrored: result.RulesErrored,
ViolationCount: violations.Count,
ViolationsBySeverity: CountBySeverity(result.Violations),
EvaluationErrorCount: result.EvaluationErrors.Count,
AnalysisWarningCount: result.AnalysisWarnings.Count),
Violations: violations,
EvaluationErrors: result.EvaluationErrors
.Select(e => new EvaluationErrorV1(e.RuleId, e.ExceptionType, e.Message, e.StackTrace))
.ToList(),
AnalysisWarnings: result.AnalysisWarnings
.Select(w => new AnalysisWarningV1(w.Code, w.Message, w.Project, w.FilePath))
.ToList());
}

/// <summary>
/// The assembly's informational version with any source-link "+commit" suffix removed
/// (e.g. "0.3.1+abc123" becomes "0.3.1"). Falls back to "0.0.0" when no version is embedded.
/// </summary>
public static string ResolveToolVersion(Assembly assembly)
{
var informational = assembly.GetCustomAttribute<AssemblyInformationalVersionAttribute>()?.InformationalVersion;
if (string.IsNullOrWhiteSpace(informational))
{
return "0.0.0";
}

var plus = informational.IndexOf('+', StringComparison.Ordinal);
return plus >= 0 ? informational[..plus] : informational;
}

// Explicit wire values rather than enum ToString(), so renaming an internal enum member cannot change the contract.
private static string MapStatus(ValidationStatus status) => status switch
{
ValidationStatus.Passed => "passed",
ValidationStatus.Failed => "failed",
ValidationStatus.PartiallyEvaluated => "partiallyEvaluated",
_ => throw new ArgumentOutOfRangeException(nameof(status), status, "Unknown validation status.")
};

private static string MapSeverity(Severity severity) => severity switch
{
Severity.Info => "info",
Severity.Warning => "warning",
Severity.Error => "error",
Severity.Critical => "critical",
_ => throw new ArgumentOutOfRangeException(nameof(severity), severity, "Unknown severity.")
};

private static ViolationV1 MapViolation(Violation v) => new(
RuleId: v.RuleId,
RuleVersion: v.RuleVersion,
Severity: MapSeverity(v.Severity),
Message: v.Message,
File: v.File,
Line: v.Line,
Column: v.Column,
Symbol: v.Symbol,
Project: v.Project,
Remediation: v.Remediation,
DocumentationReferences: v.DocumentationReferences);

private static SeverityCountsV1 CountBySeverity(IReadOnlyList<Violation> violations)
{
int Count(Severity severity) => violations.Count(v => v.Severity == severity);
return new SeverityCountsV1(
Info: Count(Severity.Info),
Warning: Count(Severity.Warning),
Error: Count(Severity.Error),
Critical: Count(Severity.Critical));
}
}
Loading
Loading