diff --git a/docs/CI.md b/docs/CI.md index fd7f607..4ccda02 100644 --- a/docs/CI.md +++ b/docs/CI.md @@ -32,7 +32,7 @@ starting point. Two details matter: | `--format` | Use | |---|---| | `console` (default) | Human-readable output on stdout, for build logs | -| `json` | Structured results for scripts and other tools | +| `json` | Structured results for scripts and other tools (versioned contract, see [RESULT_CONTRACT.md](architecture/RESULT_CONTRACT.md)) | | `sarif` | SARIF 2.1.0 for code-scanning platforms | | `html` | A standalone report page for humans | diff --git a/docs/README.md b/docs/README.md index b08684f..d1900df 100644 --- a/docs/README.md +++ b/docs/README.md @@ -16,6 +16,7 @@ Evergreen scope documents and living reference docs, updated in place and never | [`CI.md`](CI.md) | CI integration: exit codes, `--fail-on`, output formats, and copy-paste recipes for GitHub Actions, Azure Pipelines, GitLab CI, Jenkins and Bitbucket Pipelines. | | [`architecture/CORE_RULES.md`](architecture/CORE_RULES.md) | Product scope: the six policy pillars, non-goals, enforcement levels and waivers. | | [`architecture/IMPLEMENTATION_STATUS.md`](architecture/IMPLEMENTATION_STATUS.md) | Build history, decisions and gotchas. Read before making non-trivial changes. | +| [`architecture/RESULT_CONTRACT.md`](architecture/RESULT_CONTRACT.md) | The versioned JSON result contract (v1) for `validate --format json`: shape, field meanings, versioning policy. | ## Roadmap (outstanding) @@ -23,7 +24,7 @@ Documents describing work that is at least partly not yet built. | Doc | What's left | |---|---| -| [`roadmap/HIGH_LEVEL_ROADMAP.md`](roadmap/HIGH_LEVEL_ROADMAP.md) | Phase 2 (binaries, JSON/SARIF, exit codes) is mostly done. Still open: rule distribution ([#82](https://github.com/james-d12/CodeGuard/issues/82)), the result contract and CI governance (#66–#78), and server/portal/registry ([#83](https://github.com/james-d12/CodeGuard/issues/83)). | +| [`roadmap/HIGH_LEVEL_ROADMAP.md`](roadmap/HIGH_LEVEL_ROADMAP.md) | Phase 2 (binaries, JSON/SARIF, exit codes) is mostly done. Still open: rule distribution ([#82](https://github.com/james-d12/CodeGuard/issues/82)), CI governance (#67–#78), and server/portal/registry ([#83](https://github.com/james-d12/CodeGuard/issues/83)). | | [`roadmap/HIGH_LEVEL_AI_ASSISTING.md`](roadmap/HIGH_LEVEL_AI_ASSISTING.md) | Phases 0–3 and 5 are done. Still open: the MCP server ([#62](https://github.com/james-d12/CodeGuard/issues/62)) and Phase 6 ([#86](https://github.com/james-d12/CodeGuard/issues/86)). | | [`roadmap/SUPPORTING_TOOLS.md`](roadmap/SUPPORTING_TOOLS.md) | validate/explain/test/lint shipped as `codeguard rules …`. Still open: `inspect` (#59) and `generate` (#60, #61). | | [`roadmap/REPORTING_API_PROMPT.md`](roadmap/REPORTING_API_PROMPT.md) | A prompt for the Reporting API MVP design. Nothing is built yet ([#83](https://github.com/james-d12/CodeGuard/issues/83)). | diff --git a/docs/USAGE.md b/docs/USAGE.md index bb9eb9b..eaf0f6a 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -158,6 +158,11 @@ cd codeguard files. It recomputes `metadata.source.fingerprint` and `versionFingerprint` values in place, and leaves other content untouched. Rules with a broken source link are not fixed automatically. +`validate --format json` writes the versioned result contract (`schemaVersion` `1.0`). It is a +breaking change from the earlier output, which serialised the internal result model. See +[`architecture/RESULT_CONTRACT.md`](architecture/RESULT_CONTRACT.md) for the shape, status values and +the JSON Schema location. + `--rule` on `validate` and `rules test` takes an ID. Repeat the flag to select several rules, for example `--rule A --rule B`. Space-separated values are not supported. diff --git a/docs/architecture/IMPLEMENTATION_STATUS.md b/docs/architecture/IMPLEMENTATION_STATUS.md index 7f52bd3..06f5e5d 100644 --- a/docs/architecture/IMPLEMENTATION_STATUS.md +++ b/docs/architecture/IMPLEMENTATION_STATUS.md @@ -652,7 +652,8 @@ on top of. - Every `Violation` carries `RuleVersion` (`CodeGuard.Core.Results.ValidationResult.cs`), stamped from `RuleDefinition.Version` at both construction sites in `RuleEvaluator` (`EvaluateAnalyzerRule`/`CreateViolation`). This is the concrete consumer `metadata.source`'s - rejected `status` field lacked: `JsonViolationReporter` gets it for free (record serialization), + rejected `status` field lacked: `JsonViolationReporter` carries it in the v1 contract (see the + "Versioned JSON result contract" section below), and `SarifViolationReporter` stamps it onto each rule's `ReportingDescriptor` via the SARIF SDK's `properties` bag (`SetProperty("version", ...)`) - **not** a native `ReportingDescriptor.Version` property, which does not exist in `Sarif.Sdk` 5.7.0 despite an early assumption otherwise (verified @@ -716,6 +717,34 @@ rule by hand now uses `rules discover` (the engine's vocabulary) plus `rule.sche path the AI skill already follows - same precedent as the earlier `list-standards` removal (see "Things NOT done" below). +### Post-v1 addition: versioned JSON result contract (#66) + +`--format json` no longer serialises `CodeGuard.Core.Results.ValidationResult` directly. It writes +the explicit `ValidationReportV1` DTO (`CodeGuard.Reporting/Contracts/`), built by +`ValidationReportMapper.ToV1`, which adds `schemaVersion`, `toolVersion`, `durationMs`, `status` and +a `metrics` block. The JSON Schema is embedded from +`CodeGuard.Reporting/Schemas/validation-report.v1.schema.json`, and tests validate real reporter +output against it. This is a breaking change to the JSON output. Full contract: `docs/architecture/RESULT_CONTRACT.md`. + +Decisions and gotchas: + +- The DTO maps enums through explicit `switch` expressions, not `ToString()`, so renaming an + internal enum member cannot change the wire format. Status values stay `passed`, `failed` and + `partiallyEvaluated`, as before. +- `toolVersion` cannot come from `typeof(...).Assembly` inside `CodeGuard.Reporting`, because MinVer + is referenced only by `CodeGuard.Cli`. `ValidateCommand` passes its own assembly's version into + `JsonViolationReporter`. The reporter's default falls back to its own assembly. +- `durationMs` is measured by `ValidateCommand` from the start of the action until the report is + written. It is passed through the reporter constructor, not `IViolationReporter.WriteAsync`, so the + interface stays unchanged for the console, SARIF and HTML reporters. +- The schema is embedded as a resource in `CodeGuard.Reporting` with an explicit `LogicalName`. It + is not in `examples/rules/`, so `scripts/verify-nupkg-contents.sh` needs no change. It only checks for + `rules/` paths and YAML files. +- `line` and `column` can be `0` (for example `(lineSpan?.StartLinePosition.Line ?? -1) + 1` in + `RoslynTypeExtractor`), so the schema allows a minimum of `0`. +- `JsonSchema.Net` is referenced by `CodeGuard.Core.Tests` only, to validate output. Each schema is + parsed once in a static `Lazy`, because `JsonSchema.FromText` throws on a duplicate `$id`. + ## The 11 starter rules All under `rules/`, all illustrative (`Contoso.*` namespace, `illustrative: true`), matching the diff --git a/docs/architecture/RESULT_CONTRACT.md b/docs/architecture/RESULT_CONTRACT.md new file mode 100644 index 0000000..ccb32bd --- /dev/null +++ b/docs/architecture/RESULT_CONTRACT.md @@ -0,0 +1,126 @@ +# Validation result contract (v1) + +`codeguard validate --format json` writes a versioned report. It is the public contract that CI +integrations, baselines, the MCP server and the reporting API consume. It is deliberately **not** a +serialisation of the internal `CodeGuard.Core.Results.ValidationResult`: the internal model can +change without changing this output. + +| Artifact | Location | +|---|---| +| Output DTOs | `src/CodeGuard.Reporting/Contracts/ValidationReportV1.cs` | +| Mapper (`ValidationResult` to DTO) | `src/CodeGuard.Reporting/Contracts/ValidationReportMapper.cs` | +| JSON Schema (draft 2020-12, embedded resource) | `src/CodeGuard.Reporting/Schemas/validation-report.v1.schema.json` | +| Tests | `tests/CodeGuard.Core.Tests/Reporting/ValidationReport*Tests.cs` | + +The schema `$id` is `https://codeguard.internal/schemas/validation-report.v1.schema.json`. Every +object has `additionalProperties: false`, so a consumer can rely on the set of keys. + +## Shape + +```json +{ + "schemaVersion": "1.0", + "toolVersion": "0.3.1", + "status": "failed", + "evaluatedAtUtc": "2026-10-13T14:42:10.7202958+00:00", + "durationMs": 6705, + "metrics": { + "rulesEvaluated": 11, + "rulesPassed": 7, + "rulesFailed": 4, + "rulesErrored": 0, + "violationCount": 4, + "violationsBySeverity": { "info": 0, "warning": 0, "error": 3, "critical": 1 }, + "evaluationErrorCount": 0, + "analysisWarningCount": 0 + }, + "violations": [ + { + "ruleId": "DDD-ENTITY-001", + "ruleVersion": 1, + "severity": "error", + "message": "'Contoso.Domain.Entities.LegacyThing' must inherit from 'Contoso.Domain.Entity<*>'.", + "file": "Contoso.Domain/Entities/LegacyThing.cs", + "line": 5, + "column": 14, + "symbol": "Contoso.Domain.Entities.LegacyThing", + "project": "Contoso.Domain", + "remediation": "Inherit from Contoso.Domain.Entity.", + "documentationReferences": [] + } + ], + "evaluationErrors": [], + "analysisWarnings": [] +} +``` + +Optional fields (`file`, `line`, `column`, `symbol`, `project`, `remediation`, and the optional +fields of `evaluationErrors` and `analysisWarnings`) are omitted when unknown, not written as `null`. + +## Top-level fields + +| Field | Type | Meaning | +|---|---|---| +| `schemaVersion` | `"1.0"` | Version of this contract. A breaking change to the shape needs a new schema version. | +| `toolVersion` | string | Version of the CLI that produced the report. Taken from the assembly informational version with any `+commit` suffix removed. | +| `status` | enum | See below. | +| `evaluatedAtUtc` | date-time | When rule evaluation completed, ISO 8601 with offset. | +| `durationMs` | integer | Wall-clock milliseconds from the start of the `validate` command until the report is written. | +| `metrics` | object | Counts; see below. | +| `violations` | array | Violations that survived `--severity-threshold`. | +| `evaluationErrors` | array | Rules that threw during evaluation. | +| `analysisWarnings` | array | Signals that the analysis model is incomplete, e.g. a project that MSBuild could not fully load. | + +### `status` + +| Value | Meaning | +|---|---| +| `passed` | No violations reported and no evaluation errors. | +| `failed` | At least one violation reported and no evaluation errors. | +| `partiallyEvaluated` | At least one rule threw during evaluation, so results are incomplete. | + +`status` describes the report, not the process exit code. It is computed after +`--severity-threshold`, so a report with only warnings is `failed` even when `--fail-on error` +makes the command exit 0. Exit `1` is returned when a violation at or above `--fail-on` is +reported, or when any evaluation error occurred. + +### `durationMs` + +An integer number of milliseconds, measured by the CLI. It covers rule loading, model building and +evaluation, and ends just before the report is written. A library caller that does not measure +time gets `0`. + +### `metrics` + +| Field | Meaning | +|---|---| +| `rulesEvaluated` | Enabled rules that were run (after `--rule` filtering). | +| `rulesPassed` / `rulesFailed` / `rulesErrored` | Per-rule outcome counts. | +| `violationCount` | Length of `violations`. | +| `violationsBySeverity` | Counts for `info`, `warning`, `error` and `critical`. All four keys are always present. | +| `evaluationErrorCount` | Length of `evaluationErrors`. | +| `analysisWarningCount` | Length of `analysisWarnings`. | + +## Changes from the pre-v1 output + +The earlier output was the internal `ValidationResult` serialised directly. Consumers of +`--format json` must update for v1: + +- `rulesEvaluated`, `rulesPassed`, `rulesFailed` and `rulesErrored` moved under `metrics`. +- `schemaVersion`, `toolVersion`, `durationMs` and `metrics` are new. +- `status` keeps its values (`passed`, `failed`, `partiallyEvaluated`). +- Violation, evaluation error and analysis warning fields keep their names. + +## Versioning policy + +- Adding an optional field, or a new value to a documented enum, keeps `schemaVersion` `1.0`. + Consumers must ignore unknown enum values. Note that the schema's `additionalProperties: false` + means a new field also changes the schema, so the schema is updated in the same change. +- Removing or renaming a field, changing a type, or changing the meaning of a value requires a new + `schemaVersion` and a new `validation-report.vN.schema.json`. + +## Not yet in the contract + +Planned by the roadmap but not part of v1: `rulesVersion`, `repository`, `commit`, `branch`, a +`timestamp` separate from `evaluatedAtUtc`, and per-run size metrics such as projects and types +analysed. These need the run context to be passed into the reporter. diff --git a/src/CodeGuard.Cli/Commands/ValidateCommand.cs b/src/CodeGuard.Cli/Commands/ValidateCommand.cs index 85dc8cc..1699800 100644 --- a/src/CodeGuard.Cli/Commands/ValidateCommand.cs +++ b/src/CodeGuard.Cli/Commands/ValidateCommand.cs @@ -8,6 +8,7 @@ using CodeGuard.Core.Results; using CodeGuard.Reporting; using CodeGuard.Reporting.Console; +using CodeGuard.Reporting.Contracts; using CodeGuard.Reporting.Html; using CodeGuard.Reporting.Json; using CodeGuard.Reporting.Sarif; @@ -95,6 +96,7 @@ public static Command Build() command.SetAction(async (parseResult, cancellationToken) => { + var totalStopwatch = Stopwatch.StartNew(); using var loggerFactory = CliLoggerFactory.Create(CliLoggerFactory.ParseVerbosity(parseResult.GetValue(verbosityOption)!)); var logger = loggerFactory.CreateLogger(typeof(ValidateCommand)); @@ -206,7 +208,8 @@ public static Command Build() consoleOutputRedirected: Console.IsOutputRedirected, noColorEnvVar: Environment.GetEnvironmentVariable("NO_COLOR")); - var reporter = CreateReporter(parseResult.GetValue(formatOption)!, useColor); + var reporter = CreateReporter( + parseResult.GetValue(formatOption)!, useColor, totalStopwatch.Elapsed); logger.LogDebug("Writing {Format} report to {Destination}", parseResult.GetValue(formatOption), outputPath ?? "stdout"); @@ -260,9 +263,11 @@ private static ValidationResult ApplySeverityThreshold(ValidationResult result, }; } - private static IViolationReporter CreateReporter(string format, bool useColor) => format switch + // durationMs in the JSON report is measured up to this point, i.e. just before the report is written. + private static IViolationReporter CreateReporter(string format, bool useColor, TimeSpan duration) => format switch { - "json" => new JsonViolationReporter(), + "json" => new JsonViolationReporter( + ValidationReportMapper.ResolveToolVersion(typeof(ValidateCommand).Assembly), duration), "sarif" => new SarifViolationReporter(), "html" => new HtmlViolationReporter(), _ => new ConsoleViolationReporter(useColor) diff --git a/src/CodeGuard.Reporting/CodeGuard.Reporting.csproj b/src/CodeGuard.Reporting/CodeGuard.Reporting.csproj index a89cd8e..52d02d1 100644 --- a/src/CodeGuard.Reporting/CodeGuard.Reporting.csproj +++ b/src/CodeGuard.Reporting/CodeGuard.Reporting.csproj @@ -8,4 +8,8 @@ + + + + diff --git a/src/CodeGuard.Reporting/Contracts/ValidationReportMapper.cs b/src/CodeGuard.Reporting/Contracts/ValidationReportMapper.cs new file mode 100644 index 0000000..06b3ec6 --- /dev/null +++ b/src/CodeGuard.Reporting/Contracts/ValidationReportMapper.cs @@ -0,0 +1,96 @@ +using System.Reflection; +using CodeGuard.Core.Results; +using CodeGuard.RuleModel.Rules; + +namespace CodeGuard.Reporting.Contracts; + +/// Maps the internal to the public contract. +public static class ValidationReportMapper +{ + public const string SchemaVersion = "1.0"; + + public static ValidationReportV1 ToV1(ValidationResult result, TimeSpan duration, string toolVersion) + { + var violations = result.Violations.Select(MapViolation).ToList(); + + return new ValidationReportV1( + SchemaVersion: SchemaVersion, + ToolVersion: toolVersion, + Status: MapStatus(result.Status), + EvaluatedAtUtc: result.EvaluatedAtUtc, + DurationMs: (long)duration.TotalMilliseconds, + Metrics: new ValidationReportMetricsV1( + RulesEvaluated: result.RulesEvaluated, + RulesPassed: result.RulesPassed, + RulesFailed: result.RulesFailed, + RulesErrored: result.RulesErrored, + ViolationCount: violations.Count, + ViolationsBySeverity: CountBySeverity(result.Violations), + EvaluationErrorCount: result.EvaluationErrors.Count, + AnalysisWarningCount: result.AnalysisWarnings.Count), + Violations: violations, + EvaluationErrors: result.EvaluationErrors + .Select(e => new EvaluationErrorV1(e.RuleId, e.ExceptionType, e.Message, e.StackTrace)) + .ToList(), + AnalysisWarnings: result.AnalysisWarnings + .Select(w => new AnalysisWarningV1(w.Code, w.Message, w.Project, w.FilePath)) + .ToList()); + } + + /// + /// The assembly's informational version with any source-link "+commit" suffix removed + /// (e.g. "0.3.1+abc123" becomes "0.3.1"). Falls back to "0.0.0" when no version is embedded. + /// + public static string ResolveToolVersion(Assembly assembly) + { + var informational = assembly.GetCustomAttribute()?.InformationalVersion; + if (string.IsNullOrWhiteSpace(informational)) + { + return "0.0.0"; + } + + var plus = informational.IndexOf('+', StringComparison.Ordinal); + return plus >= 0 ? informational[..plus] : informational; + } + + // Explicit wire values rather than enum ToString(), so renaming an internal enum member cannot change the contract. + private static string MapStatus(ValidationStatus status) => status switch + { + ValidationStatus.Passed => "passed", + ValidationStatus.Failed => "failed", + ValidationStatus.PartiallyEvaluated => "partiallyEvaluated", + _ => throw new ArgumentOutOfRangeException(nameof(status), status, "Unknown validation status.") + }; + + private static string MapSeverity(Severity severity) => severity switch + { + Severity.Info => "info", + Severity.Warning => "warning", + Severity.Error => "error", + Severity.Critical => "critical", + _ => throw new ArgumentOutOfRangeException(nameof(severity), severity, "Unknown severity.") + }; + + private static ViolationV1 MapViolation(Violation v) => new( + RuleId: v.RuleId, + RuleVersion: v.RuleVersion, + Severity: MapSeverity(v.Severity), + Message: v.Message, + File: v.File, + Line: v.Line, + Column: v.Column, + Symbol: v.Symbol, + Project: v.Project, + Remediation: v.Remediation, + DocumentationReferences: v.DocumentationReferences); + + private static SeverityCountsV1 CountBySeverity(IReadOnlyList violations) + { + int Count(Severity severity) => violations.Count(v => v.Severity == severity); + return new SeverityCountsV1( + Info: Count(Severity.Info), + Warning: Count(Severity.Warning), + Error: Count(Severity.Error), + Critical: Count(Severity.Critical)); + } +} diff --git a/src/CodeGuard.Reporting/Contracts/ValidationReportV1.cs b/src/CodeGuard.Reporting/Contracts/ValidationReportV1.cs new file mode 100644 index 0000000..8b3fd17 --- /dev/null +++ b/src/CodeGuard.Reporting/Contracts/ValidationReportV1.cs @@ -0,0 +1,54 @@ +namespace CodeGuard.Reporting.Contracts; + +/// +/// Versioned, public JSON result contract for codeguard validate --format json. Deliberately +/// independent of the internal model, so internal +/// refactors do not change the output. Schema: Schemas/validation-report.v1.schema.json. +/// +public sealed record ValidationReportV1( + string SchemaVersion, + string ToolVersion, + string Status, + DateTimeOffset EvaluatedAtUtc, + long DurationMs, + ValidationReportMetricsV1 Metrics, + IReadOnlyList Violations, + IReadOnlyList EvaluationErrors, + IReadOnlyList AnalysisWarnings); + +public sealed record ValidationReportMetricsV1( + int RulesEvaluated, + int RulesPassed, + int RulesFailed, + int RulesErrored, + int ViolationCount, + SeverityCountsV1 ViolationsBySeverity, + int EvaluationErrorCount, + int AnalysisWarningCount); + +public sealed record SeverityCountsV1(int Info, int Warning, int Error, int Critical); + +public sealed record ViolationV1( + string RuleId, + int RuleVersion, + string Severity, + string Message, + string? File, + int? Line, + int? Column, + string? Symbol, + string? Project, + string? Remediation, + IReadOnlyList DocumentationReferences); + +public sealed record EvaluationErrorV1( + string RuleId, + string ExceptionType, + string Message, + string? StackTrace); + +public sealed record AnalysisWarningV1( + string Code, + string Message, + string? Project, + string? FilePath); diff --git a/src/CodeGuard.Reporting/Json/JsonViolationReporter.cs b/src/CodeGuard.Reporting/Json/JsonViolationReporter.cs index 843a9a7..8443d65 100644 --- a/src/CodeGuard.Reporting/Json/JsonViolationReporter.cs +++ b/src/CodeGuard.Reporting/Json/JsonViolationReporter.cs @@ -1,24 +1,40 @@ using System.Text.Json; using System.Text.Json.Serialization; using CodeGuard.Core.Results; +using CodeGuard.Reporting.Contracts; namespace CodeGuard.Reporting.Json; +/// +/// Writes the versioned contract. Output shape is documented in +/// docs/architecture/RESULT_CONTRACT.md. +/// public sealed class JsonViolationReporter : IViolationReporter { private static readonly JsonSerializerOptions Options = new() { WriteIndented = true, PropertyNamingPolicy = JsonNamingPolicy.CamelCase, - DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull, - Converters = { new JsonStringEnumConverter(JsonNamingPolicy.CamelCase) } + DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull }; + private readonly string _toolVersion; + private readonly TimeSpan _duration; + + /// Reported as toolVersion. Defaults to this assembly's version. + /// Reported as durationMs. Defaults to zero, i.e. not measured. + public JsonViolationReporter(string? toolVersion = null, TimeSpan duration = default) + { + _toolVersion = toolVersion ?? ValidationReportMapper.ResolveToolVersion(typeof(JsonViolationReporter).Assembly); + _duration = duration; + } + public string Format => "json"; public async Task WriteAsync(ValidationResult result, TextWriter writer, CancellationToken ct = default) { - var json = JsonSerializer.Serialize(result, Options); + var report = ValidationReportMapper.ToV1(result, _duration, _toolVersion); + var json = JsonSerializer.Serialize(report, Options); await writer.WriteLineAsync(json.AsMemory(), ct); } } diff --git a/src/CodeGuard.Reporting/Schemas/validation-report.v1.schema.json b/src/CodeGuard.Reporting/Schemas/validation-report.v1.schema.json new file mode 100644 index 0000000..f4441a9 --- /dev/null +++ b/src/CodeGuard.Reporting/Schemas/validation-report.v1.schema.json @@ -0,0 +1,141 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://codeguard.internal/schemas/validation-report.v1.schema.json", + "title": "CodeGuard validation report (v1)", + "description": "Output of `codeguard validate --format json`. schemaVersion identifies this contract; a breaking change to the shape requires a new major schema version.", + "type": "object", + "additionalProperties": false, + "required": [ + "schemaVersion", + "toolVersion", + "status", + "evaluatedAtUtc", + "durationMs", + "metrics", + "violations", + "evaluationErrors", + "analysisWarnings" + ], + "properties": { + "schemaVersion": { + "const": "1.0", + "description": "Version of this result contract." + }, + "toolVersion": { + "type": "string", + "minLength": 1, + "description": "Version of the CodeGuard CLI that produced the report, without any build-metadata suffix." + }, + "status": { + "type": "string", + "enum": ["passed", "failed", "partiallyEvaluated"], + "description": "passed: no violations reported and no evaluation errors. failed: at least one violation reported and no evaluation errors. partiallyEvaluated: at least one rule threw during evaluation, so results are incomplete." + }, + "evaluatedAtUtc": { + "type": "string", + "format": "date-time", + "description": "When rule evaluation completed, ISO 8601 with offset." + }, + "durationMs": { + "type": "integer", + "minimum": 0, + "description": "Wall-clock milliseconds from the start of the validate command until the report is written. Zero when the producer did not measure it." + }, + "metrics": { "$ref": "#/$defs/metrics" }, + "violations": { + "type": "array", + "items": { "$ref": "#/$defs/violation" } + }, + "evaluationErrors": { + "type": "array", + "items": { "$ref": "#/$defs/evaluationError" } + }, + "analysisWarnings": { + "type": "array", + "items": { "$ref": "#/$defs/analysisWarning" } + } + }, + "$defs": { + "severity": { + "type": "string", + "enum": ["info", "warning", "error", "critical"] + }, + "metrics": { + "type": "object", + "additionalProperties": false, + "required": [ + "rulesEvaluated", + "rulesPassed", + "rulesFailed", + "rulesErrored", + "violationCount", + "violationsBySeverity", + "evaluationErrorCount", + "analysisWarningCount" + ], + "properties": { + "rulesEvaluated": { "type": "integer", "minimum": 0 }, + "rulesPassed": { "type": "integer", "minimum": 0 }, + "rulesFailed": { "type": "integer", "minimum": 0 }, + "rulesErrored": { "type": "integer", "minimum": 0 }, + "violationCount": { "type": "integer", "minimum": 0 }, + "violationsBySeverity": { + "type": "object", + "additionalProperties": false, + "required": ["info", "warning", "error", "critical"], + "properties": { + "info": { "type": "integer", "minimum": 0 }, + "warning": { "type": "integer", "minimum": 0 }, + "error": { "type": "integer", "minimum": 0 }, + "critical": { "type": "integer", "minimum": 0 } + } + }, + "evaluationErrorCount": { "type": "integer", "minimum": 0 }, + "analysisWarningCount": { "type": "integer", "minimum": 0 } + } + }, + "violation": { + "type": "object", + "additionalProperties": false, + "required": ["ruleId", "ruleVersion", "severity", "message", "documentationReferences"], + "properties": { + "ruleId": { "type": "string", "minLength": 1 }, + "ruleVersion": { "type": "integer", "minimum": 1 }, + "severity": { "$ref": "#/$defs/severity" }, + "message": { "type": "string" }, + "file": { "type": "string", "description": "Repository-relative or absolute file path, when known." }, + "line": { "type": "integer", "minimum": 0, "description": "1-based line; 0 when the location is unknown." }, + "column": { "type": "integer", "minimum": 0, "description": "1-based column; 0 when the location is unknown." }, + "symbol": { "type": "string" }, + "project": { "type": "string" }, + "remediation": { "type": "string" }, + "documentationReferences": { + "type": "array", + "items": { "type": "string" } + } + } + }, + "evaluationError": { + "type": "object", + "additionalProperties": false, + "required": ["ruleId", "exceptionType", "message"], + "properties": { + "ruleId": { "type": "string", "minLength": 1 }, + "exceptionType": { "type": "string" }, + "message": { "type": "string" }, + "stackTrace": { "type": "string" } + } + }, + "analysisWarning": { + "type": "object", + "additionalProperties": false, + "required": ["code", "message"], + "properties": { + "code": { "type": "string", "minLength": 1 }, + "message": { "type": "string" }, + "project": { "type": "string" }, + "filePath": { "type": "string" } + } + } + } +} diff --git a/tests/CodeGuard.Core.Tests/CodeGuard.Core.Tests.csproj b/tests/CodeGuard.Core.Tests/CodeGuard.Core.Tests.csproj index 4e3efd8..f708039 100644 --- a/tests/CodeGuard.Core.Tests/CodeGuard.Core.Tests.csproj +++ b/tests/CodeGuard.Core.Tests/CodeGuard.Core.Tests.csproj @@ -13,6 +13,7 @@ all runtime; build; native; contentfiles; analyzers; buildtransitive + diff --git a/tests/CodeGuard.Core.Tests/Reporting/JsonViolationReporterTests.cs b/tests/CodeGuard.Core.Tests/Reporting/JsonViolationReporterTests.cs index a4baccf..a6e2826 100644 --- a/tests/CodeGuard.Core.Tests/Reporting/JsonViolationReporterTests.cs +++ b/tests/CodeGuard.Core.Tests/Reporting/JsonViolationReporterTests.cs @@ -42,7 +42,8 @@ public async Task WriteAsync_ProducesExpectedJsonShape() var root = document.RootElement; Assert.Equal("failed", root.GetProperty("status").GetString()); - Assert.Equal(1, root.GetProperty("rulesEvaluated").GetInt32()); + Assert.Equal(1, root.GetProperty("metrics").GetProperty("rulesEvaluated").GetInt32()); + Assert.Equal("1.0", root.GetProperty("schemaVersion").GetString()); var violation = root.GetProperty("violations")[0]; Assert.Equal("DDD-ENTITY-001", violation.GetProperty("ruleId").GetString()); @@ -83,7 +84,7 @@ public async Task WriteAsync_IncludesEvaluationErrors() using var document = JsonDocument.Parse(output); Assert.Equal("partiallyEvaluated", document.RootElement.GetProperty("status").GetString()); - Assert.Equal(1, document.RootElement.GetProperty("rulesErrored").GetInt32()); + Assert.Equal(1, document.RootElement.GetProperty("metrics").GetProperty("rulesErrored").GetInt32()); var error = document.RootElement.GetProperty("evaluationErrors")[0]; Assert.Equal("BROKEN-001", error.GetProperty("ruleId").GetString()); diff --git a/tests/CodeGuard.Core.Tests/Reporting/ValidationReportMapperTests.cs b/tests/CodeGuard.Core.Tests/Reporting/ValidationReportMapperTests.cs new file mode 100644 index 0000000..1e4d118 --- /dev/null +++ b/tests/CodeGuard.Core.Tests/Reporting/ValidationReportMapperTests.cs @@ -0,0 +1,132 @@ +using CodeGuard.Core.Results; +using CodeGuard.Reporting.Contracts; +using CodeGuard.RuleModel.Rules; + +namespace CodeGuard.Core.Tests.Reporting; + +public class ValidationReportMapperTests +{ + [Theory] + [InlineData(ValidationStatus.Passed, "passed")] + [InlineData(ValidationStatus.Failed, "failed")] + [InlineData(ValidationStatus.PartiallyEvaluated, "partiallyEvaluated")] + public void ToV1_MapsStatusToWireValue(ValidationStatus status, string expected) + { + var report = ValidationReportMapper.ToV1(CreateResult(status), TimeSpan.Zero, "1.2.3"); + + Assert.Equal(expected, report.Status); + } + + [Fact] + public void ToV1_SetsSchemaVersionAndToolVersion() + { + var report = ValidationReportMapper.ToV1(CreateResult(ValidationStatus.Passed), TimeSpan.Zero, "1.2.3"); + + Assert.Equal("1.0", report.SchemaVersion); + Assert.Equal("1.2.3", report.ToolVersion); + } + + [Fact] + public void ToV1_ConvertsDurationToWholeMilliseconds() + { + var report = ValidationReportMapper.ToV1( + CreateResult(ValidationStatus.Passed), TimeSpan.FromMilliseconds(1234.9), "1.0.0"); + + Assert.Equal(1234, report.DurationMs); + } + + [Fact] + public void ToV1_CountsMetricsAndViolationsBySeverity() + { + var result = new ValidationResult( + Status: ValidationStatus.Failed, + RulesEvaluated: 5, + RulesPassed: 2, + RulesFailed: 2, + RulesErrored: 1, + Violations: + [ + CreateViolation("R-1", Severity.Info), + CreateViolation("R-2", Severity.Error), + CreateViolation("R-2", Severity.Error), + CreateViolation("R-3", Severity.Critical) + ], + EvaluationErrors: [new RuleEvaluationError("R-4", "System.Exception", "boom", null)], + EvaluatedAtUtc: DateTimeOffset.UtcNow, + AnalysisWarnings: [new AnalysisWarning("MSBUILD-WORKSPACE", "partial load", null, null)]); + + var report = ValidationReportMapper.ToV1(result, TimeSpan.Zero, "1.0.0"); + + Assert.Equal(5, report.Metrics.RulesEvaluated); + Assert.Equal(2, report.Metrics.RulesPassed); + Assert.Equal(2, report.Metrics.RulesFailed); + Assert.Equal(1, report.Metrics.RulesErrored); + Assert.Equal(4, report.Metrics.ViolationCount); + Assert.Equal(new SeverityCountsV1(Info: 1, Warning: 0, Error: 2, Critical: 1), report.Metrics.ViolationsBySeverity); + Assert.Equal(1, report.Metrics.EvaluationErrorCount); + Assert.Equal(1, report.Metrics.AnalysisWarningCount); + } + + [Fact] + public void ToV1_MapsViolationFieldsAndSeverityWireValue() + { + var violation = new Violation( + RuleId: "DDD-ENTITY-001", + RuleVersion: 3, + Severity: Severity.Warning, + Message: "msg", + File: "LegacyThing.cs", + Line: 5, + Column: 14, + Symbol: "Contoso.Domain.Entities.LegacyThing", + Project: "Contoso.Domain", + Remediation: "fix it", + DocumentationReferences: ["docs/x.md"]); + var result = CreateResult(ValidationStatus.Failed, violations: [violation]); + + var mapped = ValidationReportMapper.ToV1(result, TimeSpan.Zero, "1.0.0").Violations.Single(); + + Assert.Equal("DDD-ENTITY-001", mapped.RuleId); + Assert.Equal(3, mapped.RuleVersion); + Assert.Equal("warning", mapped.Severity); + Assert.Equal("LegacyThing.cs", mapped.File); + Assert.Equal(5, mapped.Line); + Assert.Equal(14, mapped.Column); + Assert.Equal("Contoso.Domain.Entities.LegacyThing", mapped.Symbol); + Assert.Equal("Contoso.Domain", mapped.Project); + Assert.Equal("fix it", mapped.Remediation); + Assert.Equal(["docs/x.md"], mapped.DocumentationReferences); + } + + [Fact] + public void ResolveToolVersion_StripsBuildMetadataSuffix() + { + var version = ValidationReportMapper.ResolveToolVersion(typeof(ValidationReportMapperTests).Assembly); + + Assert.False(string.IsNullOrWhiteSpace(version)); + Assert.DoesNotContain('+', version); + } + + private static Violation CreateViolation(string ruleId, Severity severity) => new( + RuleId: ruleId, + RuleVersion: 1, + Severity: severity, + Message: "msg", + File: null, + Line: null, + Column: null, + Symbol: null, + Project: null, + Remediation: null, + DocumentationReferences: []); + + private static ValidationResult CreateResult(ValidationStatus status, IReadOnlyList? violations = null) => new( + Status: status, + RulesEvaluated: 1, + RulesPassed: 1, + RulesFailed: 0, + RulesErrored: 0, + Violations: violations ?? [], + EvaluationErrors: [], + EvaluatedAtUtc: DateTimeOffset.UtcNow); +} diff --git a/tests/CodeGuard.Core.Tests/Reporting/ValidationReportSchemaTests.cs b/tests/CodeGuard.Core.Tests/Reporting/ValidationReportSchemaTests.cs new file mode 100644 index 0000000..72da757 --- /dev/null +++ b/tests/CodeGuard.Core.Tests/Reporting/ValidationReportSchemaTests.cs @@ -0,0 +1,135 @@ +using System.Text.Json; +using CodeGuard.Core.Results; +using CodeGuard.Reporting.Json; +using CodeGuard.RuleModel.Rules; +using Json.Schema; + +namespace CodeGuard.Core.Tests.Reporting; + +public class ValidationReportSchemaTests +{ + private const string EmbeddedResourceName = "CodeGuard.Reporting.Schemas.validation-report.v1.schema.json"; + + // JsonSchema.Net registers schemas globally by $id, so parse the report schema once per process. + private static readonly Lazy ReportSchema = new(() => JsonSchema.FromText(LoadEmbeddedSchemaText())); + + [Fact] + public void EmbeddedSchema_IsValidDraft202012Document() + { + using var document = JsonDocument.Parse(LoadEmbeddedSchemaText()); + + var results = MetaSchemas.Draft202012.Evaluate(document.RootElement, new EvaluationOptions { OutputFormat = OutputFormat.List }); + + Assert.True(results.IsValid, Describe(results)); + } + + [Fact] + public async Task Output_WithViolations_ConformsToSchema() + { + var result = new ValidationResult( + Status: ValidationStatus.Failed, RulesEvaluated: 2, RulesPassed: 1, RulesFailed: 1, RulesErrored: 0, + Violations: + [ + new Violation( + RuleId: "DDD-ENTITY-001", RuleVersion: 1, Severity: Severity.Error, + Message: "msg", File: "LegacyThing.cs", Line: 5, Column: 14, + Symbol: "Contoso.Domain.Entities.LegacyThing", Project: "Contoso.Domain", + Remediation: "fix", DocumentationReferences: ["docs/a.md"]), + new Violation( + RuleId: "NAMING-001", RuleVersion: 2, Severity: Severity.Info, + Message: "msg", File: null, Line: null, Column: null, + Symbol: null, Project: null, Remediation: null, DocumentationReferences: []) + ], + EvaluationErrors: [], + EvaluatedAtUtc: DateTimeOffset.UtcNow); + + var output = await WriteAsync(result, TimeSpan.FromMilliseconds(250)); + + AssertConforms(output); + } + + [Fact] + public async Task Output_WithNoViolations_ConformsToSchema() + { + var result = new ValidationResult( + ValidationStatus.Passed, RulesEvaluated: 1, RulesPassed: 1, RulesFailed: 0, RulesErrored: 0, + Violations: [], EvaluationErrors: [], EvaluatedAtUtc: DateTimeOffset.UtcNow); + + var output = await WriteAsync(result, TimeSpan.Zero); + + AssertConforms(output); + } + + [Fact] + public async Task Output_WithEvaluationErrorsAndAnalysisWarnings_ConformsToSchema() + { + var result = new ValidationResult( + ValidationStatus.PartiallyEvaluated, RulesEvaluated: 1, RulesPassed: 0, RulesFailed: 0, RulesErrored: 1, + Violations: [], + EvaluationErrors: [new RuleEvaluationError("BROKEN-001", "System.InvalidOperationException", "boom", "at Foo.Bar()")], + EvaluatedAtUtc: DateTimeOffset.UtcNow, + AnalysisWarnings: [new AnalysisWarning("MSBUILD-WORKSPACE", "partial load", "Contoso.Domain", "/repo/x.csproj")]); + + var output = await WriteAsync(result, TimeSpan.FromSeconds(3)); + + AssertConforms(output); + + using var document = JsonDocument.Parse(output); + Assert.Equal("partiallyEvaluated", document.RootElement.GetProperty("status").GetString()); + Assert.Equal(3000, document.RootElement.GetProperty("durationMs").GetInt64()); + } + + [Fact] + public void Schema_RejectsUnknownTopLevelProperty() + { + const string document = """ + { + "schemaVersion": "1.0", "toolVersion": "1.0.0", "status": "passed", + "evaluatedAtUtc": "2026-01-01T00:00:00+00:00", "durationMs": 0, + "metrics": { + "rulesEvaluated": 0, "rulesPassed": 0, "rulesFailed": 0, "rulesErrored": 0, + "violationCount": 0, + "violationsBySeverity": { "info": 0, "warning": 0, "error": 0, "critical": 0 }, + "evaluationErrorCount": 0, "analysisWarningCount": 0 + }, + "violations": [], "evaluationErrors": [], "analysisWarnings": [], + "rulesEvaluated": 0 + } + """; + + using var parsed = JsonDocument.Parse(document); + var results = ReportSchema.Value.Evaluate(parsed.RootElement, new EvaluationOptions { OutputFormat = OutputFormat.List }); + + Assert.False(results.IsValid); + } + + private static async Task WriteAsync(ValidationResult result, TimeSpan duration) + { + var writer = new StringWriter(); + await new JsonViolationReporter("1.2.3", duration).WriteAsync(result, writer); + return writer.ToString(); + } + + private static void AssertConforms(string output) + { + using var document = JsonDocument.Parse(output); + + var results = ReportSchema.Value.Evaluate(document.RootElement, new EvaluationOptions { OutputFormat = OutputFormat.List }); + + Assert.True(results.IsValid, $"Output does not conform to validation-report.v1 schema: {Describe(results)}\n{output}"); + } + + private static string Describe(EvaluationResults results) => + string.Join("; ", (results.Details ?? []) + .Where(detail => detail.Errors is { Count: > 0 }) + .SelectMany(detail => detail.Errors!.Select(e => $"{detail.InstanceLocation}: {e.Value}"))); + + private static string LoadEmbeddedSchemaText() + { + var assembly = typeof(JsonViolationReporter).Assembly; + using var stream = assembly.GetManifestResourceStream(EmbeddedResourceName) + ?? throw new InvalidOperationException($"Embedded resource '{EmbeddedResourceName}' not found."); + using var reader = new StreamReader(stream); + return reader.ReadToEnd(); + } +}