diff --git a/docs/architecture/IMPLEMENTATION_STATUS.md b/docs/architecture/IMPLEMENTATION_STATUS.md index ce545c9..9b876b2 100644 --- a/docs/architecture/IMPLEMENTATION_STATUS.md +++ b/docs/architecture/IMPLEMENTATION_STATUS.md @@ -1060,3 +1060,37 @@ All under `rules/`, all illustrative (`Contoso.*` namespace, `illustrative: true `.claude` worktree double-discovery are all fixed, and `codeguard validate` against this repo completes end-to-end. Only the unproven multi-solution `(ProjectName, FullName)` edge case noted in gotcha #6 remains. + +### Post-v1 addition: structured expected/actual/evidence on violations (#69) + +`AssertionOutcome` gained optional `Expected`, `Actual` and `Evidence` (`IReadOnlyList?`), all +defaulting to `null`, plus a `Failure(message, expected, actual, evidence)` overload. `Violation` and +`AnalyzerViolation` gained the same optional fields, and `RuleEvaluator.CreateViolation` copies them +from the outcome. Existing constructors and call sites still compile, and `Message` text is unchanged. + +- Populated by the assertions in `CodeGuard.Evaluation.Assertions` wherever there is a natural value. + Examples: `must_inherit_from` gives the pattern as `expected`, the actual base type as `actual`, and + the type as evidence. The dependency assertions (`must_not_depend_on`, `must_only_depend_on`) give + the offending types or dependency names as evidence. The naming and namespace assertions give the + pattern as `expected` and the name as `actual`. The count and existence assertions give the expected + and actual counts, with the matched candidates as evidence. +- Quantifiers aggregate nested outcomes. `expected` is kept only when every nested failure agrees on it. + `evidence` is the failing matches, each qualified by its nested evidence (`: `). +- Evidence is capped at 50 entries (`CandidateDescriptor.MaxEvidenceEntries`, `CandidateDescriptor.CapEvidence`). + When truncated, the last entry is `...and N more`. The same cap applies to the list-valued `actual` values + (`ListOrNone`, `must_not_depend_on`'s matched names). `must_not_exist` materialises at most 51 candidates, + so its failure reports `more than 50 match(es)` and ends the evidence with `...and more`, since the total + is not counted. `must_any_match` makes one pass over its matches, evaluating each nested assertion at most + once per match. It stops at the first satisfying match, and within a match at the first failing assertion. +- Helpers `CandidateDescriptor.ListOrNone` and `CandidateDescriptor.CommonValue` are internal to the + evaluation assembly. Assertions with no sensible structured value leave the fields `null`. These are + file/directory existence + checks, `must_have_modifier`/`must_not_have_modifier` (`actual` is absent), `must_depend_on` + (`actual` is absent), `must_use_package_version` with a missing package, and every "can only be + evaluated against X" misapplication failure. The file and directory existence checks give `expected` + only. The content checks give the pattern as `expected` and the file as evidence, with no `actual`. +- Surfaced in the console reporter as indented `expected:`/`actual:`/`evidence:` lines, only when + present. The JSON v1 contract gains optional `expected`, `actual` and `evidence` properties on each + violation (`schemaVersion` stays `1.0`, see `RESULT_CONTRACT.md`). In SARIF they go into the result + property bag under the keys `expected`, `actual` and `evidence`. A `Metadata` key with the same name + takes precedence. The HTML reporter shows them under the message. diff --git a/docs/architecture/RESULT_CONTRACT.md b/docs/architecture/RESULT_CONTRACT.md index 613db4d..4323aae 100644 --- a/docs/architecture/RESULT_CONTRACT.md +++ b/docs/architecture/RESULT_CONTRACT.md @@ -60,15 +60,33 @@ object has `additionalProperties: false`, so a consumer can rely on the set of k } ``` -Optional fields (`file`, `line`, `column`, `endLine`, `endColumn`, `metadata`, `symbol`, `project`, -`remediation`, and the optional fields of `evaluationErrors` and `analysisWarnings`) are omitted when -unknown, not written as `null`. +Optional fields (`file`, `line`, `column`, `endLine`, `endColumn`, `metadata`, `expected`, `actual`, +`evidence`, `symbol`, `project`, `remediation`, and the optional fields of `evaluationErrors` and +`analysisWarnings`) are omitted when unknown, not written as `null`. Per violation, `endLine` and `endColumn` give the end of the source span when the producer knows it (currently Roslyn-backed rules). `endColumn` is 1-based and exclusive, i.e. the column just past the last character, matching SARIF `region.endColumn`. `metadata` is an optional string-to-string bag for producer-specific details; no built-in rule populates it yet. +Per violation, `expected`, `actual` and `evidence` are optional structured evidence (issue #69), so an +agent can act on a failure without parsing `message`. They are present only when the failing assertion +has a natural value for them, and `message` is unchanged either way: + +- `expected` (string): what the rule required, such as the required base type pattern or name regex. +- `actual` (string): what was found instead, such as the actual base type or name. `(none)` when a + required member or relationship is absent. +- `evidence` (array of strings): the offending references that caused the failure, such as type + full names, dependency names, file paths or project names. For quantifiers (`must_all_match`, + `must_any_match`, `must_none_match`), each entry is qualified by the match it came from + (`: `). + +`evidence` is capped at 50 entries. When a failure has more, the list holds the first 50 followed by a +final entry `...and N more`, where N is the number omitted. The cap also bounds the evidence-like parts of +`actual` for list-valued assertions. `expected` and `actual` are not otherwise limited. + +The fields are additive; `schemaVersion` stays `1.0`. + ## Top-level fields | Field | Type | Meaning | diff --git a/src/CodeGuard.Core/Evaluation/RuleEvaluator.cs b/src/CodeGuard.Core/Evaluation/RuleEvaluator.cs index 691e9be..3008fbe 100644 --- a/src/CodeGuard.Core/Evaluation/RuleEvaluator.cs +++ b/src/CodeGuard.Core/Evaluation/RuleEvaluator.cs @@ -167,7 +167,10 @@ private static void EvaluateAnalyzerRule(RuleDefinition rule, RepositoryModel mo rule.Remediation, rule.Documentation, EndLine: analyzerViolation.EndLine, - EndColumn: analyzerViolation.EndColumn)); + EndColumn: analyzerViolation.EndColumn, + Expected: analyzerViolation.Expected, + Actual: analyzerViolation.Actual, + Evidence: analyzerViolation.Evidence)); } } @@ -217,7 +220,10 @@ private static Violation CreateViolation(RuleDefinition rule, object candidate, rule.Remediation, rule.Documentation, EndLine: endLine, - EndColumn: endColumn); + EndColumn: endColumn, + Expected: outcome.Expected, + Actual: outcome.Actual, + Evidence: outcome.Evidence); } private static (string? File, int? Line, int? Column, int? EndLine, int? EndColumn, string? Symbol, string? Project) ExtractLocation(object candidate) => diff --git a/src/CodeGuard.Core/Results/ValidationResult.cs b/src/CodeGuard.Core/Results/ValidationResult.cs index ec7e0f8..910d265 100644 --- a/src/CodeGuard.Core/Results/ValidationResult.cs +++ b/src/CodeGuard.Core/Results/ValidationResult.cs @@ -37,7 +37,10 @@ public sealed record Violation( IReadOnlyList DocumentationReferences, int? EndLine = null, int? EndColumn = null, - IReadOnlyDictionary? Metadata = null); + IReadOnlyDictionary? Metadata = null, + string? Expected = null, + string? Actual = null, + IReadOnlyList? Evidence = null); /// A rule whose selector/assertion/analyzer threw instead of producing a pass/fail result. public sealed record RuleEvaluationError( diff --git a/src/CodeGuard.Evaluation/Assertions/CandidateDescriptor.cs b/src/CodeGuard.Evaluation/Assertions/CandidateDescriptor.cs index b7ca528..3dee817 100644 --- a/src/CodeGuard.Evaluation/Assertions/CandidateDescriptor.cs +++ b/src/CodeGuard.Evaluation/Assertions/CandidateDescriptor.cs @@ -25,4 +25,44 @@ internal static class CandidateDescriptor DirectoryModel directory => directory.RelativePath, _ => candidate.ToString() ?? candidate.GetType().Name }; + + /// + /// Upper bound on the number of entries in any Evidence (or evidence-like Actual) list, so a + /// failure over thousands of matches stays a bounded payload for reporters and agents. + /// + public const int MaxEvidenceEntries = 50; + + /// + /// Returns at most items. When there are more, the final entry is + /// "...and N more", where N is the number of items omitted. + /// + public static List CapEvidence(IEnumerable items, int max = MaxEvidenceEntries) + { + var all = items.ToList(); + if (all.Count <= max) + { + return all; + } + + var capped = all.Take(max).ToList(); + capped.Add($"...and {all.Count - max} more"); + return capped; + } + + /// Joins names for an Actual value, or "(none)" when empty. Bounded like evidence. + public static string ListOrNone(IEnumerable items) + { + var list = CapEvidence(items); + return list.Count == 0 ? "(none)" : string.Join(", ", list); + } + + /// + /// Aggregates nested outcome values for a quantifier: returns the value only when every nested + /// outcome agrees on it (including agreeing that it is null), otherwise null. + /// + public static string? CommonValue(IEnumerable values) + { + var distinct = values.Distinct().ToList(); + return distinct.Count == 1 ? distinct[0] : null; + } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustAllMatchAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustAllMatchAssertion.cs index 5895cc3..4f4112d 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustAllMatchAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustAllMatchAssertion.cs @@ -25,16 +25,46 @@ public sealed class MustAllMatchAssertion( public AssertionOutcome Evaluate(object candidate, RepositoryModel model) { var selector = selectorFactory(SelectorTemplateResolver.Resolve(selectorTemplate, candidate)); - var failures = selector.SelectCandidates(model) - .SelectMany(match => assertions + var failedMatches = selector.SelectCandidates(model) + .Select(match => (Match: match, Failures: assertions .Select(assertion => assertion.Evaluate(match, model)) .Where(outcome => !outcome.Passed) - .Select(outcome => $"{CandidateDescriptor.Describe(match)}: {outcome.Message}")) + .ToList())) + .Where(entry => entry.Failures.Count > 0) .ToList(); - return failures.Count == 0 - ? AssertionOutcome.Success() - : AssertionOutcome.Failure( - $"Not every match for a '{selector.Kind}' selector satisfied every nested assertion: {string.Join("; ", failures)}"); + if (failedMatches.Count == 0) + { + return AssertionOutcome.Success(); + } + + var failures = failedMatches + .SelectMany(entry => entry.Failures.Select(outcome => $"{CandidateDescriptor.Describe(entry.Match)}: {outcome.Message}")) + .ToList(); + + // Evidence is per failing match: the match's description, qualified by each nested outcome's + // own evidence when it has any, so the offending references stay traceable to their match. + var evidence = new List(); + foreach (var entry in failedMatches) + { + var description = CandidateDescriptor.Describe(entry.Match); + var nested = entry.Failures.SelectMany(outcome => outcome.Evidence ?? []).ToList(); + if (nested.Count == 0) + { + evidence.Add(description); + } + else + { + evidence.AddRange(nested.Select(item => $"{description}: {item}")); + } + } + + var nestedFailures = failedMatches.SelectMany(entry => entry.Failures).ToList(); + + return AssertionOutcome.Failure( + $"Not every match for a '{selector.Kind}' selector satisfied every nested assertion: {string.Join("; ", failures)}", + CandidateDescriptor.CommonValue(nestedFailures.Select(outcome => outcome.Expected)), + $"{failedMatches.Count} match(es) failed a nested assertion", + CandidateDescriptor.CapEvidence(evidence)); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustAnyMatchAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustAnyMatchAssertion.cs index 36a6169..4db26fe 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustAnyMatchAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustAnyMatchAssertion.cs @@ -22,11 +22,42 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) { var selector = selectorFactory(SelectorTemplateResolver.Resolve(selectorTemplate, candidate)); var matches = selector.SelectCandidates(model).ToList(); - var satisfied = matches.Any(match => assertions.All(assertion => assertion.Evaluate(match, model).Passed)); - return satisfied - ? AssertionOutcome.Success() - : AssertionOutcome.Failure( - $"Expected at least one match for a '{selector.Kind}' selector to satisfy every nested assertion, but none of the {matches.Count} match(es) did."); + // Single pass: each (match, assertion) pair is evaluated at most once. Within a match, evaluation + // stops at the first failing assertion (as assertions.All does), and the search stops at the first + // fully satisfying match, since no failure data is needed then. The failure outcomes collected here + // are the ones reported, so nothing is re-evaluated to build the failure message or evidence. + var nestedFailures = new List(matches.Count); + foreach (var match in matches) + { + var failure = FirstFailure(match, model); + if (failure is null) + { + return AssertionOutcome.Success(); + } + + nestedFailures.Add(failure); + } + + return AssertionOutcome.Failure( + $"Expected at least one match for a '{selector.Kind}' selector to satisfy every nested assertion, but none of the {matches.Count} match(es) did.", + CandidateDescriptor.CommonValue(nestedFailures.Select(outcome => outcome.Expected)), + $"0 of {matches.Count} match(es) satisfied every nested assertion", + matches.Count == 0 ? null : CandidateDescriptor.CapEvidence(matches.Select(CandidateDescriptor.Describe))); + } + + /// Returns the first nested outcome that fails for , or null when all pass. + private AssertionOutcome? FirstFailure(object match, RepositoryModel model) + { + foreach (var assertion in assertions) + { + var outcome = assertion.Evaluate(match, model); + if (!outcome.Passed) + { + return outcome; + } + } + + return null; } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustBeInNamespaceAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustBeInNamespaceAssertion.cs index 6ccc3eb..65127c9 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustBeInNamespaceAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustBeInNamespaceAssertion.cs @@ -16,6 +16,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) return GlobMatcher.IsMatch(type.Namespace, namespacePattern) ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"'{type.FullName}' must be in a namespace matching '{namespacePattern}'."); + : AssertionOutcome.Failure($"'{type.FullName}' must be in a namespace matching '{namespacePattern}'.", namespacePattern, type.Namespace, [type.FullName]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustBeInProjectAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustBeInProjectAssertion.cs index a6c117c..82add6d 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustBeInProjectAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustBeInProjectAssertion.cs @@ -16,6 +16,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) return GlobMatcher.IsMatch(type.ProjectName, projectNamePattern) ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"'{type.FullName}' must be in a project matching '{projectNamePattern}'."); + : AssertionOutcome.Failure($"'{type.FullName}' must be in a project matching '{projectNamePattern}'.", projectNamePattern, type.ProjectName, [type.FullName]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustDependOnAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustDependOnAssertion.cs index 785b5e1..45bee2d 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustDependOnAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustDependOnAssertion.cs @@ -26,6 +26,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) return hasMatch ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"Project '{project.Name}' must depend on a type matching '{requiredTypePattern}'."); + : AssertionOutcome.Failure($"Project '{project.Name}' must depend on a type matching '{requiredTypePattern}'.", requiredTypePattern, null, [project.Name]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustExistAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustExistAssertion.cs index 0935571..4ff7f84 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustExistAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustExistAssertion.cs @@ -14,6 +14,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) var selector = selectorFactory(SelectorTemplateResolver.Resolve(selectorTemplate, candidate)); return selector.SelectCandidates(model).Any() ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"Expected at least one match for a '{selector.Kind}' selector, but found none."); + : AssertionOutcome.Failure($"Expected at least one match for a '{selector.Kind}' selector, but found none.", "at least one match", "0 matches", null); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustHaveAttributeAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustHaveAttributeAssertion.cs index 480934b..f9e0e81 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustHaveAttributeAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustHaveAttributeAssertion.cs @@ -20,6 +20,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) return match is not null ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"Candidate must have attribute matching '{typePattern}'{(argument is null ? "" : $" with argument '{argument}'")}."); + : AssertionOutcome.Failure($"Candidate must have attribute matching '{typePattern}'{(argument is null ? "" : $" with argument '{argument}'")}.", argument is null ? typePattern : $"{typePattern} with argument '{argument}'", CandidateDescriptor.ListOrNone(attributes.Select(a => a.TypeName)), [CandidateDescriptor.Describe(candidate)]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustHaveConstructorAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustHaveConstructorAssertion.cs index aaad880..1503745 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustHaveConstructorAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustHaveConstructorAssertion.cs @@ -17,6 +17,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) return type.Constructors.Any(c => allowedAccessibilities.Contains(c.Accessibility)) ? AssertionOutcome.Success() : AssertionOutcome.Failure( - $"'{type.FullName}' must have a constructor with accessibility {string.Join(" or ", allowedAccessibilities)}."); + $"'{type.FullName}' must have a constructor with accessibility {string.Join(" or ", allowedAccessibilities)}.", string.Join(" or ", allowedAccessibilities), CandidateDescriptor.ListOrNone(type.Constructors.Select(c => c.Accessibility.ToString())), [type.FullName]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustHaveCountAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustHaveCountAssertion.cs index d79653d..81db940 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustHaveCountAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustHaveCountAssertion.cs @@ -25,24 +25,29 @@ public sealed class MustHaveCountAssertion( public AssertionOutcome Evaluate(object candidate, RepositoryModel model) { var selector = selectorFactory(SelectorTemplateResolver.Resolve(selectorTemplate, candidate)); - var count = selector.SelectCandidates(model).Count(); + var matches = selector.SelectCandidates(model).ToList(); + var count = matches.Count; + var evidence = matches.Count == 0 ? null : CandidateDescriptor.CapEvidence(matches.Select(CandidateDescriptor.Describe)); if (exactly is not null && count != exactly) { return AssertionOutcome.Failure( - $"Expected exactly {exactly} match(es) for a '{selector.Kind}' selector, but found {count}."); + $"Expected exactly {exactly} match(es) for a '{selector.Kind}' selector, but found {count}.", + $"exactly {exactly}", count.ToString(), evidence); } if (min is not null && count < min) { return AssertionOutcome.Failure( - $"Expected at least {min} match(es) for a '{selector.Kind}' selector, but found {count}."); + $"Expected at least {min} match(es) for a '{selector.Kind}' selector, but found {count}.", + $"at least {min}", count.ToString(), evidence); } if (max is not null && count > max) { return AssertionOutcome.Failure( - $"Expected at most {max} match(es) for a '{selector.Kind}' selector, but found {count}."); + $"Expected at most {max} match(es) for a '{selector.Kind}' selector, but found {count}.", + $"at most {max}", count.ToString(), evidence); } return AssertionOutcome.Success(); diff --git a/src/CodeGuard.Evaluation/Assertions/MustHaveDirectoryAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustHaveDirectoryAssertion.cs index 1fc5ab6..1af33d7 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustHaveDirectoryAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustHaveDirectoryAssertion.cs @@ -16,6 +16,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) return repository.Directories.Any(d => GlobMatcher.IsMatch(d.RelativePath, path)) ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"Repository must have a directory at '{path}'."); + : AssertionOutcome.Failure($"Repository must have a directory at '{path}'.", path, null, null); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustHaveFieldAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustHaveFieldAssertion.cs index 7e7bacd..f0930d4 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustHaveFieldAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustHaveFieldAssertion.cs @@ -16,6 +16,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) return type.Fields.Any(f => GlobMatcher.IsMatch(f.Name, namePattern)) ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"'{type.FullName}' must have a field matching '{namePattern}'."); + : AssertionOutcome.Failure($"'{type.FullName}' must have a field matching '{namePattern}'.", namePattern, CandidateDescriptor.ListOrNone(type.Fields.Select(f => f.Name)), [type.FullName]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustHaveFileAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustHaveFileAssertion.cs index 784e50a..a747908 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustHaveFileAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustHaveFileAssertion.cs @@ -16,6 +16,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) return repository.Files.Any(f => GlobMatcher.IsMatch(f.RelativePath, pathPattern)) ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"Repository must have a file matching '{pathPattern}'."); + : AssertionOutcome.Failure($"Repository must have a file matching '{pathPattern}'.", pathPattern, null, null); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustHaveJsonFieldAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustHaveJsonFieldAssertion.cs index 30c7bcb..10019ca 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustHaveJsonFieldAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustHaveJsonFieldAssertion.cs @@ -33,13 +33,13 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) if (fieldValue is null) { - return AssertionOutcome.Failure($"File '{file.RelativePath}' must have JSON field '{path}'."); + return AssertionOutcome.Failure($"File '{file.RelativePath}' must have JSON field '{path}'.", path, "(none)", [file.RelativePath]); } if (equals is not null && fieldValue.ToString() != equals) { return AssertionOutcome.Failure( - $"File '{file.RelativePath}' must have JSON field '{path}' equal to '{equals}' (found '{fieldValue}')."); + $"File '{file.RelativePath}' must have JSON field '{path}' equal to '{equals}' (found '{fieldValue}').", equals, fieldValue.ToString(), [file.RelativePath]); } return AssertionOutcome.Success(); diff --git a/src/CodeGuard.Evaluation/Assertions/MustHaveMethodAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustHaveMethodAssertion.cs index 3a0f7ce..dd39353 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustHaveMethodAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustHaveMethodAssertion.cs @@ -16,6 +16,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) return type.Methods.Any(m => GlobMatcher.IsMatch(m.Name, namePattern)) ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"'{type.FullName}' must have a method matching '{namePattern}'."); + : AssertionOutcome.Failure($"'{type.FullName}' must have a method matching '{namePattern}'.", namePattern, CandidateDescriptor.ListOrNone(type.Methods.Select(m => m.Name)), [type.FullName]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustHaveModifierAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustHaveModifierAssertion.cs index 6d8fcac..76c97ee 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustHaveModifierAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustHaveModifierAssertion.cs @@ -14,7 +14,7 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) { null => AssertionOutcome.Failure($"Modifier '{modifier}' is not applicable to this candidate."), true => AssertionOutcome.Success(), - false => AssertionOutcome.Failure($"Candidate must have modifier '{modifier}'.") + false => AssertionOutcome.Failure($"Candidate must have modifier '{modifier}'.", modifier, null, [CandidateDescriptor.Describe(candidate)]) }; } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustHaveMsBuildPropertyAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustHaveMsBuildPropertyAssertion.cs index 2b7baaf..32e0b82 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustHaveMsBuildPropertyAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustHaveMsBuildPropertyAssertion.cs @@ -16,13 +16,13 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) if (!project.Properties.TryGetValue(name, out var actualValue)) { - return AssertionOutcome.Failure($"Project '{project.Name}' must define MSBuild property '{name}'."); + return AssertionOutcome.Failure($"Project '{project.Name}' must define MSBuild property '{name}'.", name, "(none)", [project.Name]); } if (value is not null && !string.Equals(actualValue, value, StringComparison.OrdinalIgnoreCase)) { return AssertionOutcome.Failure( - $"Project '{project.Name}' must have MSBuild property '{name}' set to '{value}' (found '{actualValue}')."); + $"Project '{project.Name}' must have MSBuild property '{name}' set to '{value}' (found '{actualValue}').", value, actualValue, [project.Name]); } return AssertionOutcome.Success(); diff --git a/src/CodeGuard.Evaluation/Assertions/MustHaveParameterCountAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustHaveParameterCountAssertion.cs index 7bef1cf..0be790b 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustHaveParameterCountAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustHaveParameterCountAssertion.cs @@ -23,12 +23,12 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) if (min is not null && count < min) { - return AssertionOutcome.Failure($"Must have at least {min} parameter(s), found {count}."); + return AssertionOutcome.Failure($"Must have at least {min} parameter(s), found {count}.", $"at least {min} parameter(s)", count.ToString(), null); } if (max is not null && count > max) { - return AssertionOutcome.Failure($"Must have at most {max} parameter(s), found {count}."); + return AssertionOutcome.Failure($"Must have at most {max} parameter(s), found {count}.", $"at most {max} parameter(s)", count.ToString(), null); } return AssertionOutcome.Success(); diff --git a/src/CodeGuard.Evaluation/Assertions/MustHavePropertyAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustHavePropertyAssertion.cs index aba97f3..ed2041f 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustHavePropertyAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustHavePropertyAssertion.cs @@ -16,6 +16,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) return type.Properties.Any(p => GlobMatcher.IsMatch(p.Name, namePattern)) ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"'{type.FullName}' must have a property matching '{namePattern}'."); + : AssertionOutcome.Failure($"'{type.FullName}' must have a property matching '{namePattern}'.", namePattern, CandidateDescriptor.ListOrNone(type.Properties.Select(p => p.Name)), [type.FullName]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustImplementAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustImplementAssertion.cs index 1c95d42..31c2394 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustImplementAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustImplementAssertion.cs @@ -16,6 +16,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) return type.Interfaces.Any(i => GlobMatcher.IsMatch(i, interfacePattern)) ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"'{type.FullName}' must implement '{interfacePattern}'."); + : AssertionOutcome.Failure($"'{type.FullName}' must implement '{interfacePattern}'.", interfacePattern, CandidateDescriptor.ListOrNone(type.Interfaces), [type.FullName]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustInheritFromAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustInheritFromAssertion.cs index 11b7396..4351bce 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustInheritFromAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustInheritFromAssertion.cs @@ -16,6 +16,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) return type.BaseType is not null && GlobMatcher.IsMatch(type.BaseType, baseTypePattern) ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"'{type.FullName}' must inherit from '{baseTypePattern}'."); + : AssertionOutcome.Failure($"'{type.FullName}' must inherit from '{baseTypePattern}'.", baseTypePattern, type.BaseType ?? "(none)", [type.FullName]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustMatchArgumentAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustMatchArgumentAssertion.cs index 5f8920e..f320e48 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustMatchArgumentAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustMatchArgumentAssertion.cs @@ -17,11 +17,11 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) var argument = callSite.Arguments.FirstOrDefault(a => a.Index == index); if (argument is null || !argument.IsLiteral || argument.LiteralValue is null) { - return AssertionOutcome.Failure($"Call site '{callSite.InvokedMember}' must have a literal argument at index {index}."); + return AssertionOutcome.Failure($"Call site '{callSite.InvokedMember}' must have a literal argument at index {index}.", $"literal argument at index {index}", argument is null ? "(none)" : "(non-literal)", [callSite.InvokedMember]); } return AssertionRegex.IsMatch(argument.LiteralValue, pattern, Kind) ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"Argument {index} of call site '{callSite.InvokedMember}' must match '{pattern}' (found '{argument.LiteralValue}')."); + : AssertionOutcome.Failure($"Argument {index} of call site '{callSite.InvokedMember}' must match '{pattern}' (found '{argument.LiteralValue}').", pattern, argument.LiteralValue, [callSite.InvokedMember]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustMatchContentAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustMatchContentAssertion.cs index 5c729c5..d1e16ff 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustMatchContentAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustMatchContentAssertion.cs @@ -17,6 +17,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) var content = file.Content ?? File.ReadAllText(file.Path); return AssertionRegex.IsMatch(content, pattern, Kind) ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"File '{file.RelativePath}' must contain content matching '{pattern}'."); + : AssertionOutcome.Failure($"File '{file.RelativePath}' must contain content matching '{pattern}'.", pattern, null, [file.RelativePath]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustMatchFilenameAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustMatchFilenameAssertion.cs index 04ad06a..2407c93 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustMatchFilenameAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustMatchFilenameAssertion.cs @@ -17,6 +17,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) var expectedName = Path.GetFileNameWithoutExtension(type.FilePath); return type.Name == expectedName ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"'{type.FullName}' must be declared in a file named '{type.Name}.cs' (found '{Path.GetFileName(type.FilePath)}')."); + : AssertionOutcome.Failure($"'{type.FullName}' must be declared in a file named '{type.Name}.cs' (found '{Path.GetFileName(type.FilePath)}').", $"{type.Name}.cs", Path.GetFileName(type.FilePath), [type.FullName]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustMatchNameAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustMatchNameAssertion.cs index 54208d0..9cce329 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustMatchNameAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustMatchNameAssertion.cs @@ -29,6 +29,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) return AssertionRegex.IsMatch(name, regex, Kind) ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"'{name}' must match name pattern '{regex}'."); + : AssertionOutcome.Failure($"'{name}' must match name pattern '{regex}'.", regex, name, null); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustMatchNamespacePatternAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustMatchNamespacePatternAssertion.cs index 736247f..8f0cd7c 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustMatchNamespacePatternAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustMatchNamespacePatternAssertion.cs @@ -21,6 +21,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) return AssertionRegex.IsMatch(type.Namespace, regex, Kind) ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"'{type.Namespace}' must match namespace pattern '{regex}'."); + : AssertionOutcome.Failure($"'{type.Namespace}' must match namespace pattern '{regex}'.", regex, type.Namespace, [type.FullName]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustNoneMatchAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustNoneMatchAssertion.cs index 28401ad..b673c0a 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustNoneMatchAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustNoneMatchAssertion.cs @@ -29,6 +29,9 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) return offenders.Count == 0 ? AssertionOutcome.Success() : AssertionOutcome.Failure( - $"Expected no match for a '{selector.Kind}' selector to satisfy every nested assertion, but {offenders.Count} did: {string.Join(", ", offenders)}."); + $"Expected no match for a '{selector.Kind}' selector to satisfy every nested assertion, but {offenders.Count} did: {string.Join(", ", offenders)}.", + "no match satisfies every nested assertion", + $"{offenders.Count} offending match(es)", + CandidateDescriptor.CapEvidence(offenders)); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustNotBeInNamespaceAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustNotBeInNamespaceAssertion.cs index a7c9d79..61e1137 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustNotBeInNamespaceAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustNotBeInNamespaceAssertion.cs @@ -15,7 +15,7 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) } return GlobMatcher.IsMatch(type.Namespace, namespacePattern) - ? AssertionOutcome.Failure($"'{type.FullName}' must not be in a namespace matching '{namespacePattern}'.") + ? AssertionOutcome.Failure($"'{type.FullName}' must not be in a namespace matching '{namespacePattern}'.", $"namespace not matching '{namespacePattern}'", type.Namespace, [type.FullName]) : AssertionOutcome.Success(); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustNotDependOnAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustNotDependOnAssertion.cs index 6869f4f..fe0c860 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustNotDependOnAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustNotDependOnAssertion.cs @@ -19,9 +19,22 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) .Select(type => type.FullName) .ToList(); - return offendingTypes.Count == 0 - ? AssertionOutcome.Success() - : AssertionOutcome.Failure( - $"Project '{project.Name}' must not depend on '{forbiddenTypePattern}' (via: {string.Join(", ", offendingTypes)})."); + if (offendingTypes.Count == 0) + { + return AssertionOutcome.Success(); + } + + var matchedNames = project.Types + .SelectMany(DependencyTraversal.ReferencedTypeNames) + .Where(name => GlobMatcher.IsMatch(name, forbiddenTypePattern)) + .Distinct() + .OrderBy(name => name, StringComparer.Ordinal) + .ToList(); + + return AssertionOutcome.Failure( + $"Project '{project.Name}' must not depend on '{forbiddenTypePattern}' (via: {string.Join(", ", offendingTypes)}).", + $"no dependency on '{forbiddenTypePattern}'", + string.Join(", ", CandidateDescriptor.CapEvidence(matchedNames)), + CandidateDescriptor.CapEvidence(offendingTypes)); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustNotExistAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustNotExistAssertion.cs index 0301c3c..6c93f38 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustNotExistAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustNotExistAssertion.cs @@ -12,8 +12,31 @@ public sealed class MustNotExistAssertion(JsonObject selectorTemplate, Func CandidateDescriptor.MaxEvidenceEntries; + var evidence = CandidateDescriptor.CapEvidence(sample.Take(CandidateDescriptor.MaxEvidenceEntries)); + if (truncated) + { + evidence.Add("...and more"); + } + + return AssertionOutcome.Failure( + $"Expected no matches for a '{selector.Kind}' selector, but found at least one.", + "no matches", + truncated ? $"more than {CandidateDescriptor.MaxEvidenceEntries} match(es)" : $"{sample.Count} match(es)", + evidence); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustNotHaveAttributeAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustNotHaveAttributeAssertion.cs index 368f9cc..c16043e 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustNotHaveAttributeAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustNotHaveAttributeAssertion.cs @@ -20,6 +20,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) return match is null ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"Candidate must not have attribute matching '{typePattern}'{(argument is null ? "" : $" with argument '{argument}'")}."); + : AssertionOutcome.Failure($"Candidate must not have attribute matching '{typePattern}'{(argument is null ? "" : $" with argument '{argument}'")}.", argument is null ? $"no attribute matching '{typePattern}'" : $"no attribute matching '{typePattern}' with argument '{argument}'", match.TypeName, [CandidateDescriptor.Describe(candidate)]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustNotHaveDirectoryAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustNotHaveDirectoryAssertion.cs index f1ad65c..1682d28 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustNotHaveDirectoryAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustNotHaveDirectoryAssertion.cs @@ -17,6 +17,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) var match = repository.Directories.FirstOrDefault(d => GlobMatcher.IsMatch(d.RelativePath, path)); return match is null ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"Repository must not have a directory matching '{path}' (found '{match.RelativePath}')."); + : AssertionOutcome.Failure($"Repository must not have a directory matching '{path}' (found '{match.RelativePath}').", $"no directory matching '{path}'", match.RelativePath, [match.RelativePath]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustNotHaveFieldAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustNotHaveFieldAssertion.cs index 0f22acc..a168b54 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustNotHaveFieldAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustNotHaveFieldAssertion.cs @@ -17,6 +17,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) var match = type.Fields.FirstOrDefault(f => GlobMatcher.IsMatch(f.Name, namePattern)); return match is null ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"'{type.FullName}' must not have a field matching '{namePattern}' (found '{match.Name}')."); + : AssertionOutcome.Failure($"'{type.FullName}' must not have a field matching '{namePattern}' (found '{match.Name}').", $"no field matching '{namePattern}'", match.Name, [CandidateDescriptor.Describe(match)]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustNotHaveFileAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustNotHaveFileAssertion.cs index 88fbacd..a870093 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustNotHaveFileAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustNotHaveFileAssertion.cs @@ -17,6 +17,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) var match = repository.Files.FirstOrDefault(f => GlobMatcher.IsMatch(f.RelativePath, pathPattern)); return match is null ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"Repository must not have a file matching '{pathPattern}' (found '{match.RelativePath}')."); + : AssertionOutcome.Failure($"Repository must not have a file matching '{pathPattern}' (found '{match.RelativePath}').", $"no file matching '{pathPattern}'", match.RelativePath, [match.RelativePath]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustNotHaveJsonFieldAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustNotHaveJsonFieldAssertion.cs index 7d05dc9..aa529e8 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustNotHaveJsonFieldAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustNotHaveJsonFieldAssertion.cs @@ -40,6 +40,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) return AssertionOutcome.Success(); } - return AssertionOutcome.Failure($"File '{file.RelativePath}' must not have JSON field '{path}'{(equals is null ? "" : $" equal to '{equals}'")}."); + return AssertionOutcome.Failure($"File '{file.RelativePath}' must not have JSON field '{path}'{(equals is null ? "" : $" equal to '{equals}'")}.", equals is null ? $"no JSON field '{path}'" : $"no JSON field '{path}' equal to '{equals}'", fieldValue.ToString(), [file.RelativePath]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustNotHaveMethodAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustNotHaveMethodAssertion.cs index 011a0de..3fc6779 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustNotHaveMethodAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustNotHaveMethodAssertion.cs @@ -17,6 +17,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) var match = type.Methods.FirstOrDefault(m => GlobMatcher.IsMatch(m.Name, namePattern)); return match is null ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"'{type.FullName}' must not have a method matching '{namePattern}' (found '{match.Name}')."); + : AssertionOutcome.Failure($"'{type.FullName}' must not have a method matching '{namePattern}' (found '{match.Name}').", $"no method matching '{namePattern}'", match.Name, [CandidateDescriptor.Describe(match)]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustNotHaveModifierAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustNotHaveModifierAssertion.cs index 17c5549..485ba29 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustNotHaveModifierAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustNotHaveModifierAssertion.cs @@ -13,7 +13,7 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) return matches switch { null => AssertionOutcome.Failure($"Modifier '{modifier}' is not applicable to this candidate."), - true => AssertionOutcome.Failure($"Candidate must not have modifier '{modifier}'."), + true => AssertionOutcome.Failure($"Candidate must not have modifier '{modifier}'.", $"not {modifier}", modifier, [CandidateDescriptor.Describe(candidate)]), false => AssertionOutcome.Success() }; } diff --git a/src/CodeGuard.Evaluation/Assertions/MustNotHavePropertyAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustNotHavePropertyAssertion.cs index b3f22cc..b4d7911 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustNotHavePropertyAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustNotHavePropertyAssertion.cs @@ -17,6 +17,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) var match = type.Properties.FirstOrDefault(p => GlobMatcher.IsMatch(p.Name, namePattern)); return match is null ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"'{type.FullName}' must not have a property matching '{namePattern}' (found '{match.Name}')."); + : AssertionOutcome.Failure($"'{type.FullName}' must not have a property matching '{namePattern}' (found '{match.Name}').", $"no property matching '{namePattern}'", match.Name, [CandidateDescriptor.Describe(match)]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustNotImplementAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustNotImplementAssertion.cs index 774c7a4..9c4a109 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustNotImplementAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustNotImplementAssertion.cs @@ -17,6 +17,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) var match = type.Interfaces.FirstOrDefault(i => GlobMatcher.IsMatch(i, interfacePattern)); return match is null ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"'{type.FullName}' must not implement '{match}'."); + : AssertionOutcome.Failure($"'{type.FullName}' must not implement '{match}'.", $"no interface matching '{interfacePattern}'", match, [type.FullName]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustNotInheritFromAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustNotInheritFromAssertion.cs index f31da0a..e17827e 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustNotInheritFromAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustNotInheritFromAssertion.cs @@ -15,7 +15,7 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) } return type.BaseType is not null && GlobMatcher.IsMatch(type.BaseType, baseTypePattern) - ? AssertionOutcome.Failure($"'{type.FullName}' must not inherit from '{baseTypePattern}'.") + ? AssertionOutcome.Failure($"'{type.FullName}' must not inherit from '{baseTypePattern}'.", $"base type not matching '{baseTypePattern}'", type.BaseType, [type.FullName]) : AssertionOutcome.Success(); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustNotMatchContentAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustNotMatchContentAssertion.cs index 4780979..35e8f14 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustNotMatchContentAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustNotMatchContentAssertion.cs @@ -16,7 +16,7 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) var content = file.Content ?? File.ReadAllText(file.Path); return AssertionRegex.IsMatch(content, pattern, Kind) - ? AssertionOutcome.Failure($"File '{file.RelativePath}' must not contain content matching '{pattern}'.") + ? AssertionOutcome.Failure($"File '{file.RelativePath}' must not contain content matching '{pattern}'.", $"content not matching '{pattern}'", null, [file.RelativePath]) : AssertionOutcome.Success(); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustNotReferencePackageAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustNotReferencePackageAssertion.cs index 6edd42d..047135e 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustNotReferencePackageAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustNotReferencePackageAssertion.cs @@ -17,6 +17,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) var match = project.PackageReferences.FirstOrDefault(p => GlobMatcher.IsMatch(p.Id, packageIdPattern)); return match is null ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"Project '{project.Name}' must not reference package '{match.Id}'."); + : AssertionOutcome.Failure($"Project '{project.Name}' must not reference package '{match.Id}'.", $"no reference to package matching '{packageIdPattern}'", match.Id, [match.Id]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustNotReferenceProjectAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustNotReferenceProjectAssertion.cs index a9edef3..becd6d5 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustNotReferenceProjectAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustNotReferenceProjectAssertion.cs @@ -17,6 +17,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) var match = project.ProjectReferences.FirstOrDefault(reference => GlobMatcher.IsMatch(reference, projectNamePattern)); return match is null ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"Project '{project.Name}' must not reference project '{match}'."); + : AssertionOutcome.Failure($"Project '{project.Name}' must not reference project '{match}'.", $"no reference to project matching '{projectNamePattern}'", match, [match]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustOnlyDependOnAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustOnlyDependOnAssertion.cs index a075104..4bb0230 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustOnlyDependOnAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustOnlyDependOnAssertion.cs @@ -36,6 +36,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) ? AssertionOutcome.Success() : AssertionOutcome.Failure( $"Project '{project.Name}' must only depend on types matching [{string.Join(", ", allowedTypePatterns)}] " + - $"(disallowed: {string.Join(", ", offendingTypeNames)})."); + $"(disallowed: {string.Join(", ", offendingTypeNames)}).", string.Join(", ", allowedTypePatterns), null, CandidateDescriptor.CapEvidence(offendingTypeNames)); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustReferencePackageAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustReferencePackageAssertion.cs index 550f54c..8ca1152 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustReferencePackageAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustReferencePackageAssertion.cs @@ -16,6 +16,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) return project.PackageReferences.Any(p => GlobMatcher.IsMatch(p.Id, packageIdPattern)) ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"Project '{project.Name}' must reference package matching '{packageIdPattern}'."); + : AssertionOutcome.Failure($"Project '{project.Name}' must reference package matching '{packageIdPattern}'.", packageIdPattern, CandidateDescriptor.ListOrNone(project.PackageReferences.Select(p => p.Id)), [project.Name]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustReferenceProjectAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustReferenceProjectAssertion.cs index 7c8474c..18c591b 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustReferenceProjectAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustReferenceProjectAssertion.cs @@ -16,6 +16,6 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) return project.ProjectReferences.Any(reference => GlobMatcher.IsMatch(reference, projectNamePattern)) ? AssertionOutcome.Success() - : AssertionOutcome.Failure($"Project '{project.Name}' must reference a project matching '{projectNamePattern}'."); + : AssertionOutcome.Failure($"Project '{project.Name}' must reference a project matching '{projectNamePattern}'.", projectNamePattern, CandidateDescriptor.ListOrNone(project.ProjectReferences), [project.Name]); } } diff --git a/src/CodeGuard.Evaluation/Assertions/MustUsePackageVersionAssertion.cs b/src/CodeGuard.Evaluation/Assertions/MustUsePackageVersionAssertion.cs index 9975a9c..24a457b 100644 --- a/src/CodeGuard.Evaluation/Assertions/MustUsePackageVersionAssertion.cs +++ b/src/CodeGuard.Evaluation/Assertions/MustUsePackageVersionAssertion.cs @@ -30,7 +30,7 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) var matches = project.PackageReferences.Where(p => GlobMatcher.IsMatch(p.Id, packageIdPattern)).ToList(); if (matches.Count == 0) { - return AssertionOutcome.Failure($"Project '{project.Name}' does not reference a package matching '{packageIdPattern}'."); + return AssertionOutcome.Failure($"Project '{project.Name}' does not reference a package matching '{packageIdPattern}'.", packageIdPattern, "(none)", [project.Name]); } var match = ConstraintPattern.Match(constraint); @@ -43,7 +43,7 @@ public AssertionOutcome Evaluate(object candidate, RepositoryModel model) ? AssertionOutcome.Success() : AssertionOutcome.Failure( $"Project '{project.Name}' must reference '{packageIdPattern}' with version {constraint} " + - $"(found: {string.Join(", ", offending.Select(p => $"{p.Id} {p.Version}"))})."); + $"(found: {string.Join(", ", offending.Select(p => $"{p.Id} {p.Version}"))}).", constraint, string.Join(", ", offending.Select(p => $"{p.Id} {p.Version}")), offending.Select(p => p.Id).ToList()); } private static bool Satisfies(Version actual, string op, Version required) => op switch diff --git a/src/CodeGuard.Reporting/Console/ConsoleViolationReporter.cs b/src/CodeGuard.Reporting/Console/ConsoleViolationReporter.cs index 6773921..59ae854 100644 --- a/src/CodeGuard.Reporting/Console/ConsoleViolationReporter.cs +++ b/src/CodeGuard.Reporting/Console/ConsoleViolationReporter.cs @@ -24,6 +24,21 @@ public async Task WriteAsync(ValidationResult result, TextWriter writer, Cancell await writer.WriteLineAsync($"{severityTag} {violation.RuleId}: {violation.Message} ({location})"); + if (violation.Expected is not null) + { + await writer.WriteLineAsync($" expected: {violation.Expected}"); + } + + if (violation.Actual is not null) + { + await writer.WriteLineAsync($" actual: {violation.Actual}"); + } + + if (violation.Evidence is { Count: > 0 } evidence) + { + await writer.WriteLineAsync($" evidence: {string.Join(", ", evidence)}"); + } + if (violation.Remediation is not null) { await writer.WriteLineAsync($" remediation: {violation.Remediation.Trim()}"); diff --git a/src/CodeGuard.Reporting/Contracts/ValidationReportMapper.cs b/src/CodeGuard.Reporting/Contracts/ValidationReportMapper.cs index ed74335..472f8f5 100644 --- a/src/CodeGuard.Reporting/Contracts/ValidationReportMapper.cs +++ b/src/CodeGuard.Reporting/Contracts/ValidationReportMapper.cs @@ -94,7 +94,10 @@ public static string ResolveToolVersion(Assembly assembly) DocumentationReferences: v.DocumentationReferences, EndLine: v.EndLine, EndColumn: v.EndColumn, - Metadata: v.Metadata); + Metadata: v.Metadata, + Expected: v.Expected, + Actual: v.Actual, + Evidence: v.Evidence); private static SeverityCountsV1 CountBySeverity(IReadOnlyList violations) { diff --git a/src/CodeGuard.Reporting/Contracts/ValidationReportV1.cs b/src/CodeGuard.Reporting/Contracts/ValidationReportV1.cs index e7442aa..ff1b287 100644 --- a/src/CodeGuard.Reporting/Contracts/ValidationReportV1.cs +++ b/src/CodeGuard.Reporting/Contracts/ValidationReportV1.cs @@ -46,7 +46,10 @@ public sealed record ViolationV1( IReadOnlyList DocumentationReferences, int? EndLine = null, int? EndColumn = null, - IReadOnlyDictionary? Metadata = null); + IReadOnlyDictionary? Metadata = null, + string? Expected = null, + string? Actual = null, + IReadOnlyList? Evidence = null); public sealed record EvaluationErrorV1( string RuleId, diff --git a/src/CodeGuard.Reporting/Html/HtmlViolationReporter.cs b/src/CodeGuard.Reporting/Html/HtmlViolationReporter.cs index eaddcb9..62467fc 100644 --- a/src/CodeGuard.Reporting/Html/HtmlViolationReporter.cs +++ b/src/CodeGuard.Reporting/Html/HtmlViolationReporter.cs @@ -145,6 +145,21 @@ private static void AppendTable(StringBuilder sb, IReadOnlyList viola sb.Append($"
remediation: {Esc(violation.Remediation.Trim())}
"); } + if (violation.Expected is not null) + { + sb.Append($"
expected: {Esc(violation.Expected)}
"); + } + + if (violation.Actual is not null) + { + sb.Append($"
actual: {Esc(violation.Actual)}
"); + } + + if (violation.Evidence is { Count: > 0 } evidence) + { + sb.Append($"
evidence: {Esc(string.Join(", ", evidence))}
"); + } + sb.Append("\n"); sb.Append($"{Esc(location)}\n"); sb.Append($"{Esc(project)}\n"); diff --git a/src/CodeGuard.Reporting/Sarif/SarifViolationReporter.cs b/src/CodeGuard.Reporting/Sarif/SarifViolationReporter.cs index c26fb43..9948260 100644 --- a/src/CodeGuard.Reporting/Sarif/SarifViolationReporter.cs +++ b/src/CodeGuard.Reporting/Sarif/SarifViolationReporter.cs @@ -144,6 +144,24 @@ private static Result ToSarifResult(Violation violation) } } + // Structured evidence uses the reserved keys "expected"/"actual"/"evidence". A producer-supplied + // Metadata key with the same name takes precedence, so existing metadata is never overwritten. + var metadataKeys = violation.Metadata?.Keys.ToHashSet(StringComparer.Ordinal) ?? []; + if (violation.Expected is not null && !metadataKeys.Contains("expected")) + { + result.SetProperty("expected", violation.Expected); + } + + if (violation.Actual is not null && !metadataKeys.Contains("actual")) + { + result.SetProperty("actual", violation.Actual); + } + + if (violation.Evidence is { Count: > 0 } evidence && !metadataKeys.Contains("evidence")) + { + result.SetProperty("evidence", evidence.ToArray()); + } + return result; } diff --git a/src/CodeGuard.Reporting/Schemas/validation-report.v1.schema.json b/src/CodeGuard.Reporting/Schemas/validation-report.v1.schema.json index 0454443..053601f 100644 --- a/src/CodeGuard.Reporting/Schemas/validation-report.v1.schema.json +++ b/src/CodeGuard.Reporting/Schemas/validation-report.v1.schema.json @@ -133,6 +133,13 @@ "additionalProperties": { "type": "string" }, "description": "Free-form string key/value details attached by the producer. Omitted when there are none." }, + "expected": { "type": "string", "description": "What the rule required, when the failing assertion has a structured expected value. Omitted otherwise." }, + "actual": { "type": "string", "description": "What was found instead, when the failing assertion has a structured actual value. Omitted otherwise." }, + "evidence": { + "type": "array", + "items": { "type": "string" }, + "description": "Offending references that caused the failure (type, namespace, project, dependency or file names). Omitted when there are none." + }, "symbol": { "type": "string" }, "project": { "type": "string" }, "remediation": { "type": "string" }, diff --git a/src/CodeGuard.RuleModel/Analyzers/AnalyzerViolation.cs b/src/CodeGuard.RuleModel/Analyzers/AnalyzerViolation.cs index 0726561..b544921 100644 --- a/src/CodeGuard.RuleModel/Analyzers/AnalyzerViolation.cs +++ b/src/CodeGuard.RuleModel/Analyzers/AnalyzerViolation.cs @@ -8,4 +8,7 @@ public sealed record AnalyzerViolation( string? Symbol = null, string? ProjectName = null, int? EndLine = null, - int? EndColumn = null); + int? EndColumn = null, + string? Expected = null, + string? Actual = null, + IReadOnlyList? Evidence = null); diff --git a/src/CodeGuard.RuleModel/Assertions/IAssertion.cs b/src/CodeGuard.RuleModel/Assertions/IAssertion.cs index a0fa0d1..d198187 100644 --- a/src/CodeGuard.RuleModel/Assertions/IAssertion.cs +++ b/src/CodeGuard.RuleModel/Assertions/IAssertion.cs @@ -9,9 +9,24 @@ public interface IAssertion AssertionOutcome Evaluate(object candidate, RepositoryModel model); } -public sealed record AssertionOutcome(bool Passed, string? Message) +/// +/// The result of evaluating one assertion against one candidate. is the +/// human-readable explanation; , and +/// are optional structured data for agents and tooling (what the rule wanted, what was found, and the +/// offending references such as type, namespace or project names). Each is null when the assertion has +/// no natural structured value for the outcome. +/// +public sealed record AssertionOutcome( + bool Passed, + string? Message, + string? Expected = null, + string? Actual = null, + IReadOnlyList? Evidence = null) { public static AssertionOutcome Success() => new(true, null); public static AssertionOutcome Failure(string message) => new(false, message); + + public static AssertionOutcome Failure(string message, string? expected, string? actual, IReadOnlyList? evidence) => + new(false, message, expected, actual, evidence); } diff --git a/tests/CodeGuard.Core.Tests/Evaluation/RuleEvaluatorTests.cs b/tests/CodeGuard.Core.Tests/Evaluation/RuleEvaluatorTests.cs index dde337c..88d12fc 100644 --- a/tests/CodeGuard.Core.Tests/Evaluation/RuleEvaluatorTests.cs +++ b/tests/CodeGuard.Core.Tests/Evaluation/RuleEvaluatorTests.cs @@ -124,6 +124,56 @@ private static RepositoryModel BuildModel(params TypeModel[] types) return new RepositoryModel(".", [solution], [], [], [], [], [], [], [], []); } + [Fact] + public void Evaluate_CopiesExpectedActualAndEvidenceFromAssertionOutcome_ToViolation() + { + var model = BuildModel(CreateEntityType("LegacyThing", "Contoso.Other.Base")); + + var result = new RuleEvaluator().Evaluate([CreateEntityInheritsRule()], model); + + var violation = Assert.Single(result.Violations); + Assert.Equal("Contoso.Domain.Entity", violation.Expected); + Assert.Equal("Contoso.Other.Base", violation.Actual); + Assert.Equal(["Contoso.Domain.Entities.LegacyThing"], violation.Evidence); + } + + [Fact] + public void Evaluate_LeavesEvidenceNull_WhenAssertionProducesNone() + { + var model = BuildModel(CreateEntityType("Order", "Contoso.Domain.Entity")); + var rule = new RuleDefinition + { + Id = "TEST-001", + Name = "Always fails without evidence", + Severity = Severity.Error, + Target = new ClassInNamespaceSelector("Contoso.Domain.Entities"), + Assertions = [new AlwaysFailsAssertion()] + }; + + var result = new RuleEvaluator().Evaluate([rule], model); + + var violation = Assert.Single(result.Violations); + Assert.Null(violation.Expected); + Assert.Null(violation.Actual); + Assert.Null(violation.Evidence); + } + + [Fact] + public void Evaluate_AnalyzerRule_CopiesExpectedActualAndEvidence() + { + var model = new RepositoryModel(".", [], [], [], [], [], [], [], [], []); + var analyzer = new StubAnalyzer("stub-analyzer", new AnalyzerViolation( + "bad shape", "Foo.cs", 3, 1, Expected: "shape A", Actual: "shape B", Evidence: ["Foo.Bar"])); + var rule = new RuleDefinition { Id = "ANALYZER-001", Name = "Analyzer rule", Analyzer = analyzer }; + + var result = new RuleEvaluator().Evaluate([rule], model); + + var violation = Assert.Single(result.Violations); + Assert.Equal("shape A", violation.Expected); + Assert.Equal("shape B", violation.Actual); + Assert.Equal(["Foo.Bar"], violation.Evidence); + } + private sealed class AlwaysFailsAssertion : IAssertion { public string Kind => "always_fails"; diff --git a/tests/CodeGuard.Core.Tests/Reporting/ConsoleViolationReporterTests.cs b/tests/CodeGuard.Core.Tests/Reporting/ConsoleViolationReporterTests.cs index 1f92688..6307bb8 100644 --- a/tests/CodeGuard.Core.Tests/Reporting/ConsoleViolationReporterTests.cs +++ b/tests/CodeGuard.Core.Tests/Reporting/ConsoleViolationReporterTests.cs @@ -44,6 +44,59 @@ public async Task WriteAsync_PrintsViolationDetailsAndSummary() Assert.Contains("Status: Failed", output); } + [Fact] + public async Task WriteAsync_PrintsExpectedActualAndEvidence_WhenPresent() + { + var result = new ValidationResult( + Status: ValidationStatus.Failed, + RulesEvaluated: 1, + RulesPassed: 0, + RulesFailed: 1, + RulesErrored: 0, + Violations: + [ + new Violation( + RuleId: "DDD-ENTITY-001", + RuleVersion: 1, + Severity: Severity.Error, + Message: "'Contoso.Domain.Entities.LegacyThing' must inherit from 'Contoso.Domain.Entity'.", + File: "LegacyThing.cs", + Line: 5, + Column: 14, + Symbol: "Contoso.Domain.Entities.LegacyThing", + Project: "Contoso.Domain", + Remediation: null, + DocumentationReferences: [], + Expected: "Contoso.Domain.Entity", + Actual: "(none)", + Evidence: ["Contoso.Domain.Entities.LegacyThing"]) + ], + EvaluationErrors: [], + EvaluatedAtUtc: DateTimeOffset.UtcNow); + + var writer = new StringWriter(); + await new ConsoleViolationReporter().WriteAsync(result, writer); + var output = writer.ToString(); + + Assert.Contains(" expected: Contoso.Domain.Entity", output); + Assert.Contains(" actual: (none)", output); + Assert.Contains(" evidence: Contoso.Domain.Entities.LegacyThing", output); + } + + [Fact] + public async Task WriteAsync_OmitsExpectedActualAndEvidence_WhenAbsent() + { + var result = SingleErrorResult(); + + var writer = new StringWriter(); + await new ConsoleViolationReporter().WriteAsync(result, writer); + var output = writer.ToString(); + + Assert.DoesNotContain("expected:", output); + Assert.DoesNotContain("actual:", output); + Assert.DoesNotContain("evidence:", output); + } + [Fact] public async Task WriteAsync_PrintsSummaryOnly_WhenNoViolations() { diff --git a/tests/CodeGuard.Core.Tests/Reporting/SarifViolationReporterTests.cs b/tests/CodeGuard.Core.Tests/Reporting/SarifViolationReporterTests.cs index 30439a5..a45845a 100644 --- a/tests/CodeGuard.Core.Tests/Reporting/SarifViolationReporterTests.cs +++ b/tests/CodeGuard.Core.Tests/Reporting/SarifViolationReporterTests.cs @@ -235,6 +235,42 @@ public async Task WriteAsync_WithoutEndSpan_EmitsStartOnlyRegion() Assert.False(sarifResult.TryGetProperty("properties", out _)); } + [Fact] + public async Task WriteAsync_PutsExpectedActualAndEvidenceInPropertyBag() + { + var result = CreateResult(new Violation( + RuleId: "DDD-ENTITY-001", RuleVersion: 1, Severity: Severity.Error, Message: "msg", + File: "LegacyThing.cs", Line: 5, Column: 14, Symbol: null, Project: null, Remediation: null, + DocumentationReferences: [], + Expected: "Contoso.Domain.Entity<*>", Actual: "Contoso.Other.Base", + Evidence: ["Contoso.Domain.Entities.LegacyThing", "Contoso.Domain.Entities.Other"])); + + var sarifResult = await WriteSingleResultAsync(result); + + var properties = sarifResult.GetProperty("properties"); + Assert.Equal("Contoso.Domain.Entity<*>", properties.GetProperty("expected").GetString()); + Assert.Equal("Contoso.Other.Base", properties.GetProperty("actual").GetString()); + var evidence = properties.GetProperty("evidence"); + Assert.Equal(JsonValueKind.Array, evidence.ValueKind); + Assert.Equal(2, evidence.GetArrayLength()); + Assert.Equal("Contoso.Domain.Entities.LegacyThing", evidence[0].GetString()); + } + + [Fact] + public async Task WriteAsync_MetadataKeyWins_OverStructuredEvidenceKey() + { + var result = CreateResult(new Violation( + RuleId: "R-1", RuleVersion: 1, Severity: Severity.Error, Message: "msg", + File: null, Line: null, Column: null, Symbol: null, Project: null, Remediation: null, + DocumentationReferences: [], + Expected: "structured", Actual: null, Evidence: null, + Metadata: new Dictionary { ["expected"] = "from metadata" })); + + var sarifResult = await WriteSingleResultAsync(result); + + Assert.Equal("from metadata", sarifResult.GetProperty("properties").GetProperty("expected").GetString()); + } + private static ValidationResult CreateResult(Violation violation) => new( Status: ValidationStatus.Failed, RulesEvaluated: 1, RulesPassed: 0, RulesFailed: 1, RulesErrored: 0, Violations: [violation], EvaluationErrors: [], EvaluatedAtUtc: DateTimeOffset.UtcNow); diff --git a/tests/CodeGuard.Core.Tests/Reporting/ValidationReportMapperTests.cs b/tests/CodeGuard.Core.Tests/Reporting/ValidationReportMapperTests.cs index c05b2ad..a7de110 100644 --- a/tests/CodeGuard.Core.Tests/Reporting/ValidationReportMapperTests.cs +++ b/tests/CodeGuard.Core.Tests/Reporting/ValidationReportMapperTests.cs @@ -148,4 +148,42 @@ public void ToV1_MapsEndPositionAndMetadata() Assert.Equal(4, mapped.EndColumn); Assert.Equal("v", mapped.Metadata!["k"]); } + + [Fact] + public void ToV1_MapsExpectedActualAndEvidence() + { + var violation = new Violation( + RuleId: "R-1", RuleVersion: 1, Severity: Severity.Error, Message: "msg", + File: "F.cs", Line: 1, Column: 2, Symbol: null, Project: null, Remediation: null, + DocumentationReferences: [], + Expected: "Contoso.Domain.Entity<*>", Actual: "Contoso.Other.Base", + Evidence: ["Contoso.Domain.Entities.LegacyThing"]); + var result = new ValidationResult( + Status: ValidationStatus.Failed, RulesEvaluated: 1, RulesPassed: 0, RulesFailed: 1, RulesErrored: 0, + Violations: [violation], EvaluationErrors: [], EvaluatedAtUtc: DateTimeOffset.UtcNow); + + var mapped = ValidationReportMapper.ToV1(result, TimeSpan.Zero, "1.0.0").Violations.Single(); + + Assert.Equal("Contoso.Domain.Entity<*>", mapped.Expected); + Assert.Equal("Contoso.Other.Base", mapped.Actual); + Assert.Equal(["Contoso.Domain.Entities.LegacyThing"], mapped.Evidence); + } + + [Fact] + public void ToV1_LeavesEvidenceFieldsNull_WhenNotProduced() + { + var violation = new Violation( + RuleId: "R-1", RuleVersion: 1, Severity: Severity.Error, Message: "msg", + File: null, Line: null, Column: null, Symbol: null, Project: null, Remediation: null, + DocumentationReferences: []); + var result = new ValidationResult( + Status: ValidationStatus.Failed, RulesEvaluated: 1, RulesPassed: 0, RulesFailed: 1, RulesErrored: 0, + Violations: [violation], EvaluationErrors: [], EvaluatedAtUtc: DateTimeOffset.UtcNow); + + var mapped = ValidationReportMapper.ToV1(result, TimeSpan.Zero, "1.0.0").Violations.Single(); + + Assert.Null(mapped.Expected); + Assert.Null(mapped.Actual); + Assert.Null(mapped.Evidence); + } } diff --git a/tests/CodeGuard.Core.Tests/Reporting/ValidationReportSchemaTests.cs b/tests/CodeGuard.Core.Tests/Reporting/ValidationReportSchemaTests.cs index efbb2c1..830131b 100644 --- a/tests/CodeGuard.Core.Tests/Reporting/ValidationReportSchemaTests.cs +++ b/tests/CodeGuard.Core.Tests/Reporting/ValidationReportSchemaTests.cs @@ -210,6 +210,31 @@ private static string LoadEmbeddedSchemaText() return reader.ReadToEnd(); } + [Fact] + public async Task Output_WithExpectedActualAndEvidence_ConformsToSchema() + { + var result = new ValidationResult( + Status: ValidationStatus.Failed, RulesEvaluated: 1, RulesPassed: 0, RulesFailed: 1, RulesErrored: 0, + Violations: + [ + new Violation( + RuleId: "DDD-ENTITY-001", RuleVersion: 1, Severity: Severity.Error, + Message: "msg", File: "LegacyThing.cs", Line: 5, Column: 14, + Symbol: null, Project: null, Remediation: null, DocumentationReferences: [], + Expected: "Contoso.Domain.Entity", Actual: "Contoso.Other.Base", + Evidence: ["Contoso.Domain.Entities.LegacyThing"]) + ], + EvaluationErrors: [], + EvaluatedAtUtc: DateTimeOffset.UtcNow); + + var output = await WriteAsync(result, TimeSpan.Zero); + + AssertConforms(output); + Assert.Contains("\"expected\": \"Contoso.Domain.Entity\"", output); + Assert.Contains("\"actual\": \"Contoso.Other.Base\"", output); + Assert.Contains("\"evidence\"", output); + } + [Fact] public async Task Output_WithEndPositionAndMetadata_ConformsToSchema() { diff --git a/tests/CodeGuard.Evaluation.Tests/Assertions/AssertionEvidenceTests.cs b/tests/CodeGuard.Evaluation.Tests/Assertions/AssertionEvidenceTests.cs new file mode 100644 index 0000000..9318906 --- /dev/null +++ b/tests/CodeGuard.Evaluation.Tests/Assertions/AssertionEvidenceTests.cs @@ -0,0 +1,302 @@ +using System.Text.Json.Nodes; +using CodeGuard.Analysis.AnalysisModel; +using CodeGuard.Evaluation.Assertions; +using CodeGuard.RuleModel.Assertions; +using CodeGuard.RuleModel.Selectors; + +namespace CodeGuard.Evaluation.Tests.Assertions; + +/// +/// Structured Expected/Actual/Evidence carried on (issue #69). These sit +/// alongside, not instead of, the human-readable Message, which the other test classes continue to assert on. +/// +public class AssertionEvidenceTests +{ + private static readonly RepositoryModel EmptyModel = new(".", [], [], [], [], [], [], [], [], []); + private static readonly JsonObject Template = new() { ["kind"] = "stub" }; + + private sealed class StubSelector(IEnumerable candidates) : ITargetSelector + { + public string Kind => "stub"; + public IEnumerable SelectCandidates(RepositoryModel model) => candidates; + } + + private sealed class StubAssertion(Func evaluate) : IAssertion + { + public string Kind => "stub_assertion"; + + public AssertionOutcome Evaluate(object candidate, RepositoryModel model) => evaluate(candidate); + } + + [Fact] + public void MustInheritFrom_Failure_CarriesPatternAsExpected_BaseTypeAsActual_TypeAsEvidence() + { + var type = TestModels.Type("Contoso.Domain.Entities.LegacyThing", baseType: "Contoso.Other.Base"); + + var outcome = new MustInheritFromAssertion("Contoso.Domain.Entity<*>").Evaluate(type, EmptyModel); + + Assert.False(outcome.Passed); + Assert.Equal("Contoso.Domain.Entity<*>", outcome.Expected); + Assert.Equal("Contoso.Other.Base", outcome.Actual); + Assert.Equal(["Contoso.Domain.Entities.LegacyThing"], outcome.Evidence); + Assert.Contains("must inherit from", outcome.Message); + } + + [Fact] + public void MustInheritFrom_MissingBaseType_ReportsNoneAsActual() + { + var type = TestModels.Type("Contoso.Domain.Entities.LegacyThing"); + + var outcome = new MustInheritFromAssertion("Contoso.Domain.Entity<*>").Evaluate(type, EmptyModel); + + Assert.Equal("(none)", outcome.Actual); + } + + [Fact] + public void Success_HasNoStructuredEvidence() + { + var type = TestModels.Type("Contoso.Domain.Entities.Order", baseType: "Contoso.Domain.Entity"); + + var outcome = new MustInheritFromAssertion("Contoso.Domain.Entity<*>").Evaluate(type, EmptyModel); + + Assert.True(outcome.Passed); + Assert.Null(outcome.Expected); + Assert.Null(outcome.Actual); + Assert.Null(outcome.Evidence); + } + + [Fact] + public void MustNotDependOn_Failure_CarriesOffendingTypesAsEvidence() + { + var offender = TestModels.Type("Contoso.Domain.Order", interfaces: ["Forbidden.Infra.IThing"]); + var project = TestModels.Project("Contoso.Domain", types: [offender]); + + var outcome = new MustNotDependOnAssertion("Forbidden.Infra.*").Evaluate(project, EmptyModel); + + Assert.False(outcome.Passed); + Assert.Equal("no dependency on 'Forbidden.Infra.*'", outcome.Expected); + Assert.Equal("Forbidden.Infra.IThing", outcome.Actual); + Assert.Equal(["Contoso.Domain.Order"], outcome.Evidence); + } + + [Fact] + public void MustOnlyDependOn_Failure_CarriesDisallowedTypesAsEvidence() + { + var offender = TestModels.Type("Contoso.Domain.Order", interfaces: ["Forbidden.Infra.IThing"]); + var project = TestModels.Project("Contoso.Domain", types: [offender]); + + var outcome = new MustOnlyDependOnAssertion(["System.*", "Contoso.*"]).Evaluate(project, EmptyModel); + + Assert.False(outcome.Passed); + Assert.Equal("System.*, Contoso.*", outcome.Expected); + Assert.Equal(["Forbidden.Infra.IThing"], outcome.Evidence); + } + + [Fact] + public void MustMatchName_Failure_CarriesRegexAsExpected_NameAsActual() + { + var type = TestModels.Type("Contoso.Domain.OrderService"); + + var outcome = new MustMatchNameAssertion("^.+Repository$").Evaluate(type, EmptyModel); + + Assert.False(outcome.Passed); + Assert.Equal("^.+Repository$", outcome.Expected); + Assert.Equal("OrderService", outcome.Actual); + } + + [Fact] + public void MustMatchNamespacePattern_Failure_CarriesNamespaceAsActual() + { + var type = TestModels.Type("Contoso.Domain.Orders.Order"); + + var outcome = new MustMatchNamespacePatternAssertion(".*\\.Events$").Evaluate(type, EmptyModel); + + Assert.False(outcome.Passed); + Assert.Equal(".*\\.Events$", outcome.Expected); + Assert.Equal("Contoso.Domain.Orders", outcome.Actual); + } + + [Fact] + public void MustHaveCount_Failure_CarriesExpectedCountAndMatchesAsEvidence() + { + var matches = new object[] { TestModels.Type("A.One"), TestModels.Type("A.Two") }; + var assertion = new MustHaveCountAssertion(Template, _ => new StubSelector(matches), min: null, max: null, exactly: 0); + + var outcome = assertion.Evaluate(TestModels.Type("Outer"), EmptyModel); + + Assert.False(outcome.Passed); + Assert.Equal("exactly 0", outcome.Expected); + Assert.Equal("2", outcome.Actual); + Assert.Equal(["A.One", "A.Two"], outcome.Evidence); + } + + [Fact] + public void MustExist_Failure_CarriesExpectedAndActualCounts() + { + var outcome = new MustExistAssertion(Template, _ => new StubSelector([])).Evaluate(TestModels.Type("Outer"), EmptyModel); + + Assert.False(outcome.Passed); + Assert.Equal("at least one match", outcome.Expected); + Assert.Equal("0 matches", outcome.Actual); + } + + [Fact] + public void MustAllMatch_Failure_AggregatesNestedEvidenceQualifiedByMatch() + { + var nested = new StubAssertion(candidate => + { + var name = ((TypeModel)candidate).Name; + return name == "Bad" + ? AssertionOutcome.Failure("nested failed", "suffix Handler", name, [$"{name}.Extra"]) + : AssertionOutcome.Success(); + }); + var matches = new object[] { TestModels.Type("A.Good"), TestModels.Type("A.Bad") }; + var assertion = new MustAllMatchAssertion(Template, _ => new StubSelector(matches), [nested]); + + var outcome = assertion.Evaluate(TestModels.Type("Outer"), EmptyModel); + + Assert.False(outcome.Passed); + Assert.Equal("suffix Handler", outcome.Expected); + Assert.Equal("1 match(es) failed a nested assertion", outcome.Actual); + Assert.Equal(["A.Bad: Bad.Extra"], outcome.Evidence); + } + + [Fact] + public void MustAllMatch_Failure_LeavesExpectedNullWhenNestedFailuresDisagree() + { + var nested = new StubAssertion(candidate => + AssertionOutcome.Failure("nested failed", $"expected for {((TypeModel)candidate).Name}", null, null)); + var matches = new object[] { TestModels.Type("A.One"), TestModels.Type("A.Two") }; + var assertion = new MustAllMatchAssertion(Template, _ => new StubSelector(matches), [nested]); + + var outcome = assertion.Evaluate(TestModels.Type("Outer"), EmptyModel); + + Assert.False(outcome.Passed); + Assert.Null(outcome.Expected); + Assert.Equal(["A.One", "A.Two"], outcome.Evidence); + } + + [Fact] + public void MustAnyMatch_Failure_CarriesEvaluatedMatchesAsEvidence() + { + var failing = new StubAssertion(_ => AssertionOutcome.Failure("nope")); + var matches = new object[] { TestModels.Type("A.One"), TestModels.Type("A.Two") }; + var assertion = new MustAnyMatchAssertion(Template, _ => new StubSelector(matches), [failing]); + + var outcome = assertion.Evaluate(TestModels.Type("Outer"), EmptyModel); + + Assert.False(outcome.Passed); + Assert.Equal("0 of 2 match(es) satisfied every nested assertion", outcome.Actual); + Assert.Equal(["A.One", "A.Two"], outcome.Evidence); + } + + [Fact] + public void MustNoneMatch_Failure_CarriesOffendingMatchesAsEvidence() + { + var passing = new StubAssertion(_ => AssertionOutcome.Success()); + var matches = new object[] { TestModels.Type("A.One") }; + var assertion = new MustNoneMatchAssertion(Template, _ => new StubSelector(matches), [passing]); + + var outcome = assertion.Evaluate(TestModels.Type("Outer"), EmptyModel); + + Assert.False(outcome.Passed); + Assert.Equal("1 offending match(es)", outcome.Actual); + Assert.Equal(["A.One"], outcome.Evidence); + } + + [Fact] + public void MustAnyMatch_EvaluatesEachMatchAssertionPairOnce_OnFailure() + { + var calls = new List(); + var first = new StubAssertion(c => + { + calls.Add($"first:{((TypeModel)c).Name}"); + return AssertionOutcome.Success(); + }); + var second = new StubAssertion(c => + { + calls.Add($"second:{((TypeModel)c).Name}"); + return AssertionOutcome.Failure("nope"); + }); + var matches = new object[] { TestModels.Type("A.One"), TestModels.Type("A.Two"), TestModels.Type("A.Three") }; + var assertion = new MustAnyMatchAssertion(Template, _ => new StubSelector(matches), [first, second]); + + var outcome = assertion.Evaluate(TestModels.Type("Outer"), EmptyModel); + + Assert.False(outcome.Passed); + Assert.Equal(calls.Distinct().Count(), calls.Count); + Assert.Equal(["first:One", "second:One", "first:Two", "second:Two", "first:Three", "second:Three"], calls); + Assert.Equal(["A.One", "A.Two", "A.Three"], outcome.Evidence); + } + + [Fact] + public void MustAnyMatch_StopsAtFirstSatisfyingMatch() + { + var calls = new List(); + var passing = new StubAssertion(c => + { + calls.Add(((TypeModel)c).Name); + return AssertionOutcome.Success(); + }); + var matches = new object[] { TestModels.Type("A.One"), TestModels.Type("A.Two"), TestModels.Type("A.Three") }; + var assertion = new MustAnyMatchAssertion(Template, _ => new StubSelector(matches), [passing]); + + var outcome = assertion.Evaluate(TestModels.Type("Outer"), EmptyModel); + + Assert.True(outcome.Passed); + Assert.Equal(["One"], calls); + } + + [Fact] + public void MustNoneMatch_CapsEvidenceAt50_WithTrailingMoreEntry() + { + var passing = new StubAssertion(_ => AssertionOutcome.Success()); + var matches = Enumerable.Range(1, 120).Select(i => (object)TestModels.Type($"A.T{i}")).ToArray(); + var assertion = new MustNoneMatchAssertion(Template, _ => new StubSelector(matches), [passing]); + + var outcome = assertion.Evaluate(TestModels.Type("Outer"), EmptyModel); + + Assert.False(outcome.Passed); + Assert.Equal("120 offending match(es)", outcome.Actual); + Assert.Equal(51, outcome.Evidence!.Count); + Assert.Equal("A.T1", outcome.Evidence[0]); + Assert.Equal("A.T50", outcome.Evidence[49]); + Assert.Equal("...and 70 more", outcome.Evidence[50]); + } + + [Fact] + public void MustNoneMatch_LeavesEvidenceUncapped_AtExactlyTheCap() + { + var passing = new StubAssertion(_ => AssertionOutcome.Success()); + var matches = Enumerable.Range(1, 50).Select(i => (object)TestModels.Type($"A.T{i}")).ToArray(); + var assertion = new MustNoneMatchAssertion(Template, _ => new StubSelector(matches), [passing]); + + var outcome = assertion.Evaluate(TestModels.Type("Outer"), EmptyModel); + + Assert.Equal(50, outcome.Evidence!.Count); + Assert.DoesNotContain(outcome.Evidence, entry => entry.StartsWith("...and", StringComparison.Ordinal)); + } + + [Fact] + public void MustNotExist_FailureEvidence_IsBoundedAndSampleStopsEarly() + { + var yielded = 0; + IEnumerable Candidates() + { + for (var i = 1; i <= 500; i++) + { + yielded++; + yield return TestModels.Type($"A.T{i}"); + } + } + + var outcome = new MustNotExistAssertion(Template, _ => new StubSelector(Candidates())) + .Evaluate(TestModels.Type("Outer"), EmptyModel); + + Assert.False(outcome.Passed); + Assert.Equal(51, yielded); + Assert.Equal("more than 50 match(es)", outcome.Actual); + Assert.Equal(51, outcome.Evidence!.Count); + Assert.Equal("...and more", outcome.Evidence[50]); + } +}