Repository navigation
Structured expected/actual evidence on violations (#69) - #99
Merged
Merged
Conversation
AssertionOutcome, Violation and AnalyzerViolation gain optional Expected, Actual and Evidence fields (default null, additive). Assertions populate them where there is a natural value; quantifiers aggregate nested outcomes. Message text is unchanged. Surfaced in the console reporter, the JSON v1 contract (additive properties, schemaVersion stays 1.0), SARIF property bag (reserved keys expected/actual/evidence, Metadata keys take precedence) and HTML. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- must_any_match makes one pass over its matches, evaluating each nested assertion at most once per match. Failure Expected/Evidence reuse those outcomes instead of re-evaluating every match a second time. - Evidence is capped at 50 entries via CandidateDescriptor.CapEvidence, with a final "...and N more" entry when truncated. Applied to must_all_match, must_none_match, must_have_count, must_not_depend_on, must_only_depend_on and must_any_match. ListOrNone (list-valued actual) is capped too. - must_not_exist materialises at most 51 candidates, so the success path stops at the first item and the failure evidence stays bounded. Message text is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Violations carried only a free-text
Message. This adds optional structured evidence so agents can act on a failure without parsing prose (CORE_RULES Pillar 6, REFACTORING Ph7).AssertionOutcomegains optionalExpected,Actual,Evidence(IReadOnlyList<string>?) and aFailure(message, expected, actual, evidence)overload.ViolationandAnalyzerViolationgain the same optional fields.RuleEvaluator.CreateViolationcopies them across. All existing constructors and call sites still compile.must_inherit_fromgives the pattern, the actual base type and the type. Dependency assertions give the offending types or dependency names as evidence. Naming and namespace assertions give the pattern and the actual name. Count and existence assertions give expected/actual counts and the matched candidates. Quantifiers (must_all_match,must_any_match,must_none_match) aggregate nested outcomes:expectedis kept only when all nested failures agree, and evidence is qualified by the match it came from (<match>: <item>).Messagetext is unchanged everywhere.expected:/actual:/evidence:lines, only when present.expected,actual,evidenceproperties.schemaVersionstays1.0. The schema anddocs/architecture/RESULT_CONTRACT.mdare updated.expected,actual,evidence. AMetadatakey with the same name takes precedence, so existing metadata is never overwritten.docs/architecture/IMPLEMENTATION_STATUS.mdgets a Structured expected/actual evidence on violations #69 section.Tests
AssertionEvidenceTests(new):must_inherit_from,must_not_depend_on,must_only_depend_on,must_match_name,must_match_namespace_pattern,must_have_count,must_exist, quantifiers (all/any/none), and a success-has-no-evidence check.RuleEvaluatorTests: fields flow from outcome to violation, null when absent, and analyzer violations carry them.ConsoleViolationReporterTests,SarifViolationReporterTests(property bag and metadata precedence),ValidationReportMapperTests,ValidationReportSchemaTests(conforms to schema with the new fields).dotnet buildhas 0 warnings and 0 errors.dotnet testpasses across all projects.dotnet format --verify-no-changespasses.rules test(251/251) andrules validate(132/132) pass againstexamples/rules. No parser descriptor orrule.schema.jsonchanged, sosync-skill-references.shis not affected.Closes #69
🤖 Generated with Claude Code