Repository navigation
Run a prospective behaviour check on 259 advisories published after the snapshot - #62
Merged
Merged
Conversation
…he snapshot The advisory database at 9b0169eb (2026-11-15) is built into separate paths; 259 PyPI advisories absent from the pinned corpus and published after its commit date form the frozen prospective-1 batch. There are no labels, so only behaviour is reported, against the 272 dev documents of batch-03, batch-06 and batch-07. New advisories are much longer (median 515 words against 79; 71% need more than one 512-token window against 13%), continuing a rise through 2026. The CRF suggests 4.4 conditions per advisory (dev 1.1) and abstains on 7.7% (dev 43.8%); DeBERTa suggests 15.3 (dev 4.5). NOT_REQUIRED, EXCEPTION and negation-flagged items are more common.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prospective check (Definition of Done: prospective batch reported separately), behaviour only: no labels exist or will be made, so no accuracy or agreement.
scripts/experiments/prospective.sh [BUNDLE]: fetchesgithub/advisory-databaseat9b0169ebinto its own directory (pinned snapshot untouched), builds it with the same pipeline (reports/prospective/snapshot-9b0169eb.json), freezesprospective-1.json(259 PyPI advisories absent from the pinned corpus and published after 2026-10-20T03:38:30Z), predicts with the frozen CRF (live review policy) and the run-3 DeBERTa, and writessummary.mdagainst the 272 dev documents.No test data touched.