Repository navigation
SWE-bench history check: count PR mentions only since the PR was opened - #17
Merged
Merged
Conversation
The check counted every commit message with "#<number>" as naming the pull request. Django's messages cite Trac tickets the same way, and ticket 15315 (2011) is not pull request 15315 (2022), so django__django-15315 failed with a base history that is correctly pruned. A commit from before the pull request was opened cannot refer to it, so the probe now passes the task's created_at to git log --since. Multi-SWE-bench tasks carry no created_at and keep counting every message.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The history check counted every commit message with
#<number>as naming the pull request. Django cites Trac tickets the same way, sodjango__django-15315failed on a correctly pruned history: its base keeps "Fixed #15315" from 2011, ticket 15315, not PR 15315 (2022).created_attogit log --since: a commit from before the PR was opened can't refer to it.created_atand keep counting every message.Tests: 3 new probe tests against a local git repository; unit tests pass, and
django__django-15315passed in #16's Sample tasks run with this commit.