Repository navigation
Conversation
swebench-verified@1.0 already names a third-party set of 500 task directories in other registries, and nodeq.tech's task catalog pins those. node-q's generated set has different bytes (456 tasks, environments built locally), so publishing it under the same name and version would make one reference mean two datasets. It has not been published yet, so it moves to 2.0 now. Only the version changes: the task files do not carry it, and regenerating 2.0 gives the same digests as generated.lock.json.
The check counted every commit message with "#<number>" as naming the pull request. Django's messages cite Trac tickets the same way, and ticket 15315 (2011) is not pull request 15315 (2022), so django__django-15315 failed with a base history that is correctly pruned. A commit from before the pull request was opened cannot refer to it, so the probe now passes the task's created_at to git log --since. Multi-SWE-bench tasks carry no created_at and keep counting every message.
rzonyx
marked this pull request as draft
October 23, 2026 13:30
Contributor
Author
|
Closing: swebench-verified@1.0 was published before this PR, so it stays 1.0. The history-check fix moved to #17. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
swebench-verified@1.0already names a third-party set of 500 task directories in other registries, and nodeq.tech's task catalog pins those. Ours is a different set (456 tasks, environments built locally), so it moves to 2.0 before its first publish.datasets/swebench-verified/1.0→2.0;versionandtitlein source.yaml, the NOTICE heading, README and docs/licensing.md.generated.lock.json.django__django-15315's base history has "Fixed #15315" from 2011, a Trac ticket, not the PR. The probe now counts PR mentions only since the task'screated_at(Multi-SWE-bench has none and is unchanged).Tests:
validate_sources.py, the unit tests (353 OK, 3 new for the probe) andcheck_generated_tasks.py datasets/swebench-verified/2.0with hbench 0.3.5, locally.multi-swe-bench@1.0 cli__cli-7180failed once on a flaky upstream test (TestPortsUpdateVisibilityFailure403, 30 s deadline); it passed on main with the same files.