uber · behinddwalls · Jul 24, 2026
@@ -1,64 +1,17 @@
-# Scorer
+# scorer
 
-Vendor-agnostic interface for computing success probability scores for code changes.
+A `Scorer` returns the probability that a batch's build succeeds — a number between 0.0 and 1.0. It is handed the batch identity and resolves the batch's changes itself through an injected `changeset.Resolver`, so callers pass an `entity.Batch` and nothing more.
 
-## Interface
+Callers may score every batch a queue is waiting on, so implementations should be cheap. A speculation run scores each batch at most once, but it does not carry results across runs; anything expensive belongs behind the implementation's own cache.
 
-### Scorer
-
-Computes a success probability for a given change.
-
-```go
-type Scorer interface {
-    Score(ctx context.Context, change entity.Change) (float64, error)
-}
-```
-
-- **change**: A `entity.Change` identifying the code change to score.
-- **Score**: Returns a probability between 0.0 and 1.0 indicating the likelihood of a successful land. Returns an error if scoring fails.
+Like the other extensions, a `Scorer` is selected **per queue** by the wiring layer through the `Config` (queue name) and `Factory` interface.
 
 ## Implementations
 
-### Heuristic
-
-Scores a change by extracting a numeric value via a `ValueFunc` and matching it against ordered buckets. Each bucket maps a `[Min, Max]` range to a probability.
-
-```go
-s := heuristic.New(
-    []heuristic.Bucket{
-        {Min: 0, Max: 5, Score: 0.95},
-        {Min: 6, Max: 20, Score: 0.75},
-        {Min: 21, Max: 100, Score: 0.5},
-    },
-    func(ctx context.Context, change entity.Change) (int, error) {
-        // resolve the change into a numeric metric
-        return filesChanged, nil
-    },
-)
-
-score, err := s.Score(ctx, change)
-```
-
-### Composite
-
-Combines multiple named scorers into a single score using a reduce function. The reduce function receives a `map[string]float64` mapping scorer names to their scores, enabling domain-aware aggregation.
-
-Built-in reduce functions: `Min`, `Max`, `Avg`.
-
-```go
-s := composite.New(
-    map[string]scorer.Scorer{
-        "files": fileScorer,
-        "deps":  depScorer,
-    },
-    composite.Min,
-)
+**`heuristic`** scores a batch by extracting one number from its changes and matching that against ordered buckets, each mapping a `[Min, Max]` range to a probability. The extraction is a caller-supplied `ValueFunc` over the resolved `entity.BatchChanges`, so the same bucketing works for files touched, lines changed, or any other metric.
 
-score, err := s.Score(ctx, change)
-```
+**`composite`** runs several named scorers and reduces their scores to one. The reduce function receives the scores keyed by scorer name, so it can weigh sources differently rather than treating them as interchangeable; `Min`, `Max`, and `Avg` are provided.
 
-## Implementing a Backend
+## Adding a backend
 
-1. Create `extension/scorer/{backend}/` directory
-2. Implement the `Scorer` interface
-3. Accept `entity.Change` and resolve it into whatever data the implementation needs
+Create a package under `scorer/<backend>/` whose `New(...)` returns a `scorer.Scorer`, injecting whatever it needs at construction — a `changeset.Resolver` to reach the batch's changes, a metrics scope, any client. Do not add a `Config` or `Factory` implementation here; per-queue routing and the factory adapter live in the wiring layer.
@@ -22,11 +22,17 @@ import (
 	"github.com/uber/submitqueue/submitqueue/entity"
 )
 
-// Scorer computes a success probability score for a batch based on its changes.
+// Scorer computes the probability that a batch's build succeeds, based on its
+// changes.
 type Scorer interface {
-	// Score returns a probability between 0.0 and 1.0 indicating the likelihood
-	// of a successful land for the given batch. It is handed the batch identity
-	// and resolves the batch's changes itself through an injected changeset.Resolver.
+	// Score returns a probability between 0.0 and 1.0 that the given batch's
+	// build succeeds. It is handed the batch identity and resolves the batch's
+	// changes itself through an injected changeset.Resolver.
+	//
+	// Callers may score every batch a queue is waiting on, so implementations
+	// should be cheap: a speculation run scores each batch at most once, but it
+	// does not carry results over to the next run, so anything expensive to
+	// compute belongs behind the implementation's own cache.
 	Score(ctx context.Context, batch entity.Batch) (float64, error)
 }
 

@@ -0,0 +1,9 @@
+load("@rules_go//go:def.bzl", "go_library")
+
+go_library(
+    name = "go_default_library",
+    srcs = ["generator.go"],
+    importpath = "github.com/uber/submitqueue/submitqueue/extension/speculation/generator",
+    visibility = ["//visibility:public"],
+    deps = ["//submitqueue/entity:go_default_library"],
+)
@@ -0,0 +1,9 @@
+# generator
+
+The `generator` package is a piece the `standard` `Speculator` is built from: a `Generator` produces the queue's candidate paths as one stream, best first, across all heads. It is **not** controller-facing — the speculate controller only knows the `Speculator` contract, and a different `Speculator` need not split its work this way. So there is no `Config` or `Factory` here; a `Generator` is chosen when the `standard` `Speculator` is constructed.
+
+`Open` starts the stream over the queue's live batches and returns a `PathIterator`. The caller pulls one candidate at a time and the generator does only the work that answer needs. A cancelled or expired context ends the stream with its error.
+
+Candidates never repeat and never contradict a known fact. Beyond that, the order is the `Generator`'s own: it yields candidates in whatever ranking it implements, and each carries the score it ranked by — higher first, on a scale the generator defines. Consumers take the iterator in the order given and do not interpret the score. Scores mean something only within the run and are never stored.
+
+The generator offers every path in the space, including paths whose builds already ran. Suppressing finished paths is the `Allocator`'s job, since that is the piece reconciling candidates against the stored path sets.
@@ -0,0 +1,30 @@
+load("@rules_go//go:def.bzl", "go_library", "go_test")
+
+go_library(
+    name = "go_default_library",
+    srcs = [
+        "bestfirst.go",
+        "head.go",
+        "iterator.go",
+    ],
+    importpath = "github.com/uber/submitqueue/submitqueue/extension/speculation/generator/bestfirst",
+    visibility = ["//visibility:public"],
+    deps = [
+        "//submitqueue/entity:go_default_library",
+        "//submitqueue/extension/scorer:go_default_library",
+        "//submitqueue/extension/speculation/generator:go_default_library",
+    ],
+)
+
+go_test(
+    name = "go_default_test",
+    srcs = ["bestfirst_test.go"],
+    embed = [":go_default_library"],
+    deps = [
+        "//submitqueue/entity:go_default_library",
+        "//submitqueue/extension/scorer:go_default_library",
+        "//submitqueue/extension/speculation/generator:go_default_library",
+        "@com_github_stretchr_testify//assert:go_default_library",
+        "@com_github_stretchr_testify//require:go_default_library",
+    ],
+)
@@ -0,0 +1,108 @@
+# bestfirst
+
+A *head* — the batch we want to build — waiting on n unfinished dependencies has one path per combination of outcomes, 2ⁿ in all, while callers only ever want the best few. `bestfirst` hands paths out in score order without building the rest.
+
+One example runs through this whole doc: a head waiting on three dependencies, where the injected `scorer.Scorer` says each one's build succeeds with probability
+
+| dependency | P(succeeds) |
+| --- | --- |
+| A | 0.9 |
+| B | 0.8 |
+| C | 0.6 |
+
+## Scoring is multiplication
+
+A path is a set of guesses, one per dependency, and its score follows from the probability that all of its guesses come true — the product of the per-dependency probabilities.
+
+Each dependency has a *preferred* side — whichever of "succeeds" and "fails" is likelier — and an *unpreferred* side with the leftover probability. So the best path is not "assume everything succeeds"; it takes every dependency's preferred side, and a dependency that will probably fail is assumed to fail. Here all three lean "succeeds":
+
+- best path `[A✓ B✓ C✓]`: 0.9 × 0.8 × 0.6 = **0.432**
+- flip C `[A✓ B✓ C✗]`: 0.9 × 0.8 × 0.4 = **0.288**
+- flip B and C `[A✓ B✗ C✗]`: 0.9 × 0.2 × 0.4 = **0.072**
+
+Look at what one flip did to the product: flipping C swapped the factor 0.6 for 0.4, multiplying the whole product by 0.4/0.6. That is the observation everything else builds on: **every path's probability is the best path's probability times one fixed ratio (unpreferred/preferred) per flip, no matter which other flips it takes.** Check: 0.288 = 0.432 × (0.4/0.6), and 0.072 = 0.432 × (0.2/0.8) × (0.4/0.6).
+
+## Products become sums of logs
+
+Multiplying hundreds of probabilities underflows to exactly 0.0 — a wide head's paths all tie at zero and the order is lost entirely (the package doc in `bestfirst.go` shows the failure). So the code never stores the product; a score is always its logarithm, and log turns multiplication into addition: `log(x × y) = log x + log y`. Expanding the best path's product:
+
+```
+score(best) = log(0.9 × 0.8 × 0.6)
+            = log 0.9 + log 0.8 + log 0.6
+            = −0.105  − 0.223  − 0.511
+            = −0.839                        (e^−0.839 ≈ 0.432 — the product we started with)
+```
+
+A flip expands the same way. Flipping C multiplied the product by 0.4/0.6, and the log of a ratio is a difference:
+
+```
+score(flip C) = log(0.432 × 0.4/0.6)
+              = log 0.432 + (log 0.4 − log 0.6)
+              = −0.839   + (−0.41)
+              = −1.245                      (e^−1.245 ≈ 0.288)
+```
+
+That parenthesized difference, `log(unpreferred) − log(preferred)`, is exactly what the code precomputes per dependency as its `relativeScore`. It is a difference rather than just `log(unpreferred)` because the best-path score already counts the preferred side — one addition both removes it and puts the unpreferred side in. So in general:
+
+```
+score(path) = score(best path) + relativeScore of each flip the path takes
+```
+
+Two properties fall out. A `relativeScore` is never positive — the preferred side is by definition the likelier one, so the ratio is at most 1 — meaning flipping can only cost. And a side that cannot happen has probability 0, whose log is −Inf: an impossible path sinks below every possible one, and stays there however many finite terms are added.
+
+## The tree: every path is the best path plus some flips
+
+Sort the flips cheapest-first — the cheapest flip belongs to the *least* confident dependency, the assumption we mind changing least:
+
+| flip | ratio | relativeScore |
+| --- | --- | --- |
+| C | 0.4 / 0.6 ≈ 0.67 | −0.41 |
+| B | 0.2 / 0.8 = 0.25 | −1.39 |
+| A | 0.1 / 0.9 ≈ 0.11 | −2.20 |
+
+Then lay every subset of flips out as a tree, rooted at the best path. Each child either takes the next flip on top of its parent's, or trades its parent's last flip for the next costlier one — so every subset appears exactly once (probabilities on the right):
+
+```
+{}                    0.432   flip nothing — the best path
+└── {C}               0.288
+    ├── {C,B}         0.072
+    │   ├── {C,B,A}   0.008   flip everything — the worst path
+    │   └── {C,A}     0.032
+    └── {B}           0.108
+        ├── {B,A}     0.012
+        └── {A}       0.048
+```
+
+Because every flip's `relativeScore` is ≤ 0 and children only reach for later, never-cheaper flips, **a child never outscores its parent**. That is the only order the tree guarantees — `{C,B}` at 0.072 sits above `{B}` at 0.108 while scoring below it — so global order is the heap's job.
+
+A node's score is always re-summed from the best-path score in this canonical order, never carried from its parent: for `{C,A}`, −0.839 − 0.41 − 2.20 = −3.44 = log 0.032. Floating-point addition does not associate, so carrying a parent's total would give the same path different bits depending on the route taken to it.
+
+## Pulling from the heap
+
+`Open` puts every head's root — its best path — into one shared heap. Each pull hands out the top node, builds that one path, and adds its one or two children:
+
+```
+handed out      heap afterwards
+{}      0.432   {C} 0.288
+{C}     0.288   {B} 0.108 · {C,B} 0.072
+{B}     0.108   {C,B} 0.072 · {A} 0.048 · {B,A} 0.012
+{C,B}   0.072   {A} 0.048 · {C,A} 0.032 · {B,A} 0.012 · {C,B,A} 0.008
+{A}     0.048   {C,A} 0.032 · {B,A} 0.012 · {C,B,A} 0.008
+{C,A}   0.032   {B,A} 0.012 · {C,B,A} 0.008
+{B,A}   0.012   {C,B,A} 0.008
+{C,B,A} 0.008   —
+```
+
+All eight paths come out in descending order without ever being enumerated or sorted together, and the heap never holds more than four nodes. Stop after two pulls and only two paths were ever built — pulling k paths builds exactly k, and a head nobody pulls from never even sorts its flips. With several heads in the queue their roots simply compete in this same heap.
+
+Ties break on depth first (every head's root before any head's deeper nodes), then on the flips taken, then on the head; depth has to come first because a dependency at exactly 0.5 flips for free, and ordering on the head instead would let one head's tied subtree drain a caller's budget before another head's root was offered at all.
+
+What cannot be deferred is scoring: ranking heads against each other needs every head's best-path score up front, so `Open` walks every dependency of every head once. `Dependencies` holds a transitive closure, so a chain of batches has quadratically many edges in its depth, and `Open` pays that whether or not anything is pulled.
+
+## The edges
+
+- A dependency that already resolved carries a forced assumption — succeeded means the head builds on top of it, anything else means it builds without — and drops out of the search.
+- A dependency that is not a live batch cannot be scored and gets a high default, since nearly every batch a queue accepts does build successfully.
+- `bestfirst` discards nothing: pull long enough and every path comes out. It never proposes `ignored` assumptions — conflict relaxation is the `Speculator`'s call, not the generator's.
+
+The proof that the tree walk reaches every path exactly once, in score order, is on `expand` in `iterator.go`; the behavior is pinned in `bestfirst_test.go`.