Skip to content

scientific(selection): select candidate K from admitted rolling-origin predictive scores #687

Description

@seonghobae

Finding

#684 provides an owner-issued fixed-training prevalence-mean predictive log likelihood, #685 admits cutoff/leakage-safe rolling-origin partitions, and #686 binds one fitted training state plus one evaluation payload to that admitted partition. The remaining model-selection gate still chooses candidate K from the in-sample Schwarz score. There is no owner path that compares multiple candidate training fits on the same admitted rolling-origin evaluation evidence.

That leaves #680's core selection claim incomplete: TEPP can now measure one admissible predictive diagnostic, but it cannot yet make the candidate-K decision from that diagnostic without caller-side recomposition.

Required RED / repair

Add a narrow model_selection gate that:

  • accepts one admitted RollingOriginPartition, a nonempty set of owner-issued ReferenceTopicTrainingFit candidates, and one exact evaluation payload;
  • derives each candidate K from the fitted model's topic dimension; no detached caller-supplied K label;
  • rejects duplicate fitted topic dimensions rather than counting two fits of the same K as distinct candidates;
  • scores every candidate only through scientific(selection): bind predictive rows to admitted rolling-origin partition #686 rolling_origin_prevalence_mean_predictive_log_likelihood(...), preserving partition/training/evaluation identity checks and the frozen training prevalence basis;
  • selects the highest finite rolling-origin prevalence-mean predictive log likelihood, with deterministic smaller-K tie breaking;
  • remains invariant to candidate input order;
  • keeps the existing in-sample Schwarz selector unchanged as a separately named diagnostic path;
  • does not call this STM document-completion likelihood and does not let an LLM vote choose the numerical optimum.

Use actual converged CPU f64 fits for at least two candidate topic dimensions in the positive contract. Empty candidates, duplicate K, partition substitution, and non-finite diagnostics must fail closed.

Boundary

This closes only the one-window predictive candidate-selection gate. It does not complete #680: multiple rolling-origin windows, true-parameter/true-K temporal-relational recovery across replications, RMSE/bias/failure denominator/MCSE acceptance, deliberate future-leak rejection/audit evidence, Evidence source/event-time authentication (#527/#658/#671), Membership provenance (#605), relation activation (#675), backend parity, or immutable release remain separate requirements.

Refs #639 #680 #684 #685 #686.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions