Skip to content

Readiness cannot distinguish 'configured but not probed' from 'not configured' #12

Description

@crypto-a

Found in production by an operator reading policy's /readyz, twice, on two different dependencies.

What happens

Registry has three states: ready, error, unconfigured. Result.reason is deliberately not serialized, and the reason given is sound: readiness is unauthenticated at the ingress and driver errors routinely embed internal hostnames, addresses and ports.

The consequence is that add_optional_unconfigured and friends collapse two operationally different situations into one indistinguishable answer:

  • nobody configured this dependency
  • somebody configured it, it is wired, it is working, and nothing has probed it

Only the reason string differs, and the reason string never leaves the process.

How it misled somebody

In policy, legacy-proxy reports unconfigured while the bridge is demonstrably serving byte-identical traffic through the load balancer. The operator was told to verify the rollout by checking that line, and had they trusted it over an actual request they would have concluded the environment variable had not landed and rolled back a working deploy.

The same shape hit spaces earlier: the operator told a colleague the line would flip to ready once credentials were supplied. It did not and could not, because the state depends on whether a client is wired rather than on whether the value is present.

Two dependencies in one service now report unconfigured while being configured, which means the word currently carries no information there.

Why the obvious fixes are wrong

Probing the dependency from the check. For policy's legacy bridge this is actively dangerous. The kubelet allows the readiness probe 3 seconds and reads only its status code, so a slow upstream makes the response slow and fails the probe regardless of what the body says, even for a dependency registered as informational. A dependency whose failure is explicitly not supposed to affect this service would then pull every pod out of the Service endpoints.

Serializing the reason. The existing argument against it stands.

Reporting ready when configuration is present. This is the thing the current design is right to refuse. A check that passes because a URL was set proves an operator typed something.

What seems to be missing

A fourth state, or a flavour of unconfigured, meaning configured and wired, not probed. It says something true, it says something different from absent, it leaks nothing, and it costs no I/O.

Naming is the open question. configured reads as a peer of ready and might invite exactly the misreading the current design avoids; something like unprobed is uglier and harder to misread, which may be the point.

Worth deciding alongside it: whether the verdict treatment changes at all. It probably should not. This is a reporting distinction, not a readiness one.

Not blocking

policy pins the current behaviour in a test that instructs its own deletion when this lands, and its operators have been told to verify with a real request rather than with the readiness line. Filing because the gap belongs to every service in the fleet rather than to that one.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions