feat: Allow configuring tolerations and node scheduling for Kubernetes runner pods

[danhngo-lx]

Is your feature request related to a problem? Please describe.

When self-hosting the supervisor in Kubernetes mode, runner (task-run) pods can get stuck in the Pending state because they cannot be scheduled onto our worker nodes. There is currently no configuration option to resolve this without patching the supervisor.

Describe the solution you'd like to see

Configuration options to control the scheduling of all run pods, following the existing KUBERNETES_* environment variable conventions and defaulting to the current behavior so the change is non-breaking:

• KUBERNETES_WORKER_TOLERATIONS: tolerations applied to every run pod. This can reuse KUBERNETES_SCHEDULED_RUN_TOLERATIONS. For scheduled runs, these would be merged with the existing scheduled-run tolerations.
• Optionally, a way to configure general node affinity or to override/clear the injected nodetype node selector, for clusters whose worker nodes use a different labeling scheme.

When unset, no tolerations are added and the node selector is unchanged, keeping behavior identical to today.

Describe alternate solutions

• Extending KUBERNETES_SCHEDULED_RUN_TOLERATIONS to apply to all runs: simpler, but it changes the current scheduled-only semantics and would be a breaking change

Additional information

No response

[nicktrn]

Thanks for the report @danhngo-lx! This actually makes a lot of sense and was fairly simple to add.

There's now a new env var KUBERNETES_RUNNER_TOLERATIONS that can be used like so:

KUBERNETES_RUNNER_TOLERATIONS=dedicated=runs:NoSchedule,spot:NoExecute

More incl helm example in the PR.

I believe the nodetype selector shouldn't be required given the helm default of KUBERNETES_WORKER_NODETYPE_LABEL=""