fix(runner): propagate public hostname to kubernetes job pods for TF_TOKEN auth

[olemathias]

Fixes #947

Problem

When using the Kubernetes executor, OpenTofu fails to read remote state via data.terraform_remote_state with the remote backend. The runner's authentication token doesn't appear to be propagated to the pod/job that executes the plan.

Root cause:
otf-job connects to otfd via the internal Kubernetes service URL (e.g. http://otf.default:8080/).
DoOperation derives the TF_TOKEN_* credential env var name from the API client's hostname, so it sets TF_TOKEN_otf_default_8080=<token>.

But Terraform's data.terraform_remote_state can connect using the public-facing OTF hostname (e.g. otf.example.com) and looks for TF_TOKEN_otf_example_com, which is never set.

Fix

Pass the public OTF hostname to job pods as OTF_HOSTNAME.
otf-job reads it and supplies it as CredentialHostname in OperationOptions.

DoOperation now sets credential env vars for both hostnames when they differ — one for Client.Hostname() (the internal service URL) and one for CredentialHostname (the public-facing hostname). This ensures Terraform can authenticate regardless of which hostname appears in the remote backend config.

The daemon propagates the hostname automatically, so no additional configuration is needed beyond having the public hostname set in otfd.

Testing

Added unit tests covering:

• DoOperation sets only one credential env var when CredentialHostname is empty or matches Client.Hostname()
• DoOperation sets two credential env vars when CredentialHostname differs from Client.Hostname()
• SpawnOperation passes the correct value in the OTF_HOSTNAME pod env var

Note

I am not an experienced Go developer. Claude Code was used to help identify and implement this fix. That said, the fix has been verified on a real OTF deployment and is working.

Happy to make any changes if the approach needs adjusting.

[leg100]

@olemathias Thanks for the bug report and PR.

I think the heart of the issue is that there a hostname set in data.terraform_remote_state.config.hostname which must match the hostname set in the name of the TF_TOKEN_<hostname>=<key> env var.

I don't believe this issue is unique to the kubernetes executor. Any scenario in which there is a mis-match between those two settings will result in the error you're seeing.

That can occur, say, in otfd and the fork executor, if you were to set data.terraform_remote_state.config.hostname to a locally routable address, like otf.internal:8080, but then set --hostname=myotf.example.com (or vice versa).

In your case, and as you've noticed, otf-job sets the hostname in the token env var to the hostname and port derived from the otfd kubernetes service.

I'm not sure about your fix. I need to think some more on the ramifications. The --hostname flag overrides the value of the hostname in otfd, and was primarily introduced as a way for public systems like GitHub to be able to connect to an internal deployment of OTF, e.g. OTF is running on a kubernetes cluster but it has an external load balancer with a hostname which receives requests from public entities like GitHub, but the load balancer is not routable/accessible from processes on the kubernetes cluster itself, such as otf-job. Your fix may well break installations with a setup like this.

Have you considered setting data.terraform_remote_state.config.hostname to the same value as that derived from the kubernetes service?

[olemathias]

Thanks for the quick review!

I did consider setting data.terraform_remote_state.config.hostname to the Kubernetes service hostname, but it felt less intuitive than using the public OTF hostname.
Especially since this works with the default fork executor and runs outside of otf. That said, I take your point about potentially breaking existing deployments.

What if instead of replacing the TF_TOKEN_* env var, we set an additional one? That way the internal service hostname token continues to work as before, and the public hostname token is also available when --hostname is configured. Would that be an acceptable approach?

[leg100]

I did consider setting data.terraform_remote_state.config.hostname to the Kubernetes service hostname, but it felt less intuitive than using the public OTF hostname.
Especially since this works with the default fork executor and runs outside of otf. That said, I take your point about potentially breaking existing deployments.

Why would you have runs both inside and "outside of OTF"?

But in any case, on second thoughts I'm not sure this would be a viable approach because the service probably doesn't expose the endpoint as https and terraform assumes it is https.

What if instead of replacing the TF_TOKEN_* env var, we set an additional one? That way the internal service hostname token continues to work as before, and the public hostname token is also available when --hostname is configured. Would that be an acceptable approach?

Yes I think this a more valid approach. Please update your PR accordingly.

[olemathias]

@leg100 PR updated with the changes

[leg100]

What's the purpose of this test if the code under test is merely re-implemented within the test?!