kapp-controller silently drops errors when pushing CA certs/proxy settings to its sidecarexec container

[himsngh]

What steps did you take:
[A clear and concise description steps that can be used to reproduce the problem.]

1. Deploy kapp-controller such that the sidecarexec sidecar container in the same Pod is slightly slower to start listening on its RPC socket than the main kapp-controller container (e.g. under load, on a busy/upgrading cluster, or simply due to normal container start-order non-determinism — Kubernetes gives no ordering guarantee between containers in the same Pod).
2. Configure the kapp-controller-config ConfigMap/Secret with caCerts and/or proxy settings (httpProxy, httpsProxy, noProxy).
3. Let the kapp-controller Pod start. On startup, cmd/controller/run.go calls the config Reconciler.Reconcile() once synchronously, specifically to push CA certs/proxy settings to the sidecarexec container "before any tool execution happens."
4. If the sidecarexec container's RPC socket isn't listening yet when this call happens, the RPC call to OSConfig.ApplyCACerts/OSConfig.ApplyProxy fails.

What happened:
[A small description of the issue]

pkg/config/reconciler.go's Reconcile() logs the RPC failure but swallows it (// continue on) and always returns nil. Because of this:

• The startup check in run.go (if err != nil { return fmt.Errorf(...) }) never fires, even though CA certs/proxy were never actually applied to the sidecar.
• Since the config Reconciler's watches (AttachWatches) only re-trigger Reconcile on subsequent changes to the kapp-controller-config ConfigMap/Secret, if the config doesn't change again, CA certs/proxy are never retried for the lifetime of that Pod.
• Any App/PackageInstall reconciliation that depends on those CA certs (e.g. fetching from an HTTPS git/helm/imgpkg source signed by a custom CA) can then fail or hang until it hits an unrelated timeout.

What did you expect:
[A description of what was expected]

A failure to apply CA certs or proxy settings to the sidecar should be treated as a retryable error:

• The initial synchronous startup reconcile should retry (with backoff/bounded timeout) rather than silently proceeding as if configuration succeeded.
• The regular watch-triggered Reconcile() should return the error so controller-runtime's standard requeue-with-backoff mechanism retries it, consistent with how every other reconciler in this codebase (pkg/app, pkg/packageinstall, pkg/pkgrepository) handles real (non-NotFound) errors.

Anything else you would like to add:
[Additional information that will assist in solving the issue.]

Confirmed in a real cluster's kapp-controller logs — both the one-shot startup
reconcile and the very next watch-triggered reconcile (~1s later, triggered by
the initial informer sync of the config ConfigMap/Secret) failed with the same
underlying RPC dial error, showing the sidecarexec container was not yet
listening on its socket when either reconcile attempt ran:

{"level":"error","logger":"kc.controller.config","msg":"Failed applying CA certificates",
 "request":{"name":""},
 "error":"Internal run comm: dial unix /etc/kappctrl-mem-tmp/sidecarexec.sock: connect: no such file or directory"}

{"level":"error","logger":"kc.controller.config","msg":"Failed applying CA certificates",
 "request":{"name":"kapp-controller-config","namespace":"kapp-controller-system"},
 "error":"Internal run comm: dial unix /etc/kappctrl-mem-tmp/sidecarexec.sock: connect: no such file or directory"}

Both errors are logged and then discarded by Reconcile(), and since no
further config change occurred, no other retry was ever attempted for the
rest of that Pod's lifetime.

Root cause: pkg/config/reconciler.go's Reconcile() calls r.osConfig.ApplyCACerts(...) and r.osConfig.ApplyProxy(...), both of which (via pkg/sidecarexec.OSConfigClient) are net/rpc calls over a Unix domain socket to the sidecarexec container. The reconnectingRPCClient (pkg/sidecarexec/client.go) does a single dial attempt with no built-in retry — if the socket doesn't exist yet, the call fails immediately. Reconcile() currently logs this and returns nil for both calls, which silently defeats the explicit intent of the synchronous startup call in cmd/controller/run.go ("Reconcile once synchronously to ensure controller configuration ... is applied to sidecar before any tool execution happens").

Environment:

• kapp Controller version (execute kubectl get deployment -n kapp-controller kapp-controller -o yaml and the annotation is kbld.k14s.io/images):
• v0.60.4 (reproducible on develop as well)
• Kubernetes version (use kubectl version)

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.