Skip to content

DefaultX509Source stops receiving SVID rotations after a rejected X.509 context update #486

Closed

Description

@NopeCharon

Description

DefaultX509Source throws a WatcherException from its X.509 context watcher's onError. When the error comes from an update that GrpcConversionUtils.toX509Context rejects (for example a corrupted certificate or key, or a SPIFFE ID mismatch), StreamObservers calls watcher.onError from inside StreamObserver.onNext. The exception escapes onNext, gRPC cancels the call (CANCELLED: Failed to read message.), and since CANCELLED is non-retryable the watch ends permanently.

After that, the source keeps serving the SVID it already has and never applies another update. Once that SVID expires, every TLS connection built on the source fails, and only a restart recovers.

DefaultJwtSource and CachedJwtSource had the same problem with the JWT bundles watcher, fixed in #482. DefaultX509Source still has the old pattern:

@Override
public void onError(final Throwable error) {
log.log(Level.SEVERE, "Error in X509Context watcher", error);
done.countDown();
throw new WatcherException("Error in X509Context watcher", error);
}

Reproduction

A fake Workload API streams three X.509-SVID responses on one FetchX509SVID call: a valid one with hint initial, a corrupted one, then a valid one with hint rotated. DefaultX509Source.newSource(...).getX509Svid().getHint() returns initial. The third update is never delivered.

Expected behavior

As with the JWT sources: an error before the first update makes newSource fail. After initialization, a rejected update is logged, the current SVID and bundles are kept, the stream stays open, and the next valid update is applied.

Activity

  1. added a commit that references this issue on Oct 29, 2026
    ae84260
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions