Skip to content

Require Metadata-Flavor header on sidecar context endpoints - #16340

Open
shruthi713 wants to merge 1 commit into
developfrom
fix/sidecar-context-metadata-header
Open

shruthi713 wants to merge 1 commit into
developfrom
fix/sidecar-context-metadata-header

Conversation

@shruthi713

@shruthi713 shruthi713 commented Nov 19, 2026 •

Copy link
Copy Markdown

Summary

Fixes b/531578226.

The sidecar metadata server (GcpMetadataHttpHandlerInternal) checks for the Metadata-Flavor: Google header on the status and token endpoints, but /set-context and /clear-context had no such check. This PR requires the header on those two endpoints as well, and makes GcpMetadataTaskContextUtil, their only caller, send it.

Details

  • GcpMetadataHttpHandlerInternal: setContext and clearContext reject a request without the header (HTTP 403) before reading or changing any state. status() and token() are unchanged.
  • GcpMetadataTaskContextUtil: adds the header to the PUT /set-context and DELETE /clear-context requests. The util is a no-op unless the Namespaced Service Accounts feature is enabled, so deployments without it are unaffected.
  • Both sides change in the same PR because the util only logs the sidecar's response code. The caller and the sidecar run from the same image in the same pod, so they cannot be on different versions.
  • This is header parity with the token endpoint, not caller authentication. Authorization for credential provisioning is still enforced by App Fabric (PROVISION_CREDENTIAL on the namespace).

Why there is no lock on /set-context

A state lock that rejects PUT /set-context while a context is already active (until DELETE /clear-context is called) was considered to stop user plugin code from overwriting the active namespace mid-task, and left out:

  • It is bypassed in one extra HTTP call. The sidecar listens on localhost inside the pod and has no way to distinguish a request from the CDAP task-worker harness vs. user plugin code running inside the same container. If PUT /set-context rejected overwrites while a context was set, plugin code could simply call DELETE /clear-context first (unlocking it) and then call PUT /set-context with a different namespace. Nor can /clear-context be gated on a harness-only secret, because RunnableTaskLauncher runs plugin code in the same JVM as the harness.
  • It would break pod self-healing on crashes. The sidecar is a separate container that outlives the main container. Today, a stale context left behind by a failed run is harmlessly overwritten by the next PUT /set-context. With a lock, any run that failed to call /clear-context would leave the sidecar permanently stuck, rejecting every subsequent task or preview on that pod until restart. Two ways that can happen today:
    • the main container crashes or OOMs mid-task, before the finally block in TaskWorkerHttpHandlerInternal calls /clear-context;
    • DefaultPreviewRunner sets the context and then throws in prepareStart or run, which sit outside the try block that clears context on deploy failures.

Testing

  • GcpMetadataHttpHandlerInternalTest (11/11): missing header and wrong header value are rejected on both endpoints and leave the context unchanged; a lowercase header value is accepted, as on the token endpoint; a rejected /set-context keeps an existing context; GcpMetadataTaskContextUtil sets and clears the context against the real handler. The tests use a small fake of the App Fabric provision endpoint so they can assert which namespace context is active.
  • The new rejection tests fail on develop (expected 403, got 200). Removing either header check, or the header from either util call, fails a specific test.
  • Also run and passing: TaskWorkerServiceTest, RemoteConfiguratorTest, RemoteTaskExecutorTest, ArtifactLocalizerServiceTest. Verified locally on JDK 8 with Checkstyle.
  • Live-tested on a CDF Autopush Enterprise RBAC instance (feature.namespaced.service.accounts.enabled=true) with a Namespace Service Account attached via Workload Identity:
    • Direct sidecar checks inside a task-worker pod (localhost:11021): PUT /set-context and DELETE /clear-context without Metadata-Flavor: Google (or with Metadata-Flavor: AWS) returned HTTP 403; with Metadata-Flavor: Google (and lowercase metadata-flavor: google) returned HTTP 200.
    • Pipeline deploy (DataFusionQuickstart) and pipeline preview (GCSFile -> BigQueryTable) both succeeded (HTTP 200 and COMPLETED), with GcpMetadataTaskContextUtil logging HTTP 200 on both /set-context and /clear-context in task-worker and preview-runner pods (0 non-200 responses).

The sidecar metadata server checked for the Metadata-Flavor: Google header on
the status and token endpoints but not on /set-context and /clear-context.
Require it there too, and send it from GcpMetadataTaskContextUtil, the only
caller.

Authorization for credential provisioning is still enforced by App Fabric.

AI_ORCHESTRATOR=all-of-creation
@shruthi713 shruthi713 added the build Triggers github actions build label Nov 19, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a security enhancement by requiring the 'Metadata-Flavor: Google' header for the '/set-context' and '/clear-context' endpoints in GcpMetadataHttpHandlerInternal. Requests lacking this header or containing an invalid value will now be rejected with a ForbiddenException. The GcpMetadataTaskContextUtil utility has been updated to automatically attach this header, and comprehensive unit tests have been added to validate the new validation logic. There are no review comments, and the changes look solid.

@sonarqubecloud

Copy link
Copy Markdown

Comment on lines +265 to +276
private static void requireMetadataFlavorHeader(HttpRequest request)
throws ForbiddenException {
if (request.headers().contains(METADATA_FLAVOR_HEADER_KEY,
METADATA_FLAVOR_HEADER_VALUE, true)) {
return;
}
throw new ForbiddenException(
String.format("Request is missing required %s header. To access the metadata server, "
+ "you must add the %s: %s header to your request.", METADATA_FLAVOR_HEADER_KEY,
METADATA_FLAVOR_HEADER_KEY, METADATA_FLAVOR_HEADER_VALUE));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Early throw would simplify the code:

  private static void requireMetadataFlavorHeader(HttpRequest request)
      throws ForbiddenException {
    if (!request.headers().contains(METADATA_FLAVOR_HEADER_KEY,
        METADATA_FLAVOR_HEADER_VALUE, true)) {
    throw new ForbiddenException(
        String.format("Request is missing required %s header. To access the metadata server, "
            + "you must add the %s: %s header to your request.", METADATA_FLAVOR_HEADER_KEY,
            METADATA_FLAVOR_HEADER_KEY, METADATA_FLAVOR_HEADER_VALUE));
    }
  }

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build Triggers github actions build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants