Skip to content

Restrict access to the internal /v1 messaging API - #16252

Open
shruthi713 wants to merge 7 commits into
developfrom
fix/restrict-internal-v1-api
Open

shruthi713 wants to merge 7 commits into
developfrom
fix/restrict-internal-v1-api

Conversation

@shruthi713

@shruthi713 shruthi713 commented Oct 14, 2026 •

Copy link
Copy Markdown

Six logical changes, each in its own commit so it can be reviewed and reverted independently:

1. a1373ceec77 (+ 104028771cf, 2fa5ee6628b) Do not route the internal /v1 API in the router

RouterPathLookup forwarded every path outside /v3 to app-fabric, including the /v1 paths of the internal messaging, remote system operations and impersonation APIs. Those handlers are meant to be reached through service discovery only. The router now answers DONT_ROUTE (HTTP 503) for any request whose first path segment is v1.

  • The check runs before the HTTP method is parsed. AllowedMethod.valueOf throws for methods such as PATCH or OPTIONS, and the catch block routes those to app-fabric, so a check placed after it could be skipped by picking one of those methods.
  • It is a whole-segment, case-sensitive match. v1x, V1, /x/v1/... and /v3/.../v1/... are routed as before.
  • Percent-encoded spellings (/%76%31/...) and repeated slashes (///v1/..., /v1//...) are normalised by RouterServiceLookup (URI.create(uri).normalize().getPath()) before the lookup, which is the same normalisation the downstream netty-http HttpResourceHandler applies. (A double-leading slash //v1/... is parsed by RFC 2396 URI.create in both RouterServiceLookup and HttpResourceHandler with "v1" as the URI authority and "/namespaces/..." as the path, so it never resolves to /v1/... on the backend either.) This is covered by RouterServiceLookupTest.
  • The router integration tests used /v1/ as an arbitrary first path segment for their mock services. They now use /test/.

2. 349a35acce8 (+ 104028771cf) Enforce namespace authorization in the messaging topic metadata handler

MetadataHandler (/v1/namespaces/{namespace}/topics...) passed every request straight to MessagingService. Every endpoint now authorizes the caller first (via a package-private MessagingHandlerAuthorizer), before the request body is parsed or the topic is looked up, so a caller without permission gets the same answer whether or not the topic exists:

Endpoint Required
create / update (PUT), delete topic UPDATE on the namespace
get topic, list topics GET on the namespace
any of the above in the system namespace (case-insensitive), when security.internal.auth.enabled=true an internal credential, in addition to the permission above
  • Topics are authorized at the namespace level rather than per-topic: creating, updating, or deleting a topic in a namespace is modeled as modifying the namespace (StandardPermission.UPDATE, matching DraftHandler and PreferencesHttpHandler), rather than creating or deleting the namespace itself.
  • With authorization disabled the AccessEnforcer is a no-op, so standalone and unsecured deployments behave as before.
  • MessagingHandlerAuthorizer injects CConfiguration, AccessEnforcer and AuthenticationContext. It reads security.internal.auth.enabled from CConfiguration directly (same predicate as SecurityUtil.isInternalAuthEnabled) because cdap-security is only a test-scope dependency of cdap-tms.
  • Six test injectors that build the handler bind a no-op AuthenticationContext next to the no-op enforcement module they already used.

3. 4ebebf48c27 Enforce namespace authorization in the messaging store and fetch handlers

StoreHandler (POST /v1/namespaces/{namespace}/topics/{topic}/publish, /store, /rollback) and FetchHandler (POST /v1/namespaces/{namespace}/topics/{topic}/poll) passed every request straight to MessagingService (creating a temporary spill file in StoreHandler.publish and StoreHandler.store before reading the body). Both handlers now inject MessagingHandlerAuthorizer and authorize the caller on the namespace of the topic before any temporary file is created or the request body is read. Validating the namespace and topic IDs and enforcing authorization before Files.createTempFile in StoreHandler.publish and StoreHandler.store also ensures that unauthorized or malformed requests never allocate temporary files on disk.

4. d28a4ce55be Scope system topic publish exceptions to worker and preview topics

Refines MessagingHandlerAuthorizer.authorizePublish(String namespace, String topic) so that system namespace publish exceptions are tightly scoped to the exact topics that task workers and preview runners publish to over HTTP:

Endpoint Required
publish, store, rollback (StoreHandler) UPDATE on the namespace
poll (FetchHandler) GET on the namespace
any of the above in the system namespace (case-insensitive), when security.internal.auth.enabled=true an internal credential, in addition to the permission above
publish in the system namespace with the worker placeholder internal credential (empty-worker-user-credential), when security.internal.auth.enabled=true permitted without calling AccessEnforcer only when the topic is in the worker allowlist (metrics0..N-1, auditlogevent0..N-1, previewlog0, preview); rejected with 403 on all other system topics (e.g. audit, programstatusevent, metadata, dataevent, logs0..N-1)
publish to the configured preview topic (preview.messaging.topic, default preview) in the system namespace with a non-empty user credential, when security.internal.auth.enabled=true permitted without calling AccessEnforcer so PreviewRunnerService / MessagingPreviewDataPublisher can publish preview status and tracer data while SecurityRequestContext holds the previewing user's EXTERNAL credential
  • Task workers and preview runners use WorkerAuthenticationContext, which supplies empty-worker-user-credential (CredentialType.INTERNAL) when background threads (MessagingMetricsCollectionService, MessagingAuditLogWriter, PreviewTMSLogAppender, and DefaultPreviewRunner.startUp()) publish outside a user request. Scoping this exception to workerSystemTopics prevents a caller carrying empty-worker-user-credential from spoofing control-plane events on system:programstatusevent, system:audit, system:metadata, system:dataevent, etc.
  • During a preview run, RemotePreviewRequestFetcher.fetch() sets SecurityRequestContext on the PreviewRunnerService thread to the previewing user's principal (CredentialType.EXTERNAL), which is also inherited by child preview threads via InheritableThreadLocal. Allowing a caller with a non-empty credential to publish only to system:preview (configured via preview.messaging.topic) (while rejecting null / empty / placeholder credentials and rejecting all other system topics) preserves end-to-end pipeline preview on RBAC instances.

5. 9aefa3fba8c Clear user details in DefaultClientMessagingService requests

When messaging.service.enabled=true (such as on SQL-backed CDF Developer/Basic RBAC instances where MessagingHttpService runs in a dedicated messaging-0 pod, and in distributed open-source CDAP deployments):

  • Master services (appfabric, appfabricprocessor, metadata, metrics, logs, runtime) bind MessagingService to DefaultClientMessagingService (MessagingClientModule) with SystemAuthenticationContext. When an external /v3 request (e.g. deploying an app or starting/stopping a program) runs on a handler thread where AuthenticationChannelHandler has populated SecurityRequestContext with the calling user's EXTERNAL credential, SystemAuthenticationContext.getPrincipal() returns that user's EXTERNAL principal whenever SecurityRequestContext.getUserId() != null, which previously caused internal /v1 messaging calls to NamespaceId.SYSTEM to fail with 403 Only internal services can access topics of the system namespace.
  • Task workers and preview runners (TaskWorkerMessagingClientModule, PreviewRunnerMessagingClientModule) also bind MessagingService to DefaultClientMessagingService paired with WorkerAuthenticationContext when messaging.service.enabled=true (whereas they bind TaskWorkerClientMessagingService / PreviewRunnerClientMessagingService, which subclass AbstractClientMessagingService directly, when messaging.service.enabled=false).
  • AbstractClientMessagingService now routes all remote calls through protected execute(HttpRequest) and openConnection(HttpMethod, String) methods, and DefaultClientMessagingService overrides both via a shared executeWithoutUserDetails helper that saves, clears (SecurityRequestContext.resetUserDetails()), and restores the caller thread's user details in finally. As a result, master services always authenticate to MessagingHttpService as the internal cdap service principal (CDAP-Internal), and task workers / preview runners when messaging.service.enabled=true authenticate as empty-worker-user-credential on worker-allowlisted system topics even when a user credential is active on the thread.

6. 2fa5ee6628b Harden messaging authorization for mixed-case system and placeholder credentials

  • Case-insensitive system gate (MessagingHandlerAuthorizer.isSystemNamespace): NamespaceId.SYSTEM.equals(namespaceId) and NamespaceId.isReserved(String) are case-sensitive (allowing a user with namespace-creation permission to create a mixed-case SYSTEM namespace via /v3), whereas on Spanner-backed TMS (SpannerMessagingService) per-topic tables are named messaging_{namespace}_{topic} and Cloud Spanner table names are case-insensitive. The gate now checks NamespaceId.SYSTEM.getNamespace().equalsIgnoreCase(namespaceId.getNamespace()) so mixed-case spellings (SYSTEM, System, sYsTeM) cannot bypass the internal-credential requirement. (NamespaceId.isReserved and Spanner TMS case handling are tracked separately outside this PR.)
  • Reject placeholder credentials with 403 before AccessEnforcer: When security.internal.auth.enabled=true, MessagingHandlerAuthorizer now rejects both CDAP-empty-user-credential (populated by AuthenticationChannelHandler when no header is sent) and empty-worker-user-credential (outside the allowlisted worker system publishes) with UnauthorizedException (403 Forbidden, with "Only internal services can access topics of the system namespace." on system and standard namespace UnauthorizedException on user namespaces) before calling AccessEnforcer, avoiding the Base64 IllegalArgumentException in InternalAccessEnforcer (which previously surfaced as 500 on AppFabric or 400 on MessagingHttpService).
  • Shared constants & configurable EncryptionExemptionHook: Placeholder credential constants (Constants.Security.Authentication) and preview topic constants (Constants.Preview.DEFAULT_MESSAGING_TOPIC, MESSAGING_PUBLISH_URI, LOG_TOPIC_PREFIX, LOG_NUM_PARTITIONS, LOG_TOPIC) are consolidated in Constants and shared across AuthenticationChannelHandler, WorkerAuthenticationContext, PreviewConfigModule, PreviewTMSLogSubscriber, MessagingHandlerAuthorizer, and EncryptionExemptionHook (which now accepts an optional CConfiguration so custom preview.messaging.topic values stay aligned between EncryptionExemptionHook and MessagingHandlerAuthorizer).

Behavior change and architectural notes to be aware of

  • HTTP callers of MetadataHandler, StoreHandler, and FetchHandler: Remote clients (ClientMessagingService) reach these handlers over HTTP whenever messaging.service.enabled=true (SQL-backed CDF instances and distributed open-source CDAP) or from preview and task worker pods (TaskWorkerClientMessagingService, PreviewRunnerClientMessagingService). With authorization or internal authentication enabled, callers must hold UPDATE (to create/update/delete/publish/store/rollback) or GET (to get/list/poll) on the target namespace, and only internal callers may access system namespace topics (with the scoped worker allowlist and system:preview publish exceptions described above). Services using DefaultClientMessagingService automatically clear SecurityRequestContext user details around each request so they authenticate with their service/worker identity when calling MessagingHttpService.
  • RemoteAccessEnforcer cache key: RemoteAccessEnforcer keys its cache on Principal, and Principal.equals/hashCode ignore fullCredential. On Spanner-backed instances where App Fabric hosts TMS in-process (InternalAccessEnforcer), every internal token is verified without caching. On services that use RemoteAccessEnforcer (the separate messaging pod when messaging.service.enabled=true, preview service, and Dataproc DefaultRuntimeJob), a request carrying a forged username and a garbage internal credential could match a cached allow entry for that user+permission within the cache TTL (300s).
  • Dataproc DefaultRuntimeJob: DefaultRuntimeJob inside Dataproc runs MessagingHttpService bound with MasterAuthenticationContext (the server's own fixed identity). There, the check adds no security boundary inside the Dataproc cluster and makes a cached remote enforcement call (RemoteAccessEnforcer, cache TTL 300s) to App Fabric roughly once every 5 minutes (verified end to end on live Dataproc batch pipeline runs on cdap-oct-13-ent-rbac).
  • Residual worker placeholder publish scope: A forged empty-worker-user-credential header from an in-cluster pod can still publish to auditlogevent0..N-1, metrics0..N-1, previewlog0, and preview, which includes forged data-plane audit events on auditlogevent* (control-plane topics audit, programstatusevent, metadata, dataevent, and logs0..N-1 are blocked with 403).
  • Open-source CDAP (security.internal.auth.enabled=false and messaging.service.enabled=true): CDF production enables security.internal.auth.enabled=true in every environment (server.pi:476-482); in open-source CDAP deployments where security.internal.auth.enabled=false, the internal-credential check is skipped and CDAP-UserId is trusted. When messaging.service.enabled=true in OSS, namespace deletion (DefaultNamespaceAdmin / DefaultNamespaceResourceDeleter) calls DefaultClientMessagingService, which clears SecurityRequestContext user details (9aefa3fba8c) so requests execute as the master service identity (MasterAuthenticationContext) rather than the calling end user; the master service identity (or any direct caller of MetadataHandler) must hold GET and UPDATE on the namespace when authorization is enabled.

Tests

  • cdap-tms:
    • MetadataHandlerAuthorizationTest (26 tests: every endpoint x allowed / denied / system and mixed-case SYSTEM (SYSTEM, System, sYsTeM) / internal credential / placeholder credentials (CDAP-empty-user-credential, empty-worker-user-credential) / null principal / authorization disabled / ordering of the check before body parsing and topic lookup / DefaultClientMessagingService clearing user context so SystemAuthenticationContext uses its internal identity on MetadataHandler, plus negative control with a direct AbstractClientMessagingService subclass).
    • StoreAndFetchHandlerAuthorizationTest (29 tests: publish, store, rollback, and poll across no permission, missing topic, missing Content-Type, invalid topic/namespace names, zero temp-file allocation on rejected requests, UPDATE vs GET separation, cross-namespace isolation, single enforcer check per request, enforcer failure handling, system and mixed-case SYSTEM internal-credential requirement, URL-encoded %73ystem, unauthenticated HTTP requests without headers via AuthenticationChannelHandler + InternalAccessEnforcer returning 403, worker placeholder empty-worker-user-credential allowlist on metrics0 / auditlogevent0 / previewlog0 / preview vs 403 rejection on audit / programstatusevent / metadata / dataevent and user namespaces, user credential and internal credential publish to system:preview vs rejection on other system topics and system:preview store/rollback/poll, empty/unset preview.messaging.topic and empty/unset metrics/audit topic prefixes, authorized round-trip publish + poll, and DefaultClientMessagingService clearing user context under both SystemAuthenticationContext and WorkerAuthenticationContext).
    • Whole module: 118 tests, 0 failures, checkstyle and license checks run.
  • cdap-common: EncryptionExemptionHookTest (16 parameterized tests including custom and empty preview.messaging.topic), AuthenticationChannelHandlerTest (16 tests) pass.
  • cdap-gateway: RouterPathLookupTest (39), RouterServiceLookupTest (3), NettyRouterPipelineTest, NettyRouterPipelineAuthTest, NettyRouterHttpTest, NettyRouterHttpsTest pass.
  • cdap-app-fabric & cdap-watchdog: injector-building and messaging tests (DefaultPreviewRequestQueueTest, PreviewRunnerServiceTest, PreviewRunnerTwillRunnableTest, DefaultPreviewStoreTest, DefaultRuntimeJobTest, RuntimeClientServerTest, RuntimeServiceRoutingTest, RuntimeClientServiceTest, TetheringServerHandlerTest, TetheringClientHandlerTest, DefaultNamespaceAdminTest, TetheringProvisionerTest, TetheringRuntimeJobManagerTest, TestTMSLogging, MessagingMetricsCollectionServiceTest, MetricsAdminSubscriberServiceTest) pass.
  • Also exercised end to end on Kubernetes deployments across both Spanner-backed Enterprise instances (messaging.service.enabled=false, RBAC cdap-oct-13-ent-rbac and non-RBAC cdap-oct-13-ent-nonrbac) and a SQL-backed Developer RBAC instance (data.storage.implementation=postgresql, messaging.service.enabled=true, cdap-oct-7-dev-rbac with dedicated cdap-cdap-oct-7-dev-rbac-messaging-0 pod): /v1/** through the router returns 503 for every method and encoded spelling; directly against appfabric, preview-0, and messaging-0, unauthenticated or non-internal callers are rejected across MetadataHandler, StoreHandler, and FetchHandler on system (except publish to system:preview with a non-empty credential) and require namespace permissions on user namespaces; empty-worker-user-credential is permitted only on the worker allowlist topics (metrics0, auditlogevent0, previewlog0, preview) and rejected with 403 on audit, programstatusevent, and logs0; and app deployment, workflow start/stop, app deletion, pipeline preview, and Dataproc batch pipeline runs complete end to end on both Spanner-backed and SQL-backed RBAC instances.

The router forwarded every path outside /v3 to app-fabric, including the
/v1 paths of the internal messaging, remote system operations and
impersonation APIs. Those handlers are meant to be reached through
service discovery only, so the router now answers DONT_ROUTE for them.

The check runs before the HTTP method is parsed: AllowedMethod.valueOf
throws for methods such as PATCH or OPTIONS, and the catch block routes
those requests to app-fabric, so a check placed after it could be
skipped by choosing one of those methods.

The router tests used /v1/ as an arbitrary first path segment for their
mock services, so they now use /test/. New tests pin that /v1 is not
routed for any HTTP method or spelling of the segment, and that other
paths which merely contain "v1" are still routed.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request restricts the routing of internal /v1 API paths in the gateway router and introduces namespace-level authorization checks to the messaging metadata handler. When internal authentication is enabled, only internal callers are permitted to access system namespace topics. Feedback on these changes includes checking for empty URI parts in RouterPathLookup to avoid using exceptions for control flow, and adding a defensive null check on the Principal object in MetadataHandler to prevent a potential NullPointerException.

Comment on lines +175 to +178
private static boolean hasInternalCredential(Principal principal) {
Credential credential = principal.getFullCredential();
return credential != null && credential.getType() == Credential.CredentialType.INTERNAL;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If principal is null (which can happen under certain security configurations or unauthenticated requests), calling principal.getFullCredential() will throw a NullPointerException. Adding a defensive null check on principal prevents this potential runtime exception.

  private static boolean hasInternalCredential(Principal principal) {
    if (principal == null) {
      return false;
    }
    Credential credential = principal.getFullCredential();
    return credential != null && credential.getType() == Credential.CredentialType.INTERNAL;
  }

The /v1/namespaces/{namespace}/topics API of MetadataHandler passed every
request straight to the MessagingService. Authorize the caller on the
namespace of the topic before anything else is done with the request:

* creating, updating and deleting a topic requires UPDATE on the namespace
* getting and listing topics requires GET on the namespace
* when internal authentication is enabled, the topics of the system
  namespace can only be accessed by callers that present an internal
  credential

Topics are not an entity type that authorization supports, which is why
the check is made on the namespace. When security is disabled the checks
are no-ops, so standalone and unsecured deployments behave as before.

Callers that manage topics over this API on behalf of an end user, for
example namespace deletion when messaging runs as a separate service, are
now authorized as that user.

MetadataHandler now delegates authorization to MessagingHandlerAuthorizer,
which injects AuthenticationContext and AccessEnforcer. The test injectors
that build it bind a no-op AuthenticationContext next to the no-op
enforcement module they already had.
…lers

The /v1/namespaces/{namespace}/topics/{topic} endpoints of StoreHandler
(publish, store, rollback) and FetchHandler (poll) passed every request
straight to the MessagingService. Authorize the caller on the namespace
of the topic via MessagingHandlerAuthorizer before any temporary file is
created or the request body is read:

* publishing, storing and rolling back messages requires UPDATE on the
  namespace
* polling messages requires GET on the namespace
* when internal authentication is enabled, the topics of the system
  namespace can only be accessed by callers that present an internal
  credential
* background publishing to the system namespace from a task worker or
  preview runner, which presents the worker placeholder internal
  credential (empty-worker-user-credential), is permitted on publish
  without calling the access enforcer
@shruthi713
shruthi713 force-pushed the fix/restrict-internal-v1-api branch from 3a160f9 to 4ebebf4 Compare October 29, 2026 06:09
When internal authentication is enabled, authorizePublish previously
allowed the worker placeholder credential (empty-worker-user-credential)
to publish to any topic in the system namespace, and rejected publish
requests to the preview topic from preview runners executing on a
preview request thread with the user's non-internal credential.

Tighten and complete the system-namespace publish rules in
MessagingHandlerAuthorizer:

* restrict background publishing with the worker placeholder credential
  to the configured worker system topics (metrics, audit log event,
  preview log, and preview topics)
* allow publishing to the configured preview topic in the system
  namespace when the caller presents a non-empty user credential
  (RemotePreviewRequestFetcher sets the user's external credential on
  the preview runner thread before DefaultPreviewRunner and
  DefaultDataTracer publish preview status and tracer data)
When messaging.service.enabled is true (such as on SQL-backed Cloud Data Fusion instances and distributed CDAP deployments), master services, task workers, and preview runners bind MessagingService to DefaultClientMessagingService. When a service calls DefaultClientMessagingService on a thread where SecurityRequestContext holds an end-user credential from an enclosing /v3 request, SystemAuthenticationContext and WorkerAuthenticationContext return that user's external principal rather than the service's internal principal, causing internal /v1 messaging calls to the system namespace to fail with 403.

Add protected execute and openConnection hooks in AbstractClientMessagingService and override them in DefaultClientMessagingService via a shared executeWithoutUserDetails helper that saves, clears, and restores the SecurityRequestContext user details around each remote request.
return serviceLookup.getDiscoverable(new DefaultHttpRequest(VERSION, method, uri));
}

private void assertNotRouted(HttpMethod method, String uri) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heads-up: #16294 stops routing /v3Internal (except two artifact GETs used by tethering) and makes refused paths return 404 instead of 503, since clients retry a 503. The /v1 block gets the 404 automatically, so the 503 in the PR description and tests becomes 404.

@HarshitJain09 HarshitJain09 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Shruthi, this is thorough and well tested.

But, Could we split it into two PRs?

  1. Router block for /v1: the RouterPathLookup change and its tests.
  2. Authorization in the TMS handlers: MessagingHandlerAuthorizer and the related client and auth changes.

Right now it's 30 files across 6 modules. Splitting would help in a few ways:

  1. The router block is small and closes the external route on its own, so it could merge and be backported quickly while the authorization part gets a closer look.
  2. Each PR would be easier to review and test, and either one could be rolled back without the other.
  3. The overlap with #16294 would stay in the small router PR.

Also, Please add the build label so the full build and Sonar run; both have been skipped so far.
And When you split it, could each PR be a single commit? The repo only allows merge commits, so otherwise all 7 commits land in develop history as they are.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants