Repository navigation
Restrict access to the internal /v1 messaging API - #16252
shruthi713 wants to merge 7 commits into
Conversation
The router forwarded every path outside /v3 to app-fabric, including the /v1 paths of the internal messaging, remote system operations and impersonation APIs. Those handlers are meant to be reached through service discovery only, so the router now answers DONT_ROUTE for them. The check runs before the HTTP method is parsed: AllowedMethod.valueOf throws for methods such as PATCH or OPTIONS, and the catch block routes those requests to app-fabric, so a check placed after it could be skipped by choosing one of those methods. The router tests used /v1/ as an arbitrary first path segment for their mock services, so they now use /test/. New tests pin that /v1 is not routed for any HTTP method or spelling of the segment, and that other paths which merely contain "v1" are still routed.
There was a problem hiding this comment.
Code Review
This pull request restricts the routing of internal /v1 API paths in the gateway router and introduces namespace-level authorization checks to the messaging metadata handler. When internal authentication is enabled, only internal callers are permitted to access system namespace topics. Feedback on these changes includes checking for empty URI parts in RouterPathLookup to avoid using exceptions for control flow, and adding a defensive null check on the Principal object in MetadataHandler to prevent a potential NullPointerException.
| private static boolean hasInternalCredential(Principal principal) { | ||
| Credential credential = principal.getFullCredential(); | ||
| return credential != null && credential.getType() == Credential.CredentialType.INTERNAL; | ||
| } |
There was a problem hiding this comment.
If principal is null (which can happen under certain security configurations or unauthenticated requests), calling principal.getFullCredential() will throw a NullPointerException. Adding a defensive null check on principal prevents this potential runtime exception.
private static boolean hasInternalCredential(Principal principal) {
if (principal == null) {
return false;
}
Credential credential = principal.getFullCredential();
return credential != null && credential.getType() == Credential.CredentialType.INTERNAL;
}The /v1/namespaces/{namespace}/topics API of MetadataHandler passed every
request straight to the MessagingService. Authorize the caller on the
namespace of the topic before anything else is done with the request:
* creating, updating and deleting a topic requires UPDATE on the namespace
* getting and listing topics requires GET on the namespace
* when internal authentication is enabled, the topics of the system
namespace can only be accessed by callers that present an internal
credential
Topics are not an entity type that authorization supports, which is why
the check is made on the namespace. When security is disabled the checks
are no-ops, so standalone and unsecured deployments behave as before.
Callers that manage topics over this API on behalf of an end user, for
example namespace deletion when messaging runs as a separate service, are
now authorized as that user.
MetadataHandler now delegates authorization to MessagingHandlerAuthorizer,
which injects AuthenticationContext and AccessEnforcer. The test injectors
that build it bind a no-op AuthenticationContext next to the no-op
enforcement module they already had.
…lers
The /v1/namespaces/{namespace}/topics/{topic} endpoints of StoreHandler
(publish, store, rollback) and FetchHandler (poll) passed every request
straight to the MessagingService. Authorize the caller on the namespace
of the topic via MessagingHandlerAuthorizer before any temporary file is
created or the request body is read:
* publishing, storing and rolling back messages requires UPDATE on the
namespace
* polling messages requires GET on the namespace
* when internal authentication is enabled, the topics of the system
namespace can only be accessed by callers that present an internal
credential
* background publishing to the system namespace from a task worker or
preview runner, which presents the worker placeholder internal
credential (empty-worker-user-credential), is permitted on publish
without calling the access enforcer
3a160f9 to
4ebebf4
Compare
When internal authentication is enabled, authorizePublish previously allowed the worker placeholder credential (empty-worker-user-credential) to publish to any topic in the system namespace, and rejected publish requests to the preview topic from preview runners executing on a preview request thread with the user's non-internal credential. Tighten and complete the system-namespace publish rules in MessagingHandlerAuthorizer: * restrict background publishing with the worker placeholder credential to the configured worker system topics (metrics, audit log event, preview log, and preview topics) * allow publishing to the configured preview topic in the system namespace when the caller presents a non-empty user credential (RemotePreviewRequestFetcher sets the user's external credential on the preview runner thread before DefaultPreviewRunner and DefaultDataTracer publish preview status and tracer data)
When messaging.service.enabled is true (such as on SQL-backed Cloud Data Fusion instances and distributed CDAP deployments), master services, task workers, and preview runners bind MessagingService to DefaultClientMessagingService. When a service calls DefaultClientMessagingService on a thread where SecurityRequestContext holds an end-user credential from an enclosing /v3 request, SystemAuthenticationContext and WorkerAuthenticationContext return that user's external principal rather than the service's internal principal, causing internal /v1 messaging calls to the system namespace to fail with 403. Add protected execute and openConnection hooks in AbstractClientMessagingService and override them in DefaultClientMessagingService via a shared executeWithoutUserDetails helper that saves, clears, and restores the SecurityRequestContext user details around each remote request.
8f89705 to
2fa5ee6
Compare
| return serviceLookup.getDiscoverable(new DefaultHttpRequest(VERSION, method, uri)); | ||
| } | ||
|
|
||
| private void assertNotRouted(HttpMethod method, String uri) { |
There was a problem hiding this comment.
Heads-up: #16294 stops routing /v3Internal (except two artifact GETs used by tethering) and makes refused paths return 404 instead of 503, since clients retry a 503. The /v1 block gets the 404 automatically, so the 503 in the PR description and tests becomes 404.
There was a problem hiding this comment.
Thanks Shruthi, this is thorough and well tested.
But, Could we split it into two PRs?
- Router block for /v1: the RouterPathLookup change and its tests.
- Authorization in the TMS handlers: MessagingHandlerAuthorizer and the related client and auth changes.
Right now it's 30 files across 6 modules. Splitting would help in a few ways:
- The router block is small and closes the external route on its own, so it could merge and be backported quickly while the authorization part gets a closer look.
- Each PR would be easier to review and test, and either one could be rolled back without the other.
- The overlap with #16294 would stay in the small router PR.
Also, Please add the build label so the full build and Sonar run; both have been skipped so far.
And When you split it, could each PR be a single commit? The repo only allows merge commits, so otherwise all 7 commits land in develop history as they are.
Six logical changes, each in its own commit so it can be reviewed and reverted independently:
1.
a1373ceec77(+ 104028771cf,2fa5ee6628b) Do not route the internal /v1 API in the routerRouterPathLookupforwarded every path outside/v3to app-fabric, including the/v1paths of the internal messaging, remote system operations and impersonation APIs. Those handlers are meant to be reached through service discovery only. The router now answersDONT_ROUTE(HTTP 503) for any request whose first path segment isv1.AllowedMethod.valueOfthrows for methods such asPATCHorOPTIONS, and thecatchblock routes those to app-fabric, so a check placed after it could be skipped by picking one of those methods.v1x,V1,/x/v1/...and/v3/.../v1/...are routed as before./%76%31/...) and repeated slashes (///v1/...,/v1//...) are normalised byRouterServiceLookup(URI.create(uri).normalize().getPath()) before the lookup, which is the same normalisation the downstreamnetty-httpHttpResourceHandlerapplies. (A double-leading slash//v1/...is parsed by RFC 2396URI.createin bothRouterServiceLookupandHttpResourceHandlerwith"v1"as the URI authority and"/namespaces/..."as the path, so it never resolves to/v1/...on the backend either.) This is covered byRouterServiceLookupTest./v1/as an arbitrary first path segment for their mock services. They now use/test/.2.
349a35acce8(+ 104028771cf) Enforce namespace authorization in the messaging topic metadata handlerMetadataHandler(/v1/namespaces/{namespace}/topics...) passed every request straight toMessagingService. Every endpoint now authorizes the caller first (via a package-privateMessagingHandlerAuthorizer), before the request body is parsed or the topic is looked up, so a caller without permission gets the same answer whether or not the topic exists:PUT), delete topicUPDATEon the namespaceGETon the namespacesystemnamespace (case-insensitive), whensecurity.internal.auth.enabled=trueStandardPermission.UPDATE, matchingDraftHandlerandPreferencesHttpHandler), rather than creating or deleting the namespace itself.AccessEnforceris a no-op, so standalone and unsecured deployments behave as before.MessagingHandlerAuthorizerinjectsCConfiguration,AccessEnforcerandAuthenticationContext. It readssecurity.internal.auth.enabledfromCConfigurationdirectly (same predicate asSecurityUtil.isInternalAuthEnabled) becausecdap-securityis only a test-scope dependency ofcdap-tms.AuthenticationContextnext to the no-op enforcement module they already used.3.
4ebebf48c27Enforce namespace authorization in the messaging store and fetch handlersStoreHandler(POST /v1/namespaces/{namespace}/topics/{topic}/publish,/store,/rollback) andFetchHandler(POST /v1/namespaces/{namespace}/topics/{topic}/poll) passed every request straight toMessagingService(creating a temporary spill file inStoreHandler.publishandStoreHandler.storebefore reading the body). Both handlers now injectMessagingHandlerAuthorizerand authorize the caller on the namespace of the topic before any temporary file is created or the request body is read. Validating the namespace and topic IDs and enforcing authorization beforeFiles.createTempFileinStoreHandler.publishandStoreHandler.storealso ensures that unauthorized or malformed requests never allocate temporary files on disk.4.
d28a4ce55beScope system topic publish exceptions to worker and preview topicsRefines
MessagingHandlerAuthorizer.authorizePublish(String namespace, String topic)so thatsystemnamespace publish exceptions are tightly scoped to the exact topics that task workers and preview runners publish to over HTTP:publish,store,rollback(StoreHandler)UPDATEon the namespacepoll(FetchHandler)GETon the namespacesystemnamespace (case-insensitive), whensecurity.internal.auth.enabled=truepublishin thesystemnamespace with the worker placeholder internal credential (empty-worker-user-credential), whensecurity.internal.auth.enabled=trueAccessEnforceronly when the topic is in the worker allowlist (metrics0..N-1,auditlogevent0..N-1,previewlog0,preview); rejected with403on all othersystemtopics (e.g.audit,programstatusevent,metadata,dataevent,logs0..N-1)publishto the configured preview topic (preview.messaging.topic, defaultpreview) in thesystemnamespace with a non-empty user credential, whensecurity.internal.auth.enabled=trueAccessEnforcersoPreviewRunnerService/MessagingPreviewDataPublishercan publish preview status and tracer data whileSecurityRequestContextholds the previewing user'sEXTERNALcredentialWorkerAuthenticationContext, which suppliesempty-worker-user-credential(CredentialType.INTERNAL) when background threads (MessagingMetricsCollectionService,MessagingAuditLogWriter,PreviewTMSLogAppender, andDefaultPreviewRunner.startUp()) publish outside a user request. Scoping this exception toworkerSystemTopicsprevents a caller carryingempty-worker-user-credentialfrom spoofing control-plane events onsystem:programstatusevent,system:audit,system:metadata,system:dataevent, etc.RemotePreviewRequestFetcher.fetch()setsSecurityRequestContexton thePreviewRunnerServicethread to the previewing user's principal (CredentialType.EXTERNAL), which is also inherited by child preview threads viaInheritableThreadLocal. Allowing a caller with a non-empty credential to publish only tosystem:preview(configured viapreview.messaging.topic) (while rejectingnull/ empty / placeholder credentials and rejecting all othersystemtopics) preserves end-to-end pipeline preview on RBAC instances.5.
9aefa3fba8cClear user details inDefaultClientMessagingServicerequestsWhen
messaging.service.enabled=true(such as on SQL-backed CDF Developer/Basic RBAC instances whereMessagingHttpServiceruns in a dedicatedmessaging-0pod, and in distributed open-source CDAP deployments):appfabric,appfabricprocessor,metadata,metrics,logs,runtime) bindMessagingServicetoDefaultClientMessagingService(MessagingClientModule) withSystemAuthenticationContext. When an external/v3request (e.g. deploying an app or starting/stopping a program) runs on a handler thread whereAuthenticationChannelHandlerhas populatedSecurityRequestContextwith the calling user'sEXTERNALcredential,SystemAuthenticationContext.getPrincipal()returns that user'sEXTERNALprincipal wheneverSecurityRequestContext.getUserId() != null, which previously caused internal/v1messaging calls toNamespaceId.SYSTEMto fail with403 Only internal services can access topics of the system namespace.TaskWorkerMessagingClientModule,PreviewRunnerMessagingClientModule) also bindMessagingServicetoDefaultClientMessagingServicepaired withWorkerAuthenticationContextwhenmessaging.service.enabled=true(whereas they bindTaskWorkerClientMessagingService/PreviewRunnerClientMessagingService, which subclassAbstractClientMessagingServicedirectly, whenmessaging.service.enabled=false).AbstractClientMessagingServicenow routes all remote calls through protectedexecute(HttpRequest)andopenConnection(HttpMethod, String)methods, andDefaultClientMessagingServiceoverrides both via a sharedexecuteWithoutUserDetailshelper that saves, clears (SecurityRequestContext.resetUserDetails()), and restores the caller thread's user details infinally. As a result, master services always authenticate toMessagingHttpServiceas the internalcdapservice principal (CDAP-Internal), and task workers / preview runners whenmessaging.service.enabled=trueauthenticate asempty-worker-user-credentialon worker-allowlisted system topics even when a user credential is active on the thread.6.
2fa5ee6628bHarden messaging authorization for mixed-casesystemand placeholder credentialssystemgate (MessagingHandlerAuthorizer.isSystemNamespace):NamespaceId.SYSTEM.equals(namespaceId)andNamespaceId.isReserved(String)are case-sensitive (allowing a user with namespace-creation permission to create a mixed-caseSYSTEMnamespace via/v3), whereas on Spanner-backed TMS (SpannerMessagingService) per-topic tables are namedmessaging_{namespace}_{topic}and Cloud Spanner table names are case-insensitive. The gate now checksNamespaceId.SYSTEM.getNamespace().equalsIgnoreCase(namespaceId.getNamespace())so mixed-case spellings (SYSTEM,System,sYsTeM) cannot bypass the internal-credential requirement. (NamespaceId.isReservedand Spanner TMS case handling are tracked separately outside this PR.)403beforeAccessEnforcer: Whensecurity.internal.auth.enabled=true,MessagingHandlerAuthorizernow rejects bothCDAP-empty-user-credential(populated byAuthenticationChannelHandlerwhen no header is sent) andempty-worker-user-credential(outside the allowlisted worker system publishes) withUnauthorizedException(403 Forbidden, with"Only internal services can access topics of the system namespace."onsystemand standard namespaceUnauthorizedExceptionon user namespaces) before callingAccessEnforcer, avoiding the Base64IllegalArgumentExceptioninInternalAccessEnforcer(which previously surfaced as500on AppFabric or400onMessagingHttpService).EncryptionExemptionHook: Placeholder credential constants (Constants.Security.Authentication) and preview topic constants (Constants.Preview.DEFAULT_MESSAGING_TOPIC,MESSAGING_PUBLISH_URI,LOG_TOPIC_PREFIX,LOG_NUM_PARTITIONS,LOG_TOPIC) are consolidated inConstantsand shared acrossAuthenticationChannelHandler,WorkerAuthenticationContext,PreviewConfigModule,PreviewTMSLogSubscriber,MessagingHandlerAuthorizer, andEncryptionExemptionHook(which now accepts an optionalCConfigurationso custompreview.messaging.topicvalues stay aligned betweenEncryptionExemptionHookandMessagingHandlerAuthorizer).Behavior change and architectural notes to be aware of
MetadataHandler,StoreHandler, andFetchHandler: Remote clients (ClientMessagingService) reach these handlers over HTTP whenevermessaging.service.enabled=true(SQL-backed CDF instances and distributed open-source CDAP) or from preview and task worker pods (TaskWorkerClientMessagingService,PreviewRunnerClientMessagingService). With authorization or internal authentication enabled, callers must holdUPDATE(to create/update/delete/publish/store/rollback) orGET(to get/list/poll) on the target namespace, and only internal callers may accesssystemnamespace topics (with the scoped worker allowlist andsystem:previewpublish exceptions described above). Services usingDefaultClientMessagingServiceautomatically clearSecurityRequestContextuser details around each request so they authenticate with their service/worker identity when callingMessagingHttpService.RemoteAccessEnforcercache key:RemoteAccessEnforcerkeys its cache onPrincipal, andPrincipal.equals/hashCodeignorefullCredential. On Spanner-backed instances where App Fabric hosts TMS in-process (InternalAccessEnforcer), every internal token is verified without caching. On services that useRemoteAccessEnforcer(the separatemessagingpod whenmessaging.service.enabled=true,previewservice, and DataprocDefaultRuntimeJob), a request carrying a forged username and a garbage internal credential could match a cached allow entry for that user+permission within the cache TTL (300s).DefaultRuntimeJob:DefaultRuntimeJobinside Dataproc runsMessagingHttpServicebound withMasterAuthenticationContext(the server's own fixed identity). There, the check adds no security boundary inside the Dataproc cluster and makes a cached remote enforcement call (RemoteAccessEnforcer, cache TTL300s) to App Fabric roughly once every 5 minutes (verified end to end on live Dataproc batch pipeline runs oncdap-oct-13-ent-rbac).empty-worker-user-credentialheader from an in-cluster pod can still publish toauditlogevent0..N-1,metrics0..N-1,previewlog0, andpreview, which includes forged data-plane audit events onauditlogevent*(control-plane topicsaudit,programstatusevent,metadata,dataevent, andlogs0..N-1are blocked with403).security.internal.auth.enabled=falseandmessaging.service.enabled=true): CDF production enablessecurity.internal.auth.enabled=truein every environment (server.pi:476-482); in open-source CDAP deployments wheresecurity.internal.auth.enabled=false, the internal-credential check is skipped andCDAP-UserIdis trusted. Whenmessaging.service.enabled=truein OSS, namespace deletion (DefaultNamespaceAdmin/DefaultNamespaceResourceDeleter) callsDefaultClientMessagingService, which clearsSecurityRequestContextuser details (9aefa3fba8c) so requests execute as the master service identity (MasterAuthenticationContext) rather than the calling end user; the master service identity (or any direct caller ofMetadataHandler) must holdGETandUPDATEon the namespace when authorization is enabled.Tests
cdap-tms:MetadataHandlerAuthorizationTest(26 tests: every endpoint x allowed / denied /systemand mixed-caseSYSTEM(SYSTEM,System,sYsTeM) / internal credential / placeholder credentials (CDAP-empty-user-credential,empty-worker-user-credential) / null principal / authorization disabled / ordering of the check before body parsing and topic lookup /DefaultClientMessagingServiceclearing user context soSystemAuthenticationContextuses its internal identity onMetadataHandler, plus negative control with a directAbstractClientMessagingServicesubclass).StoreAndFetchHandlerAuthorizationTest(29 tests:publish,store,rollback, andpollacross no permission, missing topic, missingContent-Type, invalid topic/namespace names, zero temp-file allocation on rejected requests,UPDATEvsGETseparation, cross-namespace isolation, single enforcer check per request, enforcer failure handling,systemand mixed-caseSYSTEMinternal-credential requirement, URL-encoded%73ystem, unauthenticated HTTP requests without headers viaAuthenticationChannelHandler+InternalAccessEnforcerreturning403, worker placeholderempty-worker-user-credentialallowlist onmetrics0/auditlogevent0/previewlog0/previewvs403rejection onaudit/programstatusevent/metadata/dataeventand user namespaces, user credential and internal credential publish tosystem:previewvs rejection on othersystemtopics andsystem:previewstore/rollback/poll, empty/unsetpreview.messaging.topicand empty/unset metrics/audit topic prefixes, authorized round-trippublish+poll, andDefaultClientMessagingServiceclearing user context under bothSystemAuthenticationContextandWorkerAuthenticationContext).cdap-common:EncryptionExemptionHookTest(16 parameterized tests including custom and emptypreview.messaging.topic),AuthenticationChannelHandlerTest(16 tests) pass.cdap-gateway:RouterPathLookupTest(39),RouterServiceLookupTest(3),NettyRouterPipelineTest,NettyRouterPipelineAuthTest,NettyRouterHttpTest,NettyRouterHttpsTestpass.cdap-app-fabric&cdap-watchdog: injector-building and messaging tests (DefaultPreviewRequestQueueTest,PreviewRunnerServiceTest,PreviewRunnerTwillRunnableTest,DefaultPreviewStoreTest,DefaultRuntimeJobTest,RuntimeClientServerTest,RuntimeServiceRoutingTest,RuntimeClientServiceTest,TetheringServerHandlerTest,TetheringClientHandlerTest,DefaultNamespaceAdminTest,TetheringProvisionerTest,TetheringRuntimeJobManagerTest,TestTMSLogging,MessagingMetricsCollectionServiceTest,MetricsAdminSubscriberServiceTest) pass.messaging.service.enabled=false, RBACcdap-oct-13-ent-rbacand non-RBACcdap-oct-13-ent-nonrbac) and a SQL-backed Developer RBAC instance (data.storage.implementation=postgresql,messaging.service.enabled=true,cdap-oct-7-dev-rbacwith dedicatedcdap-cdap-oct-7-dev-rbac-messaging-0pod):/v1/**through the router returns 503 for every method and encoded spelling; directly againstappfabric,preview-0, andmessaging-0, unauthenticated or non-internal callers are rejected acrossMetadataHandler,StoreHandler, andFetchHandleronsystem(exceptpublishtosystem:previewwith a non-empty credential) and require namespace permissions on user namespaces;empty-worker-user-credentialis permitted only on the worker allowlist topics (metrics0,auditlogevent0,previewlog0,preview) and rejected with 403 onaudit,programstatusevent, andlogs0; and app deployment, workflow start/stop, app deletion, pipeline preview, and Dataproc batch pipeline runs complete end to end on both Spanner-backed and SQL-backed RBAC instances.