Skip to content

Reject reserved Hadoop keys in File plugin fileSystemProperties - #2006

Open
akkaur wants to merge 1 commit into
developfrom
restrict-reserved-fs-properties
Open

akkaur wants to merge 1 commit into
developfrom
restrict-reserved-fs-properties

Conversation

@akkaur

@akkaur akkaur commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Summary

The fileSystemProperties field of the File source, File sink and (deprecated) FTP source is copied into the Hadoop configuration used to open the file system. When the path points to Google Cloud Storage, the GCS connector reads the storage endpoint, token server, proxy, credential source, impersonation target and several implementation classes from that configuration, so a user could point the connector at an arbitrary host or identity, including during pipeline validation where it runs with the identity of the CDAP system service.

Changes

  • format-common: new FileSystemPropertiesValidator with the reserved key set (GCS API endpoint and service path, token server, HTTP transport, proxy settings, all fs.gs.auth.* credential and impersonation keys, authorization handler and delegation token binding classes, the google.cloud.* fallback prefix the connector also reads, the deprecated service account aliases, the FileSystem implementation keys and the cdap.auth.* keys used by the Google Cloud plugins). This is the same reserved set as in the Google Cloud plugins (data-integrations/google-cloud).
  • FileSourceConfig.validate() / FileSink.Conf.validate(): reserved keys are reported as a validation failure attributed to fileSystemProperties, one per offending key. In configurePipeline this runs before schema detection opens the file system.
  • FileBatchSource / FileSink / FTPBatchSource: reserved keys are dropped where the user properties are copied into the Hadoop configuration, so values resolved from macros after validation cannot reach the connector either.

Values are never inspected; a reserved key is rejected regardless of value. Other properties keep working unchanged.

Tests

  • FileSystemPropertiesValidatorTest (new): reserved / allowed keys, failure attribution, merge-point filtering.
  • FileSourceConfigTest / FileSinkConfigTest (new): validate() rejects endpoint, fallback-prefix, impersonation and plugin-internal keys and attributes the failures to fileSystemProperties; allowed keys still pass; reserved keys never reach the Hadoop configuration.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a FileSystemPropertiesValidator to prevent users from overriding reserved Hadoop file system properties (such as GCS endpoints, token servers, and proxies) which could lead to security vulnerabilities. This validation and filtering logic is integrated into FileSink, FTPBatchSource, FileBatchSource, and FileSourceConfig, and is accompanied by comprehensive unit tests. A review comment points out that in FTPBatchSource.java, parsing fileSystemProperties as JSON during validation can crash if it contains a macro or is invalid JSON. It suggests guarding against macros and catching parsing exceptions to throw an IllegalArgumentException so validation can handle it gracefully.

@akkaur
akkaur requested a review from sahusanket October 16, 2026 07:19
@akkaur

akkaur commented Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a FileSystemPropertiesValidator to validate and filter out reserved Hadoop configuration keys (such as GCS API endpoints, proxy addresses, and credentials) from user-supplied file system properties in file-based plugins, preventing security risks like credential leakage. It also adds comprehensive unit tests to verify this behavior. The review feedback correctly identifies potential NullPointerException risks in FileSink and FileBatchSource when wrapping the file system properties map in a HashMap if the properties map is null.

@akkaur
akkaur force-pushed the restrict-reserved-fs-properties branch from 975aa17 to 0677c58 Compare October 16, 2026 07:37
@akkaur

akkaur commented Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

/gemini review

1 similar comment
@akkaur

akkaur commented Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a FileSystemPropertiesValidator to prevent users from overriding reserved Hadoop configuration keys in file-based plugins, including FileSink, FTPBatchSource, and FileBatchSource. The feedback highlights potential pipeline validation crashes due to unhandled exceptions during JSON parsing in FTPBatchSource and FileSink. Additionally, the reviewer suggests refactoring removeReservedFileSystemProperties to be null-safe and return a new map, which would simplify the boilerplate code at multiple call sites.

Comment on lines +101 to +103
Map<String, String> fileSystemProperties = getFSProperties();
FileSystemPropertiesValidator.validateFileSystemProperties(fileSystemProperties, NAME_FILE_SYSTEM_PROPERTIES,
collector);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

In FileSink.java, getFSProperties() throws a ProgramFailureException (via ErrorUtils.getProgramFailureException) when JSON parsing fails. However, the catch block immediately following this try block (at line 104) only catches IllegalArgumentException. Since ProgramFailureException is not a subclass of IllegalArgumentException, any invalid JSON in fileSystemProperties will cause an unhandled exception and crash the pipeline validation instead of gracefully reporting a validation failure.

Please update the catch block at line 104 to catch Exception or RuntimeException to ensure all parsing failures are gracefully handled and reported to the collector.

@akkaur akkaur Oct 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is pre-existing: on develop, getFSProperties() already throws ProgramFailureException on invalid JSON inside the same try/catch (IllegalArgumentException) block - this change only adds the reserved-key validation call next to it. Keeping the scope to the reserved-key handling

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a FileSystemPropertiesValidator to validate and filter user-supplied Hadoop file system properties, preventing the overriding of reserved keys (such as GCS endpoints, proxies, and credentials) across FileSink, FTPBatchSource, FileBatchSource, and FileSourceConfig. The review feedback highlights a potential unhandled RuntimeException during JSON parsing in FileSink's validation, and suggests code improvements such as optimizing map operations in FTPBatchSource and adding a null check to FileSystemPropertiesValidator.removeReservedFileSystemProperties to prevent potential null pointer exceptions.

Comment on lines +101 to +102
Map<String, String> fileSystemProperties = getFSProperties();
FileSystemPropertiesValidator.validateFileSystemProperties(fileSystemProperties, NAME_FILE_SYSTEM_PROPERTIES,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

In FileSink.Conf, getFSProperties() throws a RuntimeException (via ErrorUtils.getProgramFailureException) when JSON parsing fails, rather than an IllegalArgumentException. Consequently, the catch (IllegalArgumentException e) block on line 104 will not catch parsing exceptions, causing validation to fail with an unhandled exception. Consider updating the catch block to catch RuntimeException instead.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As noted in the thread below, this try/catch block and its exception handling are pre-existing behavior on the develop branch. I am keeping this PR strictly scoped to the reserved-key validation.

@akkaur
akkaur force-pushed the restrict-reserved-fs-properties branch from 0677c58 to 69eb80e Compare October 16, 2026 08:21
@akkaur
akkaur marked this pull request as ready for review October 16, 2026 09:02
@akkaur akkaur added the build Trigger unit test build label Nov 16, 2026
@akkaur
akkaur force-pushed the restrict-reserved-fs-properties branch 2 times, most recently from 0630c26 to 0199183 Compare November 16, 2026 17:05
@sahusanket

Copy link
Copy Markdown
Contributor

Putting FileSystemPropertiesValidator in format-common makes a lot of sense as the shared home for this check, especially since data-integrations/google-cloud#1629 currently duplicates the same reserved keys and validateFileSystemProperties logic in GCPUtils (because google-cloud pins format-common 2.13.0).

One small structuring suggestion: FileSystemPropertiesValidator could also expose a helper to parse and validate a fileSystemProperties JSON string, so FileSourceConfig, FileSink.Conf (and eventually the GCS configs once cdap.plugin.version is bumped) do not each repeat the Gson parsing and FailureCollector attribution boilerplate.

The fileSystemProperties field of the File source, File sink and FTP
source is copied into the Hadoop configuration used to open the file
system. When the path points to Google Cloud Storage, the GCS connector
reads the storage endpoint, token server, proxy, credential source,
impersonation target and several implementation classes from that
configuration, so a user could point the connector at an arbitrary
host or identity, including during pipeline validation where it runs
with the identity of the CDAP system service.

Reserved keys are now rejected with a validation failure on the
fileSystemProperties field, one per offending key. As a safeguard,
reserved keys are also dropped where user properties are copied into
the Hadoop configuration, so values resolved from macros at runtime
cannot reach the connector either. The reserved key set is the same
one used by the Google Cloud plugins.
@akkaur
akkaur force-pushed the restrict-reserved-fs-properties branch from 0199183 to 9f52b73 Compare November 19, 2026 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build Trigger unit test build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants