Repository navigation
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a FileSystemPropertiesValidator to prevent users from overriding reserved Hadoop file system properties (such as GCS endpoints, token servers, and proxies) which could lead to security vulnerabilities. This validation and filtering logic is integrated into FileSink, FTPBatchSource, FileBatchSource, and FileSourceConfig, and is accompanied by comprehensive unit tests. A review comment points out that in FTPBatchSource.java, parsing fileSystemProperties as JSON during validation can crash if it contains a macro or is invalid JSON. It suggests guarding against macros and catching parsing exceptions to throw an IllegalArgumentException so validation can handle it gracefully.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces a FileSystemPropertiesValidator to validate and filter out reserved Hadoop configuration keys (such as GCS API endpoints, proxy addresses, and credentials) from user-supplied file system properties in file-based plugins, preventing security risks like credential leakage. It also adds comprehensive unit tests to verify this behavior. The review feedback correctly identifies potential NullPointerException risks in FileSink and FileBatchSource when wrapping the file system properties map in a HashMap if the properties map is null.
975aa17 to
0677c58
Compare
|
/gemini review |
1 similar comment
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces a FileSystemPropertiesValidator to prevent users from overriding reserved Hadoop configuration keys in file-based plugins, including FileSink, FTPBatchSource, and FileBatchSource. The feedback highlights potential pipeline validation crashes due to unhandled exceptions during JSON parsing in FTPBatchSource and FileSink. Additionally, the reviewer suggests refactoring removeReservedFileSystemProperties to be null-safe and return a new map, which would simplify the boilerplate code at multiple call sites.
| Map<String, String> fileSystemProperties = getFSProperties(); | ||
| FileSystemPropertiesValidator.validateFileSystemProperties(fileSystemProperties, NAME_FILE_SYSTEM_PROPERTIES, | ||
| collector); |
There was a problem hiding this comment.
In FileSink.java, getFSProperties() throws a ProgramFailureException (via ErrorUtils.getProgramFailureException) when JSON parsing fails. However, the catch block immediately following this try block (at line 104) only catches IllegalArgumentException. Since ProgramFailureException is not a subclass of IllegalArgumentException, any invalid JSON in fileSystemProperties will cause an unhandled exception and crash the pipeline validation instead of gracefully reporting a validation failure.
Please update the catch block at line 104 to catch Exception or RuntimeException to ensure all parsing failures are gracefully handled and reported to the collector.
There was a problem hiding this comment.
This is pre-existing: on develop, getFSProperties() already throws ProgramFailureException on invalid JSON inside the same try/catch (IllegalArgumentException) block - this change only adds the reserved-key validation call next to it. Keeping the scope to the reserved-key handling
There was a problem hiding this comment.
Code Review
This pull request introduces a FileSystemPropertiesValidator to validate and filter user-supplied Hadoop file system properties, preventing the overriding of reserved keys (such as GCS endpoints, proxies, and credentials) across FileSink, FTPBatchSource, FileBatchSource, and FileSourceConfig. The review feedback highlights a potential unhandled RuntimeException during JSON parsing in FileSink's validation, and suggests code improvements such as optimizing map operations in FTPBatchSource and adding a null check to FileSystemPropertiesValidator.removeReservedFileSystemProperties to prevent potential null pointer exceptions.
| Map<String, String> fileSystemProperties = getFSProperties(); | ||
| FileSystemPropertiesValidator.validateFileSystemProperties(fileSystemProperties, NAME_FILE_SYSTEM_PROPERTIES, |
There was a problem hiding this comment.
In FileSink.Conf, getFSProperties() throws a RuntimeException (via ErrorUtils.getProgramFailureException) when JSON parsing fails, rather than an IllegalArgumentException. Consequently, the catch (IllegalArgumentException e) block on line 104 will not catch parsing exceptions, causing validation to fail with an unhandled exception. Consider updating the catch block to catch RuntimeException instead.
There was a problem hiding this comment.
As noted in the thread below, this try/catch block and its exception handling are pre-existing behavior on the develop branch. I am keeping this PR strictly scoped to the reserved-key validation.
0677c58 to
69eb80e
Compare
0630c26 to
0199183
Compare
|
Putting One small structuring suggestion: |
The fileSystemProperties field of the File source, File sink and FTP source is copied into the Hadoop configuration used to open the file system. When the path points to Google Cloud Storage, the GCS connector reads the storage endpoint, token server, proxy, credential source, impersonation target and several implementation classes from that configuration, so a user could point the connector at an arbitrary host or identity, including during pipeline validation where it runs with the identity of the CDAP system service. Reserved keys are now rejected with a validation failure on the fileSystemProperties field, one per offending key. As a safeguard, reserved keys are also dropped where user properties are copied into the Hadoop configuration, so values resolved from macros at runtime cannot reach the connector either. The reserved key set is the same one used by the Google Cloud plugins.
0199183 to
9f52b73
Compare
Summary
The
fileSystemPropertiesfield of the File source, File sink and (deprecated) FTP source is copied into the Hadoop configuration used to open the file system. When the path points to Google Cloud Storage, the GCS connector reads the storage endpoint, token server, proxy, credential source, impersonation target and several implementation classes from that configuration, so a user could point the connector at an arbitrary host or identity, including during pipeline validation where it runs with the identity of the CDAP system service.Changes
format-common: newFileSystemPropertiesValidatorwith the reserved key set (GCS API endpoint and service path, token server, HTTP transport, proxy settings, allfs.gs.auth.*credential and impersonation keys, authorization handler and delegation token binding classes, thegoogle.cloud.*fallback prefix the connector also reads, the deprecated service account aliases, the FileSystem implementation keys and thecdap.auth.*keys used by the Google Cloud plugins). This is the same reserved set as in the Google Cloud plugins (data-integrations/google-cloud).FileSourceConfig.validate()/FileSink.Conf.validate(): reserved keys are reported as a validation failure attributed tofileSystemProperties, one per offending key. InconfigurePipelinethis runs before schema detection opens the file system.FileBatchSource/FileSink/FTPBatchSource: reserved keys are dropped where the user properties are copied into the Hadoop configuration, so values resolved from macros after validation cannot reach the connector either.Values are never inspected; a reserved key is rejected regardless of value. Other properties keep working unchanged.
Tests
FileSystemPropertiesValidatorTest(new): reserved / allowed keys, failure attribution, merge-point filtering.FileSourceConfigTest/FileSinkConfigTest(new):validate()rejects endpoint, fallback-prefix, impersonation and plugin-internal keys and attributes the failures tofileSystemProperties; allowed keys still pass; reserved keys never reach the Hadoop configuration.