Repository navigation
[cherrypick][6.11] Reject reserved Hadoop keys in GCS fileSystemProperties - #1640
Merged
akkaur merged 1 commit intoNov 20, 2026
Merged
Conversation
The fileSystemProperties field of the GCS source and sink is copied into the Hadoop configuration used by the GCS connector. The connector reads the storage endpoint, token server, proxy, credential source, impersonation target and several implementation classes from that configuration, so a user could point the connector at an arbitrary host or identity, including during pipeline validation where it runs with the identity of the CDAP system service. Reserved keys are now rejected with a validation failure on the fileSystemProperties field, one per offending key. As a safeguard, reserved keys are also dropped where user properties are merged with the properties set by the plugin, so values resolved from macros at runtime cannot reach the connector either. (cherry picked from commit 39702b4)
There was a problem hiding this comment.
Code Review
This pull request introduces security enhancements to prevent users from overriding critical Hadoop configuration keys through the fileSystemProperties plugin property. It defines a set of reserved keys and prefixes in GCPUtils (such as endpoints, proxies, and authentication configurations) and adds validation and filtering mechanisms to reject or strip these properties during validation and runtime. These changes are integrated into GCSBatchSink and GCSSource, and are accompanied by comprehensive unit tests. There are no review comments, and we have no additional feedback to provide.
riyaa14
self-requested a review
November 20, 2026 11:03
riyaa14
approved these changes
Nov 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #1629 (commit
39702b40b0c7e3c314af59534f7a7e2c96050ae6, merge commit6c654af5f7d5ae357436dfce1c7a1e4f45aa3daa) intorelease/0.24for CDAP 6.11.Summary
The
fileSystemPropertiesfield of the GCS source (GCSFile) and GCS sink (GCS) is copied into the Hadoop configuration used by the GCS connector. The connector reads the storage endpoint, token server, proxy, credential source, impersonation target and several implementation classes from that configuration, so a user could point the connector at an arbitrary host or identity, including during pipeline validation where it runs with the identity of the CDAP system service.Changes
GCPUtils:RESERVED_FS_PROPERTY_KEYS/RESERVED_FS_PROPERTY_PREFIXES,isReservedFileSystemProperty()andvalidateFileSystemProperties(). The reserved set covers the GCS API endpoint and service path, token server, HTTP transport, proxy settings, allfs.gs.auth.*credential and impersonation keys, authorization handler and delegation token binding classes, thegoogle.cloud.*fallback prefix the connector also reads, the deprecated service account aliases, the FileSystem implementation keys and the plugin-internalcdap.auth.*keys.GCSSourceConfig.validate()/GCSBatchSinkConfig.validate(): reserved keys are reported as a validation failure attributed tofileSystemProperties, one per offending key. InconfigurePipelinethis runs before any file system is opened.GCPUtils.getFileSystemProperties()and the sink's user-property merge: reserved keys are dropped as a safeguard so values resolved from macros after validation cannot reach the Hadoop configuration either. Precedence of all other keys is unchanged.Values are never inspected; a reserved key is rejected regardless of value. Tuning keys such as
fs.gs.inputstream.*,fs.gs.outputstream.*andmapreduce.*keep working.Tests
GCPUtilsFileSystemPropertiesTest(new): reserved / allowed keys, null and whitespace handling, failure attribution, merge-point filtering with anauto-detectconnection.GCSSourceConfigTest(new) andGCSBatchSinkTest: end-to-endvalidate()for endpoint, token server, fallback prefix, impersonation and plugin-internal keys; allowed keys still pass.