Skip to content

[cherrypick][6.11] Reject reserved Hadoop keys in GCS fileSystemProperties - #1640

Merged
akkaur merged 1 commit into
release/0.24from
cherry-pick-restrict-reserved-fs-properties-0.24
Nov 20, 2026
Merged

akkaur merged 1 commit into
release/0.24from
cherry-pick-restrict-reserved-fs-properties-0.24

Conversation

@akkaur

@akkaur akkaur commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Cherry-pick of #1629 (commit 39702b40b0c7e3c314af59534f7a7e2c96050ae6, merge commit 6c654af5f7d5ae357436dfce1c7a1e4f45aa3daa) into release/0.24 for CDAP 6.11.


Summary

The fileSystemProperties field of the GCS source (GCSFile) and GCS sink (GCS) is copied into the Hadoop configuration used by the GCS connector. The connector reads the storage endpoint, token server, proxy, credential source, impersonation target and several implementation classes from that configuration, so a user could point the connector at an arbitrary host or identity, including during pipeline validation where it runs with the identity of the CDAP system service.

Changes

  • GCPUtils: RESERVED_FS_PROPERTY_KEYS / RESERVED_FS_PROPERTY_PREFIXES, isReservedFileSystemProperty() and validateFileSystemProperties(). The reserved set covers the GCS API endpoint and service path, token server, HTTP transport, proxy settings, all fs.gs.auth.* credential and impersonation keys, authorization handler and delegation token binding classes, the google.cloud.* fallback prefix the connector also reads, the deprecated service account aliases, the FileSystem implementation keys and the plugin-internal cdap.auth.* keys.
  • GCSSourceConfig.validate() / GCSBatchSinkConfig.validate(): reserved keys are reported as a validation failure attributed to fileSystemProperties, one per offending key. In configurePipeline this runs before any file system is opened.
  • GCPUtils.getFileSystemProperties() and the sink's user-property merge: reserved keys are dropped as a safeguard so values resolved from macros after validation cannot reach the Hadoop configuration either. Precedence of all other keys is unchanged.

Values are never inspected; a reserved key is rejected regardless of value. Tuning keys such as fs.gs.inputstream.*, fs.gs.outputstream.* and mapreduce.* keep working.

Tests

  • GCPUtilsFileSystemPropertiesTest (new): reserved / allowed keys, null and whitespace handling, failure attribution, merge-point filtering with an auto-detect connection.
  • GCSSourceConfigTest (new) and GCSBatchSinkTest: end-to-end validate() for endpoint, token server, fallback prefix, impersonation and plugin-internal keys; allowed keys still pass.

The fileSystemProperties field of the GCS source and sink is copied
into the Hadoop configuration used by the GCS connector. The connector
reads the storage endpoint, token server, proxy, credential source,
impersonation target and several implementation classes from that
configuration, so a user could point the connector at an arbitrary
host or identity, including during pipeline validation where it runs
with the identity of the CDAP system service.

Reserved keys are now rejected with a validation failure on the
fileSystemProperties field, one per offending key. As a safeguard,
reserved keys are also dropped where user properties are merged with
the properties set by the plugin, so values resolved from macros at
runtime cannot reach the connector either.

(cherry picked from commit 39702b4)
@akkaur akkaur added the build Trigger unit test build label Nov 20, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces security enhancements to prevent users from overriding critical Hadoop configuration keys through the fileSystemProperties plugin property. It defines a set of reserved keys and prefixes in GCPUtils (such as endpoints, proxies, and authentication configurations) and adds validation and filtering mechanisms to reject or strip these properties during validation and runtime. These changes are integrated into GCSBatchSink and GCSSource, and are accompanied by comprehensive unit tests. There are no review comments, and we have no additional feedback to provide.

@riyaa14
riyaa14 self-requested a review November 20, 2026 11:03
@akkaur
akkaur merged commit 1ca6dd6 into release/0.24 Nov 20, 2026
29 checks passed
@akkaur
akkaur deleted the cherry-pick-restrict-reserved-fs-properties-0.24 branch November 20, 2026 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build Trigger unit test build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants