Skip to content

Upgrade Guava to 32.0.0-jre and bump minimum CDAP to 6.12.0-SNAPSHOT - #683

Open
AbhishekKumar9984 wants to merge 2 commits into
data-integrations:developfrom
cloudsufi:guava_fixes
Open

AbhishekKumar9984 wants to merge 2 commits into
data-integrations:developfrom
cloudsufi:guava_fixes

Conversation

@AbhishekKumar9984

@AbhishekKumar9984 AbhishekKumar9984 commented Oct 13, 2026 •

Copy link
Copy Markdown

Depends on: cdapio/cdap#16291 (follow-up to cdapio/cdap#16279 targeting develop)

Issue :
Resolved security vulnerabilities (CVE-2023-2976 / CVE-2020-8908) associated with older versions of com.google.guava:guava (13.0.1 / 31.x) pulled in directly and transitively across database plugins.

Root Cause :
Root and submodule POMs referenced older Guava versions (13.0.1, 31.0.1-jre, 31.1-jre), and CDAP, Hadoop, and CloudSQL socket-factory dependencies transitively brought in conflicting older versions of Guava causing compilation and test issues.

Fix :

  • Upgraded <guava.version> to 32.0.0-jre and pinned com.google.guava:guava in root <dependencyManagement> (removing redundant submodule version overrides and the e2e-tests profile override).
  • Added maven-enforcer-plugin (requireUpperBoundDeps) for com.google.guava:guava.
  • Updated <cdap.version> from 6.11.0 to 6.12.0-SNAPSHOT and tightened <cdapArtifacts> parent ranges to [6.12.0-SNAPSHOT,7.0.0-SNAPSHOT).
  • Added io.cdap.plugin.db.* to <_exportcontents> in aurora-mysql-plugin/pom.xml and generic-database-plugin/pom.xml because AuroraMysqlSource and DatabaseSource are the only two source plugins that inherit AbstractDBSource#getDBRecordType() returning io.cdap.plugin.db.DBRecord.class (all other 14 modules override getDBRecordType() with a module-specific *DBRecord in an already-exported package), which Hadoop's DBConfiguration#getInputClass() loads dynamically via Export-Package.

Testing evidence: https://docs.google.com/spreadsheets/d/1Cw6Z_k3wlks7sAC4-Kg6FTIMRL7-zxLlsJIXvVJLrb8/edit?gid=0#gid=0

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the Guava dependency version across multiple POM files to use the ${guava.version} property. The review feedback points out that specifying the version in the submodules is redundant due to parent dependency management. Additionally, the selected Guava version (32.0.0-jre) is vulnerable to CVE-2023-2976, and it is recommended to upgrade to a secure version like 32.1.3-jre and ensure consistency across all profiles.

Comment thread cloudsql-mysql-plugin/pom.xml Resolved
Comment thread cloudsql-postgresql-plugin/pom.xml Resolved
Comment thread pom.xml Resolved
Comment thread pom.xml Resolved

@dheeraj-kholia-cs dheeraj-kholia-cs left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dheeraj-kholia-cs dheeraj-kholia-cs left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@deepika-lalwani123 deepika-lalwani123 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This depends on cdapio/cdap#16279. The currently published 6.12.0-SNAPSHOT still has Guava 13, so this PR cannot be validated or merged until that PR merges and CDAP (and Twill) are published. Please call out the dependency in the PR description.

Comment thread pom.xml Resolved
Comment thread pom.xml Resolved
Comment thread cloudsql-mysql-plugin/pom.xml Resolved
@AbhishekKumar9984

Copy link
Copy Markdown
Author

@deepika-lalwani123 Updated the PR description to explicitly call out the dependency on cdapio/cdap#16279, and addressed all inline review comments.

@shruthi713 shruthi713 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Please add a line to the description on why io.cdap.plugin.db.* was added to the exports in aurora-mysql-plugin and generic-database-plugin.

@AbhishekKumar9984

Copy link
Copy Markdown
Author

@shruthi713 Thanks for the approval! Updated the PR description to explain why io.cdap.plugin.db.* was added to <_exportcontents> in aurora-mysql-plugin and generic-database-plugin.

@shruthi713

Copy link
Copy Markdown

The build check on a6b9719 failed on 18 Nov: 24 tests in 12 suites, with NoSuchMethodError: ...Service.stopAndWait() from the CDAP test framework. A new CDAP 6.12.0-SNAPSHOT was published today, so please re-run build. If it still fails, this can't merge until CDAP publishes a snapshot built with Guava 32.

fixes3

fixes34

fixes343

fixes

@deepika-lalwani123 deepika-lalwani123 left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few more points on the latest commit:

  1. Build is failing on the head commit (ca6310c): 24 of 202 tests. The failures come from the CDAP test framework, not plugin code: NoSuchMethodError: io.cdap.cdap.internal.app.worker.sidecar.ArtifactLocalizerService.stopAndWait() at TestBase.finish(TestBase.java:623), plus IncompatibleClassChangeError: Implementing class. So the 6.12.0-SNAPSHOT this build resolved still calls Guava 13 APIs. Note that cdapio/cdap#16279 was merged into the cs_guava_upgrade branch, not develop. A new snapshot was published on 19 Nov at 15:48 UTC, after this build ran (14:57 UTC). Please re-run the build. If it still fails, this can't merge until a CDAP snapshot built with Guava 32 is published.

  2. Parent artifact range is not tightened. pom.xml L379-380 still has [6.11.0-SNAPSHOT,7.0.0-SNAPSHOT) while cdap.version is now 6.12.0-SNAPSHOT. Please raise the lower bound to 6.12.0-SNAPSHOT, as was done in the ftp, servicenow and salesforce PRs.

  3. maven-bundle-plugin in the cloudsql modules. The reply on my earlier thread says the block was restored, but at head the first maven-bundle-plugin declaration is still removed from cloudsql-mysql-plugin/pom.xml and cloudsql-postgresql-plugin/pom.xml; only the second, pre-existing declaration remains. That may well be harmless since the second one already set the specific _exportcontents, but please paste the Export-Package line from the built jar before and after to confirm.

  4. io.cdap.plugin.db.* export. It is added only to aurora-mysql-plugin and generic-database-plugin. What failure showed it was needed, and why doesn't it apply to the other modules that extend the same database-commons classes (for example cloudsql-mysql-plugin, which still exports only db.source.* and db.sink.*)?

@AbhishekKumar9984 AbhishekKumar9984 changed the title guava fixes Upgrade Guava to 32.0.0-jre and bump minimum CDAP to 6.12.0-SNAPSHOT Nov 20, 2026
@AbhishekKumar9984

Copy link
Copy Markdown
Author

@deepika-lalwani123 @shruthi713 Thanks for the reviews and approvals! Addressed all points:

  1. CI build / stopAndWait() failure: Checked Central Snapshots — the latest published io.cdap.cdap:cdap:6.12.0-SNAPSHOT (6.12.0-20261119.102639-73, updated 19 Nov 15:48 UTC) still declares <guava.version>13.0.1</guava.version> because Upgrade Guava library to 32.0.0-jre cdapio/cdap#16291 (targeting develop) hasn't merged yet (updated the PR description to track Upgrade Guava library to 32.0.0-jre cdapio/cdap#16291). Pushing 998e20c triggers a fresh CI run, and we'll hold merging and re-run CI once Upgrade Guava library to 32.0.0-jre cdapio/cdap#16291 lands and publishes a Guava-32 snapshot. Locally, against a 6.12.0-SNAPSHOT built with Guava 32.0.0-jre, database-commons (52/52) and generic-database-plugin (10/10 HSQLDB ETL pipeline tests) pass cleanly.
  2. Parent artifact range: Done in 998e20c — raised both cdap-data-pipeline and cdap-data-streams lower bounds in pom.xml to [6.12.0-SNAPSHOT,7.0.0-SNAPSHOT).
  3. maven-bundle-plugin in CloudSQL modules: Confirmed the removed first maven-bundle-plugin block was a duplicate plugin declaration whose <_exportcontents> was always overridden by the second declaration in the same <plugins> list. Built both modules at base (76fe7141) and at head — the exported packages are identical (only javax.annotation removed from uses:= on cloudsql-mysql, and byte-for-byte identical package set on both):
    • cloudsql-mysql-plugin before (76fe7141):
      Export-Package: io.cdap.plugin.cloudsql.mysql;uses:="io.cdap.cdap.api.annotation,io.cdap.cdap.api.data.format,io.cdap.cdap.api.data.schema,io.cdap.cdap.etl.api,io.cdap.cdap.etl.api.batch,io.cdap.cdap.etl.api.connector,io.cdap.plugin.db.sink,io.cdap.plugin.db.source,io.cdap.plugin.mysql,javax.annotation,org.apache.hadoop.io,org.apache.hadoop.mapreduce.lib.db";version="1.13.0",io.cdap.plugin.db.source;version="1.13.0";uses:="...",io.cdap.plugin.db.sink;version="1.13.0";uses:="...",io.cdap.plugin.mysql;version="1.13.0";uses:="..."
    • cloudsql-mysql-plugin after:
      Export-Package: io.cdap.plugin.cloudsql.mysql;version="1.13.0";uses:="io.cdap.cdap.api.annotation,io.cdap.cdap.api.data.format,io.cdap.cdap.api.data.schema,io.cdap.cdap.etl.api,io.cdap.cdap.etl.api.batch,io.cdap.cdap.etl.api.connector,io.cdap.plugin.db.sink,io.cdap.plugin.db.source,io.cdap.plugin.mysql,org.apache.hadoop.io,org.apache.hadoop.mapreduce.lib.db",io.cdap.plugin.db.source;version="1.13.0";uses:="...",io.cdap.plugin.db.sink;version="1.13.0";uses:="...",io.cdap.plugin.mysql;version="1.13.0";uses:="..."
    • cloudsql-postgresql-plugin before (76fe7141) & after (identical):
      Export-Package: io.cdap.plugin.cloudsql.postgres;version="1.13.0";uses:="io.cdap.cdap.api.annotation,io.cdap.cdap.api.data.format,io.cdap.cdap.api.data.schema,io.cdap.cdap.etl.api,io.cdap.cdap.etl.api.batch,io.cdap.cdap.etl.api.connector,io.cdap.plugin.db.sink,io.cdap.plugin.db.source,io.cdap.plugin.postgres,org.apache.hadoop.io,org.apache.hadoop.mapreduce.lib.db",io.cdap.plugin.postgres;version="1.13.0";uses:="...",io.cdap.plugin.db.source;version="1.13.0";uses:="...",io.cdap.plugin.db.sink;version="1.13.0";uses:="..."
  4. Why io.cdap.plugin.db.* applies only to aurora-mysql-plugin and generic-database-plugin: AbstractDBSource#prepareRun sets mapreduce.jdbc.input.class to getDBRecordType(), which Hadoop's DBConfiguration#getInputClass() loads via Configuration#getClass(...) through the plugin's Export-Package filter. All 14 other source plugins override getDBRecordType() with a subclass in an already-exported package (e.g., CloudSQLMySQLSource and MysqlSource return io.cdap.plugin.mysql.MysqlDBRecord, exported via io.cdap.plugin.mysql.*; CloudSQLPostgreSQLSource/PostgresSource return PostgresDBRecord; OracleSource returns OracleSourceDBRecord, etc.). AuroraMysqlSource and DatabaseSource are the only two source plugins that don't override getDBRecordType() and inherit the default io.cdap.plugin.db.DBRecord.class, resulting in ClassNotFoundException: io.cdap.plugin.db.DBRecord unless io.cdap.plugin.db is exported (also updated the PR description to state this accurately).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants