Repository navigation
Validate source queries and split-by field as single read-only SELECT - #691
vishwasvaidya-cloudsufi wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a robust SQL query validation mechanism (SelectQueryValidator) across multiple database plugins to ensure that user-provided source queries are single, read-only SELECT statements, alongside enforcing read-only transactions and session rollbacks. It also improves identifier quoting to prevent SQL injection. The review feedback correctly identifies several potential NullPointerException vulnerabilities in the getTableName methods of various connectors when database or schema names are null, and suggests enhancements to the query parser to prevent multi-statement bypasses and handle empty queries more gracefully.
| List<Statement> nonEmpty = statements == null ? Collections.emptyList() | ||
| : statements.stream().filter(Objects::nonNull).collect(Collectors.toList()); | ||
| if (nonEmpty.size() == 1 && !(nonEmpty.get(0) instanceof Select)) { | ||
| throw new IllegalArgumentException(String.format( | ||
| "Only a SELECT statement is allowed. Found a %s statement.", nonEmpty.get(0).getClass().getSimpleName())); | ||
| } |
There was a problem hiding this comment.
The JSqlParser secondary check currently only validates that a single statement is a SELECT statement when nonEmpty.size() == 1. However, if a query contains multiple statements without semicolons (which is possible in some dialects like T-SQL), JSqlParser might parse them as multiple statements, resulting in nonEmpty.size() > 1. In this case, the check is bypassed. We should explicitly reject queries where nonEmpty.size() > 1 to prevent multi-statement execution bypasses.
List<Statement> nonEmpty = statements == null ? Collections.emptyList() : statements.stream().filter(Objects::nonNull).collect(Collectors.toList()); if (nonEmpty.size() > 1) { throw new IllegalArgumentException(String.format("Only a single SELECT statement is allowed. Found %d statements.", nonEmpty.size())); } if (nonEmpty.size() == 1 && !(nonEmpty.get(0) instanceof Select)) { throw new IllegalArgumentException(String.format("Only a SELECT statement is allowed. Found a %s statement.", nonEmpty.get(0).getClass().getSimpleName())); }| private void validateImportQueryIfEnabled(@Nullable String query) { | ||
| if (sourceConfig.isQueryValidationDisabled()) { | ||
| return; | ||
| } | ||
| String finalQuery = query != null && query.contains("$CONDITIONS") ? removeConditionsClause(query) : query; | ||
| try { | ||
| SelectQueryValidator.validate(finalQuery, getSqlDialect()); | ||
| } catch (IllegalArgumentException e) { | ||
| throw new IllegalArgumentException("Invalid Import Query: " + e.getMessage(), e); | ||
| } | ||
| } |
There was a problem hiding this comment.
If the query parameter is null or empty, validateImportQueryIfEnabled will call SelectQueryValidator.validate with a null query, which throws a generic IllegalArgumentException("Query must not be null."). It is safer and cleaner to perform a null/empty check first and return early, as missing query validation is already handled by the configuration's standard validation.
private void validateImportQueryIfEnabled(@Nullable String query) { if (sourceConfig.isQueryValidationDisabled() || Strings.isNullOrEmpty(query)) { return; } String finalQuery = query.contains("$CONDITIONS") ? removeConditionsClause(query) : query; try { SelectQueryValidator.validate(finalQuery, getSqlDialect()); } catch (IllegalArgumentException e) { throw new IllegalArgumentException("Invalid Import Query: " + e.getMessage(), e); } }| protected String getTableName(String database, String schema, String table) { | ||
| return schema == null ? String.format("\"%s\".\"%s\"", database, table) | ||
| : String.format("\"%s\".\"%s\".\"%s\"", database, schema, table); | ||
| return schema == null ? quoteIdentifier(database, '"') + "." + quoteIdentifier(table, '"') | ||
| : quoteIdentifier(database, '"') + "." + quoteIdentifier(schema, '"') + "." + quoteIdentifier(table, '"'); | ||
| } |
There was a problem hiding this comment.
If database is null (which can happen if the user relies on the connection's default database), quoteIdentifier(database, '"') will throw a NullPointerException. We should make getTableName null-safe by only appending and quoting non-null parts.
protected String getTableName(String database, String schema, String table) { StringBuilder sb = new StringBuilder(); if (database != null) { sb.append(quoteIdentifier(database, '"')).append("."); } if (schema != null) { sb.append(quoteIdentifier(schema, '"')).append("."); } sb.append(quoteIdentifier(table, '"')); return sb.toString(); }| protected String getTableName(String database, String schema, String table) { | ||
| return String.format("\"%s\".\"%s\"", schema, table); | ||
| return quoteIdentifier(schema, '"') + "." + quoteIdentifier(table, '"'); | ||
| } |
There was a problem hiding this comment.
If schema is null, quoteIdentifier(schema, '"') will throw a NullPointerException. We should make getTableName null-safe by checking if schema is null.
protected String getTableName(String database, String schema, String table) { return (schema == null ? "" : quoteIdentifier(schema, '"') + ".") + quoteIdentifier(table, '"'); }| protected String getTableName(String database, String schema, String table) { | ||
| return String.format("`%s`.`%s`", database, table); | ||
| return quoteIdentifier(database, '`') + "." + quoteIdentifier(table, '`'); | ||
| } |
There was a problem hiding this comment.
If database is null, quoteIdentifier(database, '`') will throw a NullPointerException. We should make getTableName null-safe by checking if database is null.
protected String getTableName(String database, String schema, String table) { return (database == null ? "" : quoteIdentifier(database, '`') + ".") + quoteIdentifier(table, '`'); }| protected String getTableName(String database, String schema, String table) { | ||
| return String.format("\"%s\".\"%s\"", schema, table); | ||
| return quoteIdentifier(schema, '"') + "." + quoteIdentifier(table, '"'); | ||
| } |
There was a problem hiding this comment.
If schema is null, quoteIdentifier(schema, '"') will throw a NullPointerException. We should make getTableName null-safe by checking if schema is null.
protected String getTableName(String database, String schema, String table) { return (schema == null ? "" : quoteIdentifier(schema, '"') + ".") + quoteIdentifier(table, '"'); }| protected String getTableName(String database, String schema, String table) { | ||
| return String.format("\"%s\".\"%s\"", schema, table); | ||
| return quoteIdentifier(schema, '"') + "." + quoteIdentifier(table, '"'); | ||
| } |
There was a problem hiding this comment.
If schema is null, quoteIdentifier(schema, '"') will throw a NullPointerException. We should make getTableName null-safe by checking if schema is null.
protected String getTableName(String database, String schema, String table) { return (schema == null ? "" : quoteIdentifier(schema, '"') + ".") + quoteIdentifier(table, '"'); }| protected String getTableName(String database, String schema, String table) { | ||
| return String.format("\"%s\".\"%s\"", schema, table); | ||
| return quoteIdentifier(schema, '"') + "." + quoteIdentifier(table, '"'); | ||
| } |
There was a problem hiding this comment.
If schema is null, quoteIdentifier(schema, '"') will throw a NullPointerException. We should make getTableName null-safe by checking if schema is null.
protected String getTableName(String database, String schema, String table) { return (schema == null ? "" : quoteIdentifier(schema, '"') + ".") + quoteIdentifier(table, '"'); }| protected String getTableName(String database, String schema, String table) { | ||
| return String.format("`%s`.`%s`", database, table); | ||
| return quoteIdentifier(database, '`') + "." + quoteIdentifier(table, '`'); | ||
| } |
There was a problem hiding this comment.
If database is null, quoteIdentifier(database, '`') will throw a NullPointerException. We should make getTableName null-safe by checking if database is null.
protected String getTableName(String database, String schema, String table) { return (database == null ? "" : quoteIdentifier(database, '`') + ".") + quoteIdentifier(table, '`'); }| protected String getTableName(String database, String schema, String table) { | ||
| return String.format("\"%s\".\"%s\".\"%s\"", database, schema, table); | ||
| return quoteIdentifier(database, '"') + "." + quoteIdentifier(schema, '"') + "." | ||
| + quoteIdentifier(table, '"'); | ||
| } |
There was a problem hiding this comment.
If database or schema is null, quoteIdentifier will throw a NullPointerException. We should make getTableName null-safe by only appending and quoting non-null parts.
protected String getTableName(String database, String schema, String table) { StringBuilder sb = new StringBuilder(); if (database != null) { sb.append(quoteIdentifier(database, '"')).append("."); } if (schema != null) { sb.append(quoteIdentifier(schema, '"')).append("."); } sb.append(quoteIdentifier(table, '"')); return sb.toString(); }
Summary
Follow-up to #688 separating source query and split-by validation into its own PR and addressing all review feedback from #688:
Dialect-aware
SelectQueryValidator(Dialectenum):SelectQueryValidator.Dialect(ANSI,TSQL,POSTGRESQL,MYSQL,MYSQL_AMBIGUOUS_BACKSLASH,DATABRICKS,UNKNOWN) and per-sourcegetSqlDialect()overrides (MysqlSourceswitches toMYSQL_AMBIGUOUS_BACKSLASHwhensqlModeorconnectionArgumentsmay overridesql_mode).--without trailing whitespace (--1),\r-terminated line comments, non-ASCII whitespace (\u00A0, etc.), lowercase keywords under Turkish locale (Locale.ROOT), PostgreSQL per-literalE'...'vs'...'/quotecontinue/ nested/* ... */breakouts,FOR KEY SHARE,LOCK IN SHARE MODE, non-SELECTstatements (COPY,EXPLAIN,REPLACE), JDBC{call ...}escapes, and T-SQL semicolon-less batches (DENY,DBCC,SET,SHUTDOWN,WAITFOR, etc.).q'...', and executable-comment checks only outside literals, comments, and quoted identifiers so ordinary literals ('Q',LIKE '%q','$A$'),$-identifiers (v$session), backslash literals onANSI/TSQL/POSTGRESQL(ESCAPE '\','C:\data\'), unquotedcall, dot-qualified columns (c.call,c.grant), andMERGE JOINare allowed.TokenMgrExceptioninvalidateColumnReferenceand wraps it inIllegalArgumentException("Must be a single column name.", cause)without repeating the field label or appending raw parser output.Single-pass configure/run validation & read-only source connection backstop:
validateQueries(collector)once inconfigurePipelineand once inprepareRun.setReadOnly(true),setAutoCommit(false),rollback()) during schema fetches and runtime splits (DataDrivenETLDBInputFormat+RollbackOnCloseConnection), blockingSELECTqueries that invoke writing functions.Connection-owned compatibility switch & documentation:
disableQueryValidation(Disable Source Query Validation, defaultfalse) onAbstractDBConnectorConfig(owned by the connection, not pipeline stages using a shared connection) and legacyDBSourceConfig.*-batchsource.mdand*-connector.mdfiles.Testing
SelectQueryValidatorTest,AbstractDBSourceTest,DatabaseSourceTest, andMysqlSourceUnitTest.DBSourceTestRuncovering configure-time (including explicit schema) and macro runtime rejections,UPDATEside-effect checks, read-only session enforcement against writing functions, anddisableQueryValidationopt-out.