Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions amazon-redshift-plugin/docs/Redshift-batchsource.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,16 +25,20 @@ Properties

**Database:** Redshift database name.

**Import Query:** The SELECT query to use to import data from the specified table.
**Import Query:** The single read-only `SELECT` query to use to import data from the specified table.
You can specify an arbitrary number of columns to import, or import all columns using \*. The Query should
contain the '$CONDITIONS' string. For example, 'SELECT * FROM table WHERE $CONDITIONS'.
The '$CONDITIONS' string will be replaced by 'splitBy' field limits specified by the bounding query.
The '$CONDITIONS' string is not required if numSplits is set to one.
Multi-statement queries, write/DDL/administrative statements, `INTO`, locking clauses (such as `FOR UPDATE`/`SHARE`),
dollar-quoted or `q'...'` literals, and executable comments are rejected, and the query is executed on a read-only
connection unless **Disable Source Query Validation** (`disableQueryValidation`) is set to `true` on the connection.

**Bounding Query:** Bounding Query should return the min and max of the values of the 'splitBy' field.
**Bounding Query:** Single read-only `SELECT` query returning the min and max of the values of the 'splitBy' field.
For example, 'SELECT MIN(id),MAX(id) FROM table'. Not required if numSplits is set to one.

**Split-By Field Name:** Field Name which will be used to generate splits. Not required if numSplits is set to one.
**Split-By Field Name:** Single column name (optionally table-qualified and/or quoted) which will be used to generate
splits. Not required if numSplits is set to one.

**Number of Splits to Generate:** Number of splits to generate.

Expand Down
5 changes: 5 additions & 0 deletions amazon-redshift-plugin/docs/Redshift-connector.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,8 @@ Properties

**Connection Arguments:** A list of arbitrary string key/value pairs as connection arguments. These arguments
will be passed to the JDBC driver as connection arguments for JDBC drivers that may need additional configurations.

**Disable Source Query Validation:** Disables read-only `SELECT` validation and read-only session enforcement for
source queries using this connection. Enable only for backward compatibility with trusted pipelines whose source
queries are rejected by the validator. Defaults to `false`.

Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ protected SchemaReader getSchemaReader(String sessionID) {

@Override
protected String getTableName(String database, String schema, String table) {
return String.format("\"%s\".\"%s\"", schema, table);
return quoteIdentifier(schema, '"') + "." + quoteIdentifier(table, '"');
}
Comment on lines 84 to 86

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If schema is null, quoteIdentifier(schema, '"') will throw a NullPointerException. We should make getTableName null-safe by checking if schema is null.

protected String getTableName(String database, String schema, String table) { return (schema == null ? "" : quoteIdentifier(schema, '"') + ".") + quoteIdentifier(table, '"'); }


@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,15 @@ public void getTableNameTest() {
CONNECTOR.getTableName("db", "schema", "table"));
}

/**
* Unit test for getTableName() with identifiers containing double quotes
*/
@Test
public void getTableNameEscapesQuotesTest() {
Assert.assertEquals("\"my\"\"schema\".\"t\"\"; DELETE FROM x; --\"",
CONNECTOR.getTableName("db", "my\"schema", "t\"; DELETE FROM x; --"));
}

@Test
public void getRandomQuery() {
Assert.assertEquals("SELECT * FROM TestData\n" +
Expand Down
8 changes: 8 additions & 0 deletions amazon-redshift-plugin/widgets/Redshift-batchsource.json
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,14 @@
"default": 5,
"minimum": 0
}
},
{
"widget-type": "hidden",
"label": "Disable Source Query Validation",
"name": "disableQueryValidation",
"widget-attributes": {
"default": "false"
}
}
]
}
Expand Down
19 changes: 19 additions & 0 deletions amazon-redshift-plugin/widgets/Redshift-connector.json
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,25 @@
"kv-delimiter": "=",
"delimiter": ";"
}
},
{
"widget-type": "radio-group",
"label": "Disable Source Query Validation",
"name": "disableQueryValidation",
"widget-attributes": {
"layout": "inline",
"default": "false",
"options": [
{
"id": "true",
"label": "True"
},
{
"id": "false",
"label": "False"
}
]
}
}
]
},
Expand Down
10 changes: 7 additions & 3 deletions aurora-mysql-plugin/docs/AuroraMysql-batchsource.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,16 +26,20 @@ Properties

**Database:** Aurora DB database name.

**Import Query:** The SELECT query to use to import data from the specified table.
**Import Query:** The single read-only `SELECT` query to use to import data from the specified table.
You can specify an arbitrary number of columns to import, or import all columns using \*. The Query should
contain the '$CONDITIONS' string. For example, 'SELECT * FROM table WHERE $CONDITIONS'.
The '$CONDITIONS' string will be replaced by 'splitBy' field limits specified by the bounding query.
The '$CONDITIONS' string is not required if numSplits is set to one.
Multi-statement queries, write/DDL/administrative statements, `INTO`, locking clauses (such as `FOR UPDATE`/`SHARE`),
dollar-quoted or `q'...'` literals, and executable comments are rejected, and the query is executed on a read-only
connection unless `disableQueryValidation` is set to `true`.

**Bounding Query:** Bounding Query should return the min and max of the values of the 'splitBy' field.
**Bounding Query:** Single read-only `SELECT` query returning the min and max of the values of the 'splitBy' field.
For example, 'SELECT MIN(id),MAX(id) FROM table'. Not required if numSplits is set to one.

**Split-By Field Name:** Field Name which will be used to generate splits. Not required if numSplits is set to one.
**Split-By Field Name:** Single column name (optionally table-qualified and/or quoted) which will be used to generate
splits. Not required if numSplits is set to one.

**Number of Splits to Generate:** Number of splits to generate.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,10 @@
import io.cdap.cdap.api.annotation.Name;
import io.cdap.cdap.api.annotation.Plugin;
import io.cdap.cdap.etl.api.batch.BatchSource;
import io.cdap.plugin.db.ConnectionConfig;
import io.cdap.plugin.db.config.DBSpecificSourceConfig;
import io.cdap.plugin.db.source.AbstractDBSource;
import io.cdap.plugin.util.SelectQueryValidator;

import java.util.Map;
import javax.annotation.Nullable;
Expand All @@ -49,6 +51,15 @@ protected String createConnectionString() {
auroraMysqlSourceConfig.host, auroraMysqlSourceConfig.port, auroraMysqlSourceConfig.database);
}

@Override
protected SelectQueryValidator.Dialect getSqlDialect() {
if (auroraMysqlSourceConfig.containsMacro(ConnectionConfig.CONNECTION_ARGUMENTS)
|| SelectQueryValidator.Dialect.mayOverrideMysqlSqlMode(auroraMysqlSourceConfig.connectionArguments)) {
return SelectQueryValidator.Dialect.MYSQL_AMBIGUOUS_BACKSLASH;
}
return SelectQueryValidator.Dialect.MYSQL;
}

/**
* Aurora DB MySQL source config.
*/
Expand Down
8 changes: 8 additions & 0 deletions aurora-mysql-plugin/widgets/AuroraMysql-batchsource.json
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,14 @@
"default": 5,
"minimum": 0
}
},
{
"widget-type": "hidden",
"label": "Disable Source Query Validation",
"name": "disableQueryValidation",
"widget-attributes": {
"default": "false"
}
}
]
}
Expand Down
10 changes: 7 additions & 3 deletions aurora-postgresql-plugin/docs/AuroraPostgres-batchsource.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,16 +26,20 @@ Properties

**Database:** Aurora DB database name.

**Import Query:** The SELECT query to use to import data from the specified table.
**Import Query:** The single read-only `SELECT` query to use to import data from the specified table.
You can specify an arbitrary number of columns to import, or import all columns using \*. The Query should
contain the '$CONDITIONS' string. For example, 'SELECT * FROM table WHERE $CONDITIONS'.
The '$CONDITIONS' string will be replaced by 'splitBy' field limits specified by the bounding query.
The '$CONDITIONS' string is not required if numSplits is set to one.
Multi-statement queries, write/DDL/administrative statements, `INTO`, locking clauses (such as `FOR UPDATE`/`SHARE`),
dollar-quoted or `q'...'` literals, and executable comments are rejected, and the query is executed on a read-only
connection unless `disableQueryValidation` is set to `true`.

**Bounding Query:** Bounding Query should return the min and max of the values of the 'splitBy' field.
**Bounding Query:** Single read-only `SELECT` query returning the min and max of the values of the 'splitBy' field.
For example, 'SELECT MIN(id),MAX(id) FROM table'. Not required if numSplits is set to one.

**Split-By Field Name:** Field Name which will be used to generate splits. Not required if numSplits is set to one.
**Split-By Field Name:** Single column name (optionally table-qualified and/or quoted) which will be used to generate
splits. Not required if numSplits is set to one.

**Number of Splits to Generate:** Number of splits to generate.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,11 @@
import io.cdap.cdap.api.annotation.Name;
import io.cdap.cdap.api.annotation.Plugin;
import io.cdap.cdap.etl.api.batch.BatchSource;
import io.cdap.plugin.db.ConnectionConfig;
import io.cdap.plugin.db.SchemaReader;
import io.cdap.plugin.db.config.DBSpecificSourceConfig;
import io.cdap.plugin.db.source.AbstractDBSource;
import io.cdap.plugin.util.SelectQueryValidator;
import org.apache.hadoop.mapreduce.lib.db.DBWritable;

import java.util.Map;
Expand Down Expand Up @@ -52,6 +54,15 @@ protected String createConnectionString() {
auroraPostgresSourceConfig.database);
}

@Override
protected SelectQueryValidator.Dialect getSqlDialect() {
if (auroraPostgresSourceConfig.containsMacro(ConnectionConfig.CONNECTION_ARGUMENTS)) {
return SelectQueryValidator.Dialect.UNKNOWN;
}
return SelectQueryValidator.Dialect.fromJdbcUrl("jdbc:postgresql:",
auroraPostgresSourceConfig.connectionArguments);
}

@Override
protected SchemaReader getSchemaReader() {
return new AuroraPostgresSchemaReader();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,14 @@
"default": 5,
"minimum": 0
}
},
{
"widget-type": "hidden",
"label": "Disable Source Query Validation",
"name": "disableQueryValidation",
"widget-attributes": {
"default": "false"
}
}
]
}
Expand Down
10 changes: 7 additions & 3 deletions cloudsql-mysql-plugin/docs/CloudSQLMySQL-batchsource.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,16 +35,20 @@ Can be found in the instance overview page.

**CloudSQL Instance Type:** Whether the CloudSQL instance to connect to is private or public. Defaults to 'Public'.

**Import Query:** The SELECT query to use to import data from the specified table.
**Import Query:** The single read-only `SELECT` query to use to import data from the specified table.
You can specify an arbitrary number of columns to import, or import all columns using \*. The Query should
contain the '$CONDITIONS' string. For example, 'SELECT * FROM table WHERE $CONDITIONS'.
The '$CONDITIONS' string will be replaced by 'splitBy' field limits specified by the bounding query.
The '$CONDITIONS' string is not required if numSplits is set to one.
Multi-statement queries, write/DDL/administrative statements, `INTO`, locking clauses (such as `FOR UPDATE`/`SHARE`),
dollar-quoted or `q'...'` literals, and executable comments are rejected, and the query is executed on a read-only
connection unless **Disable Source Query Validation** (`disableQueryValidation`) is set to `true` on the connection.

**Bounding Query:** Bounding Query should return the min and max of the values of the 'splitBy' field.
**Bounding Query:** Single read-only `SELECT` query returning the min and max of the values of the 'splitBy' field.
For example, 'SELECT MIN(id),MAX(id) FROM table'. Not required if numSplits is set to one.

**Split-By Field Name:** Field Name which will be used to generate splits. Not required if numSplits is set to one.
**Split-By Field Name:** Single column name (optionally table-qualified and/or quoted) which will be used to generate
splits. Not required if numSplits is set to one.

**Number of Splits to Generate:** Number of splits to generate.

Expand Down
4 changes: 4 additions & 0 deletions cloudsql-mysql-plugin/docs/CloudSQLMySQL-connector.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,10 @@ This is a semicolon-separated list of key-value pairs, where each pair is separa
the key and value for the argument. For example, 'key1=value1;key2=value' specifies that the connection will be
given arguments 'key1' mapped to 'value1' and the argument 'key2' mapped to 'value2'.

**Disable Source Query Validation:** Disables read-only `SELECT` validation and read-only session enforcement for
source queries using this connection. Enable only for backward compatibility with trusted pipelines whose source
queries are rejected by the validator. Defaults to `false`.

Path of the connection
----------------------
To browse, get a sample from, or get the specification for this connection through
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ protected SchemaReader getSchemaReader(String sessionID) {

@Override
protected String getTableName(String database, String schema, String table) {
return String.format("`%s`.`%s`", database, table);
return quoteIdentifier(database, '`') + "." + quoteIdentifier(table, '`');
}
Comment on lines 84 to 86

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If database is null, quoteIdentifier(database, '`') will throw a NullPointerException. We should make getTableName null-safe by checking if database is null.

protected String getTableName(String database, String schema, String table) { return (database == null ? "" : quoteIdentifier(database, '`') + ".") + quoteIdentifier(table, '`'); }


@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,13 +32,15 @@
import io.cdap.plugin.common.ConfigUtil;
import io.cdap.plugin.common.LineageRecorder;
import io.cdap.plugin.common.db.DBErrorDetailsProvider;
import io.cdap.plugin.db.ConnectionConfig;
import io.cdap.plugin.db.SchemaReader;
import io.cdap.plugin.db.config.AbstractDBSpecificSourceConfig;
import io.cdap.plugin.db.source.AbstractDBSource;
import io.cdap.plugin.mysql.MysqlDBRecord;
import io.cdap.plugin.mysql.MysqlSchemaReader;
import io.cdap.plugin.util.CloudSQLUtil;
import io.cdap.plugin.util.DBUtils;
import io.cdap.plugin.util.SelectQueryValidator;
import org.apache.hadoop.mapreduce.lib.db.DBWritable;

import java.util.Collections;
Expand Down Expand Up @@ -83,6 +85,17 @@ protected Class<? extends DBWritable> getDBRecordType() {
return MysqlDBRecord.class;
}

@Override
protected SelectQueryValidator.Dialect getSqlDialect() {
CloudSQLMySQLConnectorConfig connection = cloudsqlMysqlSourceConfig.getConnection();
if (connection != null
&& (connection.containsMacro(ConnectionConfig.CONNECTION_ARGUMENTS)
|| SelectQueryValidator.Dialect.mayOverrideMysqlSqlMode(connection.getConnectionArguments()))) {
return SelectQueryValidator.Dialect.MYSQL_AMBIGUOUS_BACKSLASH;
}
return SelectQueryValidator.Dialect.MYSQL;
}

@Override
protected String createConnectionString() {
if (CloudSQLUtil.PRIVATE_INSTANCE.equalsIgnoreCase(
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
/*
* Copyright © 2026 Cask Data, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not
* use this file except in compliance with the License. You may obtain a copy of
* the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
* WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
* License for the specific language governing permissions and limitations under
* the License.
*/

package io.cdap.plugin.cloudsql.mysql;

import org.junit.Assert;
import org.junit.Test;

/**
* Unit tests for {@link CloudSQLMySQLConnector}
*/
public class CloudSQLMySQLConnectorUnitTest {

private static final CloudSQLMySQLConnector CONNECTOR = new CloudSQLMySQLConnector(
new CloudSQLMySQLConnectorConfig("user", "password", "cloudsql-mysql", "", "public",
"project:region:instance", "db", null));

/**
* Unit test for getTableName()
*/
@Test
public void getTableNameTest() {
Assert.assertEquals("`db`.`table`",
CONNECTOR.getTableName("db", null, "table"));
}

/**
* Unit test for getTableName() with identifiers containing backticks
*/
@Test
public void getTableNameEscapesBackticksTest() {
Assert.assertEquals("`my``db`.`t``; DELETE FROM x; --`",
CONNECTOR.getTableName("my`db", null, "t`; DELETE FROM x; --"));
}
}
8 changes: 8 additions & 0 deletions cloudsql-mysql-plugin/widgets/CloudSQLMySQL-batchsink.json
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,14 @@
"default": 5,
"minimum": 0
}
},
{
"widget-type": "hidden",
"label": "Disable Source Query Validation",
"name": "disableQueryValidation",
"widget-attributes": {
"default": "false"
}
}
]
}
Expand Down
8 changes: 8 additions & 0 deletions cloudsql-mysql-plugin/widgets/CloudSQLMySQL-batchsource.json
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,14 @@
"default": 5,
"minimum": 0
}
},
{
"widget-type": "hidden",
"label": "Disable Source Query Validation",
"name": "disableQueryValidation",
"widget-attributes": {
"default": "false"
}
}
]
}
Expand Down
Loading
Loading