Skip to content

Validate source queries and split-by field as single read-only SELECT - #691

Open
vishwasvaidya-cloudsufi wants to merge 2 commits into
developfrom
source-query-validation
Open

vishwasvaidya-cloudsufi wants to merge 2 commits into
developfrom
source-query-validation

Conversation

@vishwasvaidya-cloudsufi

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #688 separating source query and split-by validation into its own PR and addressing all review feedback from #688:

  1. Dialect-aware SelectQueryValidator (Dialect enum):

    • Replaces boolean flags with SelectQueryValidator.Dialect (ANSI, TSQL, POSTGRESQL, MYSQL, MYSQL_AMBIGUOUS_BACKSLASH, DATABRICKS, UNKNOWN) and per-source getSqlDialect() overrides (MysqlSource switches to MYSQL_AMBIGUOUS_BACKSLASH when sqlMode or connectionArguments may override sql_mode).
    • Rejects MySQL -- without trailing whitespace (--1), \r-terminated line comments, non-ASCII whitespace (\u00A0, etc.), lowercase keywords under Turkish locale (Locale.ROOT), PostgreSQL per-literal E'...' vs '...' / quotecontinue / nested /* ... */ breakouts, FOR KEY SHARE, LOCK IN SHARE MODE, non-SELECT statements (COPY, EXPLAIN, REPLACE), JDBC {call ...} escapes, and T-SQL semicolon-less batches (DENY, DBCC, SET, SHUTDOWN, WAITFOR, etc.).
    • Applies dollar-quote, q'...', and executable-comment checks only outside literals, comments, and quoted identifiers so ordinary literals ('Q', LIKE '%q', '$A$'), $-identifiers (v$session), backslash literals on ANSI/TSQL/POSTGRESQL (ESCAPE '\', 'C:\data\'), unquoted call, dot-qualified columns (c.call, c.grant), and MERGE JOIN are allowed.
    • Catches TokenMgrException in validateColumnReference and wraps it in IllegalArgumentException("Must be a single column name.", cause) without repeating the field label or appending raw parser output.
  2. Single-pass configure/run validation & read-only source connection backstop:

    • Calls validateQueries(collector) once in configurePipeline and once in prepareRun.
    • Enforces read-only sessions (setReadOnly(true), setAutoCommit(false), rollback()) during schema fetches and runtime splits (DataDrivenETLDBInputFormat + RollbackOnCloseConnection), blocking SELECT queries that invoke writing functions.
  3. Connection-owned compatibility switch & documentation:

    • Adds disableQueryValidation (Disable Source Query Validation, default false) on AbstractDBConnectorConfig (owned by the connection, not pipeline stages using a shared connection) and legacy DBSourceConfig.
    • Updates connector/source widgets and documents the query restrictions and compatibility flag across all *-batchsource.md and *-connector.md files.

Testing

  • Unit tests in SelectQueryValidatorTest, AbstractDBSourceTest, DatabaseSourceTest, and MysqlSourceUnitTest.
  • Embedded HSQLDB integration tests in DBSourceTestRun covering configure-time (including explicit schema) and macro runtime rejections, UPDATE side-effect checks, read-only session enforcement against writing functions, and disableQueryValidation opt-out.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a robust SQL query validation mechanism (SelectQueryValidator) across multiple database plugins to ensure that user-provided source queries are single, read-only SELECT statements, alongside enforcing read-only transactions and session rollbacks. It also improves identifier quoting to prevent SQL injection. The review feedback correctly identifies several potential NullPointerException vulnerabilities in the getTableName methods of various connectors when database or schema names are null, and suggests enhancements to the query parser to prevent multi-statement bypasses and handle empty queries more gracefully.

Comment on lines +373 to +378
List<Statement> nonEmpty = statements == null ? Collections.emptyList()
: statements.stream().filter(Objects::nonNull).collect(Collectors.toList());
if (nonEmpty.size() == 1 && !(nonEmpty.get(0) instanceof Select)) {
throw new IllegalArgumentException(String.format(
"Only a SELECT statement is allowed. Found a %s statement.", nonEmpty.get(0).getClass().getSimpleName()));
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The JSqlParser secondary check currently only validates that a single statement is a SELECT statement when nonEmpty.size() == 1. However, if a query contains multiple statements without semicolons (which is possible in some dialects like T-SQL), JSqlParser might parse them as multiple statements, resulting in nonEmpty.size() > 1. In this case, the check is bypassed. We should explicitly reject queries where nonEmpty.size() > 1 to prevent multi-statement execution bypasses.

List<Statement> nonEmpty = statements == null ? Collections.emptyList() : statements.stream().filter(Objects::nonNull).collect(Collectors.toList()); if (nonEmpty.size() > 1) { throw new IllegalArgumentException(String.format("Only a single SELECT statement is allowed. Found %d statements.", nonEmpty.size())); } if (nonEmpty.size() == 1 && !(nonEmpty.get(0) instanceof Select)) { throw new IllegalArgumentException(String.format("Only a SELECT statement is allowed. Found a %s statement.", nonEmpty.get(0).getClass().getSimpleName())); }

Comment on lines +185 to +195
private void validateImportQueryIfEnabled(@Nullable String query) {
if (sourceConfig.isQueryValidationDisabled()) {
return;
}
String finalQuery = query != null && query.contains("$CONDITIONS") ? removeConditionsClause(query) : query;
try {
SelectQueryValidator.validate(finalQuery, getSqlDialect());
} catch (IllegalArgumentException e) {
throw new IllegalArgumentException("Invalid Import Query: " + e.getMessage(), e);
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If the query parameter is null or empty, validateImportQueryIfEnabled will call SelectQueryValidator.validate with a null query, which throws a generic IllegalArgumentException("Query must not be null."). It is safer and cleaner to perform a null/empty check first and return early, as missing query validation is already handled by the configuration's standard validation.

private void validateImportQueryIfEnabled(@Nullable String query) { if (sourceConfig.isQueryValidationDisabled() || Strings.isNullOrEmpty(query)) { return; } String finalQuery = query.contains("$CONDITIONS") ? removeConditionsClause(query) : query; try { SelectQueryValidator.validate(finalQuery, getSqlDialect()); } catch (IllegalArgumentException e) { throw new IllegalArgumentException("Invalid Import Query: " + e.getMessage(), e); } }

Comment on lines 158 to +161
protected String getTableName(String database, String schema, String table) {
return schema == null ? String.format("\"%s\".\"%s\"", database, table)
: String.format("\"%s\".\"%s\".\"%s\"", database, schema, table);
return schema == null ? quoteIdentifier(database, '"') + "." + quoteIdentifier(table, '"')
: quoteIdentifier(database, '"') + "." + quoteIdentifier(schema, '"') + "." + quoteIdentifier(table, '"');
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If database is null (which can happen if the user relies on the connection's default database), quoteIdentifier(database, '"') will throw a NullPointerException. We should make getTableName null-safe by only appending and quoting non-null parts.

protected String getTableName(String database, String schema, String table) { StringBuilder sb = new StringBuilder(); if (database != null) { sb.append(quoteIdentifier(database, '"')).append("."); } if (schema != null) { sb.append(quoteIdentifier(schema, '"')).append("."); } sb.append(quoteIdentifier(table, '"')); return sb.toString(); }

Comment on lines 84 to 86
protected String getTableName(String database, String schema, String table) {
return String.format("\"%s\".\"%s\"", schema, table);
return quoteIdentifier(schema, '"') + "." + quoteIdentifier(table, '"');
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If schema is null, quoteIdentifier(schema, '"') will throw a NullPointerException. We should make getTableName null-safe by checking if schema is null.

protected String getTableName(String database, String schema, String table) { return (schema == null ? "" : quoteIdentifier(schema, '"') + ".") + quoteIdentifier(table, '"'); }

Comment on lines 84 to 86
protected String getTableName(String database, String schema, String table) {
return String.format("`%s`.`%s`", database, table);
return quoteIdentifier(database, '`') + "." + quoteIdentifier(table, '`');
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If database is null, quoteIdentifier(database, '`') will throw a NullPointerException. We should make getTableName null-safe by checking if database is null.

protected String getTableName(String database, String schema, String table) { return (database == null ? "" : quoteIdentifier(database, '`') + ".") + quoteIdentifier(table, '`'); }

Comment on lines 88 to 90
protected String getTableName(String database, String schema, String table) {
return String.format("\"%s\".\"%s\"", schema, table);
return quoteIdentifier(schema, '"') + "." + quoteIdentifier(table, '"');
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If schema is null, quoteIdentifier(schema, '"') will throw a NullPointerException. We should make getTableName null-safe by checking if schema is null.

protected String getTableName(String database, String schema, String table) { return (schema == null ? "" : quoteIdentifier(schema, '"') + ".") + quoteIdentifier(table, '"'); }

Comment on lines 137 to 139
protected String getTableName(String database, String schema, String table) {
return String.format("\"%s\".\"%s\"", schema, table);
return quoteIdentifier(schema, '"') + "." + quoteIdentifier(table, '"');
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If schema is null, quoteIdentifier(schema, '"') will throw a NullPointerException. We should make getTableName null-safe by checking if schema is null.

protected String getTableName(String database, String schema, String table) { return (schema == null ? "" : quoteIdentifier(schema, '"') + ".") + quoteIdentifier(table, '"'); }

Comment on lines 119 to 121
protected String getTableName(String database, String schema, String table) {
return String.format("\"%s\".\"%s\"", schema, table);
return quoteIdentifier(schema, '"') + "." + quoteIdentifier(table, '"');
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If schema is null, quoteIdentifier(schema, '"') will throw a NullPointerException. We should make getTableName null-safe by checking if schema is null.

protected String getTableName(String database, String schema, String table) { return (schema == null ? "" : quoteIdentifier(schema, '"') + ".") + quoteIdentifier(table, '"'); }

Comment on lines 100 to 102
protected String getTableName(String database, String schema, String table) {
return String.format("`%s`.`%s`", database, table);
return quoteIdentifier(database, '`') + "." + quoteIdentifier(table, '`');
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If database is null, quoteIdentifier(database, '`') will throw a NullPointerException. We should make getTableName null-safe by checking if database is null.

protected String getTableName(String database, String schema, String table) { return (database == null ? "" : quoteIdentifier(database, '`') + ".") + quoteIdentifier(table, '`'); }

Comment on lines 119 to 122
protected String getTableName(String database, String schema, String table) {
return String.format("\"%s\".\"%s\".\"%s\"", database, schema, table);
return quoteIdentifier(database, '"') + "." + quoteIdentifier(schema, '"') + "."
+ quoteIdentifier(table, '"');
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If database or schema is null, quoteIdentifier will throw a NullPointerException. We should make getTableName null-safe by only appending and quoting non-null parts.

protected String getTableName(String database, String schema, String table) { StringBuilder sb = new StringBuilder(); if (database != null) { sb.append(quoteIdentifier(database, '"')).append("."); } if (schema != null) { sb.append(quoteIdentifier(schema, '"')).append("."); } sb.append(quoteIdentifier(table, '"')); return sb.toString(); }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant