Skip to content

[AURON #2723] Add native support for Spark from_unixtime - #2727

Open
Sigma-Ma wants to merge 2 commits into
apache:masterfrom
Sigma-Ma:Auron-2723-add-native-support-for-spark-from-unixtime
Open

Sigma-Ma wants to merge 2 commits into
apache:masterfrom
Sigma-Ma:Auron-2723-add-native-support-for-spark-from-unixtime

Conversation

@Sigma-Ma

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #2723

Rationale for this change

from_unixtime currently relies on Spark expression fallback. Add native execution for common constant formats.

What changes are included in this PR?

  • Convert FromUnixTime to a dedicated Spark_FromUnixTime extension function.
  • Support BIGINT columns/literals and lossless integer widening.
  • Support the default yyyy-MM-dd HH:mm:ss format, plus constant yyyy-MM-dd and yyyyMMdd.
  • Support UTC and fixed-offset session time zones under CORRECTED and EXCEPTION policies.
  • Evaluate local years 0001–9999 natively. Other values use Spark evaluation to preserve its overflow behavior.

Dynamic or unsupported formats, regional time zones, LEGACY policy and complex seconds expressions retain Spark fallback. Column pruning conversion also remains unsupported.

Keep input casts and arithmetic in Spark fallback to preserve ANSI errors. Invalid constant formats use Spark operator fallback, including when all seconds are null.

Are there any user-facing changes?

Supported calls can run natively with spark.auron.enableNativeDateTimeFunctions=true; this switch remains disabled by default.

Column inputs also require spark.auron.udf.singleChildFallback.enabled=true.

How was this patch tested?

cargo test -p datafusion-ext-functions --locked

cargo clippy -p datafusion-ext-functions --tests --no-deps --locked -- -D warnings

cargo fmt --all --check

mvn -Ppre -Pspark-3.5 -Pscala-2.12 -pl spark-extension-shims-spark test -DskipBuildNative -DskipTests=false -Dsuites=org.apache.auron.NativeConvertersSuite,org.apache.auron.AuronFunctionSuite

mvn -Ppre -Pspark-4.0 -Pscala-2.13 -pl spark-extension-shims-spark test -DskipBuildNative -DskipTests=false -Dsuites=org.apache.auron.NativeConvertersSuite,org.apache.auron.AuronFunctionSuite

mvn -Ppre -Pspark-3.5 -Pscala-2.12 -pl spark-extension,spark-extension-shims-spark spotless:check compile test-compile scalafix:scalafix -Dscalafix.mode=CHECK -Dscalafix.skipTest=true -DskipBuildNative

Was this patch authored or co-authored using generative AI tooling?

  • Yes
  • No

Generated-by: OpenAI Codex (GPT-6)

ASF guidance: https://www.apache.org/legal/generative-tooling.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add native support for Spark from_unixtime

1 participant