Repository navigation
fix(spark)!: reject file format subclasses - #1459
alexandrefimov wants to merge 2 commits into
Conversation
File format subclasses can depend on metadata that plain file reads and writes do not carry. Reject them instead of exporting a different operation. Closes substrait-io#1388 BREAKING CHANGE: Spark export rejects file-format subclasses, including path-based Delta reads. Use a format-aware exporter for these sources.
|
No actionable comments were generated in the recent review. Summary
Merge Risk:
|
Reject file-format subclasses instead of exporting them as plain CSV, Parquet or ORC. Path-based Delta reads can otherwise lose deletion vectors or column mapping and return wrong rows.
Closes #1388
BREAKING CHANGE: Spark export now rejects file-format subclasses, including DeltaParquetFileFormat. Use a format-aware exporter for those sources.