Skip to content

[AURON #2733] Report native disk spill in Spark task metrics - #2734

Draft
robreeves wants to merge 1 commit into
apache:masterfrom
robreeves:native-disk-spill-task-metrics
Draft

robreeves wants to merge 1 commit into
apache:masterfrom
robreeves:native-disk-spill-task-metrics

Conversation

@robreeves

@robreeves robreeves commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #2733

Rationale for this change

Native disk spill only shows up as Native.disk_spill_size on native nodes in the SQL tab. Spark operators add their spills to the task's diskBytesSpilled, which drives the Stage page "Spill (Disk)" column, the REST API and event logs. Auron never updates it, so a stage that spills heavily in native code reports 0 disk spill. This makes the usual way of debugging a slow stage misleading for Auron jobs.

What changes are included in this PR?

When native metrics are reported to the JVM, SparkMetricNode.add now also adds positive disk_spill_size values to the current task's diskBytesSpilled. This happens whether or not the node registered the SQL metric. incDiskBytesSpilled is private[spark], so a small helper under org.apache.spark.sql.auron makes the call.

This covers disk spill only. Memory spill is left for a follow-up because Auron's memory spill metrics don't match Spark's "Spill (Memory)" definition (in-memory size of the spilled data).

Are there any user-facing changes?

Yes. The Stage page "Spill (Disk)" column, and diskBytesSpilled in the REST API and event logs, now include native disk spill. Like the native SQL metrics, values are reported when each task's native execution finishes.

How was this patch tested?

Added SparkMetricNodeSuite, which checks that disk_spill_size updates the task's diskBytesSpilled, other metrics and non-positive values don't, and calls outside a task are ignored.

Manual e2e testing in spark-shell.

disk_spill_after

Was this patch authored or co-authored using generative AI tooling?

  • Yes
  • No

Generated-by: GitHub Copilot CLI 1.0.65 (Claude Opus 5.5)

ASF guidance: https://www.apache.org/legal/generative-tooling.html

Native disk spill was only visible as the Native.disk_spill_size SQL
metric. Add it to the task's diskBytesSpilled when native metrics are
reported, so the Stage page, REST API and event logs include it.

Generated-by: GitHub Copilot CLI 1.0.65 (Claude Opus 5.5)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added the spark label Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native disk spill is missing from Spark stage spill metrics

1 participant