Repository navigation
Conversation
Native disk spill was only visible as the Native.disk_spill_size SQL metric. Add it to the task's diskBytesSpilled when native metrics are reported, so the Stage page, REST API and event logs include it. Generated-by: GitHub Copilot CLI 1.0.65 (Claude Opus 5.5) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #2733
Rationale for this change
Native disk spill only shows up as
Native.disk_spill_sizeon native nodes in the SQL tab. Spark operators add their spills to the task'sdiskBytesSpilled, which drives the Stage page "Spill (Disk)" column, the REST API and event logs. Auron never updates it, so a stage that spills heavily in native code reports 0 disk spill. This makes the usual way of debugging a slow stage misleading for Auron jobs.What changes are included in this PR?
When native metrics are reported to the JVM,
SparkMetricNode.addnow also adds positivedisk_spill_sizevalues to the current task'sdiskBytesSpilled. This happens whether or not the node registered the SQL metric.incDiskBytesSpilledisprivate[spark], so a small helper underorg.apache.spark.sql.auronmakes the call.This covers disk spill only. Memory spill is left for a follow-up because Auron's memory spill metrics don't match Spark's "Spill (Memory)" definition (in-memory size of the spilled data).
Are there any user-facing changes?
Yes. The Stage page "Spill (Disk)" column, and
diskBytesSpilledin the REST API and event logs, now include native disk spill. Like the native SQL metrics, values are reported when each task's native execution finishes.How was this patch tested?
Added
SparkMetricNodeSuite, which checks thatdisk_spill_sizeupdates the task'sdiskBytesSpilled, other metrics and non-positive values don't, and calls outside a task are ignored.Manual e2e testing in spark-shell.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot CLI 1.0.65 (Claude Opus 5.5)
ASF guidance: https://www.apache.org/legal/generative-tooling.html