Skip to content

feat(result-service): add a service that reads results - #9356

Open
mengw15 wants to merge 10 commits into
apache:mainfrom
mengw15:feat/8995-result-service
Open

mengw15 wants to merge 10 commits into
apache:mainfrom
mengw15:feat/8995-result-service

Conversation

@mengw15

@mengw15 mengw15 commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Adds result-service, a Dropwizard service that reads execution results without a computing unit. It stands in front of Lakekeeper the way file-service stands in front of LakeFS. This PR is the module and its CI legs, so nothing calls it yet: its image, Helm chart, gateway route and local-dev entry follow in a second PR, and #8996 moves the frontend over.

Before: a result page is read through the computing unit, so a result cannot be read once its unit is gone.

flowchart TB
    browser(["Browser"])
    gateway["Gateway"]
    cu["Computing unit<br/>engine + Iceberg client"]
    lakekeeper["Lakekeeper<br/>Iceberg catalog"]
    postgres[("Postgres")]
    s3[("S3 storage")]

    browser -->|"read a result page"| gateway
    gateway -->|"websocket"| cu
    cu -->|"table location"| lakekeeper
    cu -->|"Parquet files"| s3
    lakekeeper -->|"metadata"| postgres
Loading

After (with the deployment PR and #8996): pages are read through result-service, which needs no unit. The unit still writes results and, while an execution runs, tells the browser which pages changed. Red marks what is new.

flowchart TB
    browser(["Browser"])
    gateway["Gateway"]
    rs["result-service<br/>access check<br/>Iceberg client"]
    cu["Computing unit<br/>engine"]
    lakekeeper["Lakekeeper<br/>Iceberg catalog"]
    postgres[("Postgres")]
    s3[("S3 storage")]

    browser -->|"read a result page"| gateway
    gateway -->|"HTTP"| rs
    gateway -.->|"websocket: which pages changed"| cu
    rs -->|"table location"| lakekeeper
    rs -->|"Parquet files"| s3
    cu -.->|"writes results"| s3
    lakekeeper -->|"metadata"| postgres

    classDef added stroke:#cf222e,stroke-width:2px
    class rs added
Loading
Endpoint Returns
GET /api/results/workflows/{wid}/latest-execution The caller's own latest finished execution
GET /api/results/executions/{eid} Each result of an execution: operator, port, output mode and row count, from the database alone
GET /api/results/executions/{eid}/operators/{operatorId}/pages One page, with the existing column offset, limit and search; port defaults to 0, and a visualization's content is sent whole
GET /api/results/executions/{eid}/operators/{operatorId}/statistics Per-column statistics; port defaults to 0

Two choices go beyond the issue. A read needs the caller to have started the execution, or to have access to the computing unit it ran on, the boundary these results sit behind today. Access to the workflow is not enough, public or granted, since a run reads datasets with the access of the user who started it. And pageSize is capped at 1000, because a page is read into memory in a service everyone shares.

TerminalWorkflowExecutionStatus moves from computing-unit-managing-service to WorkflowCore, so both services read one definition of an ended execution. Three follow-ups promised on #9219 ride along: getResultUriByLogicalPortId skips rows with no URI, and two Scaladoc comments are corrected.

About 2,900 of the added lines are result-service/LICENSE-binary and NOTICE-binary, which the other platform services carry too and CI checks against the dist. The code to review is mostly ResultResource.scala and its spec.

Any related issues, documentation, discussions?

Part of #8995, which the deployment PR will close. Part of #5363.

How was this PR tested?

  • ResultResourceSpec sends requests through the production Jersey and JWT stack against an embedded database and real Iceberg tables in a local Hadoop catalog, the way WorkflowCore's storage specs do. It covers the read rule on every endpoint, paging, column search, rows appended between reads and each error code.
  • 44 deliberate defects, such as a removed access check or an off-by-one page, each turned a test red.
  • Before the review changes, which left table reading alone, ran the packaged service against local Lakekeeper and RustFS on tables seeded through DocumentFactory: every endpoint, a result in a per-user warehouse, and rows appended while it stayed up.
  • smoke-boot.sh passes on the dist, check_binary_deps.py passes for its 311 jars, and NOTICE-binary is the generator's output.
  • Auth/test, WorkflowCore/test, ResultService/test and ComputingUnitManagingService/test pass, amber still compiles, and scalafmtCheck and scalafix --check pass for those modules.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (claude-opus-5-5)

Reads execution results without a computing unit, through Lakekeeper,
the way file-service reads files through LakeFS. Nothing calls it yet:
the deployment and the frontend follow.

Makes the catalog cache's bounds configurable, since this service
touches far more warehouses than a computing unit does. Also skips
rows with no URI in getResultUriByLogicalPortId and corrects two
Scaladoc comments, as promised on apache#9219.

Part of apache#8995.
@github-actions github-actions Bot added feature dependencies Pull requests that update a dependency file ci changes related to CI docs Changes related to documentations infra common labels Nov 7, 2026
@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @aglinxinyuan, @xuang7, @zyratlo
    You can notify them by mentioning @aglinxinyuan, @xuang7, @zyratlo in a comment.

@codecov-commenter

codecov-commenter commented Nov 7, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.71429% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.56%. Comparing base (4ab9bab) to head (4679a17).

Files with missing lines Patch % Lines
...cala/org/apache/texera/service/ResultService.scala 66.66% 9 Missing ⚠️
...pache/texera/service/resource/ResultResource.scala 92.85% 4 Missing and 2 partials ⚠️
...he/texera/service/ResultServiceConfiguration.scala 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #9356      +/-   ##
============================================
- Coverage     91.58%   91.56%   -0.02%     
- Complexity     5133     5157      +24     
============================================
  Files          1296     1299       +3     
  Lines         57986    58086     +100     
  Branches       7601     7615      +14     
============================================
+ Hits          53106    53189      +83     
- Misses         2945     2959      +14     
- Partials       1935     1938       +3     
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø)
agent-service 99.56% <ø> (ø)
amber 85.42% <ø> (-0.01%) ⬇️ Carriedforward from 1d7c006
computing-unit-managing-service 63.88% <ø> (-0.15%) ⬇️
config-service 87.50% <ø> (ø)
file-service 82.16% <ø> (ø)
frontend 96.45% <ø> (ø)
notebook-migration-service 83.77% <ø> (ø)
pyamber 98.38% <ø> (ø)
result-service 85.71% <85.71%> (?)
workflow-compiling-service 78.30% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 1 worse · ⚪ 12 noise (<±5%) · 0 without baseline

Compared against main 4ab9bab benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 498 0.304 19,254/30,780/30,780 us 🟢 -20.8% / 🔴 +140.3%
⚪ bs=100 sw=10 sl=64 1,026 0.626 95,818/115,194/115,194 us ⚪ within ±5% / 🔴 +40.8%
⚪ bs=1000 sw=10 sl=64 1,173 0.716 852,705/899,258/899,258 us ⚪ within ±5% / 🔴 -22.7%
Baseline details

Latest main 4ab9bab from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 498 tuples/sec 521 tuples/sec 1,084 tuples/sec -4.4% -54.1%
bs=10 sw=10 sl=64 MB/s 0.304 MB/s 0.318 MB/s 0.662 MB/s -4.4% -54.1%
bs=10 sw=10 sl=64 p50 19,254 us 18,177 us 9,548 us +5.9% +101.7%
bs=10 sw=10 sl=64 p95 30,780 us 38,852 us 12,811 us -20.8% +140.3%
bs=10 sw=10 sl=64 p99 30,780 us 38,852 us 15,432 us -20.8% +99.5%
bs=100 sw=10 sl=64 throughput 1,026 tuples/sec 1,058 tuples/sec 1,467 tuples/sec -3.0% -30.1%
bs=100 sw=10 sl=64 MB/s 0.626 MB/s 0.646 MB/s 0.895 MB/s -3.1% -30.1%
bs=100 sw=10 sl=64 p50 95,818 us 94,686 us 74,441 us +1.2% +28.7%
bs=100 sw=10 sl=64 p95 115,194 us 110,561 us 81,800 us +4.2% +40.8%
bs=100 sw=10 sl=64 p99 115,194 us 110,561 us 103,712 us +4.2% +11.1%
bs=1000 sw=10 sl=64 throughput 1,173 tuples/sec 1,165 tuples/sec 1,517 tuples/sec +0.7% -22.7%
bs=1000 sw=10 sl=64 MB/s 0.716 MB/s 0.711 MB/s 0.926 MB/s +0.7% -22.7%
bs=1000 sw=10 sl=64 p50 852,705 us 856,735 us 725,911 us -0.5% +17.5%
bs=1000 sw=10 sl=64 p95 899,258 us 897,536 us 764,152 us +0.2% +17.7%
bs=1000 sw=10 sl=64 p99 899,258 us 897,536 us 808,901 us +0.2% +11.2%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,401.32,200,128000,498,0.304,19254.11,30780.20,30780.20
1,100,10,64,20,1949.82,2000,1280000,1026,0.626,95818.06,115193.97,115193.97
2,1000,10,64,20,17045.55,20000,12800000,1173,0.716,852705.26,899257.52,899257.52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved catalog-expiry races can disrupt in-flight operations, while unbounded decoded page data can exhaust the shared service heap.

8 open issues
What changed in this PR

Adds a standalone Dropwizard service for reading stored execution results without a computing unit. Deployment and frontend integration remain follow-ups.

Changes:

  • Add authenticated endpoints for execution lookup, result metadata, pagination, and column statistics.
  • Make catalog-cache limits configurable and skip missing result URIs.
  • Add service tests, distribution licensing, and platform CI integration.
File Description
result-service/​src/​test/​scala/​org/​apache/​texera/​service/​ResultServiceRunSpec.scala Tests service initialization, resource wiring, and authentication setup.
result-service/​src/​test/​scala/​org/​apache/​texera/​service/​resource/​ResultResourceSpec.scala Tests access rules, endpoint responses, paging, and errors.
result-service/​src/​main/​scala/​org/​apache/​texera/​service/​ResultServiceConfiguration.scala Defines the Dropwizard configuration class.
result-service/​src/​main/​scala/​org/​apache/​texera/​service/​ResultService.scala Bootstraps the service, authentication, resources, and logging.
result-service/​src/​main/​scala/​org/​apache/​texera/​service/​resource/​ResultResource.scala Implements execution and result-reading endpoints.
result-service/​src/​main/​resources/​result-service-web-config.yaml Configures port 9102 and service logging.
result-service/​NOTICE-binary Adds bundled dependency notices.
result-service/​LICENSE-binary Documents licenses for packaged dependencies.
result-service/​build.sbt Defines dependencies, tests, and distribution packaging.
common/​workflow-core/​src/​test/​scala/​org/​apache/​texera/​amber/​core/​storage/​ExecutionResultUrisSpec.scala Covers null and empty result URIs.
common/​workflow-core/​src/​main/​scala/​org/​apache/​texera/​amber/​core/​tuple/​TupleUtils.scala Corrects the binary-preview documentation.
common/​workflow-core/​src/​main/​scala/​org/​apache/​texera/​amber/​core/​storage/​IcebergCatalogInstance.scala Uses configurable cache size and expiry.
common/​workflow-core/​src/​main/​scala/​org/​apache/​texera/​amber/​core/​storage/​ExecutionResultUris.scala Reuses the filtered result-URI lookup.
common/​config/​src/​test/​scala/​org/​apache/​texera/​common/​config/​StorageConfigSpec.scala Checks that catalog-cache settings load.
common/​config/​src/​main/​scala/​org/​apache/​texera/​common/​config/​StorageConfig.scala Exposes catalog-cache settings.
common/​config/​src/​main/​resources/​storage.conf Adds cache defaults and environment overrides.
common/​auth/​src/​main/​scala/​org/​apache/​texera/​auth/​util/​WorkflowAccess.scala Clarifies read-access documentation.
codecov.yml Updates the coverage-flag documentation.
build.sbt Registers the service module and its dependencies.
bin/​licensing/​check_binary_deps.py Includes the service in binary-license checks.
AGENTS.md Adds the service to the architecture map.
.github/​workflows/​build.yml Adds service test and distribution smoke-test matrix entries.
.github/​labeler.yml Maps service changes to the platform CI label.

Review effort: Max

Replaces the faked table with real ones in WorkflowCore's local Hadoop
catalog, as the other storage specs do, so ResultResource no longer
needs an injected opener.
Nothing sets them yet, and making them configurable is a separate
change from adding result-service.
Addresses review threads 2 and 6 on apache#9356. A SINGLE_SNAPSHOT port's
page keeps its strings whole, as the computing unit's result push does,
and a page is converted a row at a time instead of holding every
decoded row.
@mengw15
mengw15 requested a max review from Copilot November 12, 2026 11:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open issues
8 resolved since last review

Review effort: Lite

Addresses review threads on apache#9356. The database connection opens in
run and the resource registers by class, as in file-service, the
closest service to this one. Column search lowercases with
Locale.ROOT, so it matches in any default locale.
@mengw15
mengw15 requested a max review from Copilot November 12, 2026 12:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread result-service/build.sbt Resolved
Addresses review threads on apache#9356. PortResult's output mode and row
count are Option, as file-service models optional fields; the JSON
still carries null for them. Also drops two trailing spaces.
@mengw15
mengw15 marked this pull request as ready for review November 12, 2026 12:46
@mengw15
mengw15 requested review from bobbai00 and kunwp1 November 12, 2026 12:46

@kunwp1 kunwp1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Addresses review on apache#9356. One private query reads each stored
result's port, URI, output mode and count, and all three endpoints use
it, so a page no longer reads operator_port_executions twice and the
service has one definition of an execution's result port.
@mengw15

mengw15 commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

@bobbai00 could you take a look when you have a moment? kunwp1 has approved, and the review threads are addressed.

@bobbai00 bobbai00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 0 must-fix · 2 advisory · 3 polish: a careful module that follows file-service and the CU's read path closely; one access question is worth settling before it is deployed.

Design & architecture (1)

  • ResultResource.scala:222: should a workflow grant read every run's rows? (advisory, see inline)

Simplifications (1)

  • ResultResource.scala:65: third hand copy of the persisted terminal status codes (advisory, see inline)

Polish: 3 quick touch-ups (see inline comments).

Verification trace

Traced the four endpoints: each is role-guarded at the class (ResultResource.scala:101), the three result reads call requireReadable before any bounds check or table read, the URI always comes from operator_port_executions rather than the request, and every query is jOOQ. The bounds keep from + pageSize within Int (ResultResource.scala:165), and drop/take avoids the overflow in the CU's slice. Against ExecutionResultService.handleResultPagination, pages match row for row, with a locale-stable search, a 404 where the CU returns an empty page, and visualizations sent whole as the CU's push sends them.

@bobbai00 bobbai00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please polish accordingly. It would be great if you can add architecture diagrams of "before" and "after"

Addresses bobbai00's review on apache#9356. A run is readable by whoever
started it or can use the computing unit it ran on, the boundary these
rows have today; a workflow grant alone no longer reads others' runs.

Also moves TerminalWorkflowExecutionStatus to WorkflowCore, so this
service and computing-unit-managing-service read one definition of an
ended execution, applies two suggested comments, and counts six entry
points that open the database in run() in three run specs.
@github-actions github-actions Bot added the platform Non-amber Scala service paths label Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci changes related to CI common dependencies Pull requests that update a dependency file docs Changes related to documentations feature infra platform Non-amber Scala service paths

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants