Repository navigation
Conversation
Reads execution results without a computing unit, through Lakekeeper, the way file-service reads files through LakeFS. Nothing calls it yet: the deployment and the frontend follow. Makes the catalog cache's bounds configurable, since this service touches far more warehouses than a computing unit does. Also skips rows with no URI in getResultUriByLogicalPortId and corrects two Scaladoc comments, as promised on apache#9219. Part of apache#8995.
Automated Reviewer SuggestionsBased on the
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #9356 +/- ##
============================================
- Coverage 91.58% 91.56% -0.02%
- Complexity 5133 5157 +24
============================================
Files 1296 1299 +3
Lines 57986 58086 +100
Branches 7601 7615 +14
============================================
+ Hits 53106 53189 +83
- Misses 2945 2959 +14
- Partials 1935 1938 +3
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 498 | 0.304 | 19,254/30,780/30,780 us | 🟢 -20.8% / 🔴 +140.3% |
| ⚪ | bs=100 sw=10 sl=64 | 1,026 | 0.626 | 95,818/115,194/115,194 us | ⚪ within ±5% / 🔴 +40.8% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,173 | 0.716 | 852,705/899,258/899,258 us | ⚪ within ±5% / 🔴 -22.7% |
Baseline details
Latest main 4ab9bab from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 498 tuples/sec | 521 tuples/sec | 1,084 tuples/sec | -4.4% | -54.1% |
| bs=10 sw=10 sl=64 | MB/s | 0.304 MB/s | 0.318 MB/s | 0.662 MB/s | -4.4% | -54.1% |
| bs=10 sw=10 sl=64 | p50 | 19,254 us | 18,177 us | 9,548 us | +5.9% | +101.7% |
| bs=10 sw=10 sl=64 | p95 | 30,780 us | 38,852 us | 12,811 us | -20.8% | +140.3% |
| bs=10 sw=10 sl=64 | p99 | 30,780 us | 38,852 us | 15,432 us | -20.8% | +99.5% |
| bs=100 sw=10 sl=64 | throughput | 1,026 tuples/sec | 1,058 tuples/sec | 1,467 tuples/sec | -3.0% | -30.1% |
| bs=100 sw=10 sl=64 | MB/s | 0.626 MB/s | 0.646 MB/s | 0.895 MB/s | -3.1% | -30.1% |
| bs=100 sw=10 sl=64 | p50 | 95,818 us | 94,686 us | 74,441 us | +1.2% | +28.7% |
| bs=100 sw=10 sl=64 | p95 | 115,194 us | 110,561 us | 81,800 us | +4.2% | +40.8% |
| bs=100 sw=10 sl=64 | p99 | 115,194 us | 110,561 us | 103,712 us | +4.2% | +11.1% |
| bs=1000 sw=10 sl=64 | throughput | 1,173 tuples/sec | 1,165 tuples/sec | 1,517 tuples/sec | +0.7% | -22.7% |
| bs=1000 sw=10 sl=64 | MB/s | 0.716 MB/s | 0.711 MB/s | 0.926 MB/s | +0.7% | -22.7% |
| bs=1000 sw=10 sl=64 | p50 | 852,705 us | 856,735 us | 725,911 us | -0.5% | +17.5% |
| bs=1000 sw=10 sl=64 | p95 | 899,258 us | 897,536 us | 764,152 us | +0.2% | +17.7% |
| bs=1000 sw=10 sl=64 | p99 | 899,258 us | 897,536 us | 808,901 us | +0.2% | +11.2% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,401.32,200,128000,498,0.304,19254.11,30780.20,30780.20
1,100,10,64,20,1949.82,2000,1280000,1026,0.626,95818.06,115193.97,115193.97
2,1000,10,64,20,17045.55,20000,12800000,1173,0.716,852705.26,899257.52,899257.52There was a problem hiding this comment.
🟡 Changes recommended
Unresolved catalog-expiry races can disrupt in-flight operations, while unbounded decoded page data can exhaust the shared service heap.
8 open issues
With an expiry shorter than a table read or commit, another cache access can expire and close the… · New The row limit does not bound the memory used here:toListretains every full decoded tuple before… · New This query has no supporting index onworkflow_executions.uid/vidorworkflow_version.wid. A… · NewExecutionResultService.onResultUpdateonly persistsRESULT_COUNTfor port 0 and skips… · NewLARGE_BINARYcolumns are passed as logical names (image), butIcebergDocument.getRange… · NewconvertTuplesToJson(rows)defaults toisVisualization = false, which truncates STRING values… · New The underlyingIcebergDocument.getTableStatisticsinitializes numeric and timestamp min/max to… · NewIcebergUtil.loadTableMetadataturns every catalog exception intoNone, so a Lakekeeper timeout… · New
What changed in this PR
Adds a standalone Dropwizard service for reading stored execution results without a computing unit. Deployment and frontend integration remain follow-ups.
Changes:
- Add authenticated endpoints for execution lookup, result metadata, pagination, and column statistics.
- Make catalog-cache limits configurable and skip missing result URIs.
- Add service tests, distribution licensing, and platform CI integration.
| File | Description |
|---|---|
result-service/src/test/scala/org/apache/texera/service/ResultServiceRunSpec.scala |
Tests service initialization, resource wiring, and authentication setup. |
result-service/src/test/scala/org/apache/texera/service/resource/ResultResourceSpec.scala |
Tests access rules, endpoint responses, paging, and errors. |
result-service/src/main/scala/org/apache/texera/service/ResultServiceConfiguration.scala |
Defines the Dropwizard configuration class. |
result-service/src/main/scala/org/apache/texera/service/ResultService.scala |
Bootstraps the service, authentication, resources, and logging. |
result-service/src/main/scala/org/apache/texera/service/resource/ResultResource.scala |
Implements execution and result-reading endpoints. |
result-service/src/main/resources/result-service-web-config.yaml |
Configures port 9102 and service logging. |
result-service/NOTICE-binary |
Adds bundled dependency notices. |
result-service/LICENSE-binary |
Documents licenses for packaged dependencies. |
result-service/build.sbt |
Defines dependencies, tests, and distribution packaging. |
common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/ExecutionResultUrisSpec.scala |
Covers null and empty result URIs. |
common/workflow-core/src/main/scala/org/apache/texera/amber/core/tuple/TupleUtils.scala |
Corrects the binary-preview documentation. |
common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/IcebergCatalogInstance.scala |
Uses configurable cache size and expiry. |
common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/ExecutionResultUris.scala |
Reuses the filtered result-URI lookup. |
common/config/src/test/scala/org/apache/texera/common/config/StorageConfigSpec.scala |
Checks that catalog-cache settings load. |
common/config/src/main/scala/org/apache/texera/common/config/StorageConfig.scala |
Exposes catalog-cache settings. |
common/config/src/main/resources/storage.conf |
Adds cache defaults and environment overrides. |
common/auth/src/main/scala/org/apache/texera/auth/util/WorkflowAccess.scala |
Clarifies read-access documentation. |
codecov.yml |
Updates the coverage-flag documentation. |
build.sbt |
Registers the service module and its dependencies. |
bin/licensing/check_binary_deps.py |
Includes the service in binary-license checks. |
AGENTS.md |
Adds the service to the architecture map. |
.github/workflows/build.yml |
Adds service test and distribution smoke-test matrix entries. |
.github/labeler.yml |
Maps service changes to the platform CI label. |
Review effort: Max
Replaces the faked table with real ones in WorkflowCore's local Hadoop catalog, as the other storage specs do, so ResultResource no longer needs an injected opener.
Nothing sets them yet, and making them configurable is a separate change from adding result-service.
Addresses review threads 2 and 6 on apache#9356. A SINGLE_SNAPSHOT port's page keeps its strings whole, as the computing unit's result push does, and a page is converted a row at a time instead of holding every decoded row.
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
4 open issues
Initializing the DB connection pool ininitializeintroduces side effects during Dropewizard… · New This updated Scaladoc appears to define workflow readability as including public workflows, but… · New UsingtoLowerCasewithout a fixed locale makes filtering behavior locale-dependent (e.g., Turkish… · New Mapping ‘missing storage’ to 404 by checking a substring inIllegalArgumentException.getMessage… · New
8 resolved since last review
The row limit does not bound the memory used here:toListretains every full decoded tuple before… With an expiry shorter than a table read or commit, another cache access can expire and close the…IcebergUtil.loadTableMetadataturns every catalog exception intoNone, so a Lakekeeper timeout… The underlyingIcebergDocument.getTableStatisticsinitializes numeric and timestamp min/max to…convertTuplesToJson(rows)defaults toisVisualization = false, which truncates STRING values…LARGE_BINARYcolumns are passed as logical names (image), butIcebergDocument.getRange…ExecutionResultService.onResultUpdateonly persistsRESULT_COUNTfor port 0 and skips… This query has no supporting index onworkflow_executions.uid/vidorworkflow_version.wid. A…
Review effort: Lite
Addresses review threads on apache#9356. The database connection opens in run and the resource registers by class, as in file-service, the closest service to this one. Column search lowercases with Locale.ROOT, so it matches in any default locale.
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
5 open issues
This adds a build-wide (Global) restriction limiting all tests in the entire multi-project build to… · NewSessionHandleris a Jetty handler and is typically configured via the servlet environment (as… · NewPortResultexplicitly models nullable fields (outputModecan be null per the comment, and… · New Mapping ‘missing storage’ to 404 by checking a substring inIllegalArgumentException.getMessage… · New There is trailing whitespace at the end of this line, which can cause noisy diffs and may fail… · New
4 resolved since last review
Initializing the DB connection pool ininitializeintroduces side effects during Dropewizard… Mapping ‘missing storage’ to 404 by checking a substring inIllegalArgumentException.getMessage… UsingtoLowerCasewithout a fixed locale makes filtering behavior locale-dependent (e.g., Turkish… This updated Scaladoc appears to define workflow readability as including public workflows, but…
Review effort: Lite
Addresses review threads on apache#9356. PortResult's output mode and row count are Option, as file-service models optional fields; the JSON still carries null for them. Also drops two trailing spaces.
Addresses review on apache#9356. One private query reads each stored result's port, URI, output mode and count, and all three endpoints use it, so a page no longer reads operator_port_executions twice and the service has one definition of an execution's result port.
|
@bobbai00 could you take a look when you have a moment? kunwp1 has approved, and the review threads are addressed. |
bobbai00
left a comment
There was a problem hiding this comment.
🟡 0 must-fix · 2 advisory · 3 polish: a careful module that follows file-service and the CU's read path closely; one access question is worth settling before it is deployed.
Design & architecture (1)
ResultResource.scala:222: should a workflow grant read every run's rows? (advisory, see inline)
Simplifications (1)
ResultResource.scala:65: third hand copy of the persisted terminal status codes (advisory, see inline)
Polish: 3 quick touch-ups (see inline comments).
Verification trace
Traced the four endpoints: each is role-guarded at the class (ResultResource.scala:101), the three result reads call requireReadable before any bounds check or table read, the URI always comes from operator_port_executions rather than the request, and every query is jOOQ. The bounds keep from + pageSize within Int (ResultResource.scala:165), and drop/take avoids the overflow in the CU's slice. Against ExecutionResultService.handleResultPagination, pages match row for row, with a locale-stable search, a 404 where the CU returns an empty page, and visualizations sent whole as the CU's push sends them.
bobbai00
left a comment
There was a problem hiding this comment.
Please polish accordingly. It would be great if you can add architecture diagrams of "before" and "after"
Addresses bobbai00's review on apache#9356. A run is readable by whoever started it or can use the computing unit it ran on, the boundary these rows have today; a workflow grant alone no longer reads others' runs. Also moves TerminalWorkflowExecutionStatus to WorkflowCore, so this service and computing-unit-managing-service read one definition of an ended execution, applies two suggested comments, and counts six entry points that open the database in run() in three run specs.
…rvice # Conflicts: # build.sbt



What changes were proposed in this PR?
Adds
result-service, a Dropwizard service that reads execution results without a computing unit. It stands in front of Lakekeeper the wayfile-servicestands in front of LakeFS. This PR is the module and its CI legs, so nothing calls it yet: its image, Helm chart, gateway route and local-dev entry follow in a second PR, and #8996 moves the frontend over.Before: a result page is read through the computing unit, so a result cannot be read once its unit is gone.
flowchart TB browser(["Browser"]) gateway["Gateway"] cu["Computing unit<br/>engine + Iceberg client"] lakekeeper["Lakekeeper<br/>Iceberg catalog"] postgres[("Postgres")] s3[("S3 storage")] browser -->|"read a result page"| gateway gateway -->|"websocket"| cu cu -->|"table location"| lakekeeper cu -->|"Parquet files"| s3 lakekeeper -->|"metadata"| postgresAfter (with the deployment PR and #8996): pages are read through result-service, which needs no unit. The unit still writes results and, while an execution runs, tells the browser which pages changed. Red marks what is new.
flowchart TB browser(["Browser"]) gateway["Gateway"] rs["result-service<br/>access check<br/>Iceberg client"] cu["Computing unit<br/>engine"] lakekeeper["Lakekeeper<br/>Iceberg catalog"] postgres[("Postgres")] s3[("S3 storage")] browser -->|"read a result page"| gateway gateway -->|"HTTP"| rs gateway -.->|"websocket: which pages changed"| cu rs -->|"table location"| lakekeeper rs -->|"Parquet files"| s3 cu -.->|"writes results"| s3 lakekeeper -->|"metadata"| postgres classDef added stroke:#cf222e,stroke-width:2px class rs addedGET /api/results/workflows/{wid}/latest-executionGET /api/results/executions/{eid}GET /api/results/executions/{eid}/operators/{operatorId}/pagesportdefaults to 0, and a visualization's content is sent wholeGET /api/results/executions/{eid}/operators/{operatorId}/statisticsportdefaults to 0Two choices go beyond the issue. A read needs the caller to have started the execution, or to have access to the computing unit it ran on, the boundary these results sit behind today. Access to the workflow is not enough, public or granted, since a run reads datasets with the access of the user who started it. And
pageSizeis capped at 1000, because a page is read into memory in a service everyone shares.TerminalWorkflowExecutionStatusmoves from computing-unit-managing-service to WorkflowCore, so both services read one definition of an ended execution. Three follow-ups promised on #9219 ride along:getResultUriByLogicalPortIdskips rows with no URI, and two Scaladoc comments are corrected.About 2,900 of the added lines are
result-service/LICENSE-binaryandNOTICE-binary, which the other platform services carry too and CI checks against the dist. The code to review is mostlyResultResource.scalaand its spec.Any related issues, documentation, discussions?
Part of #8995, which the deployment PR will close. Part of #5363.
How was this PR tested?
ResultResourceSpecsends requests through the production Jersey and JWT stack against an embedded database and real Iceberg tables in a local Hadoop catalog, the way WorkflowCore's storage specs do. It covers the read rule on every endpoint, paging, column search, rows appended between reads and each error code.DocumentFactory: every endpoint, a result in a per-user warehouse, and rows appended while it stayed up.smoke-boot.shpasses on the dist,check_binary_deps.pypasses for its 311 jars, andNOTICE-binaryis the generator's output.Auth/test,WorkflowCore/test,ResultService/testandComputingUnitManagingService/testpass, amber still compiles, andscalafmtCheckandscalafix --checkpass for those modules.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-opus-5-5)