Skip to content

fix(workflow): stop rerunning a waiting wait_for_output node on resume - #7991

Open
AkshayShah03 wants to merge 1 commit into
google:mainfrom
AkshayShah03:fix/wait-for-output-superseded-rerun
Open

AkshayShah03 wants to merge 1 commit into
google:mainfrom
AkshayShah03:fix/wait-for-output-superseded-rerun

Conversation

@AkshayShah03

Copy link
Copy Markdown
Contributor

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

Problem:
When a workflow resumed after a RequestInput was answered, check_interception Case 6 reran every wait_for_output node whose recorded run had no output, route or interrupt. Those runs were executed again with the input they had already handled, so an accumulating node (the pattern the wait_for_output guide describes):

  • counted the same input twice (seen == ['B', 'C', 'B']);
  • could emit with the wrong data, for example ['a', 'a'] when it was still waiting for its second input;
  • in a loop, emitted again, ran the loop an extra lap, asked the user for the same approval a second time, and then hung.

This came in with 9d306f5, which widened the Case 6 rerun from nested Workflow nodes to every wait_for_output node so that task-mode agents rerun on resume.

Solution:
As discussed on #7905, narrow the Case 6 rerun back to the nodes that wait for new input rather than for a trigger: agents (task or chat mode, which take the user's next message) and rerun_on_resume nodes. Nested workflows are covered by the second, since they default to rerun_on_resume=True. Any other wait_for_output node falls through to the default Case 6 handling. Its next trigger starts a new run anyway, so a static run is fast-forwarded and a dynamic run still reruns, as for any other node.

docs/guides/workflow/base_node/index.md gains a paragraph on resume: a waiting node in the graph is not rerun, and a custom node that waits on the user rather than on a trigger needs rerun_on_resume=True.

A run that ended waiting without emitting any event leaves nothing to replay. That part is in #7771's marker logic, so it's a follow-up commit there (d65ab2e), as suggested on the issue.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

New tests, each parametrized for resumable and non-resumable apps:

  • test_workflow_hitl.py:

    • test_resume_does_not_rerun_superseded_waiting_run: a waiting run followed by a later run of the same node.
    • test_resume_does_not_rerun_waiting_run_before_its_next_trigger: a waiting run that is still the node's latest run.
    • test_resume_does_not_rerun_waiting_runs_from_earlier_loop_laps: the loop case.

    All 6 cases fail on main.

  • utils/test_replay_interceptor.py:

    • test_cross_turn_no_outcome_wait_for_output_node_like_plain_node: a static run is fast-forwarded and a dynamic run reruns. The static case fails on main.
    • test_cross_turn_no_outcome_waiting_agent_or_rerun_node_reruns: agents and rerun_on_resume nodes still rerun. Passes on main and here.

test_workflow_task_mode_plain_text_resume_auto_routing and the other task- and chat-mode tests pass.

Full unit suite, with this change and #7771 (rebased, with its follow-up) applied together: the same 6 failures as main (test_google_llm continuation-token tests, test_release_dependencies and test_app_details, which also fails on main when the whole suite runs). Nothing new.

Fuzzer: 1,500 random workflow graphs (loops, joins, retries, interrupts, wait_for_output and dynamic nodes), each run with and without resumability (3,000 runs). main hits 24 replay divergences. With this change and #7771 together there are 0, and nothing new fails.

Manual End-to-End (E2E) Tests:

The repro from #7905 (the simple and loop cases), with and without ResumabilityConfig(is_resumable=True): gather no longer runs a second time, done runs once with the right input, and the loop asks once.

When a workflow resumed after a RequestInput was answered, every recorded
run of a wait_for_output node that had not produced output yet was executed
again with the input it already handled. An accumulating node therefore
counted inputs twice, could emit with the wrong data, and in a loop ran the
loop again and asked the user a second time.

Case 6 of check_interception reran every wait_for_output node with no
recorded outcome. That is needed for task- and chat-mode agents, which wait
for the user's next message, and for rerun_on_resume nodes. Any other
wait_for_output node gets a new run for its next trigger, so it now falls
through to the same handling as other nodes: a static run is fast-forwarded
and a dynamic run still reruns.

Fixes google#7905
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Resuming a workflow re-runs the earlier no-output runs of a wait_for_output node, so its side effects happen twice

2 participants