Repository navigation
fix(workflow): stop rerunning a waiting wait_for_output node on resume - #7991
Open
AkshayShah03 wants to merge 1 commit into
Open
AkshayShah03 wants to merge 1 commit into
AkshayShah03 wants to merge 1 commit into
Conversation
When a workflow resumed after a RequestInput was answered, every recorded run of a wait_for_output node that had not produced output yet was executed again with the input it already handled. An accumulating node therefore counted inputs twice, could emit with the wrong data, and in a loop ran the loop again and asked the user a second time. Case 6 of check_interception reran every wait_for_output node with no recorded outcome. That is needed for task- and chat-mode agents, which wait for the user's next message, and for rerun_on_resume nodes. Any other wait_for_output node gets a new run for its next trigger, so it now falls through to the same handling as other nodes: a static run is fast-forwarded and a dynamic run still reruns. Fixes google#7905
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to Issue or Description of Change
1. Link to an existing issue (if applicable):
Problem:
When a workflow resumed after a
RequestInputwas answered,check_interceptionCase 6 reran everywait_for_outputnode whose recorded run had no output, route or interrupt. Those runs were executed again with the input they had already handled, so an accumulating node (the pattern thewait_for_outputguide describes):seen == ['B', 'C', 'B']);['a', 'a']when it was still waiting for its second input;This came in with 9d306f5, which widened the Case 6 rerun from nested
Workflownodes to everywait_for_outputnode so that task-mode agents rerun on resume.Solution:
As discussed on #7905, narrow the Case 6 rerun back to the nodes that wait for new input rather than for a trigger: agents (task or chat mode, which take the user's next message) and
rerun_on_resumenodes. Nested workflows are covered by the second, since they default torerun_on_resume=True. Any otherwait_for_outputnode falls through to the default Case 6 handling. Its next trigger starts a new run anyway, so a static run is fast-forwarded and a dynamic run still reruns, as for any other node.docs/guides/workflow/base_node/index.mdgains a paragraph on resume: a waiting node in the graph is not rerun, and a custom node that waits on the user rather than on a trigger needsrerun_on_resume=True.A run that ended waiting without emitting any event leaves nothing to replay. That part is in #7771's marker logic, so it's a follow-up commit there (d65ab2e), as suggested on the issue.
Testing Plan
Unit Tests:
New tests, each parametrized for resumable and non-resumable apps:
test_workflow_hitl.py:test_resume_does_not_rerun_superseded_waiting_run: a waiting run followed by a later run of the same node.test_resume_does_not_rerun_waiting_run_before_its_next_trigger: a waiting run that is still the node's latest run.test_resume_does_not_rerun_waiting_runs_from_earlier_loop_laps: the loop case.All 6 cases fail on
main.utils/test_replay_interceptor.py:test_cross_turn_no_outcome_wait_for_output_node_like_plain_node: a static run is fast-forwarded and a dynamic run reruns. The static case fails onmain.test_cross_turn_no_outcome_waiting_agent_or_rerun_node_reruns: agents andrerun_on_resumenodes still rerun. Passes onmainand here.test_workflow_task_mode_plain_text_resume_auto_routingand the other task- and chat-mode tests pass.Full unit suite, with this change and #7771 (rebased, with its follow-up) applied together: the same 6 failures as
main(test_google_llmcontinuation-token tests,test_release_dependenciesandtest_app_details, which also fails onmainwhen the whole suite runs). Nothing new.Fuzzer: 1,500 random workflow graphs (loops, joins, retries, interrupts,
wait_for_outputand dynamic nodes), each run with and without resumability (3,000 runs).mainhits 24 replay divergences. With this change and #7771 together there are 0, and nothing new fails.Manual End-to-End (E2E) Tests:
The repro from #7905 (the simple and loop cases), with and without
ResumabilityConfig(is_resumable=True):gatherno longer runs a second time,doneruns once with the right input, and the loop asks once.