Skip to content

Preserve partial worker models on failure and stop completion - #1295

Open
ale18V wants to merge 4 commits into
feat/training-worker-page-client-alternativefrom
fix/worker-model-recovery
Open

ale18V wants to merge 4 commits into
feat/training-worker-page-client-alternativefrom
fix/worker-model-recovery

Conversation

@ale18V

@ale18V ale18V commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

What & why

Stacked on #1287 and targeting its branch. Stop now interrupts network waits after startup without disposing the partial model, so a client waiting for participants or aggregation can stop and save its progress. Terminal messages return the model before connection cleanup, and the page waits separately for cleanup before terminating the worker.

Validation: unit/integration tests, five Chrome browser checks, lint, formatting, TypeScript checks, and the production build pass. A real-server check with two LUS browser clients and 96 images per client completed three collaborative rounds; after stopping the first client and leaving the second waiting alone for 20 seconds, they stopped in 0.83 s and 0.09 s, and both models saved. This ran on Chrome 154/Linux with software WebGL.

Technical plan

Reuse the client's existing abort mechanism once startup has handed the model to the trainer. Keep explicit success and error sends, returning the encoded model before closing connections. A stop acknowledgment cancels the response watchdog, while a separate cleanup acknowledgment allows the page to wait for shutdown after receiving the model.

Deviations from plan

None.

Todo

Remaining review suggestions are tracked in the review threads.

@ale18V
ale18V added this pull request to stack #1296 November 18, 2026 10:08
@ale18V
ale18V requested a review from JulienVig November 18, 2026 10:09
@ale18V ale18V changed the title Preserve partial worker models on failure and stop completion Simplify worker dataset inputs and preserve partial training models Nov 18, 2026
@ale18V
ale18V force-pushed the fix/worker-model-recovery branch from ccc7b55 to e241d11 Compare November 18, 2026 10:45
@ale18V ale18V changed the title Simplify worker dataset inputs and preserve partial training models Preserve partial worker models on failure and stop completion Nov 18, 2026

@JulienVig JulienVig left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a few suggestions but not blocking

Comment on lines 109 to 110
await disco?.close().catch(() => undefined);
send({

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should send before calling close: in decentralized close starts the drain timeout that defaults to 10s so it could timeout before we send the model.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in #1295. Now the worker returns the model before closing connections, so slow cleanup cannot delay the model handoff.

const roundStatus = ref<[RoundStatus, string]>();
const training = ref(false);
let worker: Worker | undefined;
let finishing = false;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're using 3 state variables to track the training state: finishing, worker and stopTimeout. To simplify things and avoid mistakes we could use a single phase variable:

interface Session {
  worker: Worker;
  closePeerTransport?: () => void;
}

type Phase =
  | { kind: "idle" }
  | { kind: "training"; session: Session }
  | { kind: "stopping"; session: Session; timeout: ReturnType<typeof setTimeout> }
  | { kind: "finishing"; session: Session };

const phase = shallowRef<Phase>({ kind: "idle" });
const isTraining = computed(() => phase.value.kind !== "idle");

These types would allow more robust transitions and avoid the different state variables from getting out of sync.
In the UI this even allows us to replace "Stop training" by "Stopping..." after clicking on the button

stopTimeout = undefined;
try {
if (message.model !== undefined) {
const model = await modelDecode(message.model);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the worker training reports an error and an inval model, this line will throw and hide the original worker error. We could report any worker error before anything else

@JulienVig

JulienVig commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

@ale18V I just tried to train a LUS model in the UI with two clients. After a few rounds, clicking stop training in the first tab stops the training immediately but then stopping the second tab takes around 30s. I also tried it on the current main branch and the second tab stops instantly. Could you look into what introduces this lag? If this is intentional then we should definitly add some UX feedback after clicking stop

Let startup finish before applying Stop, then reuse the client's existing abort mechanism to interrupt participant, peer and aggregation waits without disposing the model.

Return the model before closing connections and acknowledge cleanup separately. Cover network cancellation, deferred startup stopping, slow cleanup and saving partial models through the app.
@ale18V

ale18V commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed in #1295. Now Stop interrupts participant waits and preserves the model. In the two-tab LUS check, the second client stopped in 92 ms and both models saved.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants