Repository navigation
Conversation
ccc7b55 to
e241d11
Compare
JulienVig
left a comment
There was a problem hiding this comment.
I have a few suggestions but not blocking
| await disco?.close().catch(() => undefined); | ||
| send({ |
There was a problem hiding this comment.
Maybe we should send before calling close: in decentralized close starts the drain timeout that defaults to 10s so it could timeout before we send the model.
There was a problem hiding this comment.
Addressed in #1295. Now the worker returns the model before closing connections, so slow cleanup cannot delay the model handoff.
| const roundStatus = ref<[RoundStatus, string]>(); | ||
| const training = ref(false); | ||
| let worker: Worker | undefined; | ||
| let finishing = false; |
There was a problem hiding this comment.
We're using 3 state variables to track the training state: finishing, worker and stopTimeout. To simplify things and avoid mistakes we could use a single phase variable:
interface Session {
worker: Worker;
closePeerTransport?: () => void;
}
type Phase =
| { kind: "idle" }
| { kind: "training"; session: Session }
| { kind: "stopping"; session: Session; timeout: ReturnType<typeof setTimeout> }
| { kind: "finishing"; session: Session };
const phase = shallowRef<Phase>({ kind: "idle" });
const isTraining = computed(() => phase.value.kind !== "idle");These types would allow more robust transitions and avoid the different state variables from getting out of sync.
In the UI this even allows us to replace "Stop training" by "Stopping..." after clicking on the button
| stopTimeout = undefined; | ||
| try { | ||
| if (message.model !== undefined) { | ||
| const model = await modelDecode(message.model); |
There was a problem hiding this comment.
if the worker training reports an error and an inval model, this line will throw and hide the original worker error. We could report any worker error before anything else
|
@ale18V I just tried to train a LUS model in the UI with two clients. After a few rounds, clicking stop training in the first tab stops the training immediately but then stopping the second tab takes around 30s. I also tried it on the current main branch and the second tab stops instantly. Could you look into what introduces this lag? If this is intentional then we should definitly add some UX feedback after clicking stop |
Let startup finish before applying Stop, then reuse the client's existing abort mechanism to interrupt participant, peer and aggregation waits without disposing the model. Return the model before closing connections and acknowledge cleanup separately. Cover network cancellation, deferred startup stopping, slow cleanup and saving partial models through the app.
|
Addressed in #1295. Now Stop interrupts participant waits and preserves the model. In the two-tab LUS check, the second client stopped in 92 ms and both models saved. |
What & why
Stacked on #1287 and targeting its branch. Stop now interrupts network waits after startup without disposing the partial model, so a client waiting for participants or aggregation can stop and save its progress. Terminal messages return the model before connection cleanup, and the page waits separately for cleanup before terminating the worker.
Validation: unit/integration tests, five Chrome browser checks, lint, formatting, TypeScript checks, and the production build pass. A real-server check with two LUS browser clients and 96 images per client completed three collaborative rounds; after stopping the first client and leaving the second waiting alone for 20 seconds, they stopped in 0.83 s and 0.09 s, and both models saved. This ran on Chrome 154/Linux with software WebGL.
Technical plan
Reuse the client's existing abort mechanism once startup has handed the model to the trainer. Keep explicit success and error sends, returning the encoded model before closing connections. A stop acknowledgment cancels the response watchdog, while a separate cleanup acknowledgment allows the page to wait for shutdown after receiving the model.
Deviations from plan
None.
Todo
Remaining review suggestions are tracked in the review threads.