Repository navigation
Conversation
5c926c6 to
1b7f078
Compare
JulienVig
left a comment
There was a problem hiding this comment.
Great! Only minor comments
| ); | ||
|
|
||
| const isTraining = computed(() => trainingGenerator.value !== undefined); | ||
| const isTraining = training; |
There was a problem hiding this comment.
| const isTraining = training; | |
| const isTraining = ref(false); |
I don't think keeping both training and isTraining is necessary
| closePeerTransport = servePeerTransport(channel.port1); | ||
| peerPort = channel.port2; | ||
| } | ||
| active.onmessage = (event: MessageEvent<FromWorker>) => { |
There was a problem hiding this comment.
Maybe a separate function to init the worker would de-clutter startTraining?
| const send = (message: FromWorker): void => { | ||
| const bytes = message.type === "finished" ? message.model : undefined; | ||
| self.postMessage(message, { | ||
| transfer: bytes?.buffer instanceof ArrayBuffer ? [bytes.buffer] : [], | ||
| }); | ||
| }; | ||
| let stopRequested = false; | ||
| async function train( |
There was a problem hiding this comment.
I prefer keeping a consistent style for functions by using function, is it useful to use const here?
Also added empty lines between functions
| const send = (message: FromWorker): void => { | |
| const bytes = message.type === "finished" ? message.model : undefined; | |
| self.postMessage(message, { | |
| transfer: bytes?.buffer instanceof ArrayBuffer ? [bytes.buffer] : [], | |
| }); | |
| }; | |
| let stopRequested = false; | |
| async function train( | |
| function send(message: FromWorker): void { | |
| const bytes = message.type === "finished" ? message.model : undefined; | |
| self.postMessage(message, { | |
| transfer: bytes?.buffer instanceof ArrayBuffer ? [bytes.buffer] : [], | |
| }); | |
| }; | |
| let stopRequested = false; | |
| async function train( |
| for await (const log of epochGenerator) { | ||
| send({ type: "batch", log }); | ||
| if (stopRequested) break; | ||
| } | ||
| if (stopRequested) { | ||
| stopped = true; | ||
| break; | ||
| } |
There was a problem hiding this comment.
You could use a labeled statement here to avoid breaking multiple times
| @@ -0,0 +1,201 @@ | |||
| import { | |||
There was a problem hiding this comment.
Can you document what this file if used for?
| stopTimeout = setTimeout(() => { | ||
| if (worker !== active) return; | ||
| closeWorker(active); | ||
| toaster.info("Training stopped"); | ||
| }, 10_000); |
There was a problem hiding this comment.
The worker checks stopRequest in-between batches. Waiting for more participants or for aggregation can take longer than 10s so this timeout would trigger and throw the partial model away. I don't think that #1295 fixes this either
There was a problem hiding this comment.
Addressed in #1295. Now Stop interrupts pending network waits so the worker can return the partial model before termination.
| const msg: unknown = msgpack.decode(data); | ||
|
|
||
| if (!decentralizedMessages.isPeerMessage(msg)) { | ||
| throw new Error(`invalid message received: ${JSON.stringify(msg)}`); |
There was a problem hiding this comment.
There are many of these Error that are now thrown inside a web worker which would trigger onerror and kill the worker while it didn't use to kill training before. We should likely make this a log instead. WebSocket.onmessage also throws, etc
What & why
Training on the main thread can freeze the UI for large models. The worker now owns training, aggregation, weight serialization, and round coordination, while a peer-transport adapter connects page-owned WebRTC peers through signaling events and opaque bytes.
The benchmark in #1297 recorded 195 cases across three repetitions on Chrome 154/Linux, with the following one-peer median round-trip times for a complete WebRTC payload and its echoed response:
In the small CPU training-and-exchange workload, the median maximum page-timer lag fell from 36.1 ms to 1.1 ms, although the 64 MiB bridged WebRTC cases still had 58.2 ms median maximum lag. These results support the worker design's responsiveness benefit with modest measured exchange overhead, while remaining limited to one machine/browser, loopback networking, and a small CPU training workload.
Technical plan
training.worker.tsruns Disco with WebGL and CPU fallback, using the existing dataset loaders on files and labels passed by the page.peer_transport.tsconnects page-owned peers to the worker through a MessagePort; an injectable peer factory lets DecentralizedClient use this transport without changing its training protocol.worker_messages.tshandles startup, progress, errors, cancellation, and the final model, while TrainerDashboard manages the worker lifecycle and UI. Labeled inputs emit a singledatasetSource, using a temporary dataset for early CSV validation; evaluation constructs its own dataset and retains image filenames. Local validation covers webapp unit tests, lint, formatting, TypeScript checks, and the production build; browser E2E tests also run in CI.Deviations from plan
Following review, replaced the PageClient round-level bridge with peer transport and moved training dataset loading into the worker.
Todo
Explore ownership-aware buffer transfer to reduce copying: the isolated 64 MiB bridge round trip took about 57 ms versus near-clock-resolution transfer-only controls, suggesting copying/allocation dominates at large payload sizes. Validate other browsers/hardware, remote-network/full Disco sessions, and larger GPU models before generalizing the benchmark results. Measure peak memory and the final-model saving path separately.