Skip to content

Run training and aggregation in a worker with page-owned WebRTC - #1287

Open
ale18V wants to merge 4 commits into
mainfrom
feat/training-worker-page-client-alternative
Open

ale18V wants to merge 4 commits into
mainfrom
feat/training-worker-page-client-alternative

Conversation

@ale18V

@ale18V ale18V commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

What & why

Training on the main thread can freeze the UI for large models. The worker now owns training, aggregation, weight serialization, and round coordination, while a peer-transport adapter connects page-owned WebRTC peers through signaling events and opaque bytes.

The benchmark in #1297 recorded 195 cases across three repetitions on Chrome 154/Linux, with the following one-peer median round-trip times for a complete WebRTC payload and its echoed response:

Payload Direct WebRTC Through worker bridge Increase
1 MiB 21.85 ms 23.10 ms 5.7%
16 MiB 357.80 ms 370.05 ms 3.4%
64 MiB 1,443.20 ms 1,525.60 ms 5.7%

In the small CPU training-and-exchange workload, the median maximum page-timer lag fell from 36.1 ms to 1.1 ms, although the 64 MiB bridged WebRTC cases still had 58.2 ms median maximum lag. These results support the worker design's responsiveness benefit with modest measured exchange overhead, while remaining limited to one machine/browser, loopback networking, and a small CPU training workload.

Technical plan

training.worker.ts runs Disco with WebGL and CPU fallback, using the existing dataset loaders on files and labels passed by the page. peer_transport.ts connects page-owned peers to the worker through a MessagePort; an injectable peer factory lets DecentralizedClient use this transport without changing its training protocol. worker_messages.ts handles startup, progress, errors, cancellation, and the final model, while TrainerDashboard manages the worker lifecycle and UI. Labeled inputs emit a single datasetSource, using a temporary dataset for early CSV validation; evaluation constructs its own dataset and retains image filenames. Local validation covers webapp unit tests, lint, formatting, TypeScript checks, and the production build; browser E2E tests also run in CI.

Deviations from plan

Following review, replaced the PageClient round-level bridge with peer transport and moved training dataset loading into the worker.

Todo

Explore ownership-aware buffer transfer to reduce copying: the isolated 64 MiB bridge round trip took about 57 ms versus near-clock-resolution transfer-only controls, suggesting copying/allocation dominates at large payload sizes. Validate other browsers/hardware, remote-network/full Disco sessions, and larger GPU models before generalizing the benchmark results. Measure peak memory and the final-model saving path separately.

@ale18V
ale18V force-pushed the feat/training-worker-page-client-alternative branch from 5c926c6 to 1b7f078 Compare November 11, 2026 16:29
@ale18V
ale18V requested a review from JulienVig November 11, 2026 17:05
@ale18V ale18V changed the title Train models in a worker with main-thread decentralized aggregation Run training and aggregation in a worker with page-owned WebRTC Nov 12, 2026
@ale18V
ale18V added this pull request to stack #1296 November 18, 2026 10:08

@JulienVig JulienVig left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! Only minor comments

);

const isTraining = computed(() => trainingGenerator.value !== undefined);
const isTraining = training;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const isTraining = training;
const isTraining = ref(false);

I don't think keeping both training and isTraining is necessary

closePeerTransport = servePeerTransport(channel.port1);
peerPort = channel.port2;
}
active.onmessage = (event: MessageEvent<FromWorker>) => {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a separate function to init the worker would de-clutter startTraining?

Comment on lines +16 to +23
const send = (message: FromWorker): void => {
const bytes = message.type === "finished" ? message.model : undefined;
self.postMessage(message, {
transfer: bytes?.buffer instanceof ArrayBuffer ? [bytes.buffer] : [],
});
};
let stopRequested = false;
async function train(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer keeping a consistent style for functions by using function, is it useful to use const here?
Also added empty lines between functions

Suggested change
const send = (message: FromWorker): void => {
const bytes = message.type === "finished" ? message.model : undefined;
self.postMessage(message, {
transfer: bytes?.buffer instanceof ArrayBuffer ? [bytes.buffer] : [],
});
};
let stopRequested = false;
async function train(
function send(message: FromWorker): void {
const bytes = message.type === "finished" ? message.model : undefined;
self.postMessage(message, {
transfer: bytes?.buffer instanceof ArrayBuffer ? [bytes.buffer] : [],
});
};
let stopRequested = false;
async function train(

Comment on lines +58 to +65
for await (const log of epochGenerator) {
send({ type: "batch", log });
if (stopRequested) break;
}
if (stopRequested) {
stopped = true;
break;
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could use a labeled statement here to avoid breaking multiple times

@@ -0,0 +1,201 @@
import {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you document what this file if used for?

Comment on lines +365 to +369
stopTimeout = setTimeout(() => {
if (worker !== active) return;
closeWorker(active);
toaster.info("Training stopped");
}, 10_000);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The worker checks stopRequest in-between batches. Waiting for more participants or for aggregation can take longer than 10s so this timeout would trigger and throw the partial model away. I don't think that #1295 fixes this either

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in #1295. Now Stop interrupts pending network waits so the worker can return the partial model before termination.

const msg: unknown = msgpack.decode(data);

if (!decentralizedMessages.isPeerMessage(msg)) {
throw new Error(`invalid message received: ${JSON.stringify(msg)}`);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are many of these Error that are now thrown inside a web worker which would trigger onerror and kill the worker while it didn't use to kill training before. We should likely make this a log instead. WebSocket.onmessage also throws, etc

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants