Repository navigation
Closed
Labels
Milestone
Description
Summary
Both browser suites have failed on every push to main since 2026-11-03 (chatroom) and 2026-11-07 (editor). v2.3.0 shipped with both red. A red suite hides the next real break, so this issue makes them pass again. It is CI test health, so it rolls up under tools, not security.
Benchmark
| Field | Value |
|---|---|
| Metric | (1) Red runs in a row on main, per suite, count. (2) Failures in 5 back-to-back workflow_dispatch runs on the fix commit, per suite, count. (3) Green suite wall time, median seconds (updatedAt minus startedAt). (4) .skip( lines in apps/webapp/cypress/e2e, count. |
| Before | Read on 2026-11-11 at 0b302b222 with gh run list. (1) chatroom-e2e.yml: 15, first red 3bcc82110, 16 of 37 tests fail. editor-e2e.yml: 9, first red 61cc37c5f, 3 of 346 tests fail. (2) Not measured yet. (3) Selection: the last 10 green push runs on main before 3bcc82110. Chatroom: median 288 s, range 247-369 s, n = 7 (2026-10-21 to 2026-11-03). The agent in the repo re-posts the median and range at n = 10. Editor: median 841 s, range 698-845 s, n = 4 (every green editor run on main), a hint only. (4) 5. |
| After | Read on 2026-11-18 with gh run list, push and workflow_dispatch runs (Measured). (1) Chatroom: 16 green in a row since a3ef6e83f. Editor: 10 green in a row since 06f61ee44. Both met. (2) 0 of 5 per suite on 288d4d260 (Measured comment, 2026-11-16 15:41Z). (3) Chatroom: median 229 s, range 173–275 s, n = 10 (20% less time). Editor: median 723.5 s, range 649–795 s, n = 10 (13% less time). Both at or below the before maximum. (4) 5, each it.skip linking an open issue. |
| Condition | GitHub-hosted ubuntu-latest, branch main, push or workflow_dispatch runs. |
| How to run | gh run list --repo docs-plus/docs.plus --workflow <file> --branch main --event push --limit 60 --json conclusion,startedAt,updatedAt,headSha, then keep only conclusion == "success" and take the newest 10. --limit 10 alone returns no green run today, because the latest push runs are red. Time is updatedAt minus startedAt. For (1) and the after (3), drop --event push, because dispatch runs count too. For (2): gh workflow run <file> --repo docs-plus/docs.plus --ref main, 5 times per suite, one after the other (step 4). For (4): grep -r '\.skip(' apps/webapp/cypress/e2e | wc -l. |
| Pass rule | (1) 0, then 10 green runs in a row per suite. Push and workflow_dispatch runs both count, as the Check box says. (2) 0 per suite. (3) The after median is at or below the before maximum. A faster suite passes. The editor before has n = 4, so its range is a hint only. (4) At most 7, each linking its own open issue. Stop rule: a fix that turns a suite green by a skip the step 1 bisect did not justify is reverted. |
| Guard | A skip is allowed only after step 1 shows the spec is wrong, not the app. |
| Owner | Agent in the repo: steps 1-2 for both suites, and the counts. Maintainer: the 5 dispatches per suite, review, and the commit through commit-review. |
| Claim | "Both browser suites are green on main for 10 runs in a row: chatroom N s, editor M s median." |
Failing specs
As of 0b302b222, 2026-11-11.
| Suite | First red commit | Last green commit | Failing specs |
|---|---|---|---|
chatroom-e2e.yml |
3bcc82110 (2026-11-03) |
8e25e83df, 3 commits apart |
attachments.cy.ts (14 tests), send-and-retry.cy.ts (2 tests) |
editor-e2e.yml |
61cc37c5f (2026-11-07) |
44be02585, 48 commits apart |
copy-paste/paste-keeps-toc-ids.cy.js (3 tests) |
- Runs at
0b302b222: chatroom 43008200078, editor 43008199969. Earlier runs at24f302ae0: chatroom 42689998051, editor 42689997577. - Chatroom errors: a
cy.click()on a covered element, and a small textspanthat never leaves the DOM. - Editor errors: an array that differs from the expected one, and pasted text (
Throwaway,Other pad) that stays in the editor. - Editor was also red 3 times on 2026-11-03, then green 3 times. So the editor suite may be flaky.
- The Telegram failure notice is skipped until Make CI failures reach Telegram: set the watchdog secrets, alert on red E2E pushes #557 sets its secrets.
What to do
- Owner: agent in the repo. Find the breaking commit with
git bisect. The suites run only on path matches, so the commit before the first red run is not the last green run.- Before you bisect, run each failing spec 3 times on each end, and post the pass counts here.
- Chatroom:
git bisect start 3bcc82110 8e25e83df, withbash scripts/test-chatroom.shas the test. The first suspect for the covered-click failure isb9b99f7c9(message-list, react-virtuoso, React and Cypress bumps). - Editor:
git bisect start 61cc37c5f 44be02585, withbash scripts/test-editor.shas the test. - Fix the app or the spec, whichever is wrong.
- Owner: agent in the repo. A skip is allowed only after step 1 shows that the spec is wrong, not the app. Each skip gets
it.skipwith a link to its own open issue, as Three list cases in the editor specs fail every time and are skipped #679 does. Skip lines inapps/webapp/cypress/e2emay rise from 5 to at most 7. - Owners. Chatroom: agent in the repo. Editor: agent in the repo. The maintainer reviews and commits through
commit-review. - Owner: maintainer. Run the 5 dispatches per suite for metric (2). Both workflows set
cancel-in-progress: true, so a new run cancels the one before it. Send each dispatch only after the previous run ends, with no push tomainin between. Check that each run'sheadShais the fix commit.
Check
- Each suite has 10 runs in a row on
main, push orworkflow_dispatch, allsuccess. - On the fix commit, 5 back-to-back
gh workflow run <file> --ref mainruns per suite end with 0 failures. Each run has the fix commit as itsheadSha(step 4). - The median wall time of the 10 green runs is at or below the before maximum.
- Every skipped test links to an open issue, and skip lines are at most 7.
- The Benchmark After values are posted here.
Status 2026-11-18: every box is met. Closed.
Related
- Three list cases in the editor specs fail every time and are skipped #679: skipped editor list cases.
- Run the webapp Cypress suite in CI #276: the webapp Cypress suite in CI.
- Make CI failures reach Telegram: set the watchdog secrets, alert on red E2E pushes #557: CI failures should reach Telegram.
- [Tracking] Faster, change-aware CI/CD pipeline #554: the pipeline tracking issue.
Activity
HMarzban commentedon Nov 14, 2026
Moved to v2.3.7 at the v2.3.6 release audit. Both browser suites are still red on main. No fix, no green runs and no after are posted.
HMarzban commentedon Nov 16, 2026
Cause found, 2026-11-16
Step 1 of this issue: the cause of each failing spec, with the runs that prove it. The fixes are in the working tree, not committed. The after (10 green runs and the 5 dispatches) comes later.
Red runs, updated count
Read 2026-11-16 with
gh run list --workflow <file> --branch main.chatroom-e2e.yml3bcc82110(2026-11-03)b7888dce1editor-e2e.yml61cc37c5f(2026-11-07)b7888dce1The body counted 15 and 9 on 2026-11-11.
Test setup
mainat045a6a62f, made the same way asscripts/test-standalone-webapp.sh. It was served on port 3213.cypress/included:16.2.1, linux/arm64, Electron 146 headless. The repo was mounted read-only. Changed spec files were mounted over the originals, so each before and after used the same build.ubuntu-latestamd64. These runs are arm64. They show the same failing tests as CI.Editor:
copy-paste/paste-keeps-toc-ids.cy.js, 3 of 10 testsCause: the spec, not the app. Every chord sends CDP
modifiers: 4, which is Meta. ProseMirror'sMod-ais Meta on macOS and Ctrl on Linux. On Linux, only the browser's own SelectAll runs, and ProseMirror then reads a plain text selection. The three tests that select all and then cut or paste fail. The tests that select all and only copy pass.The spec was added in
7bf9b25e9(2026-11-06). The last green run (44be02585) did not hold it. The first red run (61cc37c5f) was the first push that did. So the spec has never passed in CI.mainmainMOD = Cypress.platform === 'darwin' ? 4 : 2The fix keeps macOS on the same value, 4.
Chatroom:
attachments.cy.ts, 14 testsCause: the spec, after a Cypress bump. Cypress 16.2.1 refuses a fixture path outside
cypress/fixtures. The spec used../../public/icons/favicon-32x32.png. The error is:Fixture path '../../public/icons/favicon-32x32.png' must resolve to a file inside the fixtures folder. Cypress went from 16.2.0 to 16.2.1 inb9b99f7c9, which is inside the first red push.Same build, same spec, only the Cypress binary changed (macOS,
CYPRESS_RUN_BINARY):Fix options, Linux:
link-icon-16.png(16 × 16)favicon-32x32.pngThe fix adds
cypress/fixtures/feed-image-32.png, a byte copy offavicon-32x32.png, and points the spec at it. A third clean run is owed. The third and later runs were void: a dev server started from the same checkout replaced the build's.next, so the page served no scripts.Chatroom:
send-and-retry.cy.ts, 2 testsCause: likely the app. The RLS retry test fails on both Cypress versions (macOS, 3 runs each, 1 failing test each run). So the Cypress bump is not its cause.
A geometry probe on Linux reproduced the failed send and read the layout 300 ms and 1,800 ms after "Failed to send" showed. Both reads were the same:
.message-feed)The Virtuoso scroller had
scrollHeight − clientHeight − scrollTop = 16. So the list stops 16 px short of its bottom, and the newest row is cut off. Only 8 px of the 24 px Retry button show. The composer does not overlap the feed. A real user sees the same cut-off row after a failed send.useSendMessage.tsflips the row tofailedwithdata.map(callback)and no scroll argument. The failed row is taller than the pending row.@virtuoso.dev/message-listwent from 1.18.1 to 1.19.4 inb9b99f7c9. Its 1.19.0 notes include "keep growing message lists following the current scroll target" and "preserve updates until scrolling and measured layout finish". The 23505 test, which fails in CI but passed in all local runs, fits the same delayed update. This is not proven yet. The A/B build with 1.18.1 is owed.Proposed fix, read from code, not run yet.
data.map(callback, autoscrollToBottomBehavior)exists in 1.18.1 and 1.19.4. The library scrolls on it only when the list was at the bottom and a row grew past it (SIZE_INCREASED). So a reader who scrolled up is not moved. The fix passes'smooth'on the three status flips that follow a network reply: failed, duplicate-key sent, and the retry result.Next
send-and-retryfix: the probe gap should go from 16 px to 0, and the spec should pass 3 runs of 3. Then run all three specs again on one build, 3 runs each.commit-review, then the maintainer's push and the 5 dispatches per suite for metric (2).Skip lines in
apps/webapp/cypress/e2eare unchanged at 5. None of the fixes adds a skip.HMarzban commentedon Nov 16, 2026
Local after, 2026-11-16
These are the local runs of the three fixes from the cause comment. The fixes are in the working tree, not committed. This is not the CI after. The 10 green runs and the 5 dispatches per suite come after the push.
Setup
045a6a62fplus the uncommitted working tree, made likescripts/test-standalone-webapp.sh. It was served on port 3213.cypress/included:16.2.1, linux/arm64, Electron 146 headless. The repo was mounted read-only, with no file overlays.apps/webapp/src/hooks/useEditorEditableState.ts(One older pad editor stays in memory after pad switches #782). It is not part of these fixes.Specs
editor/copy-paste/paste-keeps-toc-ids.cy.jschatroom/send-and-retry.cy.tschatroom/attachments.cy.tsThe editor spec is still flaky. It failed in 2 of 8 runs. Both failures came in the first 2 runs right after the build. The failing tests were "a select-all paste over the pad keeps every id" (both runs) and "a copy pasted above its original leaves the original its id" (run 1). In each one, a paste did not land within 4 s:
Throwawaystayed, or 2 of 3h2were found. That is not the Meta-chord fault. The chord fix removed the 3 failures that happened in every run before. Watch the paste timing in the 10 CI runs. If CI also shows it, it needs its own fix.Layout probe: failed send
The probe reads the layout 300 ms and 1,800 ms after "Failed to send" shows. Before: n = 1, both reads the same. After: n = 3, all three runs the same.
So the fix (
'smooth'passed todata.mapon the three status flips) lets the list follow the taller failed row. The Virtuoso 1.18.1 A/B build was not run. The fix works on 1.19.4, so that test no longer decides the fix. It would only show which release changed the behaviour.Next
commit-review, after the maintainer approves.workflow_dispatchruns per suite on the fix commit (metrics 1 to 3).Skip lines in
apps/webapp/cypress/e2e: 5, unchanged.HMarzban commentedon Nov 16, 2026
Measured, 2026-11-16: the remaining editor flake, cause proven
The three fixes from the cause comment are live since
a3ef6e83f. On that push, both suites passed: chatroom 37 tests, 36 passing, 0 failing, 1 pending (run 43851402904); editor 346 tests, 328 passing, 0 failing, 18 pending (run 43851402789).Dispatches on
a3ef6e83fBoth editor failures are the same test:
paste-keeps-toc-ids› "a copy pasted above its original leaves the original its id",Found '2', expected '3'. The editor set stopped after run 3, because the dispatch pass rule (0 failures) could no longer pass.Cause
The
/editorplayground has its own shortcut hook,useClipboardShortcuts. On every Ctrl or Cmd + C or V keydown, it clicks a hidden button a tick later. That button sends an untrustedClipboardEventwith noclipboardData, then callsexecCommand. So one chord in the spec copies twice, by two paths, and they race:data-pm-slice.clipboardData, so ProseMirror takes itscaptureCopypath. That selects a hidden wrapper outside the editor, and focus moves toBODY.The paste of that native HTML merges "Beta" into the paragraph "Body a", so no third
h2appears. Proof, from a capture listener in the test: three copy events per chord, in that order, withisTrustedtrue, false, true, and only the first carryingdata-pm-slice. macOS has no default Ctrl+C binding for step 3, which is why the spec passes there.This is test-harness code that exists only on the E2E playground route. The product has no such hook. The app is not at fault.
Fix and local before/after
The spec stops a Mod+C or Mod+V keydown at
body, so the playground's document listener never fires. ProseMirror's own listener onview.domruns first, so its behaviour does not change. Each chord also waits for editor focus (a retrying assertion).paste-keeps-toc-ids.cy.jswith a failurecypress/included:16.2.1, electron, host network, against one production build of this tree. The same command, with only the spec file swappedThe fix is commit
17eb9d797. It is local, not pushed yet: the push is the maintainer's.Left
17eb9d797. Then 10 green push runs in a row per suite, and 5 back-to-back dispatches per suite on the fix commit.test(webapp): stop the playground copy hook in the paste spec
HMarzban commentedon Nov 16, 2026
Measured, 2026-11-16: dispatches on the fix commit
The paste fix (
17eb9d797) shipped with the push of06f61ee44. The dispatches ran on288d4d260, which holds it.(2) 5 back-to-back dispatches per suite
Before the paste fix, the editor dispatches on
a3ef6e83ffailed 2 of 3, all in the same paste test (comment).Pass rule (2), 0 failures per suite: met.
(1) Green push runs in a row since the fix
a3ef6e83f,06f61ee44a3ef6e83f,06f61ee44The tip
288d4d260((build): observability) changed no file the suites watch, so it started no suite run. Pass rule (1) needs 8 more green pushes per suite.(3) Wall time
updatedAtminusstartedAt. Editor dispatch 5 waited about 50 minutes for a hosted runner; its job time (716 s, 15:27–15:39Z) is used, not its run time (3,708 s).(4) Skip lines
grep -r '\.skip(' apps/webapp/cypress/e2e | wc -l: 5, unchanged. Pass rule (4), at most 7: met.Left
test(hocuspocus): pick e2e ports below the ephemeral range
docs(changelog): record the v2.3.7 work of 2026-11-16
HMarzban commentedon Nov 16, 2026
Review, 2026-11-16 (maintainer approved).
workflow_dispatchruns, as the Check box already said. The 2026-11-16 15:41Z comment counted push runs only.main, then post the editor After values.HMarzban commentedon Nov 18, 2026
Measured, 2026-11-18: 10 green runs per suite, and close
Read with
gh run list --workflow <file> --branch main, push andworkflow_dispatchruns, as the How to run row says. Time isupdatedAtminusstartedAt. Editor dispatch 5 (run 43887636780) waited about 50 minutes for a hosted runner, so its job time (716 s) is used, as on 2026-11-16.(1) Green runs in a row
a3ef6e83f(push, 2026-11-16 10:43Z)afd06574b(push, run 44041325587)06f61ee44(push, run 43876524297)afd06574b(push, run 44041325463)Met for both suites.
(3) Wall time of the newest 10 green runs
Pass rule (3), the after median at or below the before maximum: met for both suites. The chatroom median moved from 222 s on 2026-11-16 to 229 s, because the newest 10 now include later push runs.
Editor runs, in seconds: 794, 731, 750, 649, 677, 716 (job time), 716, 701, 735, 795. Chatroom runs, in seconds: 240, 275, 195, 245, 220, 224, 236, 173, 234, 208.
(4) Skip lines
grep -r '\.skip(' apps/webapp/cypress/e2e | wc -l: 5, unchanged. One is the CDP-onlythis.skip()insaved-pad-reload.cy.ts. The fourit.skiplines link to open issues: #594 (attachments.cy.ts:340) and #679 (combined-lists.cy.js:203and:261,list-edge-cases.cy.js:114). Met.Close
All five boxes are met. Closing. The #796 row is filled.