Skip to content

Make the chatroom and editor browser suites pass on main again #730

Closed

Description

@HMarzban

Summary

Both browser suites have failed on every push to main since 2026-11-03 (chatroom) and 2026-11-07 (editor). v2.3.0 shipped with both red. A red suite hides the next real break, so this issue makes them pass again. It is CI test health, so it rolls up under tools, not security.

Benchmark

Field Value
Metric (1) Red runs in a row on main, per suite, count. (2) Failures in 5 back-to-back workflow_dispatch runs on the fix commit, per suite, count. (3) Green suite wall time, median seconds (updatedAt minus startedAt). (4) .skip( lines in apps/webapp/cypress/e2e, count.
Before Read on 2026-11-11 at 0b302b222 with gh run list. (1) chatroom-e2e.yml: 15, first red 3bcc82110, 16 of 37 tests fail. editor-e2e.yml: 9, first red 61cc37c5f, 3 of 346 tests fail. (2) Not measured yet. (3) Selection: the last 10 green push runs on main before 3bcc82110. Chatroom: median 288 s, range 247-369 s, n = 7 (2026-10-21 to 2026-11-03). The agent in the repo re-posts the median and range at n = 10. Editor: median 841 s, range 698-845 s, n = 4 (every green editor run on main), a hint only. (4) 5.
After Read on 2026-11-18 with gh run list, push and workflow_dispatch runs (Measured). (1) Chatroom: 16 green in a row since a3ef6e83f. Editor: 10 green in a row since 06f61ee44. Both met. (2) 0 of 5 per suite on 288d4d260 (Measured comment, 2026-11-16 15:41Z). (3) Chatroom: median 229 s, range 173–275 s, n = 10 (20% less time). Editor: median 723.5 s, range 649–795 s, n = 10 (13% less time). Both at or below the before maximum. (4) 5, each it.skip linking an open issue.
Condition GitHub-hosted ubuntu-latest, branch main, push or workflow_dispatch runs.
How to run gh run list --repo docs-plus/docs.plus --workflow <file> --branch main --event push --limit 60 --json conclusion,startedAt,updatedAt,headSha, then keep only conclusion == "success" and take the newest 10. --limit 10 alone returns no green run today, because the latest push runs are red. Time is updatedAt minus startedAt. For (1) and the after (3), drop --event push, because dispatch runs count too. For (2): gh workflow run <file> --repo docs-plus/docs.plus --ref main, 5 times per suite, one after the other (step 4). For (4): grep -r '\.skip(' apps/webapp/cypress/e2e | wc -l.
Pass rule (1) 0, then 10 green runs in a row per suite. Push and workflow_dispatch runs both count, as the Check box says. (2) 0 per suite. (3) The after median is at or below the before maximum. A faster suite passes. The editor before has n = 4, so its range is a hint only. (4) At most 7, each linking its own open issue. Stop rule: a fix that turns a suite green by a skip the step 1 bisect did not justify is reverted.
Guard A skip is allowed only after step 1 shows the spec is wrong, not the app.
Owner Agent in the repo: steps 1-2 for both suites, and the counts. Maintainer: the 5 dispatches per suite, review, and the commit through commit-review.
Claim "Both browser suites are green on main for 10 runs in a row: chatroom N s, editor M s median."

Failing specs

As of 0b302b222, 2026-11-11.

Suite First red commit Last green commit Failing specs
chatroom-e2e.yml 3bcc82110 (2026-11-03) 8e25e83df, 3 commits apart attachments.cy.ts (14 tests), send-and-retry.cy.ts (2 tests)
editor-e2e.yml 61cc37c5f (2026-11-07) 44be02585, 48 commits apart copy-paste/paste-keeps-toc-ids.cy.js (3 tests)

What to do

  1. Owner: agent in the repo. Find the breaking commit with git bisect. The suites run only on path matches, so the commit before the first red run is not the last green run.
    • Before you bisect, run each failing spec 3 times on each end, and post the pass counts here.
    • Chatroom: git bisect start 3bcc82110 8e25e83df, with bash scripts/test-chatroom.sh as the test. The first suspect for the covered-click failure is b9b99f7c9 (message-list, react-virtuoso, React and Cypress bumps).
    • Editor: git bisect start 61cc37c5f 44be02585, with bash scripts/test-editor.sh as the test.
    • Fix the app or the spec, whichever is wrong.
  2. Owner: agent in the repo. A skip is allowed only after step 1 shows that the spec is wrong, not the app. Each skip gets it.skip with a link to its own open issue, as Three list cases in the editor specs fail every time and are skipped #679 does. Skip lines in apps/webapp/cypress/e2e may rise from 5 to at most 7.
  3. Owners. Chatroom: agent in the repo. Editor: agent in the repo. The maintainer reviews and commits through commit-review.
  4. Owner: maintainer. Run the 5 dispatches per suite for metric (2). Both workflows set cancel-in-progress: true, so a new run cancels the one before it. Send each dispatch only after the previous run ends, with no push to main in between. Check that each run's headSha is the fix commit.

Check

  • Each suite has 10 runs in a row on main, push or workflow_dispatch, all success.
  • On the fix commit, 5 back-to-back gh workflow run <file> --ref main runs per suite end with 0 failures. Each run has the fix commit as its headSha (step 4).
  • The median wall time of the 10 green runs is at or below the before maximum.
  • Every skipped test links to an open issue, and skip lines are at most 7.
  • The Benchmark After values are posted here.

Status 2026-11-18: every box is met. Closed.

Related

Activity

  1. added this to the v2.3.6 milestone on Nov 11, 2026
  2. HMarzban commented on Nov 14, 2026

    @HMarzban
    CollaboratorAuthor

    Moved to v2.3.7 at the v2.3.6 release audit. Both browser suites are still red on main. No fix, no green runs and no after are posted.

  3. modified the milestones: v2.3.6, v2.3.7 on Nov 14, 2026
  4. HMarzban commented on Nov 16, 2026

    @HMarzban
    CollaboratorAuthor

    Cause found, 2026-11-16

    Step 1 of this issue: the cause of each failing spec, with the runs that prove it. The fixes are in the working tree, not committed. The after (10 green runs and the 5 dispatches) comes later.

    Red runs, updated count

    Read 2026-11-16 with gh run list --workflow <file> --branch main.

    Suite First red commit Red push runs in a row Latest red run
    chatroom-e2e.yml 3bcc82110 (2026-11-03) 21 43697520087 at b7888dce1
    editor-e2e.yml 61cc37c5f (2026-11-07) 15 43697520086 at b7888dce1

    The body counted 15 and 9 on 2026-11-11.

    Test setup

    • One local production build of main at 045a6a62f, made the same way as scripts/test-standalone-webapp.sh. It was served on port 3213.
    • Linux runs: Docker cypress/included:16.2.1, linux/arm64, Electron 146 headless. The repo was mounted read-only. Changed spec files were mounted over the originals, so each before and after used the same build.
    • macOS runs: the repo's own Cypress 16.2.1, Electron 146 headless.
    • CI runs on ubuntu-latest amd64. These runs are arm64. They show the same failing tests as CI.

    Editor: copy-paste/paste-keeps-toc-ids.cy.js, 3 of 10 tests

    Cause: the spec, not the app. Every chord sends CDP modifiers: 4, which is Meta. ProseMirror's Mod-a is Meta on macOS and Ctrl on Linux. On Linux, only the browser's own SelectAll runs, and ProseMirror then reads a plain text selection. The three tests that select all and then cut or paste fail. The tests that select all and only copy pass.

    The spec was added in 7bf9b25e9 (2026-11-06). The last green run (44be02585) did not hold it. The first red run (61cc37c5f) was the first push that did. So the spec has never passed in CI.

    Run Platform n Failing tests per run
    Spec as on main macOS 1 0 of 10
    Spec as on main Linux 1 3 of 10, the same 3 as CI
    Fix: MOD = Cypress.platform === 'darwin' ? 4 : 2 Linux 3 0 of 10 in each run

    The fix keeps macOS on the same value, 4.

    Chatroom: attachments.cy.ts, 14 tests

    Cause: the spec, after a Cypress bump. Cypress 16.2.1 refuses a fixture path outside cypress/fixtures. The spec used ../../public/icons/favicon-32x32.png. The error is: Fixture path '../../public/icons/favicon-32x32.png' must resolve to a file inside the fixtures folder. Cypress went from 16.2.0 to 16.2.1 in b9b99f7c9, which is inside the first red push.

    Same build, same spec, only the Cypress binary changed (macOS, CYPRESS_RUN_BINARY):

    Cypress binary n Tests Passing Failing Pending
    16.2.1 1 21 6 14 1
    16.2.0 1 21 20 0 1

    Fix options, Linux:

    Fixture n Result per run
    Existing link-icon-16.png (16 × 16) 3 1 fail: "requires two taps on spoiler feed image before opening gallery"
    A 32 × 32 copy of favicon-32x32.png 2 20 passing, 0 failing

    The fix adds cypress/fixtures/feed-image-32.png, a byte copy of favicon-32x32.png, and points the spec at it. A third clean run is owed. The third and later runs were void: a dev server started from the same checkout replaced the build's .next, so the page served no scripts.

    Chatroom: send-and-retry.cy.ts, 2 tests

    Cause: likely the app. The RLS retry test fails on both Cypress versions (macOS, 3 runs each, 1 failing test each run). So the Cypress bump is not its cause.

    A geometry probe on Linux reproduced the failed send and read the layout 300 ms and 1,800 ms after "Failed to send" showed. Both reads were the same:

    Element Top (px) Bottom (px)
    Feed (.message-feed) 284 602
    Failed message card 546 618
    Retry button 594 618
    Composer input row 603 651

    The Virtuoso scroller had scrollHeight − clientHeight − scrollTop = 16. So the list stops 16 px short of its bottom, and the newest row is cut off. Only 8 px of the 24 px Retry button show. The composer does not overlap the feed. A real user sees the same cut-off row after a failed send.

    useSendMessage.ts flips the row to failed with data.map(callback) and no scroll argument. The failed row is taller than the pending row. @virtuoso.dev/message-list went from 1.18.1 to 1.19.4 in b9b99f7c9. Its 1.19.0 notes include "keep growing message lists following the current scroll target" and "preserve updates until scrolling and measured layout finish". The 23505 test, which fails in CI but passed in all local runs, fits the same delayed update. This is not proven yet. The A/B build with 1.18.1 is owed.

    Proposed fix, read from code, not run yet. data.map(callback, autoscrollToBottomBehavior) exists in 1.18.1 and 1.19.4. The library scrolls on it only when the list was at the bottom and a row grew past it (SIZE_INCREASED). So a reader who scrolled up is not moved. The fix passes 'smooth' on the three status flips that follow a network reply: failed, duplicate-key sent, and the retry result.

    Next

    1. Test the send-and-retry fix: the probe gap should go from 16 px to 0, and the spec should pass 3 runs of 3. Then run all three specs again on one build, 3 runs each.
    2. If needed, the 1.18.1 A/B build.
    3. Commit through commit-review, then the maintainer's push and the 5 dispatches per suite for metric (2).
    4. Post the After values in the Benchmark table and add the row to v2.3.7 benchmark ledger: every quotable claim, with n, range, condition and source #796.

    Skip lines in apps/webapp/cypress/e2e are unchanged at 5. None of the fixes adds a skip.

  5. HMarzban commented on Nov 16, 2026

    @HMarzban
    CollaboratorAuthor

    Local after, 2026-11-16

    These are the local runs of the three fixes from the cause comment. The fixes are in the working tree, not committed. This is not the CI after. The 10 green runs and the 5 dispatches per suite come after the push.

    Setup

    • One local production build of 045a6a62f plus the uncommitted working tree, made like scripts/test-standalone-webapp.sh. It was served on port 3213.
    • Linux runs: Docker cypress/included:16.2.1, linux/arm64, Electron 146 headless. The repo was mounted read-only, with no file overlays.
    • The tree also held one uncommitted change from parallel work, in apps/webapp/src/hooks/useEditorEditableState.ts (One older pad editor stays in memory after pad switches #782). It is not part of these fixes.
    • Host load average during the runs: about 7 to 8.

    Specs

    Spec Before (cause comment) After: runs After: failing tests per run
    editor/copy-paste/paste-keeps-toc-ids.cy.js 3 of 10 fail 8 2, 1, then 0 in each of the last 6 runs
    chatroom/send-and-retry.cy.ts 1 of 4 fails (RLS retry) 3 0 of 4 in each run
    chatroom/attachments.cy.ts 14 of 21 fail 3 0 of 21 in each run (20 passing, 1 pending)

    The editor spec is still flaky. It failed in 2 of 8 runs. Both failures came in the first 2 runs right after the build. The failing tests were "a select-all paste over the pad keeps every id" (both runs) and "a copy pasted above its original leaves the original its id" (run 1). In each one, a paste did not land within 4 s: Throwaway stayed, or 2 of 3 h2 were found. That is not the Meta-chord fault. The chord fix removed the 3 failures that happened in every run before. Watch the paste timing in the 10 CI runs. If CI also shows it, it needs its own fix.

    Layout probe: failed send

    The probe reads the layout 300 ms and 1,800 ms after "Failed to send" shows. Before: n = 1, both reads the same. After: n = 3, all three runs the same.

    Reading Gap to the list bottom Retry button visible Element at the centre of Retry
    Before, 300 ms and 1,800 ms 16 px 8 of 24 px the composer input row
    After, 300 ms 2 px (the smooth scroll is still moving) 22 of 24 px the Retry button
    After, 1,800 ms none (0 px) 24 of 24 px the Retry button

    So the fix ('smooth' passed to data.map on the three status flips) lets the list follow the taller failed row. The Virtuoso 1.18.1 A/B build was not run. The fix works on 1.19.4, so that test no longer decides the fix. It would only show which release changed the behaviour.

    Next

    1. Commit through commit-review, after the maintainer approves.
    2. Push, then 10 green runs and 5 workflow_dispatch runs per suite on the fix commit (metrics 1 to 3).
    3. The Benchmark After values here, and the v2.3.7 benchmark ledger: every quotable claim, with n, range, condition and source #796 row.

    Skip lines in apps/webapp/cypress/e2e: 5, unchanged.

  6. HMarzban commented on Nov 16, 2026

    @HMarzban
    CollaboratorAuthor

    Measured, 2026-11-16: the remaining editor flake, cause proven

    The three fixes from the cause comment are live since a3ef6e83f. On that push, both suites passed: chatroom 37 tests, 36 passing, 0 failing, 1 pending (run 43851402904); editor 346 tests, 328 passing, 0 failing, 18 pending (run 43851402789).

    Dispatches on a3ef6e83f

    Suite Runs Failed Runs
    Chatroom 5 of 5 0 43854486425, 43855022740, 43855482526, 43856087857, 43856680080
    Editor 3 (stopped, see below) 2 43854488199 (fail), 43856348992 (pass), 43858244720 (fail)

    Both editor failures are the same test: paste-keeps-toc-ids › "a copy pasted above its original leaves the original its id", Found '2', expected '3'. The editor set stopped after run 3, because the dispatch pass rule (0 failures) could no longer pass.

    Cause

    The /editor playground has its own shortcut hook, useClipboardShortcuts. On every Ctrl or Cmd + C or V keydown, it clicks a hidden button a tick later. That button sends an untrusted ClipboardEvent with no clipboardData, then calls execCommand. So one chord in the spec copies twice, by two paths, and they race:

    1. The CDP Copy command fires a trusted copy. ProseMirror writes its slice, with data-pm-slice.
    2. The playground's untrusted copy has no clipboardData, so ProseMirror takes its captureCopy path. That selects a hidden wrapper outside the editor, and focus moves to BODY.
    3. A second trusted copy (Chrome's default Ctrl+C binding on Linux, beside the CDP command) copies that wrapper natively. It replaces the slice with native HTML that has no pad mark.

    The paste of that native HTML merges "Beta" into the paragraph "Body a", so no third h2 appears. Proof, from a capture listener in the test: three copy events per chord, in that order, with isTrusted true, false, true, and only the first carrying data-pm-slice. macOS has no default Ctrl+C binding for step 3, which is why the spec passes there.

    This is test-harness code that exists only on the E2E playground route. The product has no such hook. The app is not at fault.

    Fix and local before/after

    The spec stops a Mod+C or Mod+V keydown at body, so the playground's document listener never fires. ProseMirror's own listener on view.dom runs first, so its behaviour does not change. Each chord also waits for editor focus (a retrying assertion).

    Metric Before After Change n, condition, source
    Linux runs of paste-keeps-toc-ids.cy.js with a failure 3 of 10 0 of 10 none fail n = 10 each, back to back. cypress/included:16.2.1, electron, host network, against one production build of this tree. The same command, with only the spec file swapped

    The fix is commit 17eb9d797. It is local, not pushed yet: the push is the maintainer's.

    Left

  7. HMarzban commented on Nov 16, 2026

    @HMarzban
    CollaboratorAuthor

    Measured, 2026-11-16: dispatches on the fix commit

    The paste fix (17eb9d797) shipped with the push of 06f61ee44. The dispatches ran on 288d4d260, which holds it.

    (2) 5 back-to-back dispatches per suite

    Suite Runs Failed Tests per run
    Chatroom 43879347308, 43880006620, 43880584524, 43881250058, 43881933339 0 of 5 37: 36 passing, 0 failing, 1 pending
    Editor 43879350346, 43881369939, 43883554386, 43885539362, 43887636780 0 of 5 346: 328 passing, 0 failing, 18 pending

    Before the paste fix, the editor dispatches on a3ef6e83f failed 2 of 3, all in the same paste test (comment).

    Pass rule (2), 0 failures per suite: met.

    (1) Green push runs in a row since the fix

    Suite Push runs Green
    Chatroom a3ef6e83f, 06f61ee44 2 of 10
    Editor a3ef6e83f, 06f61ee44 2 of 10

    The tip 288d4d260 ((build): observability) changed no file the suites watch, so it started no suite run. Pass rule (1) needs 8 more green pushes per suite.

    (3) Wall time

    Suite Before (green push runs) After, push runs so far After, dispatch jobs
    Chatroom median 288 s, range 247–369 s, n = 7 186 s, 208 s (n = 2) median 224 s, range 173–236 s, n = 5
    Editor median 841 s, range 698–845 s, n = 4 (a hint) 812 s, 794 s (n = 2) median 716 s, range 649–750 s, n = 5
    • Time is updatedAt minus startedAt. Editor dispatch 5 waited about 50 minutes for a hosted runner; its job time (716 s, 15:27–15:39Z) is used, not its run time (3,708 s).
    • The editor medians sit inside the before range. The chatroom medians sit below it, so the chatroom suite runs faster than before.
    • Dispatch runs are not push runs, so the formal (3) read waits for the 10 green push runs.

    (4) Skip lines

    grep -r '\.skip(' apps/webapp/cypress/e2e | wc -l: 5, unchanged. Pass rule (4), at most 7: met.

    Left

  8. HMarzban commented on Nov 16, 2026

    @HMarzban
    CollaboratorAuthor

    Review, 2026-11-16 (maintainer approved).

    • Pass rule (1) now counts push and workflow_dispatch runs, as the Check box already said. The 2026-11-16 15:41Z comment counted push runs only.
    • Pass rule (3) is now an upper bound: the after median is at or below the before maximum. The old "inside the range" rule failed a faster suite.
    • The After row holds today's reads. Chatroom: 14 green in a row, median 222 s (173-245 s, n = 10), so met. Editor: 8 green in a row.
    • Box 2 (5 dispatches, 0 failures per suite) is ticked.
    • One step is left: 2 more green editor runs on main, then post the editor After values.
  9. HMarzban commented on Nov 18, 2026

    @HMarzban
    CollaboratorAuthor

    Measured, 2026-11-18: 10 green runs per suite, and close

    Read with gh run list --workflow <file> --branch main, push and workflow_dispatch runs, as the How to run row says. Time is updatedAt minus startedAt. Editor dispatch 5 (run 43887636780) waited about 50 minutes for a hosted runner, so its job time (716 s) is used, as on 2026-11-16.

    (1) Green runs in a row

    Suite Green in a row First Newest
    Chatroom 16 a3ef6e83f (push, 2026-11-16 10:43Z) afd06574b (push, run 44041325587)
    Editor 10 06f61ee44 (push, run 43876524297) afd06574b (push, run 44041325463)

    Met for both suites.

    (3) Wall time of the newest 10 green runs

    Suite Before (green push runs) After (newest 10 green runs) Change
    Chatroom median 288 s, range 247–369 s, n = 7 median 229 s, range 173–275 s, n = 10 20% less time
    Editor median 841 s, range 698–845 s, n = 4 (a hint) median 723.5 s, range 649–795 s, n = 10 13% less time

    Pass rule (3), the after median at or below the before maximum: met for both suites. The chatroom median moved from 222 s on 2026-11-16 to 229 s, because the newest 10 now include later push runs.

    Editor runs, in seconds: 794, 731, 750, 649, 677, 716 (job time), 716, 701, 735, 795. Chatroom runs, in seconds: 240, 275, 195, 245, 220, 224, 236, 173, 234, 208.

    (4) Skip lines

    grep -r '\.skip(' apps/webapp/cypress/e2e | wc -l: 5, unchanged. One is the CDP-only this.skip() in saved-pad-reload.cy.ts. The four it.skip lines link to open issues: #594 (attachments.cy.ts:340) and #679 (combined-lists.cy.js:203 and :261, list-edge-cases.cy.js:114). Met.

    Close

    All five boxes are met. Closing. The #796 row is filled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    DevOpsbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions