Sync ako/mxcli: nightly agent dev-loop benchmark; run --watch applies an exec change without the settle wait

.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-11-watch-settle-delay-after-exec-exited.json

@@ -0,0 +1 @@
+{"area": "cmd/mxcli", "date": "2026-11-11", "symptom": "With `run --local --watch` serving an app, a change made by `mxcli exec` was applied 4.2-5.2 s after exec had exited, although the log line said \"applied via reload in 2.0s\". Measured on a blank Mendix 11.16.0 app over eleven changes.", "cause": "The watch loop cannot see a write end. It polled the source mtime once a second and then waited for two more unchanged polls (settleSource) before building, so that a multi-file exec is never built half-written. For a writer that has already exited that is about 2.7 s of waiting for nothing.", "file": "cmd/mxcli/docker/watchwriter.go (BeginModelWrite, watchWriterGate), cmd/mxcli/docker/runlocal.go (watchAndApply wakes every 100 ms for the signal and skips the settle after one), cmd/mxcli/cmd_exec.go and cmd/mxcli/oneliner.go (the callers)", "insight": "The duration a loop prints for its own work is not the latency its user feels: subtract the two and the difference is where to look. The settle was the right answer to a writer that cannot say it is done and the wrong one for mxcli's own commands, which can. A pid-named marker plus a stamp covers the three cases without a protocol: marker present and pid alive is 'writing, do not build', stamp changed is 'done, build now', marker present and pid dead is 'done, however it ended'. Register the signal's defer before the executor's Close so it runs after the last write. Control, same app and session: an unsignalled write (touching the .mpr) still took 4.2 s, a signalled exec 2.2-2.5 s; an exec that changed nothing started no build.", "refs": ["ako/mxcli#2151"], "test": "cmd/mxcli/docker/watchwriter_test.go"}

.github/workflows/nightly.yml

@@ -5,12 +5,23 @@ on:
     # Every day at 02:00 UTC
     - cron: '0 2 * * *'
   workflow_dispatch: # Allow manual trigger
+    inputs:
+      # A manual run of everything from a branch would also publish that
+      # branch's binaries as the `nightly` release. `bench` runs the benchmark
+      # job alone, which is how a change to it is tried before it merges.
+      only:
+        description: 'Run every job, or the benchmark alone'
+        type: choice
+        options: [all, bench]
+        default: all
 
 permissions:
   contents: write
 
 jobs:
   test:
+    # On a schedule `inputs.only` is empty, so every job runs.
+    if: inputs.only != 'bench'
     runs-on: ubuntu-latest
     strategy:
       matrix:
@@ -78,6 +89,7 @@ jobs:
   # in the job log. Once this job is green, fold the version into the `test`
   # matrix and delete this job.
   test-mendix-9:
+    if: inputs.only != 'bench'
     runs-on: ubuntu-latest
     continue-on-error: true
     env:
@@ -135,6 +147,7 @@ jobs:
   # on demand with workflow_dispatch. Like upgrade-full-corpus it does not gate
   # the release: a nightly is not held back by a macOS queue flake.
   tunnel-seam-macos:
+    if: inputs.only != 'bench'
     runs-on: macos-latest
     timeout-minutes: 30
     steps:
@@ -152,6 +165,7 @@ jobs:
   # Mendix version — it runs on the committed PedApp fixture and never calls
   # mx — so it runs once, not per matrix entry. It does not gate the release.
   upgrade-full-corpus:
+    if: inputs.only != 'bench'
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
@@ -183,8 +197,133 @@ jobs:
           MXCLI_UPGRADE_SHARD: ${{ matrix.shard }}
         timeout-minutes: 45
 
+  # The agent dev-loop benchmark (docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md):
+  # a scripted session of the mxcli commands an agent runs, timed one process
+  # per command. A hosted runner's speed varies from night to night by more
+  # than most regressions, so the job does not compare tonight with last
+  # night: it downloads the previous nightly's binary and runs both
+  # interleaved on THIS runner, and reports the ratio. The record goes to the
+  # `benchmarks` branch, one file per run.
+  #
+  # Three sessions. Two time mxcli alone and start from the committed PedApp
+  # fixture. The third times the running app -- boot, a hot change, an entity
+  # change, a page check, tests -- on a project `mxcli new` creates, and so
+  # needs mxbuild and the runtime (mxcli downloads both), JDK 21 for the
+  # runtime, and a browser for the page check. It runs last: if it fails, the
+  # record of the first two is still written and committed.
+  #
+  # Does not gate the release, like upgrade-full-corpus: a slow night must not
+  # hold a nightly back. The driver exits non-zero only when a command fails
+  # or re-running a settled script changes a file.
+  bench:
+    runs-on: ubuntu-latest
+    timeout-minutes: 90
+    name: agent dev-loop benchmark
+    env:
+      # One version, the one the running-app session was developed against.
+      # Moving it is a deliberate change: boot and apply times differ between
+      # Mendix versions, so the trend restarts where it moves.
+      BENCH_MENDIX_VERSION: '11.13.0'
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-go@v7
+        with:
+          go-version: '1.27.4'
+      - name: Cache ANTLR4 JAR
+        uses: actions/cache@v6
+        with:
+          path: ~/.m2/repository/org/antlr/antlr4
+          key: antlr4-4.13.2
+      - name: Install ANTLR4
+        run: pip install 'antlr4-tools==0.2.2'
+      - name: Generate parser
+        run: make grammar
+        env:
+          ANTLR4_TOOLS_ANTLR_VERSION: '4.13.2'
+
+      # The Mendix 11 runtime launches under JDK 21; the runner's default
+      # `java` is older.
+      - uses: actions/setup-java@v6
+        with:
+          distribution: temurin
+          java-version: '21'
+
+      # `mxcli playwright check` requires the `playwright` package from the
+      # global node_modules and launches its Chromium.
+      - name: Install Playwright and Chromium
+        run: |
+          npm install -g playwright@1.63.0
+          npx --yes playwright@1.63.0 install --with-deps chromium
+
+      # First, before the `nightly` job can replace the release. That job
+      # waits for the whole `test` matrix, but nothing here relies on it: the
+      # record names the baseline's version, and with no baseline the run
+      # still measures, without ratios.
+      - name: Download the previous nightly binary
+        id: baseline
+        run: |
+          mkdir -p "$RUNNER_TEMP/baseline"
+          if gh release download nightly --repo "$GITHUB_REPOSITORY" \
+               --pattern mxcli-linux-amd64 --dir "$RUNNER_TEMP/baseline"; then
+            chmod +x "$RUNNER_TEMP/baseline/mxcli-linux-amd64"
+            echo "path=$RUNNER_TEMP/baseline/mxcli-linux-amd64" >> "$GITHUB_OUTPUT"
+          else
+            echo "No previous nightly binary: measuring without an A/B."
+          fi
+        env:
+          GH_TOKEN: ${{ github.token }}
+
+      - name: Run the benchmark
+        run: |
+          make bench BASELINE="${{ steps.baseline.outputs.path }}" \
+            BENCH_RUNTIME="$BENCH_MENDIX_VERSION" \
+            BENCH_FLAGS="-out agentbench-record.json -summary $GITHUB_STEP_SUMMARY"
+
+      - name: Upload the record
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: agent-benchmark-record
+          path: agentbench-record.json
+          if-no-files-found: ignore
+
+      # One file per run, so two runs never edit the same file; the retry is
+      # for two runs pushing at once. An orphan branch: results only, never
+      # merged. Runs from another branch (workflow_dispatch) are not history.
+      - name: Commit the record to the benchmarks branch
+        # hashFiles sees the workspace only, which is why the record is written there.
+        if: always() && github.ref == 'refs/heads/main' && hashFiles('agentbench-record.json') != ''
+        run: |
+          set -euo pipefail
+          sha=$(git rev-parse --short HEAD)
+          file="results/$(date -u +%Y/%m)/$(date -u +%Y-%m-%dT%H%M)-$sha.json"
+          git config user.name "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          if git fetch origin benchmarks:benchmarks 2>/dev/null; then
+            git worktree add "$RUNNER_TEMP/benchmarks" benchmarks
+          else
+            git worktree add --detach "$RUNNER_TEMP/benchmarks"
+            git -C "$RUNNER_TEMP/benchmarks" checkout --orphan benchmarks
+            git -C "$RUNNER_TEMP/benchmarks" rm -rf -q .
+            printf '%s\n' '# Agent dev-loop benchmark records' '' \
+              'One JSON file per nightly run, written by the `bench` job of `.github/workflows/nightly.yml`.' \
+              'Never merged into `main`. How to read a record: `docs-site/src/internals/agent-benchmark.md` on `main`.' \
+              > "$RUNNER_TEMP/benchmarks/README.md"
+          fi
+          cd "$RUNNER_TEMP/benchmarks"
+          mkdir -p "$(dirname "$file")"
+          cp "$GITHUB_WORKSPACE/agentbench-record.json" "$file"
+          git add -A
+          git commit -q -m "bench: $sha"
+          for attempt in 1 2 3; do
+            if git push origin benchmarks; then exit 0; fi
+            git pull --rebase origin benchmarks
+          done
+          exit 1
+
   nightly:
     needs: test
+    if: inputs.only != 'bench'
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v7

CHANGELOG.md

@@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ### Added
 
+- **`make bench`: a benchmark of what an agent waits for, and `MXCLI_STATS_FILE`** — a fixed session of mxcli commands (open, author entities, microflows and pages, describe, catalog refresh, check, a one-document edit, a bulk change, a re-run, a rename), one process per command, on PedApp and on PedApp with a generated app seeded into it. `make bench BENCH_RUNTIME=<Mendix version>` adds a session that validates and builds the project with mx and then runs it — `docker check`, `docker build`, boot, a hot change, an entity change, a page check, an attached and a cold test run — on a project `mxcli new` creates, with mxbuild and the runtime installed by mxcli when they are not cached. With `BASELINE=<other mxcli>` the two binaries run interleaved on the same machine and each step reports the ratio; the nightly `bench` job does that against the previous nightly and commits the record to the `benchmarks` branch. It also records counts that do not depend on the machine: unit files read and unit files changed. `MXCLI_STATS_FILE=<path>` makes any mxcli command append one JSON line per closed project connection with its unit-file reads and its writes offered and landed. See "Agent Dev-Loop Benchmark" in the docs.
 - **`mxcli test --local --reuse-build` skips the injection and the build when nothing changed** (mendixlabs/mxcli#1783, closes mendixlabs/mxcli#1782) — a local test run injects its test microflows, builds and boots; when the project, the test files and mxcli are unchanged, the deployment the previous run left is already this run's build, so `--reuse-build` boots it directly (about 12 s against 60 s, measured by the author). The decision is a content hash of the project folder (outputs, caches, `theme-cache/`, and any `.git`/`node_modules` left out), the generated test MDL, the mxcli binary and the mxbuild override, pinned to the identity of `deployment/model/model.mdp`, so a deployment rebuilt by anything else is never taken for the test build. Not under `--watch` or `--skip-build`.
 - **A `genai-runtime` skill for the runtime side of Agents Kit 2** (ako/mxcli#2068) — registering a Mendix Cloud GenAI key or an OpenAI-compatible endpoint, seeding AgentCommons agents from JSON (matched by `Title` only, so a changed JSON never reaches a stored agent), resuming Conversational UI chats, loading PgVector and Mendix Cloud knowledge bases, and traces. It collects the gotchas that `check` and `mx check` cannot see: the encryption key both the OpenAI and PgVector setup need, the `vector` extension only the UI's Test button creates, the 60–120 s indexing delay, the chat that collapses outside its page, and `StoreTraces` defaulting to false. Its examples pass `check --references` and build with 0 errors against the Agent Builder Starter App's modules.
 - **`mxcli test --attach` runs a test without building it, against an app started with `--allow-call`** (mendixlabs/mxcli#1788, mendixlabs/mxcli#1789) — `run --local --test-endpoint --allow-call` switches on an executor in the test endpoint. A `.test.mdl` test whose statements it supports (`declare`, `set`, `create`, `change`, `commit`, `delete`, `call microflow`, database `retrieve` with `where`/`first`; `@expect` with `count`, `@throws`, `@setup`, `@cleanup`) is compiled by mxcli into steps and run in the app with the Mendix runtime API, every expression evaluated by `Core.evaluateExpression`. Nothing is injected or built, so a test edit costs one HTTP round trip: on a small app the suite took 0.26 s against 3.2 s for inject-and-build, with identical verdicts. Retrieves over an association and the list aggregates `count`/`sum`/`average`/`minimum`/`maximum` over an attribute run there too. A test with any other statement is injected and built as before. Only that test is, while the rest of the run still goes straight to the app, and the output names the test and the MDL statement (`the statement if`, `a list operation (find)`). Because the runtime evaluator returns empty for an unknown name, an undeclared `$variable` and a non-existent enumeration value are refused (ERROR) before evaluating, and a model newer than the running build is refused with `mxcli run wait` as the fix under `--watch`. The route takes POST, needs the run's token and a loopback caller, and answers 403 without `--allow-call`, since it calls the microflows a test names in a system context. `.test.mdl` stays the only test language. Design discussed with the authors of #1789; measurements in `mdl-examples/spikes/evaluate-expression.mdl`.
@@ -27,6 +28,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ### Fixed
 
+- **`run --local --watch` builds a change as soon as `mxcli exec` or `mxcli -c` has finished writing it** (ako/mxcli#2151) — the watcher noticed a change within a second and then waited for the model to stay unchanged for two more, so a change was applied about 5 s after `exec` started although the build and reload took 2 s. `exec` and `-c` now tell a watching dev loop when they start and when the last write has landed: nothing is built while one is at work, however long the script, and the build starts within 100 ms of its end. Measured on a blank 11.16.0 app: 2.2-2.5 s from the start of `exec` to applied, for a microflow edit, a page edit and a new page. A change made by anything else (an editor, Studio Pro) is settled as before.
 - **A test can call a microflow that applies entity access** (ako/mxcli#2100, mendixlabs/mxcli#1805) — the generated test microflows, the legacy runner's `MxTest.TestRunner` and the wrappers `mxcli check` sees did not apply entity access, so a test (or its `@setup`) calling a production microflow that does failed with CE0114 at build time and MDL-SEC22 at check time, and the only workaround was weakening the production microflow. They now apply it, as `MxTest.RegisterEndpoint` already did. An applying microflow may not read `$latestError/Message` (CE0150), so the runner's error handlers read the message through a generated helper, `MxTest.Test_ErrorMessage`, that does not apply entity access and is removed with the test microflows; failure messages and `@throws` matching are unchanged. A test whose own error handler reads `$latestError/…` keeps a non-applying microflow, so it builds as before. Tests run in a system context, so nothing changes at run time.
 - **`run --local --watch` applies a change in about 5 s instead of 20-28 s on Mendix versions that bundle the client themselves** (ako/mxcli#2150) — measured on a blank 11.16.0 app: a microflow edit took 20-22 s and a page edit 25-28 s with a runtime restart that dropped every browser session, because each change cost a fresh `mxbuild --serve` and a cold build. mxbuild deletes the client source a build wrote, so its next build bundled a client without pages. That source is now kept while a build runs and put back before the next one: the incremental build takes about 2 s, a page edit is a reload, and an open browser tab shows it without a restart or a new login. The full build remains the fallback when the bundle still comes out incomplete. A deleted page's module stays in `dist/pages`, unused, until the next full build. On Windows the copies are read without blocking mxbuild's own deletes, and if a build still fails because `deployment/web` was in use, the kept source is given up for that run and builds go back to the full rebuild.
 - **`check` accepts the data-container names Studio Pro stores in a grid row, a control bar and a combo box option** (ako/mxcli#2109) — `describe page` → `exec` of four FactoryManagement pages was refused on its own output (MDL-BUTTON02, MDL-PAGEARG01): a show page in a selectable data grid's column passing the grid's selection (`$grid1`), a control-bar button passing the data view around its grid, and a combo box's option content reading the data view around the combo box. All three build in Studio Pro's own pages. `check` and `exec` now accept them and write what Studio Pro stores, and the pages report `Unchanged page`.