From a52aaa309cd6c96ea02651cfc40381821d9783bc Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 11 Nov 2026 20:51:15 +0000 Subject: [PATCH 1/8] docs(proposal): nightly agent dev-loop performance benchmark A deterministic, scripted benchmark of the tool-side time an agent waits for: one mxcli process per step, A/B against the previous nightly binary on the same runner, a generated large fixture, and results kept on a dedicated benchmarks branch. Complements the agent-level BENCH-001 run and the diag reports in PROPOSAL_agent_loop_efficiency. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_011oK82u4HRCKfX2BBvhgsNm --- .../PROPOSAL_agent_devloop_benchmark.md | 335 ++++++++++++++++++ docs/11-proposals/README.md | 3 +- 2 files changed, 337 insertions(+), 1 deletion(-) create mode 100644 docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md diff --git a/docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md b/docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md new file mode 100644 index 000000000..da5ed3081 --- /dev/null +++ b/docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md @@ -0,0 +1,335 @@ +--- +title: Nightly Agent Dev-Loop Performance Benchmark +status: draft +date: 2026-11-11 +author: AI-assisted design +related: + - PROPOSAL_agent_loop_efficiency + - PROPOSAL_mcp_bson_benchmark +--- + +# Proposal: Nightly Agent Dev-Loop Performance Benchmark + +**Status:** Draft +**Date:** 2026-11-11 + +## Problem Statement + +An agent building a Mendix app runs mxcli hundreds of times per session — the +four builds measured in +[PROPOSAL_agent_loop_efficiency](PROPOSAL_agent_loop_efficiency.md) made 301 to +936 invocations each — and waits for every one. That proposal's measurements +say where the waiting goes: in the largest session roughly 6 of 9 active hours +were spent waiting on commands rather than on the model, led by app restarts +and `run wait` (≥74 min), browser suites (~70 min), `docker check` (~25–35 min) +and microflow tests (~34 min), with a `run --local` boot at a 70–90 s median in +two other projects. + +Those figures were taken by hand, from individual sessions, after the fact. +Nothing tracks them continuously, so a regression in any of them ships +unnoticed: + +- The nightly workflow (`.github/workflows/nightly.yml`) runs correctness + suites only. A change that doubles `refresh catalog full`, or adds ten + seconds to a hot apply, passes every job. +- The Go benchmarks that exist (`modelsdk/codec/codec_bench_test.go`, + `modelsdk/property/property_bench_test.go`, `BenchmarkRoundtripEntity`) are + micro-benchmarks of one function. They are not run in CI, and they exclude + what the agent actually waits for: process start, opening the MPR, parsing, + reconciling and writing. +- `mxcli eval run` on the fixed `BENCH-001` brief + (`docs/14-eval/eval-bench-001.md`) measures a whole agent session — model + calls, tokens, wall time. It answers "what does a session cost", but it has + an LLM in the loop: its own write-up calls a single run "one sample of a + stochastic process with unmeasured variance". It cannot attribute a 15% + change to a commit. +- `mxcli diag loop-report` and `diag session-report` explain a session that + already happened. They are instruments for diagnosis, not a regression gate. + +This proposal adds the missing piece: a **deterministic** nightly benchmark of +the tool-side time — the same commands an agent runs, on a fixed script, with +no model involved — reporting per step how long mxcli takes and how that +compares to the previous nightly, with enough noise control that a reported +regression is worth acting on. + +It is the complement of the loop-efficiency work, not a replacement. That +proposal shows most of a session's *token* bill is the conversation being +re-read (`N × S`), which no tool-speed benchmark can see. This one covers the +wall-clock share that is the tool's, and the commands it identified as the +wall-time levers. + +### Non-goals + +- Measuring model quality, token use or number of agent turns. An LLM in the + loop adds variance far larger than any tool regression, and `mxcli eval run` + with `diag session-report` already covers that ground. +- Gating pull requests or the nightly release on timing. +- Benchmarking mxbuild or the Mendix runtime as such. Their time is recorded, + because the agent waits for it, but reported separately from mxcli's own. + +## BSON Structure + +Not applicable. The benchmark adds no document types and changes no write +path; it drives the existing CLI from outside. The generated fixture is +produced through ordinary MDL scripts, so it exercises the same writers as any +user project. + +## Proposed Design + +No MDL syntax is added. The feature is a benchmark driver, a fixture +generator, a nightly job and a results branch. + +### Principle: measure at the CLI boundary + +Each measured step is **one mxcli process**, started the way an agent starts +it (`mxcli exec script.mdl -p app.mpr`, `mxcli -p app.mpr -c "…"`). The +recorded time is the process's wall-clock time. This deliberately includes +start-up and project open, which a `testing.B` benchmark excludes and which +dominate short commands. + +The session is a fixed set of committed MDL scripts — deterministic, no LLM. + +### The scripted session + +| Phase | Step | What it times | Needs | +|---|---|---|---| +| Authoring | `new` | `mxcli new` with `--skip-build`, mxbuild already cached | mxbuild | +| | `domain` | one `exec` creating ~50 entities with attributes and associations | — | +| | `microflows` | one `exec` creating ~100 microflows | — | +| | `pages` | one `exec` creating ~50 pages (overview + edit pairs) | — | +| | `single-edit` | one `exec` changing one microflow — the most frequent agent action | — | +| | `check` | `mxcli check script.mdl -p app.mpr --references` | — | +| Reading | `open` | a trivial `-c` command: pure start-up + project open | — | +| | `describe` | `describe` of one entity, one microflow, one page | — | +| | `catalog` | `refresh catalog;` | — | +| | `catalog-full` | `refresh catalog full;` | — | +| | `catalog-after-edit` | `refresh catalog full;` after `single-edit` | — | +| | `query` | a catalog `select` and a search | — | +| Bulk change | `bulk-alter` | one `exec` altering several hundred documents | — | +| | `rename` | renaming an entity referenced from many microflows and pages | — | +| | `re-exec` | re-running the `domain`/`microflows`/`pages` scripts unchanged | — | +| Validate | `mx-check` | `mxcli docker check` | mxbuild | +| | `build` | one full build | mxbuild | +| Running app | `boot` | `mxcli run --local --detach --db-type hsqldb` until ready | runtime | +| | `hot-apply-flow` | microflow change, then `mxcli run wait` | runtime | +| | `apply-entity` | entity change (schema update), then `mxcli run wait` | runtime | +| | `page-check` | `mxcli playwright check` on one page | runtime | +| Tests | `test-cold` | `mxcli test --local --db-type hsqldb` on a small suite | runtime | +| | `test-attach` | `mxcli test --attach` against the app from `boot` | runtime | + +The step list follows the verbs the measured sessions actually ran (`exec`, +`-c`, `check`, `run wait`, `docker check`, `test`). Each step's time is also +reported multiplied by a fixed per-session call count taken from those +sessions, giving one **session-weighted total**: a step that is 50 ms slower +and runs 200 times outranks one that is 2 s slower and runs once. + +`re-exec` doubles as a correctness canary: per +[ADR-0008](../13-decisions/0008-identity-and-idempotence.md) it must write +nothing, so a non-zero write count there is a defect, not a slowdown. + +### Two project sizes + +The steps marked "—" run against two projects: + +- **small** — the blank app that `mxcli new` produces. +- **large** — a generated app of a few thousand documents. + +Project open, catalog refresh and reference checks scale with project size. +Reporting both sizes shows the *slope*, which is what an agent feels on a real +customer project and what a blank-app benchmark hides. + +### The generated fixture + +A generator emits MDL for a parameterised app — `M` modules, each with `E` +entities, `F` microflows and `P` pages, cross-module associations and calls at +a fixed density — and executes it against a blank project. + +- **Deterministic.** Names and structure derive from the parameters and a fixed + seed, so two runs produce the same model and the deterministic counters below + are comparable night to night. +- **Not committed.** Only the generator and its parameters are in the repo; the + `.mpr` is built in the job. Building it is itself the bulk-create measurement. +- **Realistic shape, not just volume.** The templates are taken from the + existing example apps under `mdl-examples/` (CRUD pages, validation + microflows, list processing), so the documents have typical size and + reference fan-out rather than being thousands of empty shells. +- **Sized by parameter.** The nightly uses one "large" preset; a local run can + pass smaller or larger values to study scaling. + +Caveat to keep in view: a fixture mxcli created has `GUID == $ID` everywhere +and contains only documents mxcli can write. It is a valid subject for timing, +and an invalid one for any identity-preservation claim (see the GUID section +of `CLAUDE.md`). + +### Noise control + +GitHub-hosted runners differ by 10–20% from run to run, so comparing tonight's +absolute time to last night's produces false alarms. Three mechanisms: + +1. **A/B against the previous nightly on the same runner.** The job downloads + the binary from the existing `nightly` release (published by the `nightly` + job) before that release is replaced, and runs both binaries interleaved — + A, B, A, B — on the same machine. The reported figure is the ratio + new/old. Runner speed cancels out. This applies to every step whose time is + mxcli's own. +2. **Median of five.** Each step runs five times against a fresh copy of the + project; the median is reported and the spread recorded, so a noisy step is + visible as noisy. +3. **Deterministic counters.** Alongside time, record values that do not depend + on the machine: units offered and written (`backend.WriteStats`), bytes + written to the project, and output size. These can be compared exactly + between nights. A step that suddenly writes 400 units instead of 40 is a + regression whether or not the runner was slow. + +Steps dominated by mxbuild or the runtime (`mx-check`, `build`, `boot`, +`apply-*`, `test-*`) get the same A/B treatment but need more care in reading: +the external tool is identical in both arms and contributes most of the time +and most of the noise, so a real mxcli regression is a smaller fraction of a +noisier number. These are also the steps the measured sessions spend the most +wall time in, so they are not optional — their thresholds are simply set wider, +from their own measured noise floor (see Test Plan). + +The driver times each process itself. mxcli's existing per-invocation session +log (`mdl/diaglog`, the source for `diag loop-report`) is collected alongside +as a cross-check and to split a step's time into its statements. + +### Results and history + +- **One JSON record per run**: commit, date, Mendix version, runner details, + fixture parameters, and per step the median, spread, A/B ratio and counters. +- **Dedicated `benchmarks` branch.** The job commits each record as its own + file (`results/YYYY/MM/-.json`). The branch is an orphan: it holds + only results, never merges into `main`, and one file per run means concurrent + runs cannot conflict. `nightly.yml` already has `contents: write`. +- **Job summary.** A table in `$GITHUB_STEP_SUMMARY` with tonight's ratios, + regressions highlighted. +- **Trend chart.** A page on the docs site reads the branch's records and plots + each step over time. This is where slow drift — invisible night to night — + shows up. + +### Alerting + +Reporting only at first. After roughly two weeks of data, set a per-step +threshold from the observed spread and have the job open (or update) one issue +when a step's A/B ratio exceeds it on two consecutive nights. A counter change +is flagged on the first night, since it carries no noise. + +### Nightly job + +A new `bench` job in `nightly.yml`: + +- runs on one Mendix version (the newest in the `test` matrix); +- is **not** in `needs:` of the `nightly` release job — the same arrangement as + `upgrade-full-corpus` and `tunnel-seam-macos` — so a benchmark failure never + holds back a release; +- downloads the previous nightly binary as its first step. The release job + waits on the `test` matrix, which takes far longer, but the job should not + rely on that ordering: if the download does not match an earlier commit, the + A/B arm is skipped and the record says so. + +### Local use + +`make bench` runs the same driver against a locally built binary, optionally +with `BASELINE=` for an A/B, and prints the same table. +A regression reported by the nightly must be reproducible this way, or it +cannot be bisected. + +## Implementation Plan + +Delivered in phases; each is useful on its own. + +### Phase 1 — driver, fixture, mxcli-only steps + +The driver, the generator, the `benchmarks` branch, the job, and every step in +the table marked "—". No runtime needed, so the job is short and the A/B signal +is clean. This phase exists to get the machinery and its noise floor right on +the easy case; it is **not** where most of the measured wall time is. + +### Phase 2 — running app and tests + +Adds `boot`, `hot-apply-flow`, `apply-entity`, `page-check`, `test-cold`, +`test-attach`, using `--db-type hsqldb` so the job needs no PostgreSQL service. +This is ordered second, ahead of validate/build, because restarts, waits and +tests are the largest measured share of command wall time. + +### Phase 3 — validate and build + +Adds `mx-check` and `build`. + +### Later, separately — a scheduled agent-level run + +`mxcli eval run` on `BENCH-001` is the agent-level counterpart and already +exists. Running it on a schedule and committing its session report to the same +`benchmarks` branch would put both views on one chart. It needs model +credentials in CI and enough repetitions to have a variance, so it is its own +decision and not part of this proposal. + +### Files to modify/create + +| File | Change | +|------|--------| +| `scripts/agentbench/` (new, Go) | Driver: runs steps, interleaves A/B, takes medians, collects counters, writes the JSON record and the summary table | +| `scripts/agentbench/gen/` (new) | Fixture generator: parameters → MDL scripts | +| `scripts/agentbench/session/*.mdl` (new) | The scripted session's MDL | +| `Makefile` | `bench` target | +| `.github/workflows/nightly.yml` | `bench` job; commit of the record to the `benchmarks` branch | +| `cmd/mxcli/` | Machine-readable write counters for a command (see Open Questions) | +| `docs-site/src/internals/` | Page describing the benchmark and showing the trend chart | + +## Version Compatibility + +No version gating. The benchmark runs on one Mendix version per night. The +session scripts must stay within syntax that version supports; running a +second version is a matrix entry, not a code change. + +## Test Plan + +- **Driver unit tests** — median and ratio arithmetic; record schema; the A/B + arm being skipped cleanly when no baseline binary is available. +- **Generator determinism** — generating twice with the same parameters yields + byte-identical MDL. +- **Session validity** — the session scripts and a small generated fixture pass + `mxcli check`, and after exec pass `mx check`, in an integration test, so the + benchmark cannot silently measure a failing script. A step that exits + non-zero fails the run rather than recording a (fast) time. +- **The benchmark detects a regression** — a control, per the repo's evidence + rule: build a binary with a deliberate delay in one step and confirm the A/B + ratio and the summary flag exactly that step. Without this, a benchmark that + always reports "no change" is indistinguishable from a working one. +- **Noise floor** — run A/B with the *same* binary in both arms and record the + ratio spread. This is the measured basis for the alert thresholds. + +## Open Questions + +1. **Write counters on the CLI.** `backend.WriteStats` (`Offered`, `Written`) + exists and drives the "Unchanged …" reporting, but this proposal has not + found a machine-readable form of it on `exec`. The driver needs one — a + flag or an environment variable that emits a final stats line. Bytes written + can be measured from outside by diffing the project directory. +2. **Large-fixture size.** A few thousand documents is a guess at "typical + customer project". The preset should be set from real numbers if they are + available. +3. **Job duration.** Five interleaved repetitions of every step at two sizes + must fit a reasonable budget (target: under 30 minutes for Phase 1). If the + large fixture makes that impossible, reduce repetitions for the slowest + steps rather than dropping the A/B. +4. **Baseline across a format change.** If a commit changes what a script + writes, the old binary runs the same script against the same starting + project, so the A/B stays valid — but if the session scripts use syntax the + previous nightly does not have, the baseline arm fails. The driver should + skip A/B for that step and say so, not fail the run. +5. **`mxcli new` in the session.** It downloads mxbuild and by default runs a + build. The benchmark pre-caches mxbuild and passes `--skip-build` so the + step times project creation only; whether that is the number worth tracking, + or the full default `new`, is a judgement call. +6. **Chart hosting.** Whether the docs-site build may read from the + `benchmarks` branch at build time, or the chart page should fetch the + records client-side. +7. **Session weights.** The per-session call counts behind the weighted total + come from four measured sessions of different kinds of work. They should be + fixed in the driver and versioned in the record, so a change of weights is + never read as a change of speed. +8. **Licence session cap.** The loop-efficiency proposal records, as an + unverified lead, that many restarts were forced by the development licence's + session cap. If true, a benchmark that boots, logs in and runs tests + repeatedly may hit the same cap and measure it. diff --git a/docs/11-proposals/README.md b/docs/11-proposals/README.md index 0619593ba..a56c4fd56 100644 --- a/docs/11-proposals/README.md +++ b/docs/11-proposals/README.md @@ -27,7 +27,7 @@ for display in this README): -## Active Proposals (122) +## Active Proposals (123) ### In Progress (partial) (17) @@ -147,6 +147,7 @@ for display in this README): | [mxcli companion, slice 2 — a long-lived mxcli the mod talks to over a Unix socket](proposal-mxcli-companion-serve.md) | Draft | Slice 2 of the companion umbrella. | | [mxcli marketplace diff — detect local modification and plan a GUID-preserving module upgrade](PROPOSAL_marketplace_module_upgrade.md) | Draft | 2026-08-11 (DESCRIBE coverage measured — §7; GUID = database identity measured — §8) | | [mxcli Playground](mxcli-playground.md) | Draft | A public GitHub repository (mendixlabs/mxcli-playground) containing a ready-to-use Mendix project pre-configured with mxcli, Claude Code ski | +| [Nightly Agent Dev-Loop Performance Benchmark](PROPOSAL_agent_devloop_benchmark.md) | Draft | An agent building a Mendix app runs mxcli hundreds of times per session — the | | [Offline synchronization configuration — the one thing an offline profile needs that MDL cannot say](PROPOSAL_offline_sync_configuration.md) | Draft | mxcli can author an offline navigation profile — PhoneOffline, | | [OpenAPI import — generate response entities and wire operations to them](PROPOSAL_openapi_response_entities.md) | Draft | Issue: ako/mxcli#1234 (mendixlabs/mxcli#280) | | [Owning the modelsdk/gen codegen — why the vendored generator cannot be adopted as-is](PROPOSAL_codegen_ownership.md) | Draft | Constraint set by the maintainer (2026-08-13): the reflection data is an | From 15d1265d757a00542b9982a00f9620189412bbbd Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 11 Nov 2026 20:59:23 +0000 Subject: [PATCH 2/8] docs(proposal): relate the benchmark to the existing read/decode budget tests Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_011oK82u4HRCKfX2BBvhgsNm --- .../PROPOSAL_agent_devloop_benchmark.md | 35 +++++++++++++------ 1 file changed, 25 insertions(+), 10 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md b/docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md index da5ed3081..5d4cee41d 100644 --- a/docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md +++ b/docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md @@ -29,9 +29,22 @@ Those figures were taken by hand, from individual sessions, after the fact. Nothing tracks them continuously, so a regression in any of them ships unnoticed: +- The recent performance fixes each left a **budget test** behind: a unit test + that counts unit-file reads or document decodes for one command on the PedApp + fixture and fails above a bound (`unit_reads_per_command_pedapp_test.go`, + `describe_decode_budget_pedapp_test.go`, + `catalog_incremental_budget_pedapp_test.go`, + `flow_build_decodes_2094_test.go`, the `reader_*` tests in `modelsdk/mpr`). + They run on every PR and are noise-free, and this proposal keeps them as the + first line of defence. They have four gaps by construction: they count reads + and decodes, not time, so CPU, parsing, SQLite and process start-up are + outside them; they run on a 375-unit fixture, while the defects they pin were + measured on 3,400- to 4,753-unit projects; they exist only for the paths + someone already found slow; and their bounds are deliberately loose, so a + partial regression passes. - The nightly workflow (`.github/workflows/nightly.yml`) runs correctness - suites only. A change that doubles `refresh catalog full`, or adds ten - seconds to a hot apply, passes every job. + suites only. A change that adds ten seconds to a hot apply, or makes project + open slower without one extra file read, passes every job. - The Go benchmarks that exist (`modelsdk/codec/codec_bench_test.go`, `modelsdk/property/property_bench_test.go`, `BenchmarkRoundtripEntity`) are micro-benchmarks of one function. They are not run in CI, and they exclude @@ -176,8 +189,9 @@ absolute time to last night's produces false alarms. Three mechanisms: project; the median is reported and the spread recorded, so a noisy step is visible as noisy. 3. **Deterministic counters.** Alongside time, record values that do not depend - on the machine: units offered and written (`backend.WriteStats`), bytes - written to the project, and output size. These can be compared exactly + on the machine: unit files read (the backend's `UnitFileReads`, the counter + the existing budget tests assert on), units offered and written + (`backend.WriteStats`), bytes written to the project, and output size. These can be compared exactly between nights. A step that suddenly writes 400 units instead of 40 is a regression whether or not the runner was slow. @@ -273,7 +287,7 @@ decision and not part of this proposal. | `scripts/agentbench/session/*.mdl` (new) | The scripted session's MDL | | `Makefile` | `bench` target | | `.github/workflows/nightly.yml` | `bench` job; commit of the record to the `benchmarks` branch | -| `cmd/mxcli/` | Machine-readable write counters for a command (see Open Questions) | +| `cmd/mxcli/` | Machine-readable read and write counters for a command (see Open Questions) | | `docs-site/src/internals/` | Page describing the benchmark and showing the trend chart | ## Version Compatibility @@ -301,11 +315,12 @@ second version is a matrix entry, not a code change. ## Open Questions -1. **Write counters on the CLI.** `backend.WriteStats` (`Offered`, `Written`) - exists and drives the "Unchanged …" reporting, but this proposal has not - found a machine-readable form of it on `exec`. The driver needs one — a - flag or an environment variable that emits a final stats line. Bytes written - can be measured from outside by diffing the project directory. +1. **Counters on the CLI.** `backend.WriteStats` (`Offered`, `Written`) and + the backend's `UnitFileReads` exist — the first drives the "Unchanged …" + reporting, the second the budget tests — but this proposal has not found a + machine-readable form of either on a command. The driver needs one — a flag + or an environment variable that emits a final stats line. Bytes written can + be measured from outside by diffing the project directory. 2. **Large-fixture size.** A few thousand documents is a guess at "typical customer project". The preset should be set from real numbers if they are available. From 52694a7d1a2e5b60a338d5ccbe67f346cb681423 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 11 Nov 2026 22:13:03 +0000 Subject: [PATCH 3/8] feat(executor): MXCLI_STATS_FILE reports a connection's reads and writes Each project connection appends one JSON line when it closes: unit files read, unit writes offered and landed. These are the counts the read- and decode-budget tests assert on in process; the agent dev-loop benchmark needs them from outside, per command, and a count does not depend on the machine it was taken on. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_011oK82u4HRCKfX2BBvhgsNm --- mdl/executor/executor.go | 2 +- mdl/executor/executor_connect.go | 6 +- mdl/executor/process_stats.go | 71 ++++++++++++++++++++ mdl/executor/process_stats_pedapp_test.go | 81 +++++++++++++++++++++++ 4 files changed, 156 insertions(+), 4 deletions(-) create mode 100644 mdl/executor/process_stats.go create mode 100644 mdl/executor/process_stats_pedapp_test.go diff --git a/mdl/executor/executor.go b/mdl/executor/executor.go index 9a98b1ec5..c68986676 100644 --- a/mdl/executor/executor.go +++ b/mdl/executor/executor.go @@ -827,7 +827,7 @@ func (e *Executor) Close() error { closeErr = fmt.Errorf("a statement that timed out was still running after %v; closing the project under it", abandonedStatementWait) } if e.backend != nil && e.backend.IsConnected() { - if err := e.backend.Disconnect(); err != nil && closeErr == nil { + if err := disconnectBackend(e.backend); err != nil && closeErr == nil { closeErr = err } e.backend = nil diff --git a/mdl/executor/executor_connect.go b/mdl/executor/executor_connect.go index c17f7bba5..812e44d81 100644 --- a/mdl/executor/executor_connect.go +++ b/mdl/executor/executor_connect.go @@ -12,7 +12,7 @@ import ( func execConnect(ctx *ExecContext, s *ast.ConnectStmt) error { if ctx.Backend != nil && ctx.Backend.IsConnected() { - if err := ctx.Backend.Disconnect(); err != nil { + if err := disconnectBackend(ctx.Backend); err != nil { fmt.Fprintf(ctx.Output, "Warning: disconnect error: %v\n", err) } } @@ -54,7 +54,7 @@ func reconnect(ctx *ExecContext) error { // Close existing connection if ctx.Backend != nil && ctx.Backend.IsConnected() { - if err := ctx.Backend.Disconnect(); err != nil { + if err := disconnectBackend(ctx.Backend); err != nil { fmt.Fprintf(ctx.Output, "Warning: disconnect error: %v\n", err) } } @@ -91,7 +91,7 @@ func execDisconnect(ctx *ExecContext) error { } } - if err := ctx.Backend.Disconnect(); err != nil { + if err := disconnectBackend(ctx.Backend); err != nil { fmt.Fprintf(ctx.Output, "Warning: disconnect error: %v\n", err) } fmt.Fprintf(ctx.Output, "Disconnected from: %s\n", ctx.MprPath) diff --git a/mdl/executor/process_stats.go b/mdl/executor/process_stats.go new file mode 100644 index 000000000..db2fafc09 --- /dev/null +++ b/mdl/executor/process_stats.go @@ -0,0 +1,71 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "encoding/json" + "os" + + "github.com/mendixlabs/mxcli/mdl/backend" +) + +// StatsFileEnv names a file to which every project connection appends one +// JSON line when it closes: the unit files it read and the unit writes it +// offered and landed. Unset, nothing is written. +// +// These are the counts the read- and decode-budget tests assert on in process +// (unitFileReads, backend.WriteStats). The agent dev-loop benchmark needs them +// from outside, per command, because it measures whole processes; a count does +// not depend on the machine, so unlike a duration it can be compared exactly +// between two runs (docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md). +// +// One line per connection rather than one per process: a command that +// reconnects, or starts child mxcli processes that inherit the variable, +// appends several, and the reader sums them. +const StatsFileEnv = "MXCLI_STATS_FILE" + +// ProcessStats is one line of the stats file. +type ProcessStats struct { + UnitFileReads int64 `json:"unit_file_reads"` + WritesOffered int `json:"writes_offered"` + WritesWritten int `json:"writes_written"` +} + +// disconnectBackend closes b's project connection and, when StatsFileEnv is +// set, appends what the connection did. Every disconnect goes through here so +// no connection's counts are lost. +// +// The reads are taken before the disconnect, which drops the reader that +// counts them; the writes after it, because the disconnect is what flushes +// deferred writes. +func disconnectBackend(b backend.FullBackend) error { + path := os.Getenv(StatsFileEnv) + if path == "" { + return b.Disconnect() + } + var stats ProcessStats + if r, ok := b.(interface{ UnitFileReads() int64 }); ok { + stats.UnitFileReads = r.UnitFileReads() + } + err := b.Disconnect() + ws := currentWriteStats(b) + stats.WritesOffered, stats.WritesWritten = ws.Offered, ws.Written + appendProcessStats(path, stats) + return err +} + +// appendProcessStats writes one line. A failure is swallowed: the file is a +// measurement aid, and a command must not fail because it could not be +// measured. +func appendProcessStats(path string, stats ProcessStats) { + line, err := json.Marshal(stats) + if err != nil { + return + } + f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644) + if err != nil { + return + } + defer f.Close() + _, _ = f.Write(append(line, '\n')) +} diff --git a/mdl/executor/process_stats_pedapp_test.go b/mdl/executor/process_stats_pedapp_test.go new file mode 100644 index 000000000..f97334726 --- /dev/null +++ b/mdl/executor/process_stats_pedapp_test.go @@ -0,0 +1,81 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bufio" + "encoding/json" + "os" + "path/filepath" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// readProcessStats returns the lines a stats file holds. +func readProcessStats(t *testing.T, path string) []ProcessStats { + t.Helper() + f, err := os.Open(path) + if os.IsNotExist(err) { + return nil + } + if err != nil { + t.Fatal(err) + } + defer f.Close() + var out []ProcessStats + sc := bufio.NewScanner(f) + for sc.Scan() { + var s ProcessStats + if err := json.Unmarshal(sc.Bytes(), &s); err != nil { + t.Fatalf("stats line %q: %v", sc.Text(), err) + } + out = append(out, s) + } + return out +} + +// A connection reports what it read and wrote when it closes, so a benchmark +// timing whole mxcli processes can also record counts that do not depend on +// the machine (PROPOSAL_agent_devloop_benchmark.md). The three connections +// below are told apart by what they did: a read-only one, one that writes, and +// one that offers the same write again and lands nothing. +func TestProcessStats_OneLinePerConnection(t *testing.T) { + stats := filepath.Join(t.TempDir(), "stats.jsonl") + t.Setenv(StatsFileEnv, stats) + + const create = "mdl 1;\ncreate or modify persistent entity MyFirstModule.StatsProbe ( Name: string(50) );" + exec, out, dir := openPedAppCopy(t) + run := func(src string) { + t.Helper() + if err := agreeExec(t, exec, src); err != nil { + t.Fatalf("%s: %v\n%s", src, err, out.String()) + } + if err := exec.Execute(&ast.DisconnectStmt{}); err != nil { + t.Fatal(err) + } + if err := exec.Execute(&ast.ConnectStmt{Path: filepath.Join(dir, "PedApp.mpr")}); err != nil { + t.Fatal(err) + } + } + run("show modules;") + run(create) + run(create) + + got := readProcessStats(t, stats) + if len(got) != 3 { + t.Fatalf("got %d stats lines, want one per closed connection (3): %+v", len(got), got) + } + if got[0].UnitFileReads == 0 { + t.Errorf("show modules read no unit files: %+v — the count is taken after the reader is gone", got[0]) + } + if got[0].WritesWritten != 0 { + t.Errorf("show modules wrote: %+v", got[0]) + } + if got[1].WritesWritten == 0 { + t.Errorf("creating an entity landed no write: %+v — the count is taken before the deferred writes flush", got[1]) + } + if got[2].WritesOffered == 0 || got[2].WritesWritten != 0 { + t.Errorf("re-running the same statement: %+v, want writes offered and none landed", got[2]) + } +} From 6742cc70f251007c182445d0ec771f1cc61f2026 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 11 Nov 2026 22:13:04 +0000 Subject: [PATCH 4/8] feat(bench): agent dev-loop benchmark, phase 1 scripts/agentbench runs a fixed session of mxcli commands, one process per command, on PedApp and on PedApp with a generated app seeded into it. Given a baseline binary it runs both interleaved on the same machine and reports each step's ratio, with unit files read and unit files changed as machine-independent counts. make bench runs it; the nightly bench job runs it against the previous nightly release and commits the record to the benchmarks branch. It does not gate the release. Phase 1 of PROPOSAL_agent_devloop_benchmark: no mxbuild, no runtime. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_011oK82u4HRCKfX2BBvhgsNm --- .github/workflows/nightly.yml | 98 +++ CHANGELOG.md | 1 + Makefile | 21 +- docs-site/src/SUMMARY.md | 1 + docs-site/src/internals/agent-benchmark.md | 161 +++++ .../PROPOSAL_agent_devloop_benchmark.md | 51 +- docs/11-proposals/README.md | 6 +- scripts/agentbench/bench.go | 597 ++++++++++++++++++ scripts/agentbench/bench_test.go | 241 +++++++ scripts/agentbench/gen/gen.go | 258 ++++++++ scripts/agentbench/gen/gen_test.go | 126 ++++ scripts/agentbench/main.go | 101 +++ scripts/agentbench/record.go | 235 +++++++ scripts/agentbench/rss_other.go | 10 + scripts/agentbench/rss_unix.go | 24 + 15 files changed, 1922 insertions(+), 9 deletions(-) create mode 100644 docs-site/src/internals/agent-benchmark.md create mode 100644 scripts/agentbench/bench.go create mode 100644 scripts/agentbench/bench_test.go create mode 100644 scripts/agentbench/gen/gen.go create mode 100644 scripts/agentbench/gen/gen_test.go create mode 100644 scripts/agentbench/main.go create mode 100644 scripts/agentbench/record.go create mode 100644 scripts/agentbench/rss_other.go create mode 100644 scripts/agentbench/rss_unix.go diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index f91e1986f..dccf4f90e 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -183,6 +183,104 @@ jobs: MXCLI_UPGRADE_SHARD: ${{ matrix.shard }} timeout-minutes: 45 + # The agent dev-loop benchmark (docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md): + # a scripted session of the mxcli commands an agent runs, timed one process + # per command. A hosted runner's speed varies from night to night by more + # than most regressions, so the job does not compare tonight with last + # night: it downloads the previous nightly's binary and runs both + # interleaved on THIS runner, and reports the ratio. The record goes to the + # `benchmarks` branch, one file per run. + # + # Needs no mxbuild (it starts from the committed PedApp fixture) and does + # not gate the release, like upgrade-full-corpus: a slow night must not hold + # a nightly back. The driver exits non-zero only when a command fails or + # re-running a settled script changes a file. + bench: + runs-on: ubuntu-latest + timeout-minutes: 60 + name: agent dev-loop benchmark + steps: + - uses: actions/checkout@v7 + - uses: actions/setup-go@v7 + with: + go-version: '1.27.4' + - name: Cache ANTLR4 JAR + uses: actions/cache@v6 + with: + path: ~/.m2/repository/org/antlr/antlr4 + key: antlr4-4.13.2 + - name: Install ANTLR4 + run: pip install 'antlr4-tools==0.2.2' + - name: Generate parser + run: make grammar + env: + ANTLR4_TOOLS_ANTLR_VERSION: '4.13.2' + + # First, before the `nightly` job can replace the release. That job + # waits for the whole `test` matrix, but nothing here relies on it: the + # record names the baseline's version, and with no baseline the run + # still measures, without ratios. + - name: Download the previous nightly binary + id: baseline + run: | + mkdir -p "$RUNNER_TEMP/baseline" + if gh release download nightly --repo "$GITHUB_REPOSITORY" \ + --pattern mxcli-linux-amd64 --dir "$RUNNER_TEMP/baseline"; then + chmod +x "$RUNNER_TEMP/baseline/mxcli-linux-amd64" + echo "path=$RUNNER_TEMP/baseline/mxcli-linux-amd64" >> "$GITHUB_OUTPUT" + else + echo "No previous nightly binary: measuring without an A/B." + fi + env: + GH_TOKEN: ${{ github.token }} + + - name: Run the benchmark + run: | + make bench BASELINE="${{ steps.baseline.outputs.path }}" \ + BENCH_FLAGS="-out agentbench-record.json -summary $GITHUB_STEP_SUMMARY" + + - name: Upload the record + if: always() + uses: actions/upload-artifact@v4 + with: + name: agent-benchmark-record + path: agentbench-record.json + if-no-files-found: ignore + + # One file per run, so two runs never edit the same file; the retry is + # for two runs pushing at once. An orphan branch: results only, never + # merged. Runs from another branch (workflow_dispatch) are not history. + - name: Commit the record to the benchmarks branch + # hashFiles sees the workspace only, which is why the record is written there. + if: always() && github.ref == 'refs/heads/main' && hashFiles('agentbench-record.json') != '' + run: | + set -euo pipefail + sha=$(git rev-parse --short HEAD) + file="results/$(date -u +%Y/%m)/$(date -u +%Y-%m-%dT%H%M)-$sha.json" + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + if git fetch origin benchmarks:benchmarks 2>/dev/null; then + git worktree add "$RUNNER_TEMP/benchmarks" benchmarks + else + git worktree add --detach "$RUNNER_TEMP/benchmarks" + git -C "$RUNNER_TEMP/benchmarks" checkout --orphan benchmarks + git -C "$RUNNER_TEMP/benchmarks" rm -rf -q . + printf '%s\n' '# Agent dev-loop benchmark records' '' \ + 'One JSON file per nightly run, written by the `bench` job of `.github/workflows/nightly.yml`.' \ + 'Never merged into `main`. How to read a record: `docs-site/src/internals/agent-benchmark.md` on `main`.' \ + > "$RUNNER_TEMP/benchmarks/README.md" + fi + cd "$RUNNER_TEMP/benchmarks" + mkdir -p "$(dirname "$file")" + cp "$GITHUB_WORKSPACE/agentbench-record.json" "$file" + git add -A + git commit -q -m "bench: $sha" + for attempt in 1 2 3; do + if git push origin benchmarks; then exit 0; fi + git pull --rebase origin benchmarks + done + exit 1 + nightly: needs: test runs-on: ubuntu-latest diff --git a/CHANGELOG.md b/CHANGELOG.md index 1f78b49c0..2e98b2803 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added +- **`make bench`: a benchmark of what an agent waits for, and `MXCLI_STATS_FILE`** — a fixed session of mxcli commands (open, author entities, microflows and pages, describe, catalog refresh, check, a one-document edit, a bulk change, a re-run, a rename), one process per command, on PedApp and on PedApp with a generated app seeded into it. With `BASELINE=` the two binaries run interleaved on the same machine and each step reports the ratio; the nightly `bench` job does that against the previous nightly and commits the record to the `benchmarks` branch. It also records counts that do not depend on the machine: unit files read and unit files changed. `MXCLI_STATS_FILE=` makes any mxcli command append one JSON line per closed project connection with its unit-file reads and its writes offered and landed. See "Agent Dev-Loop Benchmark" in the docs. - **`mxcli test --local --reuse-build` skips the injection and the build when nothing changed** (mendixlabs/mxcli#1783, closes mendixlabs/mxcli#1782) — a local test run injects its test microflows, builds and boots; when the project, the test files and mxcli are unchanged, the deployment the previous run left is already this run's build, so `--reuse-build` boots it directly (about 12 s against 60 s, measured by the author). The decision is a content hash of the project folder (outputs, caches, `theme-cache/`, and any `.git`/`node_modules` left out), the generated test MDL, the mxcli binary and the mxbuild override, pinned to the identity of `deployment/model/model.mdp`, so a deployment rebuilt by anything else is never taken for the test build. Not under `--watch` or `--skip-build`. - **A `genai-runtime` skill for the runtime side of Agents Kit 2** (ako/mxcli#2068) — registering a Mendix Cloud GenAI key or an OpenAI-compatible endpoint, seeding AgentCommons agents from JSON (matched by `Title` only, so a changed JSON never reaches a stored agent), resuming Conversational UI chats, loading PgVector and Mendix Cloud knowledge bases, and traces. It collects the gotchas that `check` and `mx check` cannot see: the encryption key both the OpenAI and PgVector setup need, the `vector` extension only the UI's Test button creates, the 60–120 s indexing delay, the chat that collapses outside its page, and `StoreTraces` defaulting to false. Its examples pass `check --references` and build with 0 errors against the Agent Builder Starter App's modules. - **`mxcli test --attach` runs a test without building it, against an app started with `--allow-call`** (mendixlabs/mxcli#1788, mendixlabs/mxcli#1789) — `run --local --test-endpoint --allow-call` switches on an executor in the test endpoint. A `.test.mdl` test whose statements it supports (`declare`, `set`, `create`, `change`, `commit`, `delete`, `call microflow`, database `retrieve` with `where`/`first`; `@expect` with `count`, `@throws`, `@setup`, `@cleanup`) is compiled by mxcli into steps and run in the app with the Mendix runtime API, every expression evaluated by `Core.evaluateExpression`. Nothing is injected or built, so a test edit costs one HTTP round trip: on a small app the suite took 0.26 s against 3.2 s for inject-and-build, with identical verdicts. Retrieves over an association and the list aggregates `count`/`sum`/`average`/`minimum`/`maximum` over an attribute run there too. A test with any other statement is injected and built as before. Only that test is, while the rest of the run still goes straight to the app, and the output names the test and the MDL statement (`the statement if`, `a list operation (find)`). Because the runtime evaluator returns empty for an unknown name, an undeclared `$variable` and a non-existent enumeration value are refused (ERROR) before evaluating, and a model newer than the running build is refused with `mxcli run wait` as the fix under `--watch`. The route takes POST, needs the run's token and a loopback caller, and answers 403 without `--allow-call`, since it calls the microflows a test names in a system context. `.test.mdl` stays the only test language. Design discussed with the authors of #1789; measurements in `mdl-examples/spikes/evaluate-expression.mdl`. diff --git a/Makefile b/Makefile index 2c4db48ab..5204826bf 100644 --- a/Makefile +++ b/Makefile @@ -36,7 +36,7 @@ GO_BUILD_FLAGS = -trimpath # Clean version for VS Code extension (must be valid semver: major.minor.patch) VSCE_VERSION = $(shell echo "$(VERSION)" | sed 's/^v//; s/-.*//' | grep -E '^[0-9]+\.[0-9]+\.[0-9]+$$' || echo "0.0.0") -.PHONY: build build-debug size release clean test ci-local test-mdl check-mdl check-skill-mdl check-conformance conformance-shrink gen-migration-reference check-migration-reference check-skill-pack-js check-findings check-wiki-pages digest-status check-tunnel-deps check-test-timeouts check-widget-versions test-integration test-integration-executor test-integration-roundtrip test-integration-parity test-integration-upgrade test-integration-other grammar completions sync-skills sync-skill-packs sync-commands sync-mods sync-lint-rules sync-changelog sync-all docs documentation docs-site docs-serve vscode-ext vscode-install source-tree sbom sbom-report lint lint-go lint-ts fmt fmt-check vet skill-index check-skill-index +.PHONY: build build-debug size release clean test ci-local test-mdl check-mdl check-skill-mdl check-conformance conformance-shrink gen-migration-reference check-migration-reference check-skill-pack-js check-findings check-wiki-pages digest-status check-tunnel-deps check-test-timeouts check-widget-versions test-integration test-integration-executor test-integration-roundtrip test-integration-parity test-integration-upgrade test-integration-other grammar completions sync-skills sync-skill-packs sync-commands sync-mods sync-lint-rules sync-changelog sync-all docs documentation docs-site docs-serve vscode-ext vscode-install source-tree sbom sbom-report lint lint-go lint-ts fmt fmt-check vet skill-index check-skill-index bench # Helper: copy file only if content differs (avoids mtime updates that invalidate go build cache) # Usage: $(call copy-if-changed,src,dst) @@ -431,6 +431,25 @@ test-integration-other: echo "integration packages: $$pkgs"; \ $(INTEGRATION_GO_TEST) -timeout 40m $$pkgs +# Agent dev-loop benchmark: a fixed, scripted session of mxcli commands -- the +# ones an agent runs while building an app -- one process per command, at two +# project sizes, no model in the loop. Needs no mxbuild: it starts from the +# committed PedApp fixture. +# +# make bench # time this checkout's binary +# make bench BASELINE=/path/to/other/mxcli # A/B against another build +# make bench BENCH_FLAGS='-sizes small -reps 3 -out record.json' +# +# With BASELINE the two binaries run interleaved on this machine and the table +# shows the ratio, which is the only figure comparable between two machines or +# two nights. The nightly `bench` job does exactly this against the previous +# nightly release and commits the record to the `benchmarks` branch. +# docs-site/src/internals/agent-benchmark.md explains how to read it. +BASELINE ?= +BENCH_FLAGS ?= +bench: build + go run ./scripts/agentbench -mxcli $(BUILD_DIR)/$(BINARY_NAME) $(if $(BASELINE),-baseline $(BASELINE)) $(BENCH_FLAGS) + # Run MDL integration tests (requires Docker and a Mendix project) # Usage: make test-mdl MPR=path/to/app.mpr MPR ?= app.mpr diff --git a/docs-site/src/SUMMARY.md b/docs-site/src/SUMMARY.md index 1390b8693..40b031e21 100644 --- a/docs-site/src/SUMMARY.md +++ b/docs-site/src/SUMMARY.md @@ -348,6 +348,7 @@ - [FTS5 Full-Text Search](internals/fts5.md) - [Reference Tracking](internals/reference-tracking.md) - [MCP Backend: Script-to-Tool-Call Cost](internals/mcp-backend-cost.md) +- [Agent Dev-Loop Benchmark](internals/agent-benchmark.md) --- diff --git a/docs-site/src/internals/agent-benchmark.md b/docs-site/src/internals/agent-benchmark.md new file mode 100644 index 000000000..74b7511dc --- /dev/null +++ b/docs-site/src/internals/agent-benchmark.md @@ -0,0 +1,161 @@ +# Agent Dev-Loop Benchmark + +An agent building an app runs mxcli hundreds of times, and waits for each run. +The agent dev-loop benchmark measures that wait directly: a fixed session of +mxcli commands, **one process per command**, timed from outside. There is no +model in the loop, so the same commit gives the same session every time. + +It complements two other things rather than replacing them: + +- the **read and decode budget tests** in the unit suite, which fail a pull + request that makes a command open more unit files than a bound. They are + exact, but they count reads, on a 370-unit project, for the commands someone + already found slow; +- **`mxcli eval run`** with `diag session-report`, which measures a whole agent + session — model calls, tokens, wall time — and therefore varies from run to + run. + +Design and rationale: `docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md`. + +## Running it + +```bash +make bench # time this checkout's binary +make bench BASELINE=/path/to/other/mxcli # A/B against another build +make bench BENCH_FLAGS='-sizes small -reps 3 -out record.json' +``` + +It needs no mxbuild and no Docker: every session starts from a copy of the +committed `testdata/pedapp` project. A default run with a baseline took 14 +minutes on a 4-CPU machine; `-sizes small -reps 1` takes under a minute and is enough to see +whether a change moved anything. + +To check a suspected regression, build the commit before it, keep that binary, +and pass it as `BASELINE`: + +```bash +git stash && make build && cp bin/mxcli /tmp/mxcli-before && git stash pop +make bench BASELINE=/tmp/mxcli-before +``` + +## The session + +The steps run in order against one project, each on the state the previous +ones left. The scripts are generated (`scripts/agentbench/gen`) and identical +on every machine. + +| Step | Command | What it shows | +|---|---|---| +| `open-cold` | `-c "show modules"`, first command on the project | building the unit index | +| `open` | the same command again | the floor under every command: start-up and project open | +| `domain` | `exec`: 25 entities and their associations | | +| `microflows` | `exec`: 50 microflows | | +| `pages` | `exec`: 50 pages | | +| `describe` | `describe` of one entity, microflow and page | | +| `catalog` | `refresh catalog` | | +| `catalog-full` | `refresh catalog full` | | +| `query` | a `select` on the catalog | | +| `check` | `check --references` of a one-microflow script | | +| `single-edit` | `exec` of that script | the most frequent thing an agent does | +| `catalog-after-edit` | `refresh catalog full` after one change | incremental refresh | +| `bulk-alter` | `exec` re-creating all 50 microflows with a changed body | | +| `re-exec` | the same script again | must change nothing | +| `rename` | `rename entity`, with references in microflows and pages | | + +It runs at two project sizes. **small** is PedApp as committed. **large** is +PedApp with a generated app executed into it first (the `seed-*` lines, run +once). The session's own scripts are the same at both sizes, so the *Scaling* +table at the end of the output shows what each command costs more purely +because the project around it is bigger. + +## Reading the output + +With a baseline, each step has these columns: + +- **new / old** — the median wall time of each binary. +- **wall ×** — new divided by old. This is the figure to act on. +- **CPU ×** — the same ratio for user plus system CPU time. CPU time is less + disturbed by a busy machine than wall time, and blind to waiting on the disk. + A step that is slower in both columns is slower; one that is slower in wall + time only is more often noise or I/O. +- **reads** — unit files the command read, as counted by mxcli itself. +- **files changed** — unit files added, rewritten with different bytes, or + removed, measured by comparing the project before and after the command. + +The last two do not depend on the machine at all. A difference between the +arms is exact, and is flagged `counts-changed` whatever the time did. + +### Why a ratio and not a time + +A shared CI runner varies from run to run by more than most regressions, so +tonight's absolute time cannot be compared with last night's. The benchmark +therefore runs the two binaries **interleaved on the same machine**, one step +at a time, each on its own copy of the project, and alternates which goes +first. The ratio is the median over the repetitions of each pair's ratio, so a +slow minute on the machine slows both halves of a pair and cancels. + +Running the same code in both arms gives the noise floor. On the machine this +page was written on (4 CPUs, 5 repetitions), most steps stayed within ±3 % and +the widest were 0.90 and 1.12; the session-weighted total read × 0.93 and +× 0.98. The default flag threshold is × 1.20, and a single night just beyond +it is worth a second run before it is worth an investigation. + +### The session-weighted total + +Each step's median is multiplied by how often a real session runs that kind of +command, and summed. The weights are rounded from the sessions measured in +`PROPOSAL_agent_loop_efficiency.md`: a one-document `exec`, a `check` and a +read each stand for many calls, the bulk steps for themselves. A record +carries `weights_version`, so a change of weights is never read as a change of +speed. + +### Flags + +| Flag | Meaning | +|---|---| +| `slower` / `faster` | wall ratio beyond the threshold | +| `counts-changed` | the two binaries read or changed a different number of files | +| `counts-unstable` | one binary's counts differed between repetitions — a deterministic session should never show this | +| `wrote-on-re-exec` | re-running a settled script changed a file. This is a defect ([Idempotent Writes](idempotent-writes.md)), and the only finding that makes the run exit non-zero besides a failing command | + +A baseline that cannot run a step — an older build that does not know a +statement the session uses — stops there. The steps before keep their ratio; +it and the later ones have none, because the two projects no longer hold the +same thing. + +## The counts on any command + +The read and write counts come from an environment variable that works for +any mxcli command: + +```bash +MXCLI_STATS_FILE=/tmp/stats.jsonl mxcli exec bulk-alter.mdl -p app.mpr +cat /tmp/stats.jsonl +# {"unit_file_reads":73,"writes_offered":49,"writes_written":49} +``` + +One line is appended per project connection closed, so a command that +reconnects, or starts child mxcli processes, appends several. `writes_written` +counts stored units rewritten after reconciliation; a newly created document +is not in it, which is why the benchmark also counts changed files from +outside. + +## The nightly job + +The `bench` job in `.github/workflows/nightly.yml` downloads the previous +nightly release's binary, runs `make bench` against it, writes the table to +the job summary, and commits the JSON record to the `benchmarks` branch as +`results/YYYY/MM/-.json`. That branch holds records only and +is never merged. + +The job does not gate the nightly release. It fails only when a command of +the session fails or a `wrote-on-re-exec` is found. + +## What it does not measure + +- **mxbuild and the running app** — `docker check`, the build, `run --local` + boot and hot apply, `mxcli test`. These are where measured sessions spend + the most wall time and are the next phases of the proposal. +- **Creating the project** — sessions start from a committed fixture, not from + `mxcli new`, which needs mxbuild. +- **Tokens and model calls** — see `mxcli eval run`. diff --git a/docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md b/docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md index 5d4cee41d..b32293e3a 100644 --- a/docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md +++ b/docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md @@ -1,6 +1,6 @@ --- title: Nightly Agent Dev-Loop Performance Benchmark -status: draft +status: partial date: 2026-11-11 author: AI-assisted design related: @@ -10,7 +10,7 @@ related: # Proposal: Nightly Agent Dev-Loop Performance Benchmark -**Status:** Draft +**Status:** Partial — Phase 1 is implemented (see "Phase 1 as built") **Date:** 2026-11-11 ## Problem Statement @@ -259,6 +259,48 @@ the table marked "—". No runtime needed, so the job is short and the A/B signa is clean. This phase exists to get the machinery and its noise floor right on the easy case; it is **not** where most of the measured wall time is. +#### Phase 1 as built + +Implemented as `scripts/agentbench` (`make bench`), the `bench` job in +`nightly.yml`, and `MXCLI_STATS_FILE`. User-facing description: +`docs-site/src/internals/agent-benchmark.md`. Where it differs from the design +above, and what building it showed: + +- **The base project is the committed PedApp fixture, not `mxcli new`.** + Creating a blank project needs `mx create-project`, i.e. mxbuild. Starting + from `testdata/pedapp` keeps Phase 1 free of mxbuild, as intended, and the + job short. The `new` step moves to the phase that brings mxbuild into the + job. PedApp is 370 units, so "small" is a small real app, not an empty one. +- **Both arms start from identical bytes with no caches.** The project copy + leaves `.mxcli/` out, so each arm builds its own unit index and catalog. That + added a step the design did not have, `open-cold`, the first command on a + project. +- **The large fixture is seeded once, by the candidate only.** The seed is + timed and recorded but has no A/B; the session that then runs on it does. +- **The large preset is about 900 units, not a few thousand.** Seeding is the + reason: on the development machine 492 generated documents take about 90 s, + while a 3,240-document seed took 4 min 46 s for its entities, 3 min 48 s for + its microflows, and had not finished its pages after 15 minutes. The cost + per statement grows with the size of the project, which is itself the kind + of finding this benchmark exists for — and is visible at 900 units already: + the same 50-page script takes 2.6 times as long there as on 370. The preset + is two flags (`-large-modules`, `-large-entities`) and should grow as that + cost comes down. +- **Counters (Open Question 1) are an environment variable.** + `MXCLI_STATS_FILE` makes every project connection append one JSON line with + its unit-file reads and its writes offered and landed. It turned out that + the write count covers reconciled rewrites of stored units but not newly + created documents, so the driver also counts changed unit files from outside, + by digest. That count needs nothing from the binary and so exists for the + baseline as well; the idempotence step asserts on it. +- **CPU time and peak memory are recorded alongside wall time.** CPU time is + the steadier of the two ratios on a shared machine. +- **The ratio is the median of per-repetition pair ratios**, not the ratio of + two medians, and the arm that goes first alternates. +- **Not built yet:** the trend chart, and the issue-opening alert. Both need + records to exist first; the alert thresholds are to be set from the first + weeks of data, as planned. + ### Phase 2 — running app and tests Adds `boot`, `hot-apply-flow`, `apply-entity`, `page-check`, `test-cold`, @@ -283,11 +325,10 @@ decision and not part of this proposal. | File | Change | |------|--------| | `scripts/agentbench/` (new, Go) | Driver: runs steps, interleaves A/B, takes medians, collects counters, writes the JSON record and the summary table | -| `scripts/agentbench/gen/` (new) | Fixture generator: parameters → MDL scripts | -| `scripts/agentbench/session/*.mdl` (new) | The scripted session's MDL | +| `scripts/agentbench/gen/` (new) | Generator: parameters → the session's MDL scripts and the large fixture's seed | | `Makefile` | `bench` target | | `.github/workflows/nightly.yml` | `bench` job; commit of the record to the `benchmarks` branch | -| `cmd/mxcli/` | Machine-readable read and write counters for a command (see Open Questions) | +| `mdl/executor/process_stats.go` (new) | `MXCLI_STATS_FILE`: read and write counters per closed connection | | `docs-site/src/internals/` | Page describing the benchmark and showing the trend chart | ## Version Compatibility diff --git a/docs/11-proposals/README.md b/docs/11-proposals/README.md index a56c4fd56..d8fe06102 100644 --- a/docs/11-proposals/README.md +++ b/docs/11-proposals/README.md @@ -30,7 +30,7 @@ for display in this README): ## Active Proposals (123) -### In Progress (partial) (17) +### In Progress (partial) (18) | Proposal | Status | Summary | |----------|--------|---------| @@ -46,6 +46,7 @@ for display in this README): | [mxcli Linter - Implementation Proposal (Revised)](PROPOSAL_mxcli_linter.md) | Partial | Add an extensible linting framework to mxcli that leverages the existing SQLite-based catalog system. | | [mxcli marketplace — Download & Manage Marketplace Modules](PROPOSAL_marketplace_modules.md) | Partial | Unblocking path #1 below has happened: the content API now returns a | | [Navigation Support in MDL](navigation-support.md) | Partial | Every Mendix project has exactly one navigation$NavigationDocument — a project-level singleton containing navigation profiles (Responsive, P | +| [Nightly Agent Dev-Loop Performance Benchmark](PROPOSAL_agent_devloop_benchmark.md) | Partial | An agent building a Mendix app runs mxcli hundreds of times per session — the | | [Podman Support as Docker Alternative](PROPOSAL_podman_support.md) | Partial | Docker Desktop requires a paid subscription for larger organizations. | | [SHOW/DESCRIBE/USE Building Blocks](show-describe-building-blocks.md) | Partial | Document type: Pages$BuildingBlock (NOT Forms$BuildingBlock — the reader now | | [Skill packs — shipping a skill that carries more than prose](PROPOSAL_skill_packs.md) | Partial | mxcli ships 65 skills today and every one of them is a single Markdown file. | @@ -105,7 +106,7 @@ for display in this README): | [VS Code Search — Quick Pick + Workspace Symbol](PROPOSAL_vscode_search.md) | Proposed | Full-text search exists in mxcli (mxcli search) but is only accessible via the terminal. | | [Workflow Improvements: ALTER WORKFLOW + Cross-References](PROPOSAL_workflow_improvements.md) | Proposed | Workflow support in mxcli has full CREATE/DESCRIBE/DROP/SHOW coverage with 13 activity types and BSON round-trip fidelity. | -### Draft (52) +### Draft (51) | Proposal | Status | Summary | |----------|--------|---------| @@ -147,7 +148,6 @@ for display in this README): | [mxcli companion, slice 2 — a long-lived mxcli the mod talks to over a Unix socket](proposal-mxcli-companion-serve.md) | Draft | Slice 2 of the companion umbrella. | | [mxcli marketplace diff — detect local modification and plan a GUID-preserving module upgrade](PROPOSAL_marketplace_module_upgrade.md) | Draft | 2026-08-11 (DESCRIBE coverage measured — §7; GUID = database identity measured — §8) | | [mxcli Playground](mxcli-playground.md) | Draft | A public GitHub repository (mendixlabs/mxcli-playground) containing a ready-to-use Mendix project pre-configured with mxcli, Claude Code ski | -| [Nightly Agent Dev-Loop Performance Benchmark](PROPOSAL_agent_devloop_benchmark.md) | Draft | An agent building a Mendix app runs mxcli hundreds of times per session — the | | [Offline synchronization configuration — the one thing an offline profile needs that MDL cannot say](PROPOSAL_offline_sync_configuration.md) | Draft | mxcli can author an offline navigation profile — PhoneOffline, | | [OpenAPI import — generate response entities and wire operations to them](PROPOSAL_openapi_response_entities.md) | Draft | Issue: ako/mxcli#1234 (mendixlabs/mxcli#280) | | [Owning the modelsdk/gen codegen — why the vendored generator cannot be adopted as-is](PROPOSAL_codegen_ownership.md) | Draft | Constraint set by the maintainer (2026-08-13): the reflection data is an | diff --git a/scripts/agentbench/bench.go b/scripts/agentbench/bench.go new file mode 100644 index 000000000..6e8b26bc0 --- /dev/null +++ b/scripts/agentbench/bench.go @@ -0,0 +1,597 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bufio" + "bytes" + "crypto/sha256" + "encoding/json" + "errors" + "fmt" + "io" + "io/fs" + "os" + "os/exec" + "path/filepath" + "sort" + "strings" + "time" + + "github.com/mendixlabs/mxcli/scripts/agentbench/gen" +) + +// statsFileEnv is executor.StatsFileEnv, repeated so the driver does not link +// the code it measures. +const statsFileEnv = "MXCLI_STATS_FILE" + +// A step is one mxcli process, started as an agent starts it. The steps of a +// session run in order against one project, each on the state the previous +// ones left. +type step struct { + name string + // args is the command line after the binary; {P} is the project's .mpr and + // {S} the directory holding the session's scripts. + args []string + // weight is how many times a real session runs this kind of command for + // each time the benchmark does; see weights below. + weight int + // noWrites marks a step that must change no unit file: the idempotence + // canary. A candidate that writes there has a defect, not a slowdown. + noWrites bool +} + +// weightsVersion identifies the weight column. A record carries it so a change +// of weights is never read as a change of speed. +const weightsVersion = 1 + +// The session. Weights are per-session call counts rounded from the builds +// measured in docs/11-proposals/PROPOSAL_agent_loop_efficiency.md (`exec` 85 to +// 201 calls, `-c` 111 to 120, `check` 63 to 90 per session): a one-document +// exec, a check and a read each stand for many, the bulk steps for themselves. +func sessionSteps() []step { + const read = "describe entity Bench.Item02; describe microflow Bench.ACT_Item02_Save; describe page Bench.Item02_Overview" + return []step{ + // The first command on a project builds its unit index; the second is + // what every later command pays to start and open. + {name: "open-cold", args: []string{"-p", "{P}", "-c", "show modules"}, weight: 1}, + {name: "open", args: []string{"-p", "{P}", "-c", "show modules"}, weight: 60}, + {name: "domain", args: []string{"exec", "{S}/domain.mdl", "-p", "{P}"}, weight: 1}, + {name: "microflows", args: []string{"exec", "{S}/microflows.mdl", "-p", "{P}"}, weight: 1}, + {name: "pages", args: []string{"exec", "{S}/pages.mdl", "-p", "{P}"}, weight: 1}, + {name: "describe", args: []string{"-p", "{P}", "-c", read}, weight: 60}, + {name: "catalog", args: []string{"-p", "{P}", "-c", "refresh catalog"}, weight: 5}, + {name: "catalog-full", args: []string{"-p", "{P}", "-c", "refresh catalog full"}, weight: 1}, + {name: "query", args: []string{"-p", "{P}", "-c", "select count(*) from CATALOG.ENTITIES"}, weight: 20}, + {name: "check", args: []string{"check", "{S}/single-edit.mdl", "-p", "{P}", "--references"}, weight: 90}, + {name: "single-edit", args: []string{"exec", "{S}/single-edit.mdl", "-p", "{P}"}, weight: 150}, + {name: "catalog-after-edit", args: []string{"-p", "{P}", "-c", "refresh catalog full"}, weight: 10}, + {name: "bulk-alter", args: []string{"exec", "{S}/bulk-alter.mdl", "-p", "{P}"}, weight: 1}, + {name: "re-exec", args: []string{"exec", "{S}/bulk-alter.mdl", "-p", "{P}"}, weight: 1, noWrites: true}, + {name: "rename", args: []string{"-p", "{P}", "-c", "rename entity Bench.Item01 to Item01Renamed"}, weight: 1}, + } +} + +// writeSession writes the session's scripts for an app of the given size. +func writeSession(dir string, entities int) error { + p := gen.Params{Prefix: "Bench", Modules: 1, Entities: entities} + for name, src := range map[string]string{ + "domain.mdl": gen.Domain(p), + "microflows.mdl": gen.Microflows(p), + "pages.mdl": gen.Pages(p), + "single-edit.mdl": gen.SingleEdit(p), + "bulk-alter.mdl": gen.BulkAlter(p), + } { + if err := os.WriteFile(filepath.Join(dir, name), []byte(src), 0o644); err != nil { + return err + } + } + return nil +} + +// A sample is one measured process. +type sample struct { + WallMS float64 `json:"wall_ms"` + // CPUMS is user plus system time of the process. It is less sensitive to a + // busy runner than wall time, and blind to waiting on the disk. + CPUMS float64 `json:"cpu_ms"` + RSSMB float64 `json:"rss_mb"` + Counts *counts `json:"counts,omitempty"` + // FilesChanged is how many unit files the command added, rewrote or + // removed, measured from outside by comparing the project before and + // after. Unlike Counts it needs nothing from the binary, so it exists for + // a baseline too, and it sees a created document, which mxcli's own write + // count (reconciled rewrites of stored units) does not. + FilesChanged int `json:"files_changed"` + // StdoutBytes is how much the command printed: what an agent then pays to + // read. + StdoutBytes int `json:"stdout_bytes"` +} + +// counts are a process's machine-independent figures, summed over the +// connections it and its children closed. nil when the binary does not report +// them (a baseline older than the stats file). +type counts struct { + UnitFileReads int64 `json:"unit_file_reads"` + WritesOffered int `json:"writes_offered"` + WritesWritten int `json:"writes_written"` +} + +// runStep runs one process and measures it. A non-zero exit is an error +// carrying the tail of the output: a failed command's time is not a +// measurement. +func runStep(bin string, args []string, project, scripts, scratch string) (sample, error) { + resolved := make([]string, len(args)) + for i, a := range args { + a = strings.ReplaceAll(a, "{P}", project) + resolved[i] = strings.ReplaceAll(a, "{S}", scripts) + } + statsPath := filepath.Join(scratch, "stats.jsonl") + _ = os.Remove(statsPath) + + before := unitDigests(filepath.Dir(project)) + cmd := exec.Command(bin, resolved...) + cmd.Env = append(os.Environ(), statsFileEnv+"="+statsPath, "NO_COLOR=1") + var stdout, stderr bytes.Buffer + cmd.Stdout, cmd.Stderr = &stdout, &stderr + start := time.Now() + err := cmd.Run() + wall := time.Since(start) + if err != nil { + return sample{}, fmt.Errorf("%s %s: %w\n%s", filepath.Base(bin), strings.Join(resolved, " "), err, tail(stdout.String()+stderr.String(), 20)) + } + s := sample{ + WallMS: float64(wall.Microseconds()) / 1000, + CPUMS: float64((cmd.ProcessState.UserTime() + cmd.ProcessState.SystemTime()).Microseconds()) / 1000, + StdoutBytes: stdout.Len(), + Counts: readCounts(statsPath), + FilesChanged: changedFiles(before, unitDigests(filepath.Dir(project))), + } + s.RSSMB = maxRSSMB(cmd.ProcessState) + return s, nil +} + +func readCounts(path string) *counts { + f, err := os.Open(path) + if err != nil { + return nil + } + defer f.Close() + var total counts + sc := bufio.NewScanner(f) + for sc.Scan() { + var c counts + if json.Unmarshal(sc.Bytes(), &c) == nil { + total.UnitFileReads += c.UnitFileReads + total.WritesOffered += c.WritesOffered + total.WritesWritten += c.WritesWritten + } + } + return &total +} + +func tail(s string, lines int) string { + all := strings.Split(strings.TrimRight(s, "\n"), "\n") + if len(all) > lines { + all = all[len(all)-lines:] + } + return strings.Join(all, "\n") +} + +// copyProject copies a project directory, leaving out .mxcli — the unit index +// and catalog a previous command cached there. Both arms must start from the +// same bytes and build their own caches, or the arm whose binary wrote the +// snapshot starts warm. +func copyProject(src, dst string) error { + return filepath.WalkDir(src, func(path string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + rel, err := filepath.Rel(src, path) + if err != nil { + return err + } + if d.IsDir() { + if d.Name() == ".mxcli" || d.Name() == ".git" { + return filepath.SkipDir + } + return os.MkdirAll(filepath.Join(dst, rel), 0o755) + } + if !d.Type().IsRegular() { + return nil + } + in, err := os.Open(path) + if err != nil { + return err + } + defer in.Close() + out, err := os.Create(filepath.Join(dst, rel)) + if err != nil { + return err + } + if _, err := io.Copy(out, in); err != nil { + out.Close() + return err + } + return out.Close() + }) +} + +// findMPR returns the one .mpr in dir. +func findMPR(dir string) (string, error) { + matches, err := filepath.Glob(filepath.Join(dir, "*.mpr")) + if err != nil { + return "", err + } + if len(matches) != 1 { + return "", fmt.Errorf("%s: want exactly one .mpr, found %d", dir, len(matches)) + } + return filepath.Base(matches[0]), nil +} + +// countUnits is how many unit files a v2 project holds: the size a record +// reports for the project a session ran against. +func countUnits(dir string) int { + n := 0 + _ = filepath.WalkDir(filepath.Join(dir, "mprcontents"), func(_ string, d fs.DirEntry, err error) error { + if err == nil && d.Type().IsRegular() { + n++ + } + return nil + }) + return n +} + +// unitDigests maps every unit file of a v2 project to a digest of its bytes. +func unitDigests(dir string) map[string][sha256.Size]byte { + out := map[string][sha256.Size]byte{} + _ = filepath.WalkDir(filepath.Join(dir, "mprcontents"), func(path string, d fs.DirEntry, err error) error { + if err != nil || !d.Type().IsRegular() { + return nil + } + if data, err := os.ReadFile(path); err == nil { + out[path] = sha256.Sum256(data) + } + return nil + }) + return out +} + +// changedFiles counts the files added, rewritten with different bytes, or +// removed between two unitDigests. +func changedFiles(before, after map[string][sha256.Size]byte) int { + n := 0 + for path, digest := range after { + if old, ok := before[path]; !ok || old != digest { + n++ + } + } + for path := range before { + if _, ok := after[path]; !ok { + n++ + } + } + return n +} + +// dist summarises one step's samples in one arm. +type dist struct { + Median float64 `json:"median"` + Min float64 `json:"min"` + Max float64 `json:"max"` + Samples []float64 `json:"samples"` +} + +func newDist(v []float64) dist { + d := dist{Samples: v, Median: median(v)} + if len(v) > 0 { + d.Min, d.Max = v[0], v[0] + for _, x := range v { + d.Min, d.Max = min(d.Min, x), max(d.Max, x) + } + } + return d +} + +func median(v []float64) float64 { + if len(v) == 0 { + return 0 + } + s := append([]float64(nil), v...) + sort.Float64s(s) + if n := len(s); n%2 == 1 { + return s[n/2] + } else { + return (s[n/2-1] + s[n/2]) / 2 + } +} + +// pairedRatio is the median of candidate/baseline over the repetitions both +// arms completed. Each pair ran back to back on the same machine, so a slow +// minute on the runner slows both and cancels; the ratio of two medians taken +// minutes apart would not have that property. +func pairedRatio(candidate, baseline []float64) float64 { + n := min(len(candidate), len(baseline)) + ratios := make([]float64, 0, n) + for i := 0; i < n; i++ { + if baseline[i] > 0 { + ratios = append(ratios, candidate[i]/baseline[i]) + } + } + return median(ratios) +} + +// armResult is one arm's measurements of one step. +type armResult struct { + Wall dist `json:"wall_ms"` + CPU dist `json:"cpu_ms"` + RSS dist `json:"rss_mb"` + // Counts are the first repetition's. CountsStable is false when a later + // repetition disagreed — which a deterministic session should never show. + Counts *counts `json:"counts,omitempty"` + FilesChanged int `json:"files_changed"` + CountsStable bool `json:"counts_stable"` + StdoutBytes int `json:"stdout_bytes"` +} + +func newArmResult(samples []sample) *armResult { + if len(samples) == 0 { + return nil + } + var wall, cpu, rss []float64 + r := &armResult{ + Counts: samples[0].Counts, FilesChanged: samples[0].FilesChanged, + CountsStable: true, StdoutBytes: samples[0].StdoutBytes, + } + for _, s := range samples { + wall, cpu, rss = append(wall, s.WallMS), append(cpu, s.CPUMS), append(rss, s.RSSMB) + if s.FilesChanged != r.FilesChanged || + (s.Counts == nil) != (r.Counts == nil) || (s.Counts != nil && *s.Counts != *r.Counts) { + r.CountsStable = false + } + } + r.Wall, r.CPU, r.RSS = newDist(wall), newDist(cpu), newDist(rss) + return r +} + +// stepResult is one step of one session, both arms. +type stepResult struct { + Name string `json:"name"` + Weight int `json:"weight"` + Candidate *armResult `json:"candidate"` + Baseline *armResult `json:"baseline,omitempty"` + // WallRatio and CPURatio are candidate/baseline; 0 when there is no + // baseline measurement of this step. + WallRatio float64 `json:"wall_ratio,omitempty"` + CPURatio float64 `json:"cpu_ratio,omitempty"` + // Flags are what a reader should look at: "slower", "faster", + // "counts-changed", "counts-unstable", "wrote-on-re-exec". + Flags []string `json:"flags,omitempty"` +} + +// sessionResult is the scripted session at one project size. +type sessionResult struct { + Size string `json:"size"` + // Units is how many unit files the project held when the session started. + Units int `json:"units"` + // Seed is the one-off build of the large fixture, candidate only: it is + // run once, so it is a trend figure, not an A/B. + Seed []seedResult `json:"seed,omitempty"` + Steps []stepResult `json:"steps"` + // BaselineError is why the baseline arm stopped, when it did. Steps from + // that one on have no ratio: the baseline's project no longer holds what + // the candidate's does. + BaselineError string `json:"baseline_error,omitempty"` + // Weighted is the session's steps multiplied by their weights, in + // milliseconds of median wall time: one figure for "what a session costs". + Weighted weighted `json:"weighted"` +} + +type seedResult struct { + Name string `json:"name"` + sample +} + +type weighted struct { + CandidateMS float64 `json:"candidate_ms"` + BaselineMS float64 `json:"baseline_ms,omitempty"` + Ratio float64 `json:"ratio,omitempty"` +} + +// config is one benchmark run. +type config struct { + candidate, baseline string + base string // the starting project's directory + work string + reps int + sessionEntities int + large gen.Params + sizes []string + threshold float64 + log io.Writer +} + +// errInvariant is returned, after the record is complete, when the candidate +// broke something a benchmark run can state exactly rather than estimate. +var errInvariant = errors.New("the candidate broke an invariant") + +// runSession measures the session at one size. snapshot is the project every +// repetition of both arms starts from. +func runSession(cfg config, size, snapshot, scripts string) (sessionResult, error) { + mpr, err := findMPR(snapshot) + if err != nil { + return sessionResult{}, err + } + steps := sessionSteps() + res := sessionResult{Size: size, Units: countUnits(snapshot)} + candidate := make([][]sample, len(steps)) + baseline := make([][]sample, len(steps)) + baselineLive := cfg.baseline != "" + + for rep := 0; rep < cfg.reps; rep++ { + dirs := map[string]string{} + arms := []string{"candidate"} + if baselineLive { + arms = append(arms, "baseline") + // Alternate which arm goes first, so neither always runs on the + // page cache the other warmed. + if rep%2 == 1 { + arms[0], arms[1] = arms[1], arms[0] + } + } + for _, arm := range arms { + dir := filepath.Join(cfg.work, fmt.Sprintf("%s-%s-%d", size, arm, rep)) + if err := copyProject(snapshot, dir); err != nil { + return res, err + } + dirs[arm] = dir + } + for i, st := range steps { + for _, arm := range arms { + if arm == "baseline" && !baselineLive { + continue + } + bin := cfg.candidate + if arm == "baseline" { + bin = cfg.baseline + } + s, err := runStep(bin, st.args, filepath.Join(dirs[arm], mpr), scripts, cfg.work) + switch { + case err != nil && arm == "candidate": + return res, fmt.Errorf("%s, step %s: %w", size, st.name, err) + case err != nil: + // The baseline may not know a statement the session uses. + // That is not this run's failure, but its project has now + // diverged, so no later step of it is comparable. + baselineLive = false + res.BaselineError = fmt.Sprintf("step %s: %v", st.name, err) + fmt.Fprintf(cfg.log, " baseline stopped at %s; later steps have no A/B\n", st.name) + case arm == "candidate": + candidate[i] = append(candidate[i], s) + default: + baseline[i] = append(baseline[i], s) + } + } + } + for _, dir := range dirs { + _ = os.RemoveAll(dir) + } + fmt.Fprintf(cfg.log, " %s: repetition %d of %d done\n", size, rep+1, cfg.reps) + } + + var invariant bool + for i, st := range steps { + sr := stepResult{Name: st.name, Weight: st.weight, Candidate: newArmResult(candidate[i])} + res.Weighted.CandidateMS += sr.Candidate.Wall.Median * float64(st.weight) + if b := newArmResult(baseline[i]); b != nil { + sr.Baseline = b + sr.WallRatio = pairedRatio(sr.Candidate.Wall.Samples, b.Wall.Samples) + sr.CPURatio = pairedRatio(sr.Candidate.CPU.Samples, b.CPU.Samples) + res.Weighted.BaselineMS += b.Wall.Median * float64(st.weight) + switch { + case sr.WallRatio > cfg.threshold: + sr.Flags = append(sr.Flags, "slower") + case sr.WallRatio > 0 && sr.WallRatio < 1/cfg.threshold: + sr.Flags = append(sr.Flags, "faster") + } + if sr.Candidate.FilesChanged != b.FilesChanged || + (sr.Candidate.Counts != nil && b.Counts != nil && *sr.Candidate.Counts != *b.Counts) { + sr.Flags = append(sr.Flags, "counts-changed") + } + } + if !sr.Candidate.CountsStable { + sr.Flags = append(sr.Flags, "counts-unstable") + } + if st.noWrites && sr.Candidate.FilesChanged != 0 { + sr.Flags = append(sr.Flags, "wrote-on-re-exec") + invariant = true + } + res.Steps = append(res.Steps, sr) + } + // A weighted ratio only means something over the same steps in both arms. + if res.BaselineError == "" && res.Weighted.BaselineMS > 0 { + res.Weighted.Ratio = res.Weighted.CandidateMS / res.Weighted.BaselineMS + } else { + res.Weighted.BaselineMS = 0 + } + if invariant { + return res, errInvariant + } + return res, nil +} + +// seedLarge builds the large fixture: a copy of the base project with the +// generated app executed into it, by the candidate. Its three scripts are +// timed once each. +func seedLarge(cfg config, scripts string) (string, []seedResult, error) { + dir := filepath.Join(cfg.work, "large-snapshot") + if err := copyProject(cfg.base, dir); err != nil { + return "", nil, err + } + mpr, err := findMPR(dir) + if err != nil { + return "", nil, err + } + var out []seedResult + for _, part := range []struct{ name, src string }{ + {"seed-domain", gen.Domain(cfg.large)}, + {"seed-microflows", gen.Microflows(cfg.large)}, + {"seed-pages", gen.Pages(cfg.large)}, + } { + script := filepath.Join(scripts, part.name+".mdl") + if err := os.WriteFile(script, []byte(part.src), 0o644); err != nil { + return "", nil, err + } + s, err := runStep(cfg.candidate, []string{"exec", script, "-p", "{P}"}, filepath.Join(dir, mpr), scripts, cfg.work) + if err != nil { + return "", nil, fmt.Errorf("%s: %w", part.name, err) + } + fmt.Fprintf(cfg.log, " %s: %.1fs\n", part.name, s.WallMS/1000) + out = append(out, seedResult{Name: part.name, sample: s}) + } + return dir, out, nil +} + +// run is the whole benchmark: every requested size, in order. +func run(cfg config) ([]sessionResult, error) { + scripts := filepath.Join(cfg.work, "scripts") + if err := os.MkdirAll(scripts, 0o755); err != nil { + return nil, err + } + if err := writeSession(scripts, cfg.sessionEntities); err != nil { + return nil, err + } + var results []sessionResult + var invariant bool + for _, size := range cfg.sizes { + fmt.Fprintf(cfg.log, "%s session\n", size) + var snapshot string + var seed []seedResult + switch size { + case "small": + snapshot = filepath.Join(cfg.work, "small-snapshot") + if err := copyProject(cfg.base, snapshot); err != nil { + return results, err + } + case "large": + var err error + if snapshot, seed, err = seedLarge(cfg, scripts); err != nil { + return results, err + } + default: + return results, fmt.Errorf("unknown size %q (want small or large)", size) + } + res, err := runSession(cfg, size, snapshot, scripts) + res.Seed = seed + if errors.Is(err, errInvariant) { + invariant = true + } else if err != nil { + return results, err + } + results = append(results, res) + } + if invariant { + return results, errInvariant + } + return results, nil +} diff --git a/scripts/agentbench/bench_test.go b/scripts/agentbench/bench_test.go new file mode 100644 index 000000000..f92f059c2 --- /dev/null +++ b/scripts/agentbench/bench_test.go @@ -0,0 +1,241 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build unix + +package main + +import ( + "errors" + "fmt" + "io" + "os" + "path/filepath" + "slices" + "strings" + "testing" +) + +// fakeMxcli writes a shell script that stands in for mxcli, so the driver's +// own behaviour — interleaving, pairing, flagging, giving up on a baseline — +// is tested without a project or a build. It takes about 30 ms, so that the +// scheduler's jitter is small beside it, exits 0 and reports fixed counts, +// except: +// +// slowOn a command line containing this sleeps first +// failOn a command line containing this exits non-zero +// writeOn a command line containing this rewrites a unit file with new bytes +func fakeMxcli(t *testing.T, name, slowOn, failOn, writeOn string) string { + t.Helper() + script := fmt.Sprintf(`#!/bin/sh +if [ "$1" = version ]; then echo "mxcli version %[1]s"; exit 0; fi +match() { [ -n "$1" ] && case "$ARGS" in *"$1"*) return 0;; esac; return 1; } +ARGS="$*" +project="" +while [ $# -gt 0 ]; do if [ "$1" = -p ]; then project="$2"; fi; shift; done +sleep 0.03 +if match %[2]q; then sleep 0.25; fi +if match %[3]q; then echo "unknown statement" >&2; exit 3; fi +if match %[4]q; then date +%%s%%N >> "$(dirname "$project")/mprcontents/unit.mxunit"; fi +if [ -n "$MXCLI_STATS_FILE" ]; then + echo '{"unit_file_reads":7,"writes_offered":2,"writes_written":1}' >> "$MXCLI_STATS_FILE" +fi +echo ok +`, name, slowOn, failOn, writeOn) + path := filepath.Join(t.TempDir(), name) + if err := os.WriteFile(path, []byte(script), 0o755); err != nil { + t.Fatal(err) + } + return path +} + +// fakeProject is the smallest thing the driver accepts as a base project. +func fakeProject(t *testing.T) string { + t.Helper() + dir := t.TempDir() + if err := os.MkdirAll(filepath.Join(dir, "mprcontents"), 0o755); err != nil { + t.Fatal(err) + } + for name, body := range map[string]string{"App.mpr": "mpr", "mprcontents/unit.mxunit": "unit"} { + if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o644); err != nil { + t.Fatal(err) + } + } + return dir +} + +func testConfig(t *testing.T, candidate, baseline string) config { + t.Helper() + return config{ + candidate: candidate, baseline: baseline, base: fakeProject(t), work: t.TempDir(), + reps: 3, sessionEntities: 1, sizes: []string{"small"}, threshold: 1.5, log: io.Discard, + } +} + +func stepNamed(t *testing.T, s sessionResult, name string) stepResult { + t.Helper() + for _, st := range s.Steps { + if st.Name == name { + return st + } + } + t.Fatalf("no step %q in the session", name) + return stepResult{} +} + +// The benchmark's reason to exist: a candidate that is slower in one step is +// reported slower in that step and in no other. The control is the baseline, +// which is the same script without the delay — so a driver that flagged +// nothing, or everything, fails here. +func TestRun_FlagsExactlyTheStepThatGotSlower(t *testing.T) { + t.Parallel() + candidate := fakeMxcli(t, "candidate", "rename entity", "", "") + baseline := fakeMxcli(t, "baseline", "", "", "") + sessions, err := run(testConfig(t, candidate, baseline)) + if err != nil { + t.Fatal(err) + } + var slower []string + for _, st := range sessions[0].Steps { + if slices.Contains(st.Flags, "slower") { + slower = append(slower, st.Name) + } + } + if want := []string{"rename"}; !slices.Equal(slower, want) { + t.Fatalf("flagged slower: %v, want %v", slower, want) + } + if r := stepNamed(t, sessions[0], "rename").WallRatio; r < 3 { + t.Errorf("rename ratio %.2f, want a 250 ms delay on a 30 ms command to read well above 3", r) + } +} + +// A baseline that cannot run a step — an older build that does not know a +// statement the session uses — is not this run's failure. The steps before it +// keep their A/B; it and the ones after have none, because the baseline's +// project no longer holds what the candidate's does. +func TestRun_BaselineThatFailsLosesItsLaterSteps(t *testing.T) { + t.Parallel() + candidate := fakeMxcli(t, "candidate", "", "", "") + baseline := fakeMxcli(t, "baseline", "", "refresh catalog full", "") + sessions, err := run(testConfig(t, candidate, baseline)) + if err != nil { + t.Fatalf("a failing baseline failed the run: %v", err) + } + s := sessions[0] + if !strings.Contains(s.BaselineError, "catalog-full") { + t.Errorf("baseline error %q does not name the step it stopped at", s.BaselineError) + } + if stepNamed(t, s, "pages").Baseline == nil { + t.Error("pages ran before the failure and lost its baseline") + } + for _, name := range []string{"catalog-full", "single-edit", "rename"} { + if st := stepNamed(t, s, name); st.Baseline != nil || st.WallRatio != 0 { + t.Errorf("%s has a baseline measurement after the baseline stopped: %+v", name, st.Baseline) + } + if st := stepNamed(t, s, name); len(st.Candidate.Wall.Samples) != 3 { + t.Errorf("%s: candidate has %d samples, want all 3 repetitions", name, len(st.Candidate.Wall.Samples)) + } + } + if s.Weighted.Ratio != 0 { + t.Errorf("weighted ratio %.2f over steps only one arm ran", s.Weighted.Ratio) + } +} + +// A candidate that fails a step fails the run: a failed command returns fast, +// and recording that as a time would report a regression as a speed-up. +func TestRun_CandidateThatFailsFailsTheRun(t *testing.T) { + t.Parallel() + candidate := fakeMxcli(t, "candidate", "", "pages.mdl", "") + _, err := run(testConfig(t, candidate, "")) + if err == nil || !strings.Contains(err.Error(), "step pages") || !strings.Contains(err.Error(), "unknown statement") { + t.Fatalf("got %v, want an error naming the step and carrying the command's output", err) + } +} + +// Re-running a settled script must change no unit file. A candidate that does +// is reported with its record intact, and the run ends in error. +func TestRun_WritingOnReExecIsAnInvariantFailure(t *testing.T) { + t.Parallel() + writes := fakeMxcli(t, "candidate", "", "", "bulk-alter.mdl") + sessions, err := run(testConfig(t, writes, "")) + if !errors.Is(err, errInvariant) { + t.Fatalf("got %v, want errInvariant", err) + } + if len(sessions) != 1 { + t.Fatalf("the record was dropped with the error: %d sessions", len(sessions)) + } + if st := stepNamed(t, sessions[0], "re-exec"); !slices.Contains(st.Flags, "wrote-on-re-exec") || st.Candidate.FilesChanged != 1 { + t.Errorf("re-exec: flags %v, files changed %d", st.Flags, st.Candidate.FilesChanged) + } + + // Control: the same session with a candidate that leaves the file alone. + quiet := fakeMxcli(t, "candidate", "", "", "") + if _, err := run(testConfig(t, quiet, "")); err != nil { + t.Fatalf("a candidate that writes nothing on re-exec: %v", err) + } +} + +// Counts come from the stats file and are summed over its lines; a step whose +// count differs between the arms is flagged whatever its time did. +func TestRun_CountsAreRecordedAndComparedAcrossArms(t *testing.T) { + t.Parallel() + candidate := fakeMxcli(t, "candidate", "", "", "pages.mdl") + baseline := fakeMxcli(t, "baseline", "", "", "") + sessions, err := run(testConfig(t, candidate, baseline)) + if err != nil { + t.Fatal(err) + } + open := stepNamed(t, sessions[0], "open") + if c := open.Candidate.Counts; c == nil || c.UnitFileReads != 7 || c.WritesWritten != 1 { + t.Errorf("open counts %+v, want the stats file's 7 reads and 1 write", c) + } + if slices.Contains(open.Flags, "counts-changed") { + t.Errorf("open flagged counts-changed with equal counts in both arms") + } + if pages := stepNamed(t, sessions[0], "pages"); !slices.Contains(pages.Flags, "counts-changed") { + t.Errorf("pages changed a file in one arm only and is not flagged: %v", pages.Flags) + } +} + +func TestPairedRatioAndMedian(t *testing.T) { + if got := median([]float64{5, 1, 3}); got != 3 { + t.Errorf("median of three: %v", got) + } + if got := median([]float64{4, 1, 3, 2}); got != 2.5 { + t.Errorf("median of four: %v", got) + } + if got := median(nil); got != 0 { + t.Errorf("median of none: %v", got) + } + // Repetition 2 ran while the machine was busy: both arms doubled. The + // paired ratio is unmoved; the ratio of the medians would be too, here, + // but only the pairing guarantees it. + if got := pairedRatio([]float64{110, 220, 110}, []float64{100, 200, 100}); got < 1.099 || got > 1.101 { + t.Errorf("paired ratio: %v, want 1.1", got) + } + // The baseline stopped after one repetition: only that pair counts. + if got := pairedRatio([]float64{150, 100, 100}, []float64{100}); got != 1.5 { + t.Errorf("paired ratio over unequal lengths: %v, want 1.5", got) + } +} + +// A copy must not carry the caches a previous command left in the project: +// the arm whose binary wrote them would start warm. +func TestCopyProject_LeavesTheCachesBehind(t *testing.T) { + src := fakeProject(t) + if err := os.MkdirAll(filepath.Join(src, ".mxcli"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(src, ".mxcli", "unit-index-2.idx"), []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + dst := filepath.Join(t.TempDir(), "copy") + if err := copyProject(src, dst); err != nil { + t.Fatal(err) + } + if _, err := os.Stat(filepath.Join(dst, ".mxcli")); !os.IsNotExist(err) { + t.Errorf(".mxcli was copied (%v)", err) + } + if _, err := os.Stat(filepath.Join(dst, "mprcontents", "unit.mxunit")); err != nil { + t.Errorf("the unit file was not copied: %v", err) + } +} diff --git a/scripts/agentbench/gen/gen.go b/scripts/agentbench/gen/gen.go new file mode 100644 index 000000000..39489ce16 --- /dev/null +++ b/scripts/agentbench/gen/gen.go @@ -0,0 +1,258 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Package gen emits the MDL the agent dev-loop benchmark executes: the +// authoring scripts of its scripted session and the seed of its large fixture. +// +// The output is a pure function of Params — no clock, no randomness, no map +// iteration — so the same parameters produce the same bytes on every machine. +// That is what makes the benchmark's machine-independent counters (unit files +// read, units written) comparable from one night to the next. See +// docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md. +// +// Every statement is `create or modify`, so a script can be executed twice: +// the second run is the benchmark's idempotence step, which must write nothing. +package gen + +import ( + "fmt" + "strings" +) + +// Params sizes one generated app. +type Params struct { + // Prefix names the modules: Prefix01, Prefix02, … (or just Prefix when + // Modules is 1). + Prefix string + // Modules is how many modules to generate. + Modules int + // Entities is how many entities each module has. Every entity brings two + // microflows (validate, save), one overview page and one edit page. + Entities int + // Revision changes the body of every microflow without changing its name + // or signature. Revision 0 and revision 1 of the same Params are the + // "before" and "after" of a bulk change. + Revision int +} + +// Documents is how many project documents Params generates, the size figure a +// record reports: a domain model per module, and two microflows and two pages +// per entity. +func (p Params) Documents() int { + return p.Modules * (1 + 4*p.Entities) +} + +// ModuleName is the name of module i (0-based). +func (p Params) ModuleName(i int) string { + if p.Modules == 1 { + return p.Prefix + } + return fmt.Sprintf("%s%02d", p.Prefix, i+1) +} + +// EntityName is the name of entity j (0-based) in any module. +func EntityName(j int) string { return fmt.Sprintf("Item%02d", j+1) } + +const header = "mdl 1;\n" + +// Domain is the script that creates every module, its entities and their +// associations: each entity refers to the one before it, and the first entity +// of each module to the first entity of the previous module, so references +// cross module boundaries at a fixed density. +func Domain(p Params) string { + var sb strings.Builder + sb.WriteString(header) + for i := 0; i < p.Modules; i++ { + mod := p.ModuleName(i) + fmt.Fprintf(&sb, "\ncreate module if not exists %s;\n", mod) + for j := 0; j < p.Entities; j++ { + ent := EntityName(j) + fmt.Fprintf(&sb, ` +/** %s %d of module %s. */ +@position(%d, %d) +create or modify persistent entity %s.%s ( + Name: string(200) not null, + Description: string(unlimited), + Amount: integer default 0, + Price: decimal default 0, + IsActive: boolean default true, + DueDate: datetime +); +`, "Benchmark entity", j+1, mod, 100+(j%8)*260, 100+(j/8)*220, mod, ent) + } + for j := 1; j < p.Entities; j++ { + fmt.Fprintf(&sb, "create or modify association %s.%s_%s from %s.%s to %s.%s;\n", + mod, EntityName(j), EntityName(j-1), mod, EntityName(j), mod, EntityName(j-1)) + } + if i > 0 { + prev := p.ModuleName(i - 1) + fmt.Fprintf(&sb, "create or modify association %s.%s_%s%s from %s.%s to %s.%s;\n", + mod, EntityName(0), prev, EntityName(0), mod, EntityName(0), prev, EntityName(0)) + } + } + return sb.String() +} + +// Microflows is the script that creates a validation microflow and a save +// microflow for every entity. The save calls the validation, and the save of +// each entity but the first also calls the save of the entity before it, so +// renaming one entity or re-creating one microflow has callers to resolve. +func Microflows(p Params) string { + var sb strings.Builder + sb.WriteString(header) + for i := 0; i < p.Modules; i++ { + for j := 0; j < p.Entities; j++ { + sb.WriteString(Validate(p, i, j)) + sb.WriteString(Save(p, i, j)) + } + } + return sb.String() +} + +// Validate is the `create or modify` of one entity's validation microflow. +func Validate(p Params, module, entity int) string { + mod, ent := p.ModuleName(module), EntityName(entity) + return fmt.Sprintf(` +/** + * Validates a %[2]s before it is saved (revision %[3]d). + */ +create or modify microflow %[1]s.VAL_%[2]s_Save ( + $Item: %[1]s.%[2]s +) +returns boolean +begin + declare $IsValid boolean = true; + + if $Item/Name = empty then + validation feedback $Item/Name message 'Name is required'; + set $IsValid = false; + end if; + + if $Item/Amount < %[3]d then + validation feedback $Item/Amount message 'Amount is too low'; + set $IsValid = false; + end if; + + if $Item/Price < 0 then + validation feedback $Item/Price message 'Price cannot be negative'; + set $IsValid = false; + end if; + + return $IsValid; +end; +`, mod, ent, p.Revision) +} + +// Save is the `create or modify` of one entity's save microflow. +func Save(p Params, module, entity int) string { + mod, ent := p.ModuleName(module), EntityName(entity) + return fmt.Sprintf(` +/** + * Validates and commits a %[2]s (revision %[3]d). + */ +create or modify microflow %[1]s.ACT_%[2]s_Save ( + $Item: %[1]s.%[2]s +) +returns boolean +begin + $IsValid = call microflow %[1]s.VAL_%[2]s_Save(Item = $Item); + + if $IsValid then + change $Item (Description = 'Saved at revision %[3]d'); + commit $Item; + end if; + + return $IsValid; +end; +`, mod, ent, p.Revision) +} + +// Pages is the script that creates an edit page and an overview page for every +// entity. The overview's grid opens the edit page, and the edit page's save +// button calls the entity's save microflow. +func Pages(p Params) string { + var sb strings.Builder + sb.WriteString(header) + for i := 0; i < p.Modules; i++ { + mod := p.ModuleName(i) + for j := 0; j < p.Entities; j++ { + ent := EntityName(j) + fmt.Fprintf(&sb, ` +create or modify page %[1]s.%[2]s_NewEdit folder 'Pages' +( + params: ( $Item: %[1]s.%[2]s ), + title: 'Edit %[2]s', + layout: Atlas_Core.PopupLayout +) +{ + layoutgrid mainGrid { + row { + column (desktopwidth: autofill) { + dataview dataView1 (datasource: $Item) { + textbox txtName (label: 'Name', attribute: Name) + textarea txtDescription (label: 'Description', attribute: Description) + textbox txtAmount (label: 'Amount', attribute: Amount) + textbox txtPrice (label: 'Price', attribute: Price) + datepicker dpDueDate (label: 'Due date', attribute: DueDate) + + footer { + actionbutton btnSave (caption: 'Save', action: call microflow %[1]s.ACT_%[2]s_Save(Item = $Item), buttonstyle: success) + actionbutton btnCancel (caption: 'Cancel', action: cancel changes) + } + } + } + } + } +}; + +create or modify page %[1]s.%[2]s_Overview folder 'Pages' +( + title: '%[2]s overview', + layout: Atlas_Core.Atlas_Default +) +{ + layoutgrid mainGrid { + row { + column (desktopwidth: 12) { + dynamictext heading (content: '%[2]s', rendermode: H2) + datagrid grid1 ( + datasource: database %[1]s.%[2]s + ) { + column (attribute: Name, caption: 'Name') { + textfilter filterName + } + column (attribute: Amount, caption: 'Amount') { + numberfilter filterAmount + } + column (attribute: DueDate, caption: 'Due date') { + datefilter filterDueDate + } + column (caption: 'Actions') { + actionbutton btnEdit (caption: 'Edit', action: show page %[1]s.%[2]s_NewEdit(Item = $currentObject)) + actionbutton btnDelete (caption: 'Delete', action: delete, buttonstyle: danger) + } + } + } + } + } +}; +`, mod, ent) + } + } + return sb.String() +} + +// SingleEdit is the smallest change an agent makes: one microflow re-created +// with a changed body. +func SingleEdit(p Params) string { + q := p + q.Revision = p.Revision + 1 + return header + Save(q, 0, 0) +} + +// BulkAlter re-creates every microflow at the next revision: one statement per +// document, each changing it. +func BulkAlter(p Params) string { + q := p + q.Revision = p.Revision + 1 + return Microflows(q) +} diff --git a/scripts/agentbench/gen/gen_test.go b/scripts/agentbench/gen/gen_test.go new file mode 100644 index 000000000..5a5b950be --- /dev/null +++ b/scripts/agentbench/gen/gen_test.go @@ -0,0 +1,126 @@ +// SPDX-License-Identifier: Apache-2.0 + +package gen + +import ( + "bytes" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" + modelsdkbackend "github.com/mendixlabs/mxcli/mdl/backend/modelsdk" + "github.com/mendixlabs/mxcli/mdl/executor" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +func TestDocumentsCountsWhatIsGenerated(t *testing.T) { + p := Params{Prefix: "Seed", Modules: 3, Entities: 4} + if got, want := p.Documents(), 3*(1+4*4); got != want { + t.Fatalf("Documents() = %d, want %d", got, want) + } + if got := strings.Count(Microflows(p), "create or modify microflow "); got != 3*4*2 { + t.Errorf("%d microflows generated, want two per entity", got) + } + if got := strings.Count(Pages(p), "create or modify page "); got != 3*4*2 { + t.Errorf("%d pages generated, want two per entity", got) + } + if got := strings.Count(Domain(p), " persistent entity "); got != 3*4 { + t.Errorf("%d entities generated", got) + } + // One module is named by the bare prefix: the session's scripts and its + // read steps both say `Bench.Item01`. + if got := (Params{Prefix: "Bench", Modules: 1}).ModuleName(0); got != "Bench" { + t.Errorf("single module named %q", got) + } +} + +// A bulk change must change every microflow and nothing about its identity: +// same names, different bodies. +func TestBulkAlterChangesEveryMicroflowBody(t *testing.T) { + p := Params{Prefix: "Bench", Modules: 1, Entities: 3} + before, after := Microflows(p), BulkAlter(p) + if before == after { + t.Fatal("the bulk change is the script it changes") + } + names := func(src string) []string { + var out []string + for _, line := range strings.Split(src, "\n") { + if strings.HasPrefix(line, "create or modify microflow ") { + out = append(out, line) + } + } + return out + } + b, a := names(before), names(after) + if len(b) != 6 || strings.Join(b, "\n") != strings.Join(a, "\n") { + t.Errorf("the bulk change renamed or dropped a microflow:\n%v\n%v", b, a) + } +} + +// The benchmark must not time a failing script. This runs the session's +// scripts, and a two-module seed, through the real executor on the PedApp +// fixture — the project the nightly job starts from — and then checks the +// property the re-exec step asserts: a settled script lands no write. +func TestGeneratedScriptsExecuteAndSettleOnPedApp(t *testing.T) { + src := filepath.Join("..", "..", "..", "testdata", "pedapp") + if _, err := os.Stat(filepath.Join(src, "PedApp.mpr")); err != nil { + t.Fatalf("the PedApp fixture is the benchmark's base project and is missing: %v", err) + } + dir := t.TempDir() + if err := os.CopyFS(dir, os.DirFS(src)); err != nil { + t.Fatal(err) + } + _ = os.RemoveAll(filepath.Join(dir, ".mxcli")) + + out := &bytes.Buffer{} + exec := executor.New(out) + exec.SetBackendFactory(func() backend.FullBackend { return modelsdkbackend.New() }) + if err := exec.Execute(&ast.ConnectStmt{Path: filepath.Join(dir, "PedApp.mpr")}); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(func() { _ = exec.Close() }) + + run := func(name, script string) { + t.Helper() + prog, errs := visitor.Build(script) + if len(errs) > 0 { + t.Fatalf("%s does not parse: %v", name, errs[0]) + } + if err := exec.ExecuteProgram(prog); err != nil { + t.Fatalf("%s: %v\n%s", name, err, out.String()) + } + } + written := func() int { + t.Helper() + r, ok := exec.Backend().(backend.WriteStatsReporter) + if !ok { + t.Fatalf("backend %T does not report its writes", exec.Backend()) + } + return r.WriteStats().Written + } + + session := Params{Prefix: "Bench", Modules: 1, Entities: 2} + run("domain", Domain(session)) + run("microflows", Microflows(session)) + run("pages", Pages(session)) + run("single-edit", SingleEdit(session)) + + before := written() + run("bulk-alter", BulkAlter(session)) + if written() == before { + t.Fatal("control: the bulk change landed no write, so the settled re-run below would prove nothing") + } + before = written() + run("re-exec", BulkAlter(session)) + if got := written() - before; got != 0 { + t.Errorf("re-running the settled bulk change landed %d writes, want 0\n%s", got, out.String()) + } + + seed := Params{Prefix: "Seed", Modules: 2, Entities: 2} + run("seed-domain", Domain(seed)) + run("seed-microflows", Microflows(seed)) + run("seed-pages", Pages(seed)) +} diff --git a/scripts/agentbench/main.go b/scripts/agentbench/main.go new file mode 100644 index 000000000..d9b5f4458 --- /dev/null +++ b/scripts/agentbench/main.go @@ -0,0 +1,101 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Command agentbench measures how long mxcli takes to do what an agent asks of +// it while building an app: a fixed, scripted session of mxcli commands, one +// process per command, with no model in the loop. +// +// Given a second binary (-baseline) it runs both interleaved on the same +// machine and reports the ratio, which is what makes a figure from a shared CI +// runner comparable from one night to the next. It also records counts that do +// not depend on the machine at all: unit files read and unit writes landed. +// +// make bench # the binary in bin/ +// make bench BASELINE=/path/to/another/mxcli # A/B against another build +// +// Design and reading guide: docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md +// and docs-site/src/internals/agent-benchmark.md. +package main + +import ( + "errors" + "flag" + "fmt" + "os" + "strings" +) + +func main() { + var cfg config + var sizes, out, summary string + flag.StringVar(&cfg.candidate, "mxcli", "bin/mxcli", "the mxcli to measure") + flag.StringVar(&cfg.baseline, "baseline", "", "a second mxcli to compare against, interleaved on this machine (optional)") + flag.StringVar(&cfg.base, "base", "testdata/pedapp", "directory of the project every session starts from (MPR v2, one .mpr)") + flag.StringVar(&cfg.work, "work", "", "working directory (default: a temporary one, removed afterwards)") + flag.IntVar(&cfg.reps, "reps", 5, "repetitions of the session per arm; the median is reported") + flag.IntVar(&cfg.sessionEntities, "session-entities", 25, "entities the session authors (each brings 2 microflows and 2 pages)") + flag.IntVar(&cfg.large.Modules, "large-modules", 12, "modules seeded into the large project") + flag.IntVar(&cfg.large.Entities, "large-entities", 10, "entities per seeded module (each brings 2 microflows and 2 pages)") + flag.StringVar(&sizes, "sizes", "small,large", "project sizes to run the session at") + flag.Float64Var(&cfg.threshold, "threshold", 1.20, "wall-time ratio above which a step is flagged slower (and below its inverse, faster)") + flag.StringVar(&out, "out", "", "write the JSON record here") + flag.StringVar(&summary, "summary", "", "append the Markdown summary here (e.g. $GITHUB_STEP_SUMMARY); it is always printed") + flag.Parse() + + cfg.large.Prefix = "Seed" + cfg.sizes = strings.Split(sizes, ",") + cfg.log = os.Stderr + if cfg.reps < 1 { + fatal(errors.New("-reps must be at least 1")) + } + if cfg.work == "" { + dir, err := os.MkdirTemp("", "agentbench-") + if err != nil { + fatal(err) + } + defer os.RemoveAll(dir) + cfg.work = dir + } else if err := os.MkdirAll(cfg.work, 0o755); err != nil { + fatal(err) + } + // A relative binary path would be resolved against each child's directory. + var err error + if cfg.candidate, err = absBinary(cfg.candidate); err != nil { + fatal(err) + } + if cfg.baseline != "" { + if cfg.baseline, err = absBinary(cfg.baseline); err != nil { + fatal(err) + } + } + + sessions, runErr := run(cfg) + if runErr != nil && !errors.Is(runErr, errInvariant) { + fatal(runErr) + } + rec := newRecord(cfg, sessions) + md := rec.markdown() + fmt.Print(md) + if summary != "" { + f, err := os.OpenFile(summary, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644) + if err != nil { + fatal(err) + } + _, _ = f.WriteString(md) + f.Close() + } + if out != "" { + if err := rec.write(out); err != nil { + fatal(err) + } + } + if runErr != nil { + // The record is written first: the night an invariant breaks is the + // night its numbers are wanted. + fatal(fmt.Errorf("%w: see the flagged steps above", runErr)) + } +} + +func fatal(err error) { + fmt.Fprintln(os.Stderr, "agentbench:", err) + os.Exit(1) +} diff --git a/scripts/agentbench/record.go b/scripts/agentbench/record.go new file mode 100644 index 000000000..54c747a1e --- /dev/null +++ b/scripts/agentbench/record.go @@ -0,0 +1,235 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "fmt" + "os" + "os/exec" + "path/filepath" + "runtime" + "strings" + "time" +) + +// recordSchema is bumped when a field changes meaning, so a chart reading a +// year of records can tell which ones it understands. +const recordSchema = 1 + +// record is one benchmark run: what the nightly job commits to the +// `benchmarks` branch, one file per run. +type record struct { + Schema int `json:"schema"` + Date string `json:"date"` + Commit string `json:"commit,omitempty"` + Candidate string `json:"candidate"` + Baseline string `json:"baseline,omitempty"` + Runner runner `json:"runner"` + Params params `json:"params"` + WeightsVersion int `json:"weights_version"` + Sessions []sessionResult `json:"sessions"` +} + +type runner struct { + OS string `json:"os"` + Arch string `json:"arch"` + CPUs int `json:"cpus"` + CPU string `json:"cpu,omitempty"` +} + +type params struct { + Base string `json:"base"` + Reps int `json:"reps"` + SessionEntities int `json:"session_entities"` + LargeModules int `json:"large_modules"` + LargeEntities int `json:"large_entities"` + LargeDocuments int `json:"large_documents"` + Threshold float64 `json:"threshold"` +} + +func newRecord(cfg config, sessions []sessionResult) record { + rec := record{ + Schema: recordSchema, + Date: time.Now().UTC().Format(time.RFC3339), + Commit: os.Getenv("GITHUB_SHA"), + Candidate: binaryVersion(cfg.candidate), + Runner: runner{OS: runtime.GOOS, Arch: runtime.GOARCH, CPUs: runtime.NumCPU(), CPU: cpuModel()}, + Params: params{ + Base: filepath.Base(cfg.base), Reps: cfg.reps, SessionEntities: cfg.sessionEntities, + LargeModules: cfg.large.Modules, LargeEntities: cfg.large.Entities, + LargeDocuments: cfg.large.Documents(), Threshold: cfg.threshold, + }, + WeightsVersion: weightsVersion, + Sessions: sessions, + } + if cfg.baseline != "" { + rec.Baseline = binaryVersion(cfg.baseline) + } + return rec +} + +func (r record) write(path string) error { + data, err := json.MarshalIndent(r, "", " ") + if err != nil { + return err + } + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return err + } + return os.WriteFile(path, append(data, '\n'), 0o644) +} + +// absBinary resolves a binary path and confirms there is something to run. +func absBinary(path string) (string, error) { + abs, err := filepath.Abs(path) + if err != nil { + return "", err + } + if _, err := os.Stat(abs); err != nil { + return "", fmt.Errorf("no mxcli at %s: %w", path, err) + } + return abs, nil +} + +// binaryVersion is what `mxcli version` prints, the line that names the build. +func binaryVersion(bin string) string { + out, err := exec.Command(bin, "version").Output() + if err != nil { + return "unknown" + } + for _, line := range strings.Split(string(out), "\n") { + if strings.Contains(line, "version") { + return strings.TrimSpace(line) + } + } + return strings.TrimSpace(string(out)) +} + +// cpuModel is the processor's name where the platform exposes it cheaply. It +// is in the record so a step change that follows a runner-fleet change can be +// told from one that follows a commit. +func cpuModel() string { + data, err := os.ReadFile("/proc/cpuinfo") + if err != nil { + return "" + } + for _, line := range strings.Split(string(data), "\n") { + if name, ok := strings.CutPrefix(line, "model name"); ok { + return strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(name), ":")) + } + } + return "" +} + +// markdown is the run as a table per session: what a person reads in the job +// summary or the terminal. +func (r record) markdown() string { + var sb strings.Builder + sb.WriteString("## Agent dev-loop benchmark\n\n") + fmt.Fprintf(&sb, "- candidate: `%s`\n", r.Candidate) + if r.Baseline != "" { + fmt.Fprintf(&sb, "- baseline: `%s`\n", r.Baseline) + } + fmt.Fprintf(&sb, "- median of %d, %d CPUs", r.Params.Reps, r.Runner.CPUs) + if r.Runner.CPU != "" { + fmt.Fprintf(&sb, " (%s)", r.Runner.CPU) + } + sb.WriteString("\n") + ab := r.Baseline != "" + + for _, s := range r.Sessions { + fmt.Fprintf(&sb, "\n### %s project — %d units\n\n", s.Size, s.Units) + for _, seed := range s.Seed { + fmt.Fprintf(&sb, "- %s: %s (once, candidate only)\n", seed.Name, ms(seed.WallMS)) + } + if len(s.Seed) > 0 { + sb.WriteString("\n") + } + if ab { + sb.WriteString("| step | new | old | wall × | CPU × | reads new / old | files changed new / old | |\n") + sb.WriteString("|---|---:|---:|---:|---:|---:|---:|---|\n") + } else { + sb.WriteString("| step | wall | spread | CPU | RSS | reads | files changed | |\n") + sb.WriteString("|---|---:|---:|---:|---:|---:|---:|---|\n") + } + for _, st := range s.Steps { + c := st.Candidate + flags := strings.Join(st.Flags, ", ") + switch { + case ab && st.Baseline != nil: + fmt.Fprintf(&sb, "| %s | %s | %s | %.2f | %.2f | %s / %s | %d / %d | %s |\n", + st.Name, ms(c.Wall.Median), ms(st.Baseline.Wall.Median), st.WallRatio, st.CPURatio, + reads(c.Counts), reads(st.Baseline.Counts), c.FilesChanged, st.Baseline.FilesChanged, flags) + case ab: + fmt.Fprintf(&sb, "| %s | %s | — | — | — | %s / — | %d / — | %s |\n", + st.Name, ms(c.Wall.Median), reads(c.Counts), c.FilesChanged, flags) + default: + fmt.Fprintf(&sb, "| %s | %s | %s – %s | %s | %.0f MB | %s | %d | %s |\n", + st.Name, ms(c.Wall.Median), ms(c.Wall.Min), ms(c.Wall.Max), ms(c.CPU.Median), c.RSS.Median, + reads(c.Counts), c.FilesChanged, flags) + } + } + fmt.Fprintf(&sb, "\nSession-weighted total: **%s**", ms(s.Weighted.CandidateMS)) + if s.Weighted.Ratio > 0 { + fmt.Fprintf(&sb, " against %s — **× %.2f**", ms(s.Weighted.BaselineMS), s.Weighted.Ratio) + } + sb.WriteString("\n") + if s.BaselineError != "" { + fmt.Fprintf(&sb, "\nThe baseline stopped, so later steps have no A/B: `%s`\n", firstLine(s.BaselineError)) + } + } + sb.WriteString(r.scaling()) + return sb.String() +} + +// scaling is the same session's steps at the two sizes, side by side: what +// each command costs more on a larger project. The session's own scripts are +// identical at both sizes, so the multiple is the project's doing. +func (r record) scaling() string { + var small, large *sessionResult + for i := range r.Sessions { + switch r.Sessions[i].Size { + case "small": + small = &r.Sessions[i] + case "large": + large = &r.Sessions[i] + } + } + if small == nil || large == nil || small.Units == 0 { + return "" + } + var sb strings.Builder + fmt.Fprintf(&sb, "\n### Scaling — the large project has × %.1f the units\n\n", float64(large.Units)/float64(small.Units)) + sb.WriteString("| step | small | large | × |\n|---|---:|---:|---:|\n") + for i, st := range small.Steps { + if i >= len(large.Steps) || st.Candidate.Wall.Median == 0 { + continue + } + l := large.Steps[i].Candidate.Wall.Median + fmt.Fprintf(&sb, "| %s | %s | %s | %.1f |\n", st.Name, ms(st.Candidate.Wall.Median), ms(l), l/st.Candidate.Wall.Median) + } + return sb.String() +} + +func ms(v float64) string { + if v >= 10000 { + return fmt.Sprintf("%.1f s", v/1000) + } + if v >= 1000 { + return fmt.Sprintf("%.2f s", v/1000) + } + return fmt.Sprintf("%.0f ms", v) +} + +func reads(c *counts) string { + if c == nil { + return "n/a" + } + return fmt.Sprint(c.UnitFileReads) +} + +func firstLine(s string) string { + line, _, _ := strings.Cut(s, "\n") + return line +} diff --git a/scripts/agentbench/rss_other.go b/scripts/agentbench/rss_other.go new file mode 100644 index 000000000..c527be253 --- /dev/null +++ b/scripts/agentbench/rss_other.go @@ -0,0 +1,10 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build !unix + +package main + +import "os" + +// maxRSSMB is not measured on this platform. +func maxRSSMB(*os.ProcessState) float64 { return 0 } diff --git a/scripts/agentbench/rss_unix.go b/scripts/agentbench/rss_unix.go new file mode 100644 index 000000000..076c8c916 --- /dev/null +++ b/scripts/agentbench/rss_unix.go @@ -0,0 +1,24 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build unix + +package main + +import ( + "os" + "runtime" + "syscall" +) + +// maxRSSMB is the process's peak resident memory in MiB. +func maxRSSMB(ps *os.ProcessState) float64 { + ru, ok := ps.SysUsage().(*syscall.Rusage) + if !ok { + return 0 + } + // ru_maxrss is kilobytes on Linux and bytes on macOS. + if runtime.GOOS == "darwin" { + return float64(ru.Maxrss) / (1 << 20) + } + return float64(ru.Maxrss) / (1 << 10) +} From 5722248a6ecec96d8d096673aa872d6bddabbefc Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 11 Nov 2026 22:44:02 +0000 Subject: [PATCH 5/8] =?UTF-8?q?feat(bench):=20running-app=20session=20?= =?UTF-8?q?=E2=80=94=20boot,=20hot=20apply,=20entity=20change,=20tests?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 2 of PROPOSAL_agent_devloop_benchmark. -runtime-version (make bench BENCH_RUNTIME=) adds a session on a running app: boot, a page check, a microflow change applied hot, an attached test run, an entity change applied by restart, stop, and a cold test run. The project is created by mxcli new, since the PedApp fixture cannot be built; mxcli installs mxbuild and the runtime when they are not cached. The arms take turns in this session instead of alternating step by step, because a boot leaves a runtime up, and each arm's app is stopped whatever happened. A change to the running app is one step of two commands, the exec and the run wait. A session that fails no longer drops the record of the ones that completed. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_011oK82u4HRCKfX2BBvhgsNm --- .github/workflows/nightly.yml | 36 +- CHANGELOG.md | 2 +- Makefile | 15 +- docs-site/src/internals/agent-benchmark.md | 64 ++- .../PROPOSAL_agent_devloop_benchmark.md | 34 +- scripts/agentbench/bench.go | 379 +++++++++++++----- scripts/agentbench/bench_test.go | 110 ++++- scripts/agentbench/gen/gen.go | 30 ++ scripts/agentbench/gen/gen_test.go | 31 ++ scripts/agentbench/main.go | 29 +- scripts/agentbench/record.go | 10 +- 11 files changed, 612 insertions(+), 128 deletions(-) diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index dccf4f90e..b07dc08b9 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -191,14 +191,25 @@ jobs: # interleaved on THIS runner, and reports the ratio. The record goes to the # `benchmarks` branch, one file per run. # - # Needs no mxbuild (it starts from the committed PedApp fixture) and does - # not gate the release, like upgrade-full-corpus: a slow night must not hold - # a nightly back. The driver exits non-zero only when a command fails or - # re-running a settled script changes a file. + # Three sessions. Two time mxcli alone and start from the committed PedApp + # fixture. The third times the running app -- boot, a hot change, an entity + # change, a page check, tests -- on a project `mxcli new` creates, and so + # needs mxbuild and the runtime (mxcli downloads both), JDK 21 for the + # runtime, and a browser for the page check. It runs last: if it fails, the + # record of the first two is still written and committed. + # + # Does not gate the release, like upgrade-full-corpus: a slow night must not + # hold a nightly back. The driver exits non-zero only when a command fails + # or re-running a settled script changes a file. bench: runs-on: ubuntu-latest - timeout-minutes: 60 + timeout-minutes: 90 name: agent dev-loop benchmark + env: + # One version, the one the running-app session was developed against. + # Moving it is a deliberate change: boot and apply times differ between + # Mendix versions, so the trend restarts where it moves. + BENCH_MENDIX_VERSION: '11.13.0' steps: - uses: actions/checkout@v7 - uses: actions/setup-go@v7 @@ -216,6 +227,20 @@ jobs: env: ANTLR4_TOOLS_ANTLR_VERSION: '4.13.2' + # The Mendix 11 runtime launches under JDK 21; the runner's default + # `java` is older. + - uses: actions/setup-java@v6 + with: + distribution: temurin + java-version: '21' + + # `mxcli playwright check` requires the `playwright` package from the + # global node_modules and launches its Chromium. + - name: Install Playwright and Chromium + run: | + npm install -g playwright@1.63.0 + npx --yes playwright@1.63.0 install --with-deps chromium + # First, before the `nightly` job can replace the release. That job # waits for the whole `test` matrix, but nothing here relies on it: the # record names the baseline's version, and with no baseline the run @@ -237,6 +262,7 @@ jobs: - name: Run the benchmark run: | make bench BASELINE="${{ steps.baseline.outputs.path }}" \ + BENCH_RUNTIME="$BENCH_MENDIX_VERSION" \ BENCH_FLAGS="-out agentbench-record.json -summary $GITHUB_STEP_SUMMARY" - name: Upload the record diff --git a/CHANGELOG.md b/CHANGELOG.md index 2e98b2803..53f22bb5b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added -- **`make bench`: a benchmark of what an agent waits for, and `MXCLI_STATS_FILE`** — a fixed session of mxcli commands (open, author entities, microflows and pages, describe, catalog refresh, check, a one-document edit, a bulk change, a re-run, a rename), one process per command, on PedApp and on PedApp with a generated app seeded into it. With `BASELINE=` the two binaries run interleaved on the same machine and each step reports the ratio; the nightly `bench` job does that against the previous nightly and commits the record to the `benchmarks` branch. It also records counts that do not depend on the machine: unit files read and unit files changed. `MXCLI_STATS_FILE=` makes any mxcli command append one JSON line per closed project connection with its unit-file reads and its writes offered and landed. See "Agent Dev-Loop Benchmark" in the docs. +- **`make bench`: a benchmark of what an agent waits for, and `MXCLI_STATS_FILE`** — a fixed session of mxcli commands (open, author entities, microflows and pages, describe, catalog refresh, check, a one-document edit, a bulk change, a re-run, a rename), one process per command, on PedApp and on PedApp with a generated app seeded into it. `make bench BENCH_RUNTIME=` adds a session on a running app — boot, a hot change, an entity change, a page check, an attached and a cold test run — on a project `mxcli new` creates, with mxbuild and the runtime installed by mxcli when they are not cached. With `BASELINE=` the two binaries run interleaved on the same machine and each step reports the ratio; the nightly `bench` job does that against the previous nightly and commits the record to the `benchmarks` branch. It also records counts that do not depend on the machine: unit files read and unit files changed. `MXCLI_STATS_FILE=` makes any mxcli command append one JSON line per closed project connection with its unit-file reads and its writes offered and landed. See "Agent Dev-Loop Benchmark" in the docs. - **`mxcli test --local --reuse-build` skips the injection and the build when nothing changed** (mendixlabs/mxcli#1783, closes mendixlabs/mxcli#1782) — a local test run injects its test microflows, builds and boots; when the project, the test files and mxcli are unchanged, the deployment the previous run left is already this run's build, so `--reuse-build` boots it directly (about 12 s against 60 s, measured by the author). The decision is a content hash of the project folder (outputs, caches, `theme-cache/`, and any `.git`/`node_modules` left out), the generated test MDL, the mxcli binary and the mxbuild override, pinned to the identity of `deployment/model/model.mdp`, so a deployment rebuilt by anything else is never taken for the test build. Not under `--watch` or `--skip-build`. - **A `genai-runtime` skill for the runtime side of Agents Kit 2** (ako/mxcli#2068) — registering a Mendix Cloud GenAI key or an OpenAI-compatible endpoint, seeding AgentCommons agents from JSON (matched by `Title` only, so a changed JSON never reaches a stored agent), resuming Conversational UI chats, loading PgVector and Mendix Cloud knowledge bases, and traces. It collects the gotchas that `check` and `mx check` cannot see: the encryption key both the OpenAI and PgVector setup need, the `vector` extension only the UI's Test button creates, the 60–120 s indexing delay, the chat that collapses outside its page, and `StoreTraces` defaulting to false. Its examples pass `check --references` and build with 0 errors against the Agent Builder Starter App's modules. - **`mxcli test --attach` runs a test without building it, against an app started with `--allow-call`** (mendixlabs/mxcli#1788, mendixlabs/mxcli#1789) — `run --local --test-endpoint --allow-call` switches on an executor in the test endpoint. A `.test.mdl` test whose statements it supports (`declare`, `set`, `create`, `change`, `commit`, `delete`, `call microflow`, database `retrieve` with `where`/`first`; `@expect` with `count`, `@throws`, `@setup`, `@cleanup`) is compiled by mxcli into steps and run in the app with the Mendix runtime API, every expression evaluated by `Core.evaluateExpression`. Nothing is injected or built, so a test edit costs one HTTP round trip: on a small app the suite took 0.26 s against 3.2 s for inject-and-build, with identical verdicts. Retrieves over an association and the list aggregates `count`/`sum`/`average`/`minimum`/`maximum` over an attribute run there too. A test with any other statement is injected and built as before. Only that test is, while the rest of the run still goes straight to the app, and the output names the test and the MDL statement (`the statement if`, `a list operation (find)`). Because the runtime evaluator returns empty for an unknown name, an undeclared `$variable` and a non-existent enumeration value are refused (ERROR) before evaluating, and a model newer than the running build is refused with `mxcli run wait` as the fix under `--watch`. The route takes POST, needs the run's token and a loopback caller, and answers 403 without `--allow-call`, since it calls the microflows a test names in a system context. `.test.mdl` stays the only test language. Design discussed with the authors of #1789; measurements in `mdl-examples/spikes/evaluate-expression.mdl`. diff --git a/Makefile b/Makefile index 5204826bf..78439c2a9 100644 --- a/Makefile +++ b/Makefile @@ -433,12 +433,19 @@ test-integration-other: # Agent dev-loop benchmark: a fixed, scripted session of mxcli commands -- the # ones an agent runs while building an app -- one process per command, at two -# project sizes, no model in the loop. Needs no mxbuild: it starts from the -# committed PedApp fixture. +# project sizes, no model in the loop. Those sessions need no mxbuild: they +# start from the committed PedApp fixture. # # make bench # time this checkout's binary # make bench BASELINE=/path/to/other/mxcli # A/B against another build # make bench BENCH_FLAGS='-sizes small -reps 3 -out record.json' +# make bench BENCH_RUNTIME=11.13.0 # also the running-app session +# +# BENCH_RUNTIME adds the session on a running app -- boot, a hot change, an +# entity change, a page check, tests -- on a project `mxcli new` creates for +# that Mendix version. mxcli downloads mxbuild and the runtime when they are +# not cached (about 800 MB); it needs JDK 21, and node with playwright and a +# Chromium for the page check (BENCH_FLAGS=-page-check=false to leave it out). # # With BASELINE the two binaries run interleaved on this machine and the table # shows the ratio, which is the only figure comparable between two machines or @@ -446,9 +453,11 @@ test-integration-other: # nightly release and commits the record to the `benchmarks` branch. # docs-site/src/internals/agent-benchmark.md explains how to read it. BASELINE ?= +BENCH_RUNTIME ?= BENCH_FLAGS ?= bench: build - go run ./scripts/agentbench -mxcli $(BUILD_DIR)/$(BINARY_NAME) $(if $(BASELINE),-baseline $(BASELINE)) $(BENCH_FLAGS) + go run ./scripts/agentbench -mxcli $(BUILD_DIR)/$(BINARY_NAME) $(if $(BASELINE),-baseline $(BASELINE)) \ + $(if $(BENCH_RUNTIME),-runtime-version $(BENCH_RUNTIME)) $(BENCH_FLAGS) # Run MDL integration tests (requires Docker and a Mendix project) # Usage: make test-mdl MPR=path/to/app.mpr diff --git a/docs-site/src/internals/agent-benchmark.md b/docs-site/src/internals/agent-benchmark.md index 74b7511dc..3ef8a482f 100644 --- a/docs-site/src/internals/agent-benchmark.md +++ b/docs-site/src/internals/agent-benchmark.md @@ -23,10 +23,11 @@ Design and rationale: `docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md`. make bench # time this checkout's binary make bench BASELINE=/path/to/other/mxcli # A/B against another build make bench BENCH_FLAGS='-sizes small -reps 3 -out record.json' +make bench BENCH_RUNTIME=11.13.0 # also the running-app session ``` -It needs no mxbuild and no Docker: every session starts from a copy of the -committed `testdata/pedapp` project. A default run with a baseline took 14 +Without `BENCH_RUNTIME` it needs no mxbuild and no Docker: every session +starts from a copy of the committed `testdata/pedapp` project. A default run with a baseline took 14 minutes on a 4-CPU machine; `-sizes small -reps 1` takes under a minute and is enough to see whether a change moved anything. @@ -68,6 +69,49 @@ once). The session's own scripts are the same at both sizes, so the *Scaling* table at the end of the output shows what each command costs more purely because the project around it is bigger. +## The running-app session + +`BENCH_RUNTIME=` adds a third session: what an agent waits for +once the app is up. Measured sessions spend more wall time here than in every +mxcli-only command together. + +| Step | Commands | What it shows | +|---|---|---| +| `boot` | `run --local --watch --detach --db-type hsqldb --test-endpoint --allow-call` | until the app serves: build, runtime start, schema | +| `page-check` | `playwright check` | a text verdict for the home page | +| `hot-apply` | `exec` of a one-microflow change, then `run wait` | a behavioural change, live without a restart | +| `test-attach` | `test --attach` | a test run in the running app, nothing built | +| `apply-entity` | `exec` adding an attribute, then `run wait` | a structural change: runtime restart and schema update | +| `stop` | `run stop` | | +| `test-cold` | `test --local --db-type hsqldb` | a test run that boots its own runtime | + +It differs from the other two sessions in four ways: + +- **The project is created by `mxcli new`**, with the session's entities, + microflows and pages authored into it. PedApp cannot be built — the fixture + carries no theme. `new` and the authoring are timed once, by the candidate + (the `new` and `author` lines). +- **mxcli installs what it needs.** mxbuild and the runtime for the version + are downloaded when they are not cached, before anything is timed. The + machine has to bring JDK 21, and node with the `playwright` package and a + Chromium for `page-check`; `BENCH_FLAGS=-page-check=false` leaves that step + out. +- **The arms take turns.** A boot leaves a runtime up, and two at once would + share the ports and the processor they are timed on. So one binary runs its + whole session and is stopped, then the other. The pairing is by repetition, + minutes apart instead of seconds. +- **Read the wall ratio only.** The time is spent in mxbuild and the runtime, + which are the same in both arms; the CPU column covers the mxcli process + alone and says little here. A real mxcli regression is a smaller share of a + noisier figure than in the other sessions — with the same code in both arms + the ratios ran from 0.92 to 1.04 over three repetitions. + +`boot` and `stop` each report 7 files changed: the test endpoint's documents, +added when the app starts and removed when it stops. + +The database is the runtime's built-in HSQLDB, so no PostgreSQL is needed. +Figures are for one Mendix version and do not carry over to another. + ## Reading the output With a baseline, each step has these columns: @@ -143,19 +187,21 @@ outside. ## The nightly job The `bench` job in `.github/workflows/nightly.yml` downloads the previous -nightly release's binary, runs `make bench` against it, writes the table to +nightly release's binary, runs all three sessions against it, writes the table to the job summary, and commits the JSON record to the `benchmarks` branch as `results/YYYY/MM/-.json`. That branch holds records only and is never merged. The job does not gate the nightly release. It fails only when a command of -the session fails or a `wrote-on-re-exec` is found. +a session fails or a `wrote-on-re-exec` is found. The running-app session runs +last, and a session that fails does not take the completed ones with it: their +record is still written and committed. ## What it does not measure -- **mxbuild and the running app** — `docker check`, the build, `run --local` - boot and hot apply, `mxcli test`. These are where measured sessions spend - the most wall time and are the next phases of the proposal. -- **Creating the project** — sessions start from a committed fixture, not from - `mxcli new`, which needs mxbuild. +- **Validation and the full build** — `docker check` and `docker build`, the + next phase of the proposal. +- **A PostgreSQL-backed app** — the running-app session uses HSQLDB. +- **`mxcli new` as an A/B** — it is timed once, by the candidate, as a trend + figure. - **Tokens and model calls** — see `mxcli eval run`. diff --git a/docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md b/docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md index b32293e3a..5f68cc150 100644 --- a/docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md +++ b/docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md @@ -10,7 +10,7 @@ related: # Proposal: Nightly Agent Dev-Loop Performance Benchmark -**Status:** Partial — Phase 1 is implemented (see "Phase 1 as built") +**Status:** Partial — Phases 1 and 2 are implemented (see "Phase 1 as built", "Phase 2 as built") **Date:** 2026-11-11 ## Problem Statement @@ -308,6 +308,38 @@ Adds `boot`, `hot-apply-flow`, `apply-entity`, `page-check`, `test-cold`, This is ordered second, ahead of validate/build, because restarts, waits and tests are the largest measured share of command wall time. +#### Phase 2 as built + +The running-app session of `scripts/agentbench`, switched on with +`-runtime-version` (`make bench BENCH_RUNTIME=11.13.0`) and run by the nightly +job. Steps: `boot`, `page-check`, `hot-apply`, `test-attach`, `apply-entity`, +`stop`, `test-cold`. Where it differs from the design, and what it showed: + +- **The project comes from `mxcli new`.** The PedApp fixture cannot be built + (it has no theme), so the `new` step deferred from Phase 1 arrives here, as + preparation: timed once, by the candidate. The session's app is authored + into it before the snapshot. +- **mxcli installs mxbuild and the runtime**, before anything is timed. The + job adds JDK 21 and a Playwright Chromium. +- **The arms take turns** instead of alternating step by step, because a boot + leaves a runtime up. A teardown stops each arm's app whatever happened. +- **A change to the running app is one step of two commands**, the `exec` and + the `run wait`: that pair is what the agent waits through. +- **Runtime steps do get an A/B**, as the revised design said, and it is + usable: with the same code in both arms, ratios ran from 0.92 to 1.04 over + three repetitions on the development machine. +- **`test-attach` uses `--allow-call`**, the path that runs a test in the + running app without building it. The inject-and-build path of `--attach` is + not measured separately. +- **First figures** (Mendix 11.13.0, 4 CPUs, 25 authored entities): boot 27 s, + hot apply 7.7 s, entity change 19 s, cold test run 25 s, attached test run + 0.13 s, page check 3.1 s, `mxcli new` 37 s. They also settle a Phase 1 + caveat: the generated app builds and boots, so it is valid for mxbuild, not + only for `mxcli check`. +- **One version.** The nightly pins the Mendix version the session was + developed against. Open Question 8 (the licence session cap) did not show in + these short sessions. + ### Phase 3 — validate and build Adds `mx-check` and `build`. diff --git a/scripts/agentbench/bench.go b/scripts/agentbench/bench.go index 6e8b26bc0..194d6193a 100644 --- a/scripts/agentbench/bench.go +++ b/scripts/agentbench/bench.go @@ -5,6 +5,7 @@ package main import ( "bufio" "bytes" + "context" "crypto/sha256" "encoding/json" "errors" @@ -14,6 +15,7 @@ import ( "os" "os/exec" "path/filepath" + "slices" "sort" "strings" "time" @@ -30,9 +32,12 @@ const statsFileEnv = "MXCLI_STATS_FILE" // ones left. type step struct { name string - // args is the command line after the binary; {P} is the project's .mpr and - // {S} the directory holding the session's scripts. - args []string + // cmds are the command lines after the binary, run one after the other + // and measured as one: {P} is the project's .mpr and {S} the directory + // holding the session's scripts. Most steps are one command. A change to a + // running app is two, the edit and the wait for it to be live, because + // that pair is what the agent sits through. + cmds [][]string // weight is how many times a real session runs this kind of command for // each time the benchmark does; see weights below. weight int @@ -41,6 +46,9 @@ type step struct { noWrites bool } +// one is a step of a single command. +func one(args ...string) [][]string { return [][]string{args} } + // weightsVersion identifies the weight column. A record carries it so a change // of weights is never read as a change of speed. const weightsVersion = 1 @@ -54,21 +62,21 @@ func sessionSteps() []step { return []step{ // The first command on a project builds its unit index; the second is // what every later command pays to start and open. - {name: "open-cold", args: []string{"-p", "{P}", "-c", "show modules"}, weight: 1}, - {name: "open", args: []string{"-p", "{P}", "-c", "show modules"}, weight: 60}, - {name: "domain", args: []string{"exec", "{S}/domain.mdl", "-p", "{P}"}, weight: 1}, - {name: "microflows", args: []string{"exec", "{S}/microflows.mdl", "-p", "{P}"}, weight: 1}, - {name: "pages", args: []string{"exec", "{S}/pages.mdl", "-p", "{P}"}, weight: 1}, - {name: "describe", args: []string{"-p", "{P}", "-c", read}, weight: 60}, - {name: "catalog", args: []string{"-p", "{P}", "-c", "refresh catalog"}, weight: 5}, - {name: "catalog-full", args: []string{"-p", "{P}", "-c", "refresh catalog full"}, weight: 1}, - {name: "query", args: []string{"-p", "{P}", "-c", "select count(*) from CATALOG.ENTITIES"}, weight: 20}, - {name: "check", args: []string{"check", "{S}/single-edit.mdl", "-p", "{P}", "--references"}, weight: 90}, - {name: "single-edit", args: []string{"exec", "{S}/single-edit.mdl", "-p", "{P}"}, weight: 150}, - {name: "catalog-after-edit", args: []string{"-p", "{P}", "-c", "refresh catalog full"}, weight: 10}, - {name: "bulk-alter", args: []string{"exec", "{S}/bulk-alter.mdl", "-p", "{P}"}, weight: 1}, - {name: "re-exec", args: []string{"exec", "{S}/bulk-alter.mdl", "-p", "{P}"}, weight: 1, noWrites: true}, - {name: "rename", args: []string{"-p", "{P}", "-c", "rename entity Bench.Item01 to Item01Renamed"}, weight: 1}, + {name: "open-cold", cmds: one("-p", "{P}", "-c", "show modules"), weight: 1}, + {name: "open", cmds: one("-p", "{P}", "-c", "show modules"), weight: 60}, + {name: "domain", cmds: one("exec", "{S}/domain.mdl", "-p", "{P}"), weight: 1}, + {name: "microflows", cmds: one("exec", "{S}/microflows.mdl", "-p", "{P}"), weight: 1}, + {name: "pages", cmds: one("exec", "{S}/pages.mdl", "-p", "{P}"), weight: 1}, + {name: "describe", cmds: one("-p", "{P}", "-c", read), weight: 60}, + {name: "catalog", cmds: one("-p", "{P}", "-c", "refresh catalog"), weight: 5}, + {name: "catalog-full", cmds: one("-p", "{P}", "-c", "refresh catalog full"), weight: 1}, + {name: "query", cmds: one("-p", "{P}", "-c", "select count(*) from CATALOG.ENTITIES"), weight: 20}, + {name: "check", cmds: one("check", "{S}/single-edit.mdl", "-p", "{P}", "--references"), weight: 90}, + {name: "single-edit", cmds: one("exec", "{S}/single-edit.mdl", "-p", "{P}"), weight: 150}, + {name: "catalog-after-edit", cmds: one("-p", "{P}", "-c", "refresh catalog full"), weight: 10}, + {name: "bulk-alter", cmds: one("exec", "{S}/bulk-alter.mdl", "-p", "{P}"), weight: 1}, + {name: "re-exec", cmds: one("exec", "{S}/bulk-alter.mdl", "-p", "{P}"), weight: 1, noWrites: true}, + {name: "rename", cmds: one("-p", "{P}", "-c", "rename entity Bench.Item01 to Item01Renamed"), weight: 1}, } } @@ -81,6 +89,8 @@ func writeSession(dir string, entities int) error { "pages.mdl": gen.Pages(p), "single-edit.mdl": gen.SingleEdit(p), "bulk-alter.mdl": gen.BulkAlter(p), + "entity-edit.mdl": gen.EntityEdit(p), + "bench.test.mdl": gen.Tests(p), } { if err := os.WriteFile(filepath.Join(dir, name), []byte(src), 0o644); err != nil { return err @@ -117,37 +127,59 @@ type counts struct { WritesWritten int `json:"writes_written"` } -// runStep runs one process and measures it. A non-zero exit is an error -// carrying the tail of the output: a failed command's time is not a -// measurement. -func runStep(bin string, args []string, project, scripts, scratch string) (sample, error) { - resolved := make([]string, len(args)) - for i, a := range args { - a = strings.ReplaceAll(a, "{P}", project) - resolved[i] = strings.ReplaceAll(a, "{S}", scripts) - } - statsPath := filepath.Join(scratch, "stats.jsonl") - _ = os.Remove(statsPath) +// commandTimeout bounds one command. A hung boot must fail the run with its +// output, not sit until the CI job is cancelled with none. +const commandTimeout = 20 * time.Minute +// statsSeq numbers the stats files, one per command: a detached `run` appends +// its line when it stops, minutes later, and must not land in the file of +// whichever step is running by then. +var statsSeq int + +// runStep runs a step's commands in order and measures them as one. A non-zero +// exit is an error carrying the tail of the output: a failed command's time is +// not a measurement. +func runStep(bin string, cmds [][]string, project, scripts, scratch string) (sample, error) { before := unitDigests(filepath.Dir(project)) - cmd := exec.Command(bin, resolved...) - cmd.Env = append(os.Environ(), statsFileEnv+"="+statsPath, "NO_COLOR=1") - var stdout, stderr bytes.Buffer - cmd.Stdout, cmd.Stderr = &stdout, &stderr - start := time.Now() - err := cmd.Run() - wall := time.Since(start) - if err != nil { - return sample{}, fmt.Errorf("%s %s: %w\n%s", filepath.Base(bin), strings.Join(resolved, " "), err, tail(stdout.String()+stderr.String(), 20)) - } - s := sample{ - WallMS: float64(wall.Microseconds()) / 1000, - CPUMS: float64((cmd.ProcessState.UserTime() + cmd.ProcessState.SystemTime()).Microseconds()) / 1000, - StdoutBytes: stdout.Len(), - Counts: readCounts(statsPath), - FilesChanged: changedFiles(before, unitDigests(filepath.Dir(project))), + var s sample + for _, args := range cmds { + resolved := make([]string, len(args)) + for i, a := range args { + a = strings.ReplaceAll(a, "{P}", project) + resolved[i] = strings.ReplaceAll(a, "{S}", scripts) + } + statsSeq++ + statsPath := filepath.Join(scratch, fmt.Sprintf("stats-%d.jsonl", statsSeq)) + + ctx, cancel := context.WithTimeout(context.Background(), commandTimeout) + cmd := exec.CommandContext(ctx, bin, resolved...) + cmd.Env = append(os.Environ(), statsFileEnv+"="+statsPath, "NO_COLOR=1") + // A detached child may keep the output pipes open; do not wait for it. + cmd.WaitDelay = 5 * time.Second + var stdout, stderr bytes.Buffer + cmd.Stdout, cmd.Stderr = &stdout, &stderr + start := time.Now() + err := cmd.Run() + wall := time.Since(start) + cancel() + if err != nil && !errors.Is(err, exec.ErrWaitDelay) { + return sample{}, fmt.Errorf("%s %s: %w\n%s", filepath.Base(bin), strings.Join(resolved, " "), err, tail(stdout.String()+stderr.String(), 20)) + } + s.WallMS += float64(wall.Microseconds()) / 1000 + s.CPUMS += float64((cmd.ProcessState.UserTime() + cmd.ProcessState.SystemTime()).Microseconds()) / 1000 + s.RSSMB = max(s.RSSMB, maxRSSMB(cmd.ProcessState)) + s.StdoutBytes += stdout.Len() + if c := readCounts(statsPath); c != nil { + if s.Counts == nil { + s.Counts = &counts{} + } + s.Counts.UnitFileReads += c.UnitFileReads + s.Counts.WritesOffered += c.WritesOffered + s.Counts.WritesWritten += c.WritesWritten + } + _ = os.Remove(statsPath) } - s.RSSMB = maxRSSMB(cmd.ProcessState) + s.FilesChanged = changedFiles(before, unitDigests(filepath.Dir(project))) return s, nil } @@ -197,7 +229,9 @@ func copyProject(src, dst string) error { } return os.MkdirAll(filepath.Join(dst, rel), 0o755) } - if !d.Type().IsRegular() { + // `mxcli new` links its own binary into the project root: 100 MB a + // copy, and not part of the project. + if !d.Type().IsRegular() || rel == "mxcli" { return nil } in, err := os.Open(path) @@ -372,6 +406,8 @@ type stepResult struct { // sessionResult is the scripted session at one project size. type sessionResult struct { Size string `json:"size"` + // Reps is how many repetitions each arm ran. + Reps int `json:"reps"` // Units is how many unit files the project held when the session started. Units int `json:"units"` // Seed is the one-off build of the large fixture, candidate only: it is @@ -407,29 +443,48 @@ type config struct { sessionEntities int large gen.Params sizes []string - threshold float64 - log io.Writer + // runtimeVersion, when set, adds the running-app session on a project + // created by `mxcli new` for that Mendix version. + runtimeVersion string + runtimeReps int + pageCheck bool + threshold float64 + log io.Writer } // errInvariant is returned, after the record is complete, when the candidate // broke something a benchmark run can state exactly rather than estimate. var errInvariant = errors.New("the candidate broke an invariant") -// runSession measures the session at one size. snapshot is the project every -// repetition of both arms starts from. -func runSession(cfg config, size, snapshot, scripts string) (sessionResult, error) { +// A session is a list of steps run in order against one project. +type session struct { + name string + steps []step + reps int + // armsInTurn runs one arm's whole session before the other's, instead of + // alternating the arms step by step. The running-app session needs it: its + // steps leave a runtime up, and two of them at once would share the ports + // and the processor they are being timed on. + armsInTurn bool + // teardown is run, unmeasured, after each arm's repetition, whatever + // happened in it: a runtime left behind would fail the next boot. + teardown []string +} + +// runSession measures one session. snapshot is the project every repetition of +// both arms starts from. +func runSession(cfg config, def session, snapshot, scripts string) (sessionResult, error) { mpr, err := findMPR(snapshot) if err != nil { return sessionResult{}, err } - steps := sessionSteps() - res := sessionResult{Size: size, Units: countUnits(snapshot)} + size, steps := def.name, def.steps + res := sessionResult{Size: size, Reps: def.reps, Units: countUnits(snapshot)} candidate := make([][]sample, len(steps)) baseline := make([][]sample, len(steps)) baselineLive := cfg.baseline != "" - for rep := 0; rep < cfg.reps; rep++ { - dirs := map[string]string{} + for rep := 0; rep < def.reps; rep++ { arms := []string{"candidate"} if baselineLive { arms = append(arms, "baseline") @@ -439,44 +494,86 @@ func runSession(cfg config, size, snapshot, scripts string) (sessionResult, erro arms[0], arms[1] = arms[1], arms[0] } } - for _, arm := range arms { - dir := filepath.Join(cfg.work, fmt.Sprintf("%s-%s-%d", size, arm, rep)) - if err := copyProject(snapshot, dir); err != nil { - return res, err + binOf := func(arm string) string { + if arm == "baseline" { + return cfg.baseline + } + return cfg.candidate + } + dirs := map[string]string{} + prepare := func(arm string) error { + dirs[arm] = filepath.Join(cfg.work, fmt.Sprintf("%s-%s-%d", size, arm, rep)) + return copyProject(snapshot, dirs[arm]) + } + // finish stops what the arm started and removes its project. + finish := func(arm string) { + if def.teardown != nil { + _, _ = runStep(binOf(arm), [][]string{def.teardown}, filepath.Join(dirs[arm], mpr), scripts, cfg.work) + } + _ = os.RemoveAll(dirs[arm]) + } + // runOne runs step i in one arm. Its error is the candidate's: a + // baseline that fails only retires the baseline. + runOne := func(i int, arm string) error { + if arm == "baseline" && !baselineLive { + return nil + } + st := steps[i] + s, err := runStep(binOf(arm), st.cmds, filepath.Join(dirs[arm], mpr), scripts, cfg.work) + switch { + case err != nil && arm == "candidate": + return fmt.Errorf("%s, step %s: %w", size, st.name, err) + case err != nil: + // The baseline may not know a statement the session uses. + // That is not this run's failure, but its project has now + // diverged, so no later step of it is comparable. + baselineLive = false + res.BaselineError = fmt.Sprintf("step %s: %v", st.name, err) + fmt.Fprintf(cfg.log, " baseline stopped at %s; later steps have no A/B\n", st.name) + case arm == "candidate": + candidate[i] = append(candidate[i], s) + default: + baseline[i] = append(baseline[i], s) } - dirs[arm] = dir + return nil } - for i, st := range steps { + + var stepErr error + if def.armsInTurn { for _, arm := range arms { - if arm == "baseline" && !baselineLive { - continue + if stepErr = prepare(arm); stepErr == nil { + for i := range steps { + if stepErr = runOne(i, arm); stepErr != nil { + break + } + } } - bin := cfg.candidate - if arm == "baseline" { - bin = cfg.baseline + finish(arm) + if stepErr != nil { + break } - s, err := runStep(bin, st.args, filepath.Join(dirs[arm], mpr), scripts, cfg.work) - switch { - case err != nil && arm == "candidate": - return res, fmt.Errorf("%s, step %s: %w", size, st.name, err) - case err != nil: - // The baseline may not know a statement the session uses. - // That is not this run's failure, but its project has now - // diverged, so no later step of it is comparable. - baselineLive = false - res.BaselineError = fmt.Sprintf("step %s: %v", st.name, err) - fmt.Fprintf(cfg.log, " baseline stopped at %s; later steps have no A/B\n", st.name) - case arm == "candidate": - candidate[i] = append(candidate[i], s) - default: - baseline[i] = append(baseline[i], s) + } + } else { + for _, arm := range arms { + if stepErr = prepare(arm); stepErr != nil { + break } } + for i := 0; i < len(steps) && stepErr == nil; i++ { + for _, arm := range arms { + if stepErr = runOne(i, arm); stepErr != nil { + break + } + } + } + for arm := range dirs { + finish(arm) + } } - for _, dir := range dirs { - _ = os.RemoveAll(dir) + if stepErr != nil { + return res, stepErr } - fmt.Fprintf(cfg.log, " %s: repetition %d of %d done\n", size, rep+1, cfg.reps) + fmt.Fprintf(cfg.log, " %s: repetition %d of %d done\n", size, rep+1, def.reps) } var invariant bool @@ -542,7 +639,7 @@ func seedLarge(cfg config, scripts string) (string, []seedResult, error) { if err := os.WriteFile(script, []byte(part.src), 0o644); err != nil { return "", nil, err } - s, err := runStep(cfg.candidate, []string{"exec", script, "-p", "{P}"}, filepath.Join(dir, mpr), scripts, cfg.work) + s, err := runStep(cfg.candidate, one("exec", script, "-p", "{P}"), filepath.Join(dir, mpr), scripts, cfg.work) if err != nil { return "", nil, fmt.Errorf("%s: %w", part.name, err) } @@ -552,7 +649,83 @@ func seedLarge(cfg config, scripts string) (string, []seedResult, error) { return dir, out, nil } -// run is the whole benchmark: every requested size, in order. +// runtimeSteps is the running-app session: what an agent waits for once the +// app is up. Weights are rounded from the same measured builds as the others +// (`run wait` 95 calls in the largest, restarts 150, microflow test runs 35). +func runtimeSteps(pageCheck bool) []step { + steps := []step{ + // Until the app serves: the build, the runtime start, the schema. + {name: "boot", cmds: one("run", "--local", "--watch", "--detach", "--db-type", "hsqldb", + "--test-endpoint", "--allow-call", "-p", "{P}"), weight: 10}, + {name: "page-check", cmds: one("playwright", "check", "-p", "{P}"), weight: 30}, + // A microflow change, live: the model is reloaded, the runtime stays. + {name: "hot-apply", cmds: [][]string{ + {"exec", "{S}/single-edit.mdl", "-p", "{P}"}, + {"run", "wait", "-p", "{P}"}, + }, weight: 60}, + {name: "test-attach", cmds: one("test", "{S}/bench.test.mdl", "-p", "{P}", "--attach"), weight: 30}, + // An entity change, live: the runtime restarts and updates the schema. + {name: "apply-entity", cmds: [][]string{ + {"exec", "{S}/entity-edit.mdl", "-p", "{P}"}, + {"run", "wait", "-p", "{P}"}, + }, weight: 20}, + {name: "stop", cmds: one("run", "stop", "-p", "{P}"), weight: 10}, + // A test run that boots its own runtime, as one does with no app up. + {name: "test-cold", cmds: one("test", "{S}/bench.test.mdl", "-p", "{P}", "--local", "--db-type", "hsqldb"), weight: 5}, + } + if !pageCheck { + steps = slices.DeleteFunc(steps, func(s step) bool { return s.name == "page-check" }) + } + return steps +} + +// prepareRuntime creates the project the running-app session starts from: +// `mxcli new` for the requested Mendix version, with the session's app +// authored into it. The fixture the other sessions use cannot be built — it +// carries no theme — so this one has to be made. `mxcli new` installs mxbuild +// when it is not cached; that is done first and separately, so the download +// is in neither the `new` figure nor the first boot. +func prepareRuntime(cfg config, scripts string) (string, []seedResult, error) { + untimed := func(what string, args ...string) error { + fmt.Fprintf(cfg.log, " %s\n", what) + out, err := exec.Command(cfg.candidate, args...).CombinedOutput() + if err != nil { + return fmt.Errorf("%s: %w\n%s", what, err, tail(string(out), 20)) + } + return nil + } + if err := untimed("installing mxbuild "+cfg.runtimeVersion+" if it is not cached", "setup", "mxbuild", "--version", cfg.runtimeVersion); err != nil { + return "", nil, err + } + dir := filepath.Join(cfg.work, "runtime-snapshot") + var seed []seedResult + start := time.Now() + if err := untimed("creating the project", "new", "BenchApp", "--version", cfg.runtimeVersion, "--output-dir", dir); err != nil { + return "", nil, err + } + seed = append(seed, seedResult{Name: "new", sample: sample{WallMS: float64(time.Since(start).Microseconds()) / 1000}}) + mpr, err := findMPR(dir) + if err != nil { + return "", nil, err + } + project := filepath.Join(dir, mpr) + s, err := runStep(cfg.candidate, [][]string{ + {"exec", "{S}/domain.mdl", "-p", "{P}"}, + {"exec", "{S}/microflows.mdl", "-p", "{P}"}, + {"exec", "{S}/pages.mdl", "-p", "{P}"}, + }, project, scripts, cfg.work) + if err != nil { + return "", nil, fmt.Errorf("authoring the app: %w", err) + } + seed = append(seed, seedResult{Name: "author", sample: s}) + if err := untimed("caching the runtime", "run", "--local", "--setup", "--db-type", "hsqldb", "-p", project); err != nil { + return "", nil, err + } + return dir, seed, nil +} + +// run is the whole benchmark: every requested session, in order. When a +// session fails, the ones that completed are still returned. func run(cfg config) ([]sessionResult, error) { scripts := filepath.Join(cfg.work, "scripts") if err := os.MkdirAll(scripts, 0o755); err != nil { @@ -561,27 +734,37 @@ func run(cfg config) ([]sessionResult, error) { if err := writeSession(scripts, cfg.sessionEntities); err != nil { return nil, err } + names := cfg.sizes + if cfg.runtimeVersion != "" { + names = append(slices.Clone(names), "runtime") + } var results []sessionResult var invariant bool - for _, size := range cfg.sizes { - fmt.Fprintf(cfg.log, "%s session\n", size) + for _, name := range names { + fmt.Fprintf(cfg.log, "%s session\n", name) + def := session{name: name, steps: sessionSteps(), reps: cfg.reps} var snapshot string var seed []seedResult - switch size { + var err error + switch name { case "small": snapshot = filepath.Join(cfg.work, "small-snapshot") - if err := copyProject(cfg.base, snapshot); err != nil { - return results, err - } + err = copyProject(cfg.base, snapshot) case "large": - var err error - if snapshot, seed, err = seedLarge(cfg, scripts); err != nil { - return results, err + snapshot, seed, err = seedLarge(cfg, scripts) + case "runtime": + def = session{ + name: name, steps: runtimeSteps(cfg.pageCheck), reps: cfg.runtimeReps, + armsInTurn: true, teardown: []string{"run", "stop", "-p", "{P}"}, } + snapshot, seed, err = prepareRuntime(cfg, scripts) default: - return results, fmt.Errorf("unknown size %q (want small or large)", size) + err = fmt.Errorf("unknown size %q (want small or large)", name) + } + if err != nil { + return results, err } - res, err := runSession(cfg, size, snapshot, scripts) + res, err := runSession(cfg, def, snapshot, scripts) res.Seed = seed if errors.Is(err, errInvariant) { invariant = true diff --git a/scripts/agentbench/bench_test.go b/scripts/agentbench/bench_test.go index f92f059c2..db5922541 100644 --- a/scripts/agentbench/bench_test.go +++ b/scripts/agentbench/bench_test.go @@ -25,13 +25,30 @@ import ( // failOn a command line containing this exits non-zero // writeOn a command line containing this rewrites a unit file with new bytes func fakeMxcli(t *testing.T, name, slowOn, failOn, writeOn string) string { + t.Helper() + return fakeMxcliLogging(t, name, slowOn, failOn, writeOn, "") +} + +// fakeMxcliLogging is fakeMxcli that also appends " " to +// callLog for every command, and creates a project for `new`, so the order in +// which the driver ran two binaries can be read back. +func fakeMxcliLogging(t *testing.T, name, slowOn, failOn, writeOn, callLog string) string { t.Helper() script := fmt.Sprintf(`#!/bin/sh if [ "$1" = version ]; then echo "mxcli version %[1]s"; exit 0; fi match() { [ -n "$1" ] && case "$ARGS" in *"$1"*) return 0;; esac; return 1; } ARGS="$*" +if [ -n %[5]q ]; then echo "%[1]s $ARGS" >> %[5]q; fi project="" -while [ $# -gt 0 ]; do if [ "$1" = -p ]; then project="$2"; fi; shift; done +outdir="" +while [ $# -gt 0 ]; do + if [ "$1" = -p ]; then project="$2"; fi + if [ "$1" = --output-dir ]; then outdir="$2"; fi + shift +done +if [ -n "$outdir" ]; then + mkdir -p "$outdir/mprcontents" && echo mpr > "$outdir/App.mpr" && echo unit > "$outdir/mprcontents/unit.mxunit" +fi sleep 0.03 if match %[2]q; then sleep 0.25; fi if match %[3]q; then echo "unknown statement" >&2; exit 3; fi @@ -40,7 +57,7 @@ if [ -n "$MXCLI_STATS_FILE" ]; then echo '{"unit_file_reads":7,"writes_offered":2,"writes_written":1}' >> "$MXCLI_STATS_FILE" fi echo ok -`, name, slowOn, failOn, writeOn) +`, name, slowOn, failOn, writeOn, callLog) path := filepath.Join(t.TempDir(), name) if err := os.WriteFile(path, []byte(script), 0o755); err != nil { t.Fatal(err) @@ -239,3 +256,92 @@ func TestCopyProject_LeavesTheCachesBehind(t *testing.T) { t.Errorf("the unit file was not copied: %v", err) } } + +// The running-app session leaves a runtime up between its steps, so the two +// arms must not be interleaved: one arm's whole session, its stop, then the +// other's. And the stop must happen whatever the session did — here the +// baseline fails half way — or the next boot finds the ports taken. +func TestRun_RuntimeSessionRunsArmsInTurnAndAlwaysStops(t *testing.T) { + t.Parallel() + callLog := filepath.Join(t.TempDir(), "calls.log") + candidate := fakeMxcliLogging(t, "candidate", "", "", "", callLog) + baseline := fakeMxcliLogging(t, "baseline", "", "entity-edit.mdl", "", callLog) + cfg := testConfig(t, candidate, baseline) + cfg.sizes, cfg.runtimeVersion, cfg.runtimeReps, cfg.pageCheck = nil, "11.13.0", 1, true + + sessions, err := run(cfg) + if err != nil { + t.Fatal(err) + } + if len(sessions) != 1 || sessions[0].Size != "runtime" { + t.Fatalf("sessions: %+v, want the runtime session alone", sessions) + } + s := sessions[0] + var seeds []string + for _, seed := range s.Seed { + seeds = append(seeds, seed.Name) + } + if !slices.Equal(seeds, []string{"new", "author"}) { + t.Errorf("seed steps %v, want the project's creation and its authoring", seeds) + } + + data, err := os.ReadFile(callLog) + if err != nil { + t.Fatal(err) + } + // The session proper starts at the first boot; before it is preparation. + var arms, stops []string + started := false + for _, line := range strings.Split(strings.TrimSpace(string(data)), "\n") { + arm, cmd, _ := strings.Cut(line, " ") + if strings.HasPrefix(cmd, "run --local --watch") { + started = true + } + if !started { + if arm != "candidate" { + t.Errorf("the baseline took part in preparing the project: %s", line) + } + continue + } + if len(arms) == 0 || arms[len(arms)-1] != arm { + arms = append(arms, arm) + } + if strings.HasPrefix(cmd, "run stop") { + stops = append(stops, arm) + } + } + if !slices.Equal(arms, []string{"candidate", "baseline"}) { + t.Errorf("arms ran in the order %v, want each arm's whole session in turn\n%s", arms, data) + } + // The candidate stops twice (its own stop step, then the teardown); the + // baseline never reached its stop step, so its one stop is the teardown. + if !slices.Equal(stops, []string{"candidate", "candidate", "baseline"}) { + t.Errorf("run stop was called by %v, want the baseline stopped after it failed\n%s", stops, data) + } + + if st := stepNamed(t, s, "hot-apply"); st.Baseline == nil { + t.Error("hot-apply ran before the baseline failed and has no baseline") + } else if one := stepNamed(t, s, "test-attach").Candidate.Wall.Median; st.Candidate.Wall.Median < 1.3*one { + // Two commands, the edit and the wait, measured as one step. + t.Errorf("hot-apply took %.0f ms against %.0f ms for a one-command step, want the two commands summed", st.Candidate.Wall.Median, one) + } + for _, name := range []string{"apply-entity", "stop", "test-cold"} { + if stepNamed(t, s, name).Baseline != nil { + t.Errorf("%s has a baseline measurement after the baseline failed", name) + } + } +} + +// Without the page check — a machine with no browser — the session is the +// same minus that step. +func TestRuntimeSteps_PageCheckIsOptional(t *testing.T) { + with, without := runtimeSteps(true), runtimeSteps(false) + if len(with) != len(without)+1 { + t.Fatalf("%d steps with the page check, %d without", len(with), len(without)) + } + for _, st := range without { + if st.name == "page-check" { + t.Error("page-check is still in the session") + } + } +} diff --git a/scripts/agentbench/gen/gen.go b/scripts/agentbench/gen/gen.go index 39489ce16..b5c77ada3 100644 --- a/scripts/agentbench/gen/gen.go +++ b/scripts/agentbench/gen/gen.go @@ -256,3 +256,33 @@ func BulkAlter(p Params) string { q.Revision = p.Revision + 1 return Microflows(q) } + +// EntityEdit is the smallest structural change: one attribute added to one +// entity. A running app applies it by restarting and updating its schema, +// where a microflow change is reloaded in place. +func EntityEdit(p Params) string { + return fmt.Sprintf("%salter entity %s.%s add attribute if not exists Remark: string(200);\n", + header, p.ModuleName(0), EntityName(0)) +} + +// Tests is a two-test suite over the first entity's validation microflow, in +// the statements a test endpoint runs without building anything. +func Tests(p Params) string { + mod, ent := p.ModuleName(0), EntityName(0) + return fmt.Sprintf(`/** + * @test A complete item passes validation + * @expect $result = true + */ +$item = create %[1]s.%[2]s (Name = 'Widget', Amount = 5, Price = 1); +$result = call microflow %[1]s.VAL_%[2]s_Save(Item = $item); +/ + +/** + * @test An item without a name is rejected + * @expect $result = false + */ +$item = create %[1]s.%[2]s (Amount = 5, Price = 1); +$result = call microflow %[1]s.VAL_%[2]s_Save(Item = $item); +/ +`, mod, ent) +} diff --git a/scripts/agentbench/gen/gen_test.go b/scripts/agentbench/gen/gen_test.go index 5a5b950be..651a889fd 100644 --- a/scripts/agentbench/gen/gen_test.go +++ b/scripts/agentbench/gen/gen_test.go @@ -119,8 +119,39 @@ func TestGeneratedScriptsExecuteAndSettleOnPedApp(t *testing.T) { t.Errorf("re-running the settled bulk change landed %d writes, want 0\n%s", got, out.String()) } + // The structural edit of the running-app session, and its re-run: the + // attribute is there the second time, and the statement says so itself. + before = written() + run("entity-edit", EntityEdit(session)) + if written() == before { + t.Error("adding an attribute landed no write") + } + before = written() + run("entity-edit again", EntityEdit(session)) + if got := written() - before; got != 0 { + t.Errorf("adding the same attribute again landed %d writes", got) + } + seed := Params{Prefix: "Seed", Modules: 2, Entities: 2} run("seed-domain", Domain(seed)) run("seed-microflows", Microflows(seed)) run("seed-pages", Pages(seed)) } + +// The test suite must name what the session authored, or every test in it +// fails for a reason that has nothing to do with speed. +func TestTestsNameTheFirstEntitysValidation(t *testing.T) { + p := Params{Prefix: "Bench", Modules: 1, Entities: 2} + suite := Tests(p) + if got := strings.Count(suite, "@test "); got != 2 { + t.Errorf("%d tests in the suite, want 2", got) + } + for _, want := range []string{"create Bench.Item01 (", "call microflow Bench.VAL_Item01_Save("} { + if !strings.Contains(suite, want) { + t.Errorf("the suite does not contain %q", want) + } + } + if !strings.Contains(Microflows(p), "create or modify microflow Bench.VAL_Item01_Save (") { + t.Error("the microflow the suite calls is not one the session creates") + } +} diff --git a/scripts/agentbench/main.go b/scripts/agentbench/main.go index d9b5f4458..3f4298733 100644 --- a/scripts/agentbench/main.go +++ b/scripts/agentbench/main.go @@ -21,6 +21,7 @@ import ( "flag" "fmt" "os" + "path/filepath" "strings" ) @@ -36,16 +37,21 @@ func main() { flag.IntVar(&cfg.large.Modules, "large-modules", 12, "modules seeded into the large project") flag.IntVar(&cfg.large.Entities, "large-entities", 10, "entities per seeded module (each brings 2 microflows and 2 pages)") flag.StringVar(&sizes, "sizes", "small,large", "project sizes to run the session at") + flag.StringVar(&cfg.runtimeVersion, "runtime-version", "", "also run the running-app session (boot, hot apply, tests) on a project created by `mxcli new` for this Mendix version, e.g. 11.13.0; installs mxbuild and the runtime if they are not cached") + flag.IntVar(&cfg.runtimeReps, "runtime-reps", 3, "repetitions of the running-app session per arm") + flag.BoolVar(&cfg.pageCheck, "page-check", true, "include the page-check step in the running-app session (needs node with playwright and a Chromium)") flag.Float64Var(&cfg.threshold, "threshold", 1.20, "wall-time ratio above which a step is flagged slower (and below its inverse, faster)") flag.StringVar(&out, "out", "", "write the JSON record here") flag.StringVar(&summary, "summary", "", "append the Markdown summary here (e.g. $GITHUB_STEP_SUMMARY); it is always printed") flag.Parse() cfg.large.Prefix = "Seed" - cfg.sizes = strings.Split(sizes, ",") + if sizes != "" { + cfg.sizes = strings.Split(sizes, ",") + } cfg.log = os.Stderr - if cfg.reps < 1 { - fatal(errors.New("-reps must be at least 1")) + if cfg.reps < 1 || cfg.runtimeReps < 1 { + fatal(errors.New("-reps and -runtime-reps must be at least 1")) } if cfg.work == "" { dir, err := os.MkdirTemp("", "agentbench-") @@ -54,8 +60,15 @@ func main() { } defer os.RemoveAll(dir) cfg.work = dir - } else if err := os.MkdirAll(cfg.work, 0o755); err != nil { - fatal(err) + } else { + abs, err := filepath.Abs(cfg.work) + if err != nil { + fatal(err) + } + cfg.work = abs + if err := os.MkdirAll(cfg.work, 0o755); err != nil { + fatal(err) + } } // A relative binary path would be resolved against each child's directory. var err error @@ -68,8 +81,10 @@ func main() { } } + // A session that fails does not take the completed ones with it: their + // record is written, and the run still ends in error. sessions, runErr := run(cfg) - if runErr != nil && !errors.Is(runErr, errInvariant) { + if runErr != nil && len(sessions) == 0 { fatal(runErr) } rec := newRecord(cfg, sessions) @@ -91,7 +106,7 @@ func main() { if runErr != nil { // The record is written first: the night an invariant breaks is the // night its numbers are wanted. - fatal(fmt.Errorf("%w: see the flagged steps above", runErr)) + fatal(runErr) } } diff --git a/scripts/agentbench/record.go b/scripts/agentbench/record.go index 54c747a1e..8345d2a8d 100644 --- a/scripts/agentbench/record.go +++ b/scripts/agentbench/record.go @@ -45,6 +45,7 @@ type params struct { LargeModules int `json:"large_modules"` LargeEntities int `json:"large_entities"` LargeDocuments int `json:"large_documents"` + RuntimeVersion string `json:"runtime_version,omitempty"` Threshold float64 `json:"threshold"` } @@ -59,6 +60,7 @@ func newRecord(cfg config, sessions []sessionResult) record { Base: filepath.Base(cfg.base), Reps: cfg.reps, SessionEntities: cfg.sessionEntities, LargeModules: cfg.large.Modules, LargeEntities: cfg.large.Entities, LargeDocuments: cfg.large.Documents(), Threshold: cfg.threshold, + RuntimeVersion: cfg.runtimeVersion, }, WeightsVersion: weightsVersion, Sessions: sessions, @@ -131,7 +133,7 @@ func (r record) markdown() string { if r.Baseline != "" { fmt.Fprintf(&sb, "- baseline: `%s`\n", r.Baseline) } - fmt.Fprintf(&sb, "- median of %d, %d CPUs", r.Params.Reps, r.Runner.CPUs) + fmt.Fprintf(&sb, "- %d CPUs", r.Runner.CPUs) if r.Runner.CPU != "" { fmt.Fprintf(&sb, " (%s)", r.Runner.CPU) } @@ -139,7 +141,11 @@ func (r record) markdown() string { ab := r.Baseline != "" for _, s := range r.Sessions { - fmt.Fprintf(&sb, "\n### %s project — %d units\n\n", s.Size, s.Units) + if s.Size == "runtime" { + fmt.Fprintf(&sb, "\n### running app, Mendix %s — %d units, median of %d\n\n", r.Params.RuntimeVersion, s.Units, s.Reps) + } else { + fmt.Fprintf(&sb, "\n### %s project — %d units, median of %d\n\n", s.Size, s.Units, s.Reps) + } for _, seed := range s.Seed { fmt.Fprintf(&sb, "- %s: %s (once, candidate only)\n", seed.Name, ms(seed.WallMS)) } From 8299d1c6806afbeb3f5dacb82a318d78ae1ecc4f Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 11 Nov 2026 22:49:49 +0000 Subject: [PATCH 6/8] fix(run): build as soon as exec has finished writing the model The watch loop noticed a change within a second and then waited for the model to stay unchanged for two more polls, so a change made by mxcli exec was applied about 5 s after exec started although the build and reload took 2 s. exec and -c now leave a pid-named marker while they run and touch a stamp when the last write has landed. The loop does not build while a live marker is there and builds within 100 ms of the stamp, without the settle. A writer that says nothing is settled as before. Closes ako/mxcli#2151 Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_011oK82sKAKMxKadjp4EraKh --- ...-watch-settle-delay-after-exec-exited.json | 1 + CHANGELOG.md | 1 + cmd/mxcli/cmd_exec.go | 7 + cmd/mxcli/docker/runlocal.go | 30 +++- cmd/mxcli/docker/watchwriter.go | 133 ++++++++++++++++++ cmd/mxcli/docker/watchwriter_test.go | 101 +++++++++++++ cmd/mxcli/oneliner.go | 5 + 7 files changed, 276 insertions(+), 2 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/cmd-mxcli/2026-11-11-watch-settle-delay-after-exec-exited.json create mode 100644 cmd/mxcli/docker/watchwriter.go create mode 100644 cmd/mxcli/docker/watchwriter_test.go diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-11-watch-settle-delay-after-exec-exited.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-11-watch-settle-delay-after-exec-exited.json new file mode 100644 index 000000000..cdba58fe1 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-11-watch-settle-delay-after-exec-exited.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-11-11", "symptom": "With `run --local --watch` serving an app, a change made by `mxcli exec` was applied 4.2-5.2 s after exec had exited, although the log line said \"applied via reload in 2.0s\". Measured on a blank Mendix 11.16.0 app over eleven changes.", "cause": "The watch loop cannot see a write end. It polled the source mtime once a second and then waited for two more unchanged polls (settleSource) before building, so that a multi-file exec is never built half-written. For a writer that has already exited that is about 2.7 s of waiting for nothing.", "file": "cmd/mxcli/docker/watchwriter.go (BeginModelWrite, watchWriterGate), cmd/mxcli/docker/runlocal.go (watchAndApply wakes every 100 ms for the signal and skips the settle after one), cmd/mxcli/cmd_exec.go and cmd/mxcli/oneliner.go (the callers)", "insight": "The duration a loop prints for its own work is not the latency its user feels: subtract the two and the difference is where to look. The settle was the right answer to a writer that cannot say it is done and the wrong one for mxcli's own commands, which can. A pid-named marker plus a stamp covers the three cases without a protocol: marker present and pid alive is 'writing, do not build', stamp changed is 'done, build now', marker present and pid dead is 'done, however it ended'. Register the signal's defer before the executor's Close so it runs after the last write. Control, same app and session: an unsignalled write (touching the .mpr) still took 4.2 s, a signalled exec 2.2-2.5 s; an exec that changed nothing started no build.", "refs": ["ako/mxcli#2151"], "test": "cmd/mxcli/docker/watchwriter_test.go"} diff --git a/CHANGELOG.md b/CHANGELOG.md index d5af6761b..66bfa3536 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **`run --local --watch` builds a change as soon as `mxcli exec` or `mxcli -c` has finished writing it** (ako/mxcli#2151) — the watcher noticed a change within a second and then waited for the model to stay unchanged for two more, so a change was applied about 5 s after `exec` started although the build and reload took 2 s. `exec` and `-c` now tell a watching dev loop when they start and when the last write has landed: nothing is built while one is at work, however long the script, and the build starts within 100 ms of its end. Measured on a blank 11.16.0 app: 2.2-2.5 s from the start of `exec` to applied, for a microflow edit, a page edit and a new page. A change made by anything else (an editor, Studio Pro) is settled as before. - **`run --local --watch` applies a change in about 5 s instead of 20-28 s on Mendix versions that bundle the client themselves** (ako/mxcli#2150) — measured on a blank 11.16.0 app: a microflow edit took 20-22 s and a page edit 25-28 s with a runtime restart that dropped every browser session, because each change cost a fresh `mxbuild --serve` and a cold build. mxbuild deletes the client source a build wrote, so its next build bundled a client without pages. That source is now kept while a build runs and put back before the next one: the incremental build takes about 2 s, a page edit is a reload, and an open browser tab shows it without a restart or a new login. The full build remains the fallback when the bundle still comes out incomplete. A deleted page's module stays in `dist/pages`, unused, until the next full build. On Windows the copies are read without blocking mxbuild's own deletes, and if a build still fails because `deployment/web` was in use, the kept source is given up for that run and builds go back to the full rebuild. - **`check` accepts the data-container names Studio Pro stores in a grid row, a control bar and a combo box option** (ako/mxcli#2109) — `describe page` → `exec` of four FactoryManagement pages was refused on its own output (MDL-BUTTON02, MDL-PAGEARG01): a show page in a selectable data grid's column passing the grid's selection (`$grid1`), a control-bar button passing the data view around its grid, and a combo box's option content reading the data view around the combo box. All three build in Studio Pro's own pages. `check` and `exec` now accept them and write what Studio Pro stores, and the pages report `Unchanged page`. - **A binding to an outer data view keeps that data view** (ako/mxcli#2107) — a markdown viewer or an attribute-based visibility condition inside a nested data view, bound to the outer one's object, was described as a bare qualified name and rewritten against the nearest data view: CE7247, or "has no attribute". `$dvOuter.Attr` and `$dvOuter/Assoc/Attr` now work on a pluggable widget's attribute property and in `Visible: … in (…)`, as on input widgets, and `describe` prints them; the six FactoryManagement pages report `Unchanged page`. diff --git a/cmd/mxcli/cmd_exec.go b/cmd/mxcli/cmd_exec.go index 73d1579f3..af7e356f9 100644 --- a/cmd/mxcli/cmd_exec.go +++ b/cmd/mxcli/cmd_exec.go @@ -5,6 +5,7 @@ package main import ( "errors" "fmt" + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" "os" "path/filepath" @@ -93,6 +94,12 @@ Example: os.Exit(1) } + // A watching dev loop is told when this script starts writing and when + // the last write has landed, so it builds at once instead of waiting + // for the model to go quiet (ako/mxcli#2151). Deferred first, so it + // runs after exec.Close below. + defer docker.BeginModelWrite(projectPath)() + exec, logger := newLoggedExecutor("exec") defer logger.Close() defer exec.Close() diff --git a/cmd/mxcli/docker/runlocal.go b/cmd/mxcli/docker/runlocal.go index a1660afde..ff92bd9fb 100644 --- a/cmd/mxcli/docker/runlocal.go +++ b/cmd/mxcli/docker/runlocal.go @@ -1531,7 +1531,14 @@ func watchAndApply(opts LocalRunOptions, serve *ServeServer, rt *LocalRuntime, b holds := &watchHoldGate{projectPath: opts.ProjectPath} holdAnnounced := false resync := false - ticker := time.NewTicker(opts.PollInterval) + // The loop wakes at the writer-signal interval and looks at the source only + // every PollInterval, unless a writer has just said it is done + // (watchwriter.go). + writers := newWatchWriterGate(opts.ProjectPath) + writerDone := false + wake := min(opts.PollInterval, watchWriterPoll) + nextScan := time.Now().Add(opts.PollInterval) + ticker := time.NewTicker(wake) defer ticker.Stop() for { @@ -1546,6 +1553,18 @@ func watchAndApply(opts LocalRunOptions, serve *ServeServer, rt *LocalRuntime, b // re-arms this channel, so only an unasked-for exit lands here. return runtimeStoppedError(rt) case <-ticker.C: + // An mxcli command that is writing the model says so: nothing is + // built while it is at work, and its change is built as soon as it + // is done instead of after the settle. + writing, finished := writers.poll() + if writing { + continue + } + writerDone = writerDone || finished + if !writerDone && time.Now().Before(nextScan) { + continue + } + nextScan = time.Now().Add(opts.PollInterval) source := func() time.Time { return sourceMTimeExcept(opts.ProjectPath, scaffold) } now := source() // Another mxcli process (a `test --attach`) may be changing the @@ -1579,6 +1598,8 @@ func watchAndApply(opts LocalRunOptions, serve *ServeServer, rt *LocalRuntime, b holdAnnounced = false resync = true } + settled := writerDone + writerDone = false if !now.After(last) && !resync { continue } @@ -1594,7 +1615,12 @@ func watchAndApply(opts LocalRunOptions, serve *ServeServer, rt *LocalRuntime, b // the stale build is what you are left with, with no way out the tool // offers. Settling first means the watcher can only ever observe a // model that has stopped changing. - now = settleSource(source, now, opts.PollInterval, sigCh) + // + // A writer that announced itself and has finished is the exception: + // the model has stopped changing because the writer says so. + if !settled { + now = settleSource(source, now, opts.PollInterval, sigCh) + } if now.IsZero() { // Interrupted while settling. fmt.Fprintln(w, "\nShutting down...") diff --git a/cmd/mxcli/docker/watchwriter.go b/cmd/mxcli/docker/watchwriter.go new file mode 100644 index 000000000..5e84c6a60 --- /dev/null +++ b/cmd/mxcli/docker/watchwriter.go @@ -0,0 +1,133 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "os" + "path/filepath" + "strconv" + "time" +) + +// watchwriter.go lets an mxcli command that writes the model tell a +// `run --local --watch` dev loop when it starts and when it is done +// (ako/mxcli#2151). +// +// The watcher cannot see a write end, so it waits for the model source to stay +// unchanged for two polls before it builds (settleSource), on top of up to one +// poll to notice the change at all. Measured on a blank Mendix 11.16.0 app over +// eleven changes: 4.2-5.2 s from the end of `mxcli exec` to "applied", of which +// the build and the reload were 1.9-2.5 s. The rest was the watcher waiting for +// a writer that had already exited. +// +// `exec` knows both moments. While it runs it leaves a marker named after its +// pid; the watcher does not build while a live marker is there, which holds +// for a script of any length where the settle only guesses. When it is done it +// removes the marker and touches a stamp, and the watcher builds at once. +// +// Nothing changes for a writer that says nothing — an editor, Studio Pro, an +// older mxcli: its change is noticed and settled as before. A writer that dies +// without cleaning up is a marker whose pid is gone, which reads as done. + +// watchWriterPoll is how often the watcher looks for a writer's signal. It is +// a ReadDir of a directory that is almost always empty. +const watchWriterPoll = 100 * time.Millisecond + +const ( + watchWriterDirName = "watch-writers" + watchWriterStamp = "done" +) + +func watchWriterDir(projectPath string) string { + return filepath.Join(filepath.Dir(projectPath), ".mxcli", watchWriterDirName) +} + +// watchRunLive reports whether a `run --local --watch` is serving the project. +// A seam for tests. +var watchRunLive = func(projectPath string) bool { + st, err := ReadRunState(projectPath) + return err == nil && st != nil && st.Watch && st.Ended.IsZero() && PidAlive(st.PID) +} + +// BeginModelWrite tells a watching dev loop that this process is about to +// change the project, and returns the function that tells it the change is +// complete. Without a watching dev loop it does nothing and leaves no file. +// +// Call the returned function once the last write has landed. A process that +// exits without calling it is noticed by its pid, and its change is then built +// after the usual settle. +func BeginModelWrite(projectPath string) (done func()) { + if projectPath == "" || !watchRunLive(projectPath) { + return func() {} + } + dir := watchWriterDir(projectPath) + if err := os.MkdirAll(dir, 0o755); err != nil { + return func() {} + } + marker := filepath.Join(dir, strconv.Itoa(os.Getpid())) + if err := os.WriteFile(marker, nil, 0o644); err != nil { + return func() {} + } + return func() { + // The stamp first: with the marker gone and no stamp yet, a poll in + // between would see a writer that vanished without a word. + stamp := filepath.Join(dir, watchWriterStamp) + now := time.Now() + if err := os.WriteFile(stamp, []byte(now.Format(time.RFC3339Nano)), 0o644); err == nil { + _ = os.Chtimes(stamp, now, now) + } + _ = os.Remove(marker) + } +} + +// watchWriterGate is the watcher's side. +type watchWriterGate struct { + projectPath string + stamp time.Time // the stamp as of the last poll with no writer + sawWriter bool // a live writer was seen since then +} + +// newWatchWriterGate starts from the stamp as it is, so a writer that finished +// before the loop began watching is not taken for a new one. +func newWatchWriterGate(projectPath string) *watchWriterGate { + g := &watchWriterGate{projectPath: projectPath} + g.stamp = g.readStamp() + return g +} + +func (g *watchWriterGate) readStamp() time.Time { + fi, err := os.Stat(filepath.Join(watchWriterDir(g.projectPath), watchWriterStamp)) + if err != nil { + return time.Time{} + } + return fi.ModTime() +} + +// poll reports whether a writer is at work now, and whether one finished since +// the last poll that reported none at work. finished is reported once. +func (g *watchWriterGate) poll() (writing, finished bool) { + dir := watchWriterDir(g.projectPath) + entries, _ := os.ReadDir(dir) + for _, e := range entries { + pid, err := strconv.Atoi(e.Name()) + if err != nil { + continue // the stamp + } + if holdAlive(pid) { + writing = true + continue + } + // Its owner is gone without removing it: the write is over, however + // it ended. + _ = os.Remove(filepath.Join(dir, e.Name())) + g.sawWriter = true + } + if writing { + g.sawWriter = true + return true, false + } + stamp := g.readStamp() + finished = g.sawWriter || !stamp.Equal(g.stamp) + g.stamp, g.sawWriter = stamp, false + return false, finished +} diff --git a/cmd/mxcli/docker/watchwriter_test.go b/cmd/mxcli/docker/watchwriter_test.go new file mode 100644 index 000000000..637aff29b --- /dev/null +++ b/cmd/mxcli/docker/watchwriter_test.go @@ -0,0 +1,101 @@ +// SPDX-License-Identifier: Apache-2.0 + +// The watcher waited ~2.7 s after `mxcli exec` had exited before it built, +// because it cannot see a write end and settles instead (ako/mxcli#2151). +package docker + +import ( + "os" + "path/filepath" + "strconv" + "testing" +) + +func writerProject(t *testing.T, watching bool) string { + t.Helper() + project := filepath.Join(t.TempDir(), "App.mpr") + prev := watchRunLive + watchRunLive = func(string) bool { return watching } + t.Cleanup(func() { watchRunLive = prev }) + return project +} + +// The signal the watcher builds on: nothing while the writer runs, "finished" +// exactly once when it is done. +func TestWatchWriter_FinishedIsReportedOnceAfterTheWrite(t *testing.T) { + project := writerProject(t, true) + g := newWatchWriterGate(project) + + if writing, finished := g.poll(); writing || finished { + t.Fatalf("no writer yet: poll = writing %v, finished %v", writing, finished) + } + done := BeginModelWrite(project) + if writing, finished := g.poll(); !writing || finished { + t.Fatalf("while the writer runs the watcher must wait: poll = writing %v, finished %v", writing, finished) + } + done() + if writing, finished := g.poll(); writing || !finished { + t.Fatalf("after the write: poll = writing %v, finished %v, want finished", writing, finished) + } + if writing, finished := g.poll(); writing || finished { + t.Fatalf("finished must be reported once: poll = writing %v, finished %v", writing, finished) + } +} + +// A write that begins and ends between two polls is still seen: the stamp +// outlives the marker. +func TestWatchWriter_WriteBetweenTwoPollsIsSeen(t *testing.T) { + project := writerProject(t, true) + g := newWatchWriterGate(project) + g.poll() + + BeginModelWrite(project)() + if writing, finished := g.poll(); writing || !finished { + t.Fatalf("poll = writing %v, finished %v, want finished", writing, finished) + } +} + +// A writer that exited without saying so is over all the same; the watcher +// must not wait on it for ever. +func TestWatchWriter_DeadWriterCountsAsFinished(t *testing.T) { + project := writerProject(t, true) + g := newWatchWriterGate(project) + dir := watchWriterDir(project) + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + marker := filepath.Join(dir, strconv.Itoa(424242)) + if err := os.WriteFile(marker, nil, 0o644); err != nil { + t.Fatal(err) + } + prev := holdAlive + holdAlive = func(int) bool { return false } + t.Cleanup(func() { holdAlive = prev }) + + if writing, finished := g.poll(); writing || !finished { + t.Fatalf("poll = writing %v, finished %v, want finished", writing, finished) + } + if _, err := os.Stat(marker); !os.IsNotExist(err) { + t.Fatalf("a dead writer's marker must be removed: %v", err) + } +} + +// A stamp left by an earlier run is not a new write. +func TestWatchWriter_StampFromBeforeTheLoopIsNotASignal(t *testing.T) { + project := writerProject(t, true) + BeginModelWrite(project)() + + g := newWatchWriterGate(project) + if writing, finished := g.poll(); writing || finished { + t.Fatalf("poll = writing %v, finished %v, want neither", writing, finished) + } +} + +// Without a watching dev loop, a write leaves nothing behind in .mxcli/. +func TestWatchWriter_NoWatchingLoopNoFiles(t *testing.T) { + project := writerProject(t, false) + BeginModelWrite(project)() + if _, err := os.Stat(watchWriterDir(project)); !os.IsNotExist(err) { + t.Fatalf("no dev loop is watching, so no marker directory should exist: %v", err) + } +} diff --git a/cmd/mxcli/oneliner.go b/cmd/mxcli/oneliner.go index fef5bf553..19ff0be48 100644 --- a/cmd/mxcli/oneliner.go +++ b/cmd/mxcli/oneliner.go @@ -5,6 +5,7 @@ package main import ( "errors" "fmt" + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" "io" "os" "strings" @@ -37,6 +38,10 @@ func runCommandLine(cmd *cobra.Command, commands, projectPath string, continueOn return 1 } + // As in `exec`: a watching dev loop builds as soon as these statements + // are done (ako/mxcli#2151). Deferred first, so it runs after exec.Close. + defer docker.BeginModelWrite(projectPath)() + exec, logger := newLoggedExecutor("batch") defer logger.Close() defer exec.Close() From 79ba0eae5eb6f6da8732e20aed6f8149bca2a226 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 11 Nov 2026 23:12:30 +0000 Subject: [PATCH 7/8] =?UTF-8?q?feat(bench):=20validate=20and=20build=20ste?= =?UTF-8?q?ps=20=E2=80=94=20docker=20check,=20docker=20build?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 3 of PROPOSAL_agent_devloop_benchmark. The running-app session now starts with mx-check (mxcli docker check) and build (mxcli docker build --skip-check, so the check is not timed twice), on the same buildable project the boot uses. Neither needs a Docker daemon. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_011oK82u4HRCKfX2BBvhgsNm --- CHANGELOG.md | 2 +- docs-site/src/internals/agent-benchmark.md | 12 ++++++++---- .../PROPOSAL_agent_devloop_benchmark.md | 16 +++++++++++++++- scripts/agentbench/bench.go | 14 +++++++++++--- scripts/agentbench/bench_test.go | 19 +++++++++++++++++++ 5 files changed, 54 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ec78c8b81..4d82cbf75 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added -- **`make bench`: a benchmark of what an agent waits for, and `MXCLI_STATS_FILE`** — a fixed session of mxcli commands (open, author entities, microflows and pages, describe, catalog refresh, check, a one-document edit, a bulk change, a re-run, a rename), one process per command, on PedApp and on PedApp with a generated app seeded into it. `make bench BENCH_RUNTIME=` adds a session on a running app — boot, a hot change, an entity change, a page check, an attached and a cold test run — on a project `mxcli new` creates, with mxbuild and the runtime installed by mxcli when they are not cached. With `BASELINE=` the two binaries run interleaved on the same machine and each step reports the ratio; the nightly `bench` job does that against the previous nightly and commits the record to the `benchmarks` branch. It also records counts that do not depend on the machine: unit files read and unit files changed. `MXCLI_STATS_FILE=` makes any mxcli command append one JSON line per closed project connection with its unit-file reads and its writes offered and landed. See "Agent Dev-Loop Benchmark" in the docs. +- **`make bench`: a benchmark of what an agent waits for, and `MXCLI_STATS_FILE`** — a fixed session of mxcli commands (open, author entities, microflows and pages, describe, catalog refresh, check, a one-document edit, a bulk change, a re-run, a rename), one process per command, on PedApp and on PedApp with a generated app seeded into it. `make bench BENCH_RUNTIME=` adds a session that validates and builds the project with mx and then runs it — `docker check`, `docker build`, boot, a hot change, an entity change, a page check, an attached and a cold test run — on a project `mxcli new` creates, with mxbuild and the runtime installed by mxcli when they are not cached. With `BASELINE=` the two binaries run interleaved on the same machine and each step reports the ratio; the nightly `bench` job does that against the previous nightly and commits the record to the `benchmarks` branch. It also records counts that do not depend on the machine: unit files read and unit files changed. `MXCLI_STATS_FILE=` makes any mxcli command append one JSON line per closed project connection with its unit-file reads and its writes offered and landed. See "Agent Dev-Loop Benchmark" in the docs. - **`mxcli test --local --reuse-build` skips the injection and the build when nothing changed** (mendixlabs/mxcli#1783, closes mendixlabs/mxcli#1782) — a local test run injects its test microflows, builds and boots; when the project, the test files and mxcli are unchanged, the deployment the previous run left is already this run's build, so `--reuse-build` boots it directly (about 12 s against 60 s, measured by the author). The decision is a content hash of the project folder (outputs, caches, `theme-cache/`, and any `.git`/`node_modules` left out), the generated test MDL, the mxcli binary and the mxbuild override, pinned to the identity of `deployment/model/model.mdp`, so a deployment rebuilt by anything else is never taken for the test build. Not under `--watch` or `--skip-build`. - **A `genai-runtime` skill for the runtime side of Agents Kit 2** (ako/mxcli#2068) — registering a Mendix Cloud GenAI key or an OpenAI-compatible endpoint, seeding AgentCommons agents from JSON (matched by `Title` only, so a changed JSON never reaches a stored agent), resuming Conversational UI chats, loading PgVector and Mendix Cloud knowledge bases, and traces. It collects the gotchas that `check` and `mx check` cannot see: the encryption key both the OpenAI and PgVector setup need, the `vector` extension only the UI's Test button creates, the 60–120 s indexing delay, the chat that collapses outside its page, and `StoreTraces` defaulting to false. Its examples pass `check --references` and build with 0 errors against the Agent Builder Starter App's modules. - **`mxcli test --attach` runs a test without building it, against an app started with `--allow-call`** (mendixlabs/mxcli#1788, mendixlabs/mxcli#1789) — `run --local --test-endpoint --allow-call` switches on an executor in the test endpoint. A `.test.mdl` test whose statements it supports (`declare`, `set`, `create`, `change`, `commit`, `delete`, `call microflow`, database `retrieve` with `where`/`first`; `@expect` with `count`, `@throws`, `@setup`, `@cleanup`) is compiled by mxcli into steps and run in the app with the Mendix runtime API, every expression evaluated by `Core.evaluateExpression`. Nothing is injected or built, so a test edit costs one HTTP round trip: on a small app the suite took 0.26 s against 3.2 s for inject-and-build, with identical verdicts. Retrieves over an association and the list aggregates `count`/`sum`/`average`/`minimum`/`maximum` over an attribute run there too. A test with any other statement is injected and built as before. Only that test is, while the rest of the run still goes straight to the app, and the output names the test and the MDL statement (`the statement if`, `a list operation (find)`). Because the runtime evaluator returns empty for an unknown name, an undeclared `$variable` and a non-existent enumeration value are refused (ERROR) before evaluating, and a model newer than the running build is refused with `mxcli run wait` as the fix under `--watch`. The route takes POST, needs the run's token and a loopback caller, and answers 403 without `--allow-call`, since it calls the microflows a test names in a system context. `.test.mdl` stays the only test language. Design discussed with the authors of #1789; measurements in `mdl-examples/spikes/evaluate-expression.mdl`. diff --git a/docs-site/src/internals/agent-benchmark.md b/docs-site/src/internals/agent-benchmark.md index 3ef8a482f..79a94f81e 100644 --- a/docs-site/src/internals/agent-benchmark.md +++ b/docs-site/src/internals/agent-benchmark.md @@ -71,12 +71,14 @@ because the project around it is bigger. ## The running-app session -`BENCH_RUNTIME=` adds a third session: what an agent waits for -once the app is up. Measured sessions spend more wall time here than in every +`BENCH_RUNTIME=` adds a third session: validating and building +the project with mx, and what an agent waits for once the app is up. Measured sessions spend more wall time here than in every mxcli-only command together. | Step | Commands | What it shows | |---|---|---| +| `mx-check` | `docker check` | validation by mx itself, on a temporary copy of the project | +| `build` | `docker build --skip-check` | the full build to a deployable package | | `boot` | `run --local --watch --detach --db-type hsqldb --test-endpoint --allow-call` | until the app serves: build, runtime start, schema | | `page-check` | `playwright check` | a text verdict for the home page | | `hot-apply` | `exec` of a one-microflow change, then `run wait` | a behavioural change, live without a restart | @@ -105,6 +107,8 @@ It differs from the other two sessions in four ways: alone and says little here. A real mxcli regression is a smaller share of a noisier figure than in the other sessions — with the same code in both arms the ratios ran from 0.92 to 1.04 over three repetitions. +- **Neither `mx-check` nor `build` needs a Docker daemon**, despite the + command names: both run mx and MxBuild directly. `boot` and `stop` each report 7 files changed: the test endpoint's documents, added when the app starts and removed when it stops. @@ -199,8 +203,8 @@ record is still written and committed. ## What it does not measure -- **Validation and the full build** — `docker check` and `docker build`, the - next phase of the proposal. +- **A check or build after a change** — `mx-check` and `build` run once, on + the project as authored, before the app boots. - **A PostgreSQL-backed app** — the running-app session uses HSQLDB. - **`mxcli new` as an A/B** — it is timed once, by the candidate, as a trend figure. diff --git a/docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md b/docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md index 5f68cc150..5ca0eac11 100644 --- a/docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md +++ b/docs/11-proposals/PROPOSAL_agent_devloop_benchmark.md @@ -10,7 +10,7 @@ related: # Proposal: Nightly Agent Dev-Loop Performance Benchmark -**Status:** Partial — Phases 1 and 2 are implemented (see "Phase 1 as built", "Phase 2 as built") +**Status:** Partial — Phases 1 to 3 are implemented (see the "as built" sections); the trend chart and the alert are not **Date:** 2026-11-11 ## Problem Statement @@ -344,6 +344,20 @@ job. Steps: `boot`, `page-check`, `hot-apply`, `test-attach`, `apply-entity`, Adds `mx-check` and `build`. +#### Phase 3 as built + +Two steps at the head of the running-app session, since they need the same +buildable project: `mx-check` (`mxcli docker check`) and `build` +(`mxcli docker build --skip-check`, so the check is not timed twice). Neither +needs a Docker daemon. First figures on the development machine (Mendix +11.13.0, 476 units): check 8 s, build 73 s — which makes `build` the longest +step of the session by a factor of two, and adds about 75 s to every +repetition of each arm. + +Both run once per repetition, on the project as authored. A check after each +change — how an agent actually uses it — is not measured separately; the +session-weighted total stands in for it through the step's weight. + ### Later, separately — a scheduled agent-level run `mxcli eval run` on `BENCH-001` is the agent-level counterpart and already diff --git a/scripts/agentbench/bench.go b/scripts/agentbench/bench.go index 194d6193a..a400d132a 100644 --- a/scripts/agentbench/bench.go +++ b/scripts/agentbench/bench.go @@ -33,8 +33,8 @@ const statsFileEnv = "MXCLI_STATS_FILE" type step struct { name string // cmds are the command lines after the binary, run one after the other - // and measured as one: {P} is the project's .mpr and {S} the directory - // holding the session's scripts. Most steps are one command. A change to a + // and measured as one: {P} is the project's .mpr, {D} its directory and + // {S} the directory holding the session's scripts. Most steps are one command. A change to a // running app is two, the edit and the wait for it to be live, because // that pair is what the agent sits through. cmds [][]string @@ -146,6 +146,7 @@ func runStep(bin string, cmds [][]string, project, scripts, scratch string) (sam resolved := make([]string, len(args)) for i, a := range args { a = strings.ReplaceAll(a, "{P}", project) + a = strings.ReplaceAll(a, "{D}", filepath.Dir(project)) resolved[i] = strings.ReplaceAll(a, "{S}", scripts) } statsSeq++ @@ -651,9 +652,16 @@ func seedLarge(cfg config, scripts string) (string, []seedResult, error) { // runtimeSteps is the running-app session: what an agent waits for once the // app is up. Weights are rounded from the same measured builds as the others -// (`run wait` 95 calls in the largest, restarts 150, microflow test runs 35). +// (`run wait` 95 calls in the largest, restarts 150, `docker check` 84, +// microflow test runs 35). func runtimeSteps(pageCheck bool) []step { steps := []step{ + // Validation by mx itself, on a temporary copy of the project: what an + // agent runs after a batch of changes, before it trusts them. + {name: "mx-check", cmds: one("docker", "check", "-p", "{P}"), weight: 80}, + // The full build to a deployable package. --skip-check, or it would + // time the check a second time. + {name: "build", cmds: one("docker", "build", "-p", "{P}", "--skip-check", "-o", "{D}/bench-package"), weight: 2}, // Until the app serves: the build, the runtime start, the schema. {name: "boot", cmds: one("run", "--local", "--watch", "--detach", "--db-type", "hsqldb", "--test-endpoint", "--allow-call", "-p", "{P}"), weight: 10}, diff --git a/scripts/agentbench/bench_test.go b/scripts/agentbench/bench_test.go index db5922541..a5e612d11 100644 --- a/scripts/agentbench/bench_test.go +++ b/scripts/agentbench/bench_test.go @@ -332,6 +332,25 @@ func TestRun_RuntimeSessionRunsArmsInTurnAndAlwaysStops(t *testing.T) { } } +// The session validates and builds before it boots, and the build does not +// repeat the check it follows. +func TestRuntimeSteps_ValidateAndBuildComeBeforeBoot(t *testing.T) { + var names []string + for _, st := range runtimeSteps(true) { + names = append(names, st.name) + } + if want := []string{"mx-check", "build", "boot"}; len(names) < 3 || !slices.Equal(names[:3], want) { + t.Fatalf("the session starts %v, want %v", names, want) + } + build := runtimeSteps(true)[1].cmds[0] + if !slices.Contains(build, "--skip-check") { + t.Errorf("the build step %v runs the check again, which mx-check already timed", build) + } + if !slices.Contains(build, "{D}/bench-package") { + t.Errorf("the build step %v does not write its package inside the arm's own project copy", build) + } +} + // Without the page check — a machine with no browser — the session is the // same minus that step. func TestRuntimeSteps_PageCheckIsOptional(t *testing.T) { From 24c0aab9a5b3fb88e95adb21310b0a8a014b62d7 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 11 Nov 2026 23:22:13 +0000 Subject: [PATCH 8/8] ci(nightly): a manual run can be the benchmark job alone A workflow_dispatch of the whole nightly from a branch also publishes that branch's binaries as the nightly release. The 'only' input runs the bench job by itself; a scheduled run has no inputs and runs everything. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_011oK82u4HRCKfX2BBvhgsNm --- .github/workflows/nightly.yml | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index b07dc08b9..05e20690c 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -5,12 +5,23 @@ on: # Every day at 02:00 UTC - cron: '0 2 * * *' workflow_dispatch: # Allow manual trigger + inputs: + # A manual run of everything from a branch would also publish that + # branch's binaries as the `nightly` release. `bench` runs the benchmark + # job alone, which is how a change to it is tried before it merges. + only: + description: 'Run every job, or the benchmark alone' + type: choice + options: [all, bench] + default: all permissions: contents: write jobs: test: + # On a schedule `inputs.only` is empty, so every job runs. + if: inputs.only != 'bench' runs-on: ubuntu-latest strategy: matrix: @@ -78,6 +89,7 @@ jobs: # in the job log. Once this job is green, fold the version into the `test` # matrix and delete this job. test-mendix-9: + if: inputs.only != 'bench' runs-on: ubuntu-latest continue-on-error: true env: @@ -135,6 +147,7 @@ jobs: # on demand with workflow_dispatch. Like upgrade-full-corpus it does not gate # the release: a nightly is not held back by a macOS queue flake. tunnel-seam-macos: + if: inputs.only != 'bench' runs-on: macos-latest timeout-minutes: 30 steps: @@ -152,6 +165,7 @@ jobs: # Mendix version — it runs on the committed PedApp fixture and never calls # mx — so it runs once, not per matrix entry. It does not gate the release. upgrade-full-corpus: + if: inputs.only != 'bench' runs-on: ubuntu-latest strategy: fail-fast: false @@ -309,6 +323,7 @@ jobs: nightly: needs: test + if: inputs.only != 'bench' runs-on: ubuntu-latest steps: - uses: actions/checkout@v7