Repository navigation
Sync ako/mxcli: nightly agent dev-loop benchmark; run --watch applies an exec change without the settle wait - #1826
Merged
Merged
Conversation
A deterministic, scripted benchmark of the tool-side time an agent waits for: one mxcli process per step, A/B against the previous nightly binary on the same runner, a generated large fixture, and results kept on a dedicated benchmarks branch. Complements the agent-level BENCH-001 run and the diag reports in PROPOSAL_agent_loop_efficiency. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011oK82u4HRCKfX2BBvhgsNm
…et tests Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011oK82u4HRCKfX2BBvhgsNm
Each project connection appends one JSON line when it closes: unit files read, unit writes offered and landed. These are the counts the read- and decode-budget tests assert on in process; the agent dev-loop benchmark needs them from outside, per command, and a count does not depend on the machine it was taken on. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011oK82u4HRCKfX2BBvhgsNm
scripts/agentbench runs a fixed session of mxcli commands, one process per command, on PedApp and on PedApp with a generated app seeded into it. Given a baseline binary it runs both interleaved on the same machine and reports each step's ratio, with unit files read and unit files changed as machine-independent counts. make bench runs it; the nightly bench job runs it against the previous nightly release and commits the record to the benchmarks branch. It does not gate the release. Phase 1 of PROPOSAL_agent_devloop_benchmark: no mxbuild, no runtime. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011oK82u4HRCKfX2BBvhgsNm
Phase 2 of PROPOSAL_agent_devloop_benchmark. -runtime-version (make bench BENCH_RUNTIME=<version>) adds a session on a running app: boot, a page check, a microflow change applied hot, an attached test run, an entity change applied by restart, stop, and a cold test run. The project is created by mxcli new, since the PedApp fixture cannot be built; mxcli installs mxbuild and the runtime when they are not cached. The arms take turns in this session instead of alternating step by step, because a boot leaves a runtime up, and each arm's app is stopped whatever happened. A change to the running app is one step of two commands, the exec and the run wait. A session that fails no longer drops the record of the ones that completed. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011oK82u4HRCKfX2BBvhgsNm
The watch loop noticed a change within a second and then waited for the model to stay unchanged for two more polls, so a change made by mxcli exec was applied about 5 s after exec started although the build and reload took 2 s. exec and -c now leave a pid-named marker while they run and touch a stamp when the last write has landed. The loop does not build while a live marker is there and builds within 100 ms of the stamp, without the settle. A writer that says nothing is settled as before. Closes #2151 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011oK82sKAKMxKadjp4EraKh
Phase 3 of PROPOSAL_agent_devloop_benchmark. The running-app session now starts with mx-check (mxcli docker check) and build (mxcli docker build --skip-check, so the check is not timed twice), on the same buildable project the boot uses. Neither needs a Docker daemon. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011oK82u4HRCKfX2BBvhgsNm
A workflow_dispatch of the whole nightly from a branch also publishes that branch's binaries as the nightly release. The 'only' input runs the bench job by itself; a scheduled run has no inputs and runs everything. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011oK82u4HRCKfX2BBvhgsNm
…in-4d6q38 # Conflicts: # CHANGELOG.md
Agent dev-loop benchmark: proposal and phases 1–3 (make bench, nightly job)
fix(run): build as soon as exec has finished writing the model
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Syncs
ako/mxcli:maininto upstream: 8 commits since #1825 — a nightly benchmark of the tool-side time an agent waits for, and a fasterrun --watchapply aftermxcli exec.run --watch: build as soon as exec has finished writing
The watch loop noticed a change within a second, then waited for the model to stay unchanged for two more polls, so a change made by
mxcli execwas applied about 5 s after exec started although the build and reload took 2 s.execand-cnow leave a pid-named marker while they run and touch a stamp when the last write has landed: the loop doesn't build while a live marker is there, and builds within 100 ms of the stamp without the settle. A writer that says nothing is settled as before.Agent dev-loop benchmark (nightly, non-gating)
Proposal included (
PROPOSAL_agent_devloop_benchmark): a deterministic, scripted measure of the tool-side time an agent waits for, A/B against the previous nightly binary on the same runner.scripts/agentbenchruns a fixed session of mxcli commands, one process per command, on PedApp and on PedApp with a generated app seeded into it. Given a baseline binary it runs both interleaved and reports each step's ratio, plus unit files read and changed as machine-independent counts.make benchruns it.-runtime-version(make bench BENCH_RUNTIME=<version>), a running-app session: boot, a page check, a microflow change applied hot, an attached test run, an entity change applied by restart, stop, and a cold test run, on a project made bymxcli new. Each arm's app is stopped whatever happens, and a failed session keeps the records of the completed ones.docker checkanddocker build --skip-check. Neither needs a Docker daemon.MXCLI_STATS_FILE. Each project connection appends one JSON line when it closes: unit files read, unit writes offered and landed — the counts the read- and decode-budget tests assert in process, now available per command from outside.benchjob in the nightly runs it against the previous nightly release and commits the record to an orphanbenchmarksbranch, one file per run. It does not gate the release. A manual nightly run can be the bench job alone (onlyinput), since a fullworkflow_dispatchfrom a branch also publishes that branch's binaries as the nightly release