Repository navigation
Conversation
There was a problem hiding this comment.
English
The core execution authority model is solid — especially Stopped::verify requiring physical cleanup evidence, single-transaction generation + Head CAS + receipt in FileRepository, and OnceCell on Host::shutdown so failed cleanups stay unconfirmed across retries.
Before merging, there are a few real reachable issues, a hot-path overhead concern, and test/cleanup items to address:
Blocking issues
-
Background recovery mutates out-of-grant sessions and revoked runs (
queue.rs+driver_assembly.rs)Owner::reconcile_ready(plugins/agent/native/src/owner/queue.rs) scansready_runs_with_causeacross the entireRootStorewithout filtering bycheckpoint_execution(). If another session in the same Root (e.g.,sessions[1]outside the grant) has an unfinished run, startup automatically callswake_run->reserve_driver->drive_local->run_with_capabilities.- In
run_with_capabilities(plugins/agent/native/src/owner/driver_assembly.rs),current_work_scopefails withSession is outside the execution grant, and the error handler immediately callsself.unavailable_current(self.durable.clone(), claim, AuthorityStage::WorkspaceScope, ...). Becauseself.durableis the unscoped store,current_authority_unavailable(plugins/agent/engine/src/driver/preflight.rs) writes durable facts (Blocked(CurrentAuthorityUnavailable), cell/tool resolution, orTerminal::Failed) into a session outside the grant. - The same write-after-revocation happens for the granted session itself if revocation lands right before
current_work_scopeorrestored_data(driver_assembly.rspreflight error path), andadmit_queuedrepeatedly spawns failing workers for out-of-grant queued entries on every durable change. - Fix: Filter
reconcile_ready/wake_run/admit_queuedto the granted session whencheckpoint_execution()is active, and do not route execution-authority failures intounavailable_current.
-
Workspace- and session-mutating RPCs bypass
Executionchecksagent/saveFile(plugins/agent/native/src/owner/files.rs) writes directly to workspace files and only intersectscapture_sandbox— it never callsrequire_execution(&input.session_id)orwork_scope. Under a writable grant, a client can write through an out-of-grant session or continue writing afterrevoke()during the 250 ms poll window.- Similarly,
agent/changeSession(plugins/agent/native/src/owner/lifecycle.rs, including archive/remove),editQueue,create,fork,metadata,selectBackend,changeGoal, andschedulehave no execution gate. - Fix: Enforce the execution check centrally at the mutation dispatch boundary (e.g.,
plugins/agent/native/src/service.rs) rather than patching individual handlers piecemeal.
Performance & hot-path concerns
- Write locks on read-only
observe()and 4–5xobserve()amplification per tool callFileRepository::observe(platform/event-log/src/checkpoints/execution/files.rs) always opensTransactionBehavior::Immediateand commits on the single-threadedsynchronous=FULLworker, even on the common read-onlyActivepath. Read first and only take a write transaction when transitioning an expiredactiverow todraining.- Because
RunAccess::fresh()callsrequire_execution()->authority.observe(), a single tool call triggers 4–5 serialobserve()calls (check_tool->check_context->live_configuration->fresh, thenlive_configurationagain, thensandbox_floor, thentarget_for_context->check_context+fresh), andauthorize_model_requesttriggers 2 back-to-back. Once a remote linearizable authority adapter is plugged in, this will multiply latency on every step. - In
ExecutingRepository::commit(platform/event-log/src/checkpoints/execution/repository.rs),authority::commitalready checkedreceiptbeforehand, yetExecutingRepository::commitcallsreceipt+execution.check()(observe) +authority.commit(which checks both receipt and status atomically anyway). Checking localcancellation.is_cancelled()and going straight toauthority.commit(cancelling locally on inactive-generation failure) avoids two redundant round-trips without weakening fencing.
Test quality (53.5% of PR diff)
- Rework tests around observable behavior and drop implementation-coupled checks
- Tests make up >50% of the diff (1,080 / 2,018 added lines), yet
plugins/agent/native/src/owner/driver/tests/model_recovery/execution.rsreaches intoowner.runs.state().controlsto grab internalRunAccessand callsowner.restored_data(&wider)on a mutated in-memory struct instead of exercising real behavior. The only tool actually executed istodo_read. platform/event-log/tests/checkpoint_execution.rsalso injects a raw SQLiteCREATE TRIGGERagainst internal tablecheckpoint_execution_commits(generation_check_head_and_receipt_are_one_transaction) and spawns dummyshprocesses (ProcessEnvironment) that only test the test's own mockEnvironment.- Please replace those implementation-coupled tests with smaller, higher-confidence behavioral coverage for the actual gaps:
- Real persisted unfinished run in an out-of-grant session and a real persisted over-privileged run during Host startup recovery (would catch issue #1 directly).
- A real file/network tool or
additional_permissionsrequest hitting the grant ceiling at the Host boundary. - Autonomous
Host::watch_executiondrain/shutdown on revoke/expiry without manually callingstop_execution().
- Tests make up >50% of the diff (1,080 / 2,018 added lines), yet
Code hygiene & nits
- SQL & matching: Use explicit column lists on
INSERT INTOstatements inplatform/event-log/src/checkpoints/execution/files.rsand unpack the dense single-line blocks inclaim/renew. Inplatform/event-log/src/root_store/checkpoints/authority.rs, matching directly on(¤t, expected)avoids the three booleans (same/empty_local/activating) and the.expect("activating has a configuration"). - Ownership & timeouts:
ExecutionAuthoritymethods takeGrantby value, causing deep clones of the entire sandbox/tool policy on every 250 ms poll and everycheck(); taking&GrantorArc<Grant>avoids this. Also consider adding the same timeout toself.authority.stopped(stopped)inExecution::stopthatobserveandrevokealready have.
简体中文
核心的执行权模型和状态机边界设计得不错——特别是用 Stopped::verify 做不可伪造的物理停机证据、FileRepository 里把 generation + Head CAS + receipt 放进同一个 SQL 事务,以及 Host::shutdown 用 OnceCell 防止首次清理失败后重复调用误报成功。
合入前有几个已核实的可达问题、热路径开销问题,以及测试和代码细节需要处理:
阻塞问题
-
后台恢复队列会唤醒非授权 Session 和已撤销 Run,并用无隔离的
self.durable写脏持久状态(queue.rs+driver_assembly.rs)Owner::reconcile_ready(plugins/agent/native/src/owner/queue.rs)会扫描整个RootStore里所有 Session 的ready_runs_with_cause,没有按checkpoint_execution()过滤。只要同 Root 里的其他 Session(比如 grant 之外的sessions[1])留有未终结的 Run,Host 启动时就会自动调wake_run->reserve_driver->drive_local->run_with_capabilities。run_with_capabilities(plugins/agent/native/src/owner/driver_assembly.rs)在current_work_scope命中Session is outside the execution grant后,直接把错误交给self.unavailable_current(self.durable.clone(), claim, AuthorityStage::WorkspaceScope, ...)。因为self.durable是未绑定 work custody 的底层 store,current_authority_unavailable(plugins/agent/engine/src/driver/preflight.rs)会直接往 grant 之外的 Session 写入持久事件(Blocked(CurrentAuthorityUnavailable)、cell/tool 结算,甚至Terminal::Failed)。- 同样的“丢权后误写”也会发生在已授权 Session 自己被 revoke 的窗口里(
current_work_scope或restored_data失败后走 preflight 错误分支进unavailable_current);另外admit_queued也会在每次 durable 变更时反复给非授权 Session 的排队项 spawn 失败 worker。 - 修复建议:受管模式下在
reconcile_ready/wake_run/admit_queued只处理 grant 绑定的目标 Session;driver_assembly.rs遇到 execution authority 失效或越界错误时直接退出,不要走unavailable_current写库。
-
直接写工作区和改 Session 状态的 RPC 漏拦了执行权检查
agent/saveFile(plugins/agent/native/src/owner/files.rs)直接写磁盘文件,只做了capture_sandbox交集,完全没调require_execution(&input.session_id)或work_scope。在可写 grant 下,既可以通过 grant 外的 Session 写磁盘,也能在revoke()后的 250 ms 轮询窗口内继续修改工作区。- 同样,
agent/changeSession(plugins/agent/native/src/owner/lifecycle.rs,含归档/永久删除)、editQueue、create、fork、metadata、selectBackend、changeGoal、schedule都没有做 execution 检查。 - 修复建议:在 mutation 分发边界(如
plugins/agent/native/src/service.rs)统一收口,而不是在个别 handler 里零散补洞。
性能与热路径问题
- 只读
observe()抢 SQLite 写锁 + 单次工具调用 4–5 倍observe()放大FileRepository::observe(platform/event-log/src/checkpoints/execution/files.rs)在绝大多数纯读Active路径上也开了TransactionBehavior::Immediate+tx.commit()(跑在synchronous=FULL的单线程 worker 上)。建议先读,只有真的观察到过期需要把active改成draining时再升写事务。- 由于
RunAccess::fresh()每次都会调require_execution()->authority.observe(),单次工具调用会串行触发 4–5 次observe()(check_tool->check_context->live_configuration->fresh,紧接着再调live_configuration、sandbox_floor、target_for_context->check_context+fresh);authorize_model_request也是fresh()下一行立刻live_configuration()。接上真正远程线性化控制面后,每次调用的延迟会被成倍放大。 ExecutingRepository::commit(platform/event-log/src/checkpoints/execution/repository.rs)在authority::commit已经查过一次receipt之后,又做了一次receipt+execution.check()(observe)+authority.commit(事务内本身就会原子校验 receipt 和 status)。这里只需先判本地cancellation.is_cancelled(),直接进authority.commit(因非 active 失败时顺手 cancel 本地 token),能省掉两轮冗余往返且不削弱 fencing。
测试质量(测试占比 53.5%)
- 删掉耦合实现细节的测试,按真实外部行为重构
- 本次 diff 中测试占了 53.5%(1,080 / 2,018 新增行),但
plugins/agent/native/src/owner/driver/tests/model_recovery/execution.rs却钻进owner.runs.state().controls拿内部RunAccess,在内存里改wider.spec直接调内部函数owner.restored_data(&wider);全程真跑的工具只有todo_read。 platform/event-log/tests/checkpoint_execution.rs里也混了直接开repository.sqlite注入CREATE TRIGGER ... ON checkpoint_execution_commits的表结构耦合测试,以及靠 spawnsh进程测测试自己 mockEnvironment的ProcessEnvironment。- 建议拔掉这些实现细节测试,换成体量更小、信心更高的端到端行为用例,覆盖当前空白:
- 库里真实持久化了非授权 Session 的未完成 Run、以及越权旧 Run 时,Host 启动恢复的真实行为(能直接抓出问题 1)。
- 真实文件/网络工具或带
additional_permissions的调用在 Host 边界撞上 grant ceiling 被拒。 - 不手动调
stop_execution(),单纯 revoke / expire 后Host::watch_execution自动排空并关闭 Host。
- 本次 diff 中测试占了 53.5%(1,080 / 2,018 新增行),但
代码整洁度与杂项
- SQL 与状态匹配:
platform/event-log/src/checkpoints/execution/files.rs里的INSERT INTO建议补全显式列名,同时把claim/renew里挤成超长单行的代码拆开;platform/event-log/src/root_store/checkpoints/authority.rs里直接对(¤t, expected)做match解构,可以消掉same/empty_local/activating三个布尔和.expect("activating has a configuration")。 - 所有权与超时:
ExecutionAuthority的方法全部按值接收Grant,导致每 250 ms 轮询和每次check()都在深拷贝整份 sandbox 规则与工具列表,建议改为&Grant或Arc<Grant>;另外Execution::stop里最后一步self.authority.stopped(stopped)建议也加上和observe/revoke一致的超时保护。
Add a durable execution authority, generation-fenced checkpoint commits, Host admission and permission ceilings, and confirmed stop-before-handoff. Cover concurrent claims, lost replies, expiry, cleanup failures and real Host model/tool/checkpoint behavior. Refs apache#2370 Generated-by: Codex
Reject captured sandbox and approval ceilings that exceed the active execution grant. Cover the restore boundary in the real Host fixture and document it in both guides. Generated-by: Codex
c3bd8dd to
fe79528
Compare
Summary
Shared checkpoint storage needs a corresponding execution authority: a worker that loses its grant must stop admitting work and must not advance the Session Head. This adds a control-plane-neutral execution contract and connects it to real native Host execution and checkpoint publication.
Refs #2370. Builds on #6929; this PR does not close the broader cloud-session work.
Scope and rollout
This is opt-in through trusted Rust bootstrap (
Execution::attach,Storage::with_execution, andRootStore::open_with_checkpoint_storage). It currently binds one native Session in its original Root/deployment/namespace. Default local operation does not enable execution grants. The reference coordinator uses a private local directory and SQLite; it is not a production distributed lease service.Cross-Root preparation/activation, an external single-run protocol, automatic failover, and CLI/UI configuration remain separate follow-ups. External executor backends are rejected in execution-managed mode. Cooperative cancellation cannot undo third-party side effects or isolate unmanaged workspace writers. The supervisor owns renewal and confirmed handoff.
Verification
Validated on a macOS arm64 Mac mini against
rust-preview@540e5609d; Rust builds and tests were not run on the editing machine. Jobs were limited to 2, incremental compilation was disabled, and tests ran serially.cargo nextest run --locked -p maka-event-log -p maka-host-runtime -p maka-provider-protocol --test-threads 1: 104 passed, one existing manual performance test skipped.test(execution_grant_) | test(permission_boundary) | test(original_wait_read_failure) | test(compaction_checkpoint_survives_stop) | test(queued_run_keeps_native_tools_after_host_replacement): 8 passed (167 other tests not selected). This includes real model/tool execution under a narrower grant, revocation, confirmed Host drain, and preserved permission boundaries after upstream integration.git diff --checkpassed. Strict all-targets/no-deps Clippy passed for event-log, host-runtime and provider-protocol. The small provider-protocol redundant-default cleanup is a separate commit.540e5609dreproduced the identical diagnostic messages and file paths, with no added diagnostics. Final sources were restored after this comparison; all 28 changed file hashes match the submitted commit.The full Native suite, full repository CI, Linux/Windows execution, and a performance baseline were not run in this validation. No CI checks, timeouts, or test assertions were relaxed.
AI use
Tool(s) and scope: OpenAI Codex assisted with implementation, tests, documentation, upstream integration, and validation. The commits include
Generated-by: Codextrailers.Checklist
Does this PR entail a change in behavior?