Skip to content

feat(agent): enforce control-plane session execution grants - #7085

Open
MicroGery wants to merge 2 commits into
apache:rust-previewfrom
MicroGery:codex/rust-session-execution-authority
Open

MicroGery wants to merge 2 commits into
apache:rust-previewfrom
MicroGery:codex/rust-session-execution-authority

Conversation

@MicroGery

Copy link
Copy Markdown
Contributor

Summary

Shared checkpoint storage needs a corresponding execution authority: a worker that loses its grant must stop admitting work and must not advance the Session Head. This adds a control-plane-neutral execution contract and connects it to real native Host execution and checkpoint publication.

  • Bind each grant to a logical Session, activation ID, generation, exact starting Head, and sandbox/tool ceilings. Persist original claim and renewal results, including rejected requests.
  • Provide a durable FileRepository reference coordinator. Generation validation, expected-Head CAS, and the original publication receipt commit in one transaction; unrestricted writes cannot bypass a managed Session's authority.
  • Gate message/run admission and live model/tool authorization, including approved filesystem/network permissions. Refuse to restore a Run whose captured or approvable rights exceed the grant. Invalid or unavailable authority cancels local work and initiates Host cleanup.
  • Require confirmed environment stop and drain before admitting a successor. Lease expiry and lost contact alone never authorize takeover; cleanup failures remain unconfirmed on repeated shutdown.

Refs #2370. Builds on #6929; this PR does not close the broader cloud-session work.

Scope and rollout

This is opt-in through trusted Rust bootstrap (Execution::attach, Storage::with_execution, and RootStore::open_with_checkpoint_storage). It currently binds one native Session in its original Root/deployment/namespace. Default local operation does not enable execution grants. The reference coordinator uses a private local directory and SQLite; it is not a production distributed lease service.

Cross-Root preparation/activation, an external single-run protocol, automatic failover, and CLI/UI configuration remain separate follow-ups. External executor backends are rejected in execution-managed mode. Cooperative cancellation cannot undo third-party side effects or isolate unmanaged workspace writers. The supervisor owns renewal and confirmed handoff.

Verification

Validated on a macOS arm64 Mac mini against rust-preview@540e5609d; Rust builds and tests were not run on the editing machine. Jobs were limited to 2, incremental compilation was disabled, and tests ran serially.

  • cargo nextest run --locked -p maka-event-log -p maka-host-runtime -p maka-provider-protocol --test-threads 1: 104 passed, one existing manual performance test skipped.
  • Native library filter test(execution_grant_) | test(permission_boundary) | test(original_wait_read_failure) | test(compaction_checkpoint_survives_stop) | test(queued_run_keeps_native_tools_after_host_replacement): 8 passed (167 other tests not selected). This includes real model/tool execution under a narrower grant, revocation, confirmed Host drain, and preserved permission boundaries after upstream integration.
  • The wider captured-Run regression failed without the restoration check and passed with it.
  • Formatting for the four affected crates, ASF source headers, and git diff --check passed. Strict all-targets/no-deps Clippy passed for event-log, host-runtime and provider-protocol. The small provider-protocol redundant-default cleanup is a separate commit.
  • Native all-targets/no-deps strict Clippy remains blocked by 20 existing diagnostics. The same command against unmodified 540e5609d reproduced the identical diagnostic messages and file paths, with no added diagnostics. Final sources were restored after this comparison; all 28 changed file hashes match the submitted commit.

The full Native suite, full repository CI, Linux/Windows execution, and a performance baseline were not run in this validation. No CI checks, timeouts, or test assertions were relaxed.

AI use

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: OpenAI Codex assisted with implementation, tests, documentation, upstream integration, and validation. The commits include Generated-by: Codex trailers.

Checklist

  • Tests cover execution claims, fencing, durable replies, physical stop, and real Host behavior.
  • Format, typecheck and the affected suites pass locally — see the exact remote validation scope and inherited Clippy limitations above.

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No

@github-actions github-actions Bot added the effort/XL Under 2500 readable lines label Nov 21, 2026

@M4n5ter M4n5ter left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

English

The core execution authority model is solid — especially Stopped::verify requiring physical cleanup evidence, single-transaction generation + Head CAS + receipt in FileRepository, and OnceCell on Host::shutdown so failed cleanups stay unconfirmed across retries.

Before merging, there are a few real reachable issues, a hot-path overhead concern, and test/cleanup items to address:

Blocking issues

  1. Background recovery mutates out-of-grant sessions and revoked runs (queue.rs + driver_assembly.rs)

    • Owner::reconcile_ready (plugins/agent/native/src/owner/queue.rs) scans ready_runs_with_cause across the entire RootStore without filtering by checkpoint_execution(). If another session in the same Root (e.g., sessions[1] outside the grant) has an unfinished run, startup automatically calls wake_run -> reserve_driver -> drive_local -> run_with_capabilities.
    • In run_with_capabilities (plugins/agent/native/src/owner/driver_assembly.rs), current_work_scope fails with Session is outside the execution grant, and the error handler immediately calls self.unavailable_current(self.durable.clone(), claim, AuthorityStage::WorkspaceScope, ...). Because self.durable is the unscoped store, current_authority_unavailable (plugins/agent/engine/src/driver/preflight.rs) writes durable facts (Blocked(CurrentAuthorityUnavailable), cell/tool resolution, or Terminal::Failed) into a session outside the grant.
    • The same write-after-revocation happens for the granted session itself if revocation lands right before current_work_scope or restored_data (driver_assembly.rs preflight error path), and admit_queued repeatedly spawns failing workers for out-of-grant queued entries on every durable change.
    • Fix: Filter reconcile_ready / wake_run / admit_queued to the granted session when checkpoint_execution() is active, and do not route execution-authority failures into unavailable_current.
  2. Workspace- and session-mutating RPCs bypass Execution checks

    • agent/saveFile (plugins/agent/native/src/owner/files.rs) writes directly to workspace files and only intersects capture_sandbox — it never calls require_execution(&input.session_id) or work_scope. Under a writable grant, a client can write through an out-of-grant session or continue writing after revoke() during the 250 ms poll window.
    • Similarly, agent/changeSession (plugins/agent/native/src/owner/lifecycle.rs, including archive/remove), editQueue, create, fork, metadata, selectBackend, changeGoal, and schedule have no execution gate.
    • Fix: Enforce the execution check centrally at the mutation dispatch boundary (e.g., plugins/agent/native/src/service.rs) rather than patching individual handlers piecemeal.

Performance & hot-path concerns

  1. Write locks on read-only observe() and 4–5x observe() amplification per tool call
    • FileRepository::observe (platform/event-log/src/checkpoints/execution/files.rs) always opens TransactionBehavior::Immediate and commits on the single-threaded synchronous=FULL worker, even on the common read-only Active path. Read first and only take a write transaction when transitioning an expired active row to draining.
    • Because RunAccess::fresh() calls require_execution() -> authority.observe(), a single tool call triggers 4–5 serial observe() calls (check_tool -> check_context -> live_configuration -> fresh, then live_configuration again, then sandbox_floor, then target_for_context -> check_context + fresh), and authorize_model_request triggers 2 back-to-back. Once a remote linearizable authority adapter is plugged in, this will multiply latency on every step.
    • In ExecutingRepository::commit (platform/event-log/src/checkpoints/execution/repository.rs), authority::commit already checked receipt beforehand, yet ExecutingRepository::commit calls receipt + execution.check() (observe) + authority.commit (which checks both receipt and status atomically anyway). Checking local cancellation.is_cancelled() and going straight to authority.commit (cancelling locally on inactive-generation failure) avoids two redundant round-trips without weakening fencing.

Test quality (53.5% of PR diff)

  1. Rework tests around observable behavior and drop implementation-coupled checks
    • Tests make up >50% of the diff (1,080 / 2,018 added lines), yet plugins/agent/native/src/owner/driver/tests/model_recovery/execution.rs reaches into owner.runs.state().controls to grab internal RunAccess and calls owner.restored_data(&wider) on a mutated in-memory struct instead of exercising real behavior. The only tool actually executed is todo_read.
    • platform/event-log/tests/checkpoint_execution.rs also injects a raw SQLite CREATE TRIGGER against internal table checkpoint_execution_commits (generation_check_head_and_receipt_are_one_transaction) and spawns dummy sh processes (ProcessEnvironment) that only test the test's own mock Environment.
    • Please replace those implementation-coupled tests with smaller, higher-confidence behavioral coverage for the actual gaps:
      1. Real persisted unfinished run in an out-of-grant session and a real persisted over-privileged run during Host startup recovery (would catch issue #1 directly).
      2. A real file/network tool or additional_permissions request hitting the grant ceiling at the Host boundary.
      3. Autonomous Host::watch_execution drain/shutdown on revoke/expiry without manually calling stop_execution().

Code hygiene & nits

  • SQL & matching: Use explicit column lists on INSERT INTO statements in platform/event-log/src/checkpoints/execution/files.rs and unpack the dense single-line blocks in claim/renew. In platform/event-log/src/root_store/checkpoints/authority.rs, matching directly on (&current, expected) avoids the three booleans (same/empty_local/activating) and the .expect("activating has a configuration").
  • Ownership & timeouts: ExecutionAuthority methods take Grant by value, causing deep clones of the entire sandbox/tool policy on every 250 ms poll and every check(); taking &Grant or Arc<Grant> avoids this. Also consider adding the same timeout to self.authority.stopped(stopped) in Execution::stop that observe and revoke already have.
简体中文

核心的执行权模型和状态机边界设计得不错——特别是用 Stopped::verify 做不可伪造的物理停机证据、FileRepository 里把 generation + Head CAS + receipt 放进同一个 SQL 事务,以及 Host::shutdown 用 OnceCell 防止首次清理失败后重复调用误报成功。

合入前有几个已核实的可达问题、热路径开销问题,以及测试和代码细节需要处理:

阻塞问题

  1. 后台恢复队列会唤醒非授权 Session 和已撤销 Run,并用无隔离的 self.durable 写脏持久状态(queue.rs + driver_assembly.rs)

    • Owner::reconcile_ready(plugins/agent/native/src/owner/queue.rs)会扫描整个 RootStore 里所有 Session 的 ready_runs_with_cause,没有按 checkpoint_execution() 过滤。只要同 Root 里的其他 Session(比如 grant 之外的 sessions[1])留有未终结的 Run,Host 启动时就会自动调 wake_run -> reserve_driver -> drive_local -> run_with_capabilities。
    • run_with_capabilities(plugins/agent/native/src/owner/driver_assembly.rs)在 current_work_scope 命中 Session is outside the execution grant 后,直接把错误交给 self.unavailable_current(self.durable.clone(), claim, AuthorityStage::WorkspaceScope, ...)。因为 self.durable 是未绑定 work custody 的底层 store,current_authority_unavailable(plugins/agent/engine/src/driver/preflight.rs)会直接往 grant 之外的 Session 写入持久事件(Blocked(CurrentAuthorityUnavailable)、cell/tool 结算,甚至 Terminal::Failed)。
    • 同样的“丢权后误写”也会发生在已授权 Session 自己被 revoke 的窗口里(current_work_scope 或 restored_data 失败后走 preflight 错误分支进 unavailable_current);另外 admit_queued 也会在每次 durable 变更时反复给非授权 Session 的排队项 spawn 失败 worker。
    • 修复建议:受管模式下在 reconcile_ready / wake_run / admit_queued 只处理 grant 绑定的目标 Session;driver_assembly.rs 遇到 execution authority 失效或越界错误时直接退出,不要走 unavailable_current 写库。
  2. 直接写工作区和改 Session 状态的 RPC 漏拦了执行权检查

    • agent/saveFile(plugins/agent/native/src/owner/files.rs)直接写磁盘文件,只做了 capture_sandbox 交集,完全没调 require_execution(&input.session_id) 或 work_scope。在可写 grant 下,既可以通过 grant 外的 Session 写磁盘,也能在 revoke() 后的 250 ms 轮询窗口内继续修改工作区。
    • 同样,agent/changeSession(plugins/agent/native/src/owner/lifecycle.rs,含归档/永久删除)、editQueue、create、fork、metadata、selectBackend、changeGoal、schedule 都没有做 execution 检查。
    • 修复建议:在 mutation 分发边界(如 plugins/agent/native/src/service.rs)统一收口,而不是在个别 handler 里零散补洞。

性能与热路径问题

  1. 只读 observe() 抢 SQLite 写锁 + 单次工具调用 4–5 倍 observe() 放大
    • FileRepository::observe(platform/event-log/src/checkpoints/execution/files.rs)在绝大多数纯读 Active 路径上也开了 TransactionBehavior::Immediate + tx.commit()(跑在 synchronous=FULL 的单线程 worker 上)。建议先读,只有真的观察到过期需要把 active 改成 draining 时再升写事务。
    • 由于 RunAccess::fresh() 每次都会调 require_execution() -> authority.observe(),单次工具调用会串行触发 4–5 次 observe()(check_tool -> check_context -> live_configuration -> fresh,紧接着再调 live_configuration、sandbox_floor、target_for_context -> check_context + fresh);authorize_model_request 也是 fresh() 下一行立刻 live_configuration()。接上真正远程线性化控制面后,每次调用的延迟会被成倍放大。
    • ExecutingRepository::commit(platform/event-log/src/checkpoints/execution/repository.rs)在 authority::commit 已经查过一次 receipt 之后,又做了一次 receipt + execution.check()(observe)+ authority.commit(事务内本身就会原子校验 receipt 和 status)。这里只需先判本地 cancellation.is_cancelled(),直接进 authority.commit(因非 active 失败时顺手 cancel 本地 token),能省掉两轮冗余往返且不削弱 fencing。

测试质量(测试占比 53.5%)

  1. 删掉耦合实现细节的测试,按真实外部行为重构
    • 本次 diff 中测试占了 53.5%(1,080 / 2,018 新增行),但 plugins/agent/native/src/owner/driver/tests/model_recovery/execution.rs 却钻进 owner.runs.state().controls 拿内部 RunAccess,在内存里改 wider.spec 直接调内部函数 owner.restored_data(&wider);全程真跑的工具只有 todo_read。
    • platform/event-log/tests/checkpoint_execution.rs 里也混了直接开 repository.sqlite 注入 CREATE TRIGGER ... ON checkpoint_execution_commits 的表结构耦合测试,以及靠 spawn sh 进程测测试自己 mock Environment 的 ProcessEnvironment。
    • 建议拔掉这些实现细节测试,换成体量更小、信心更高的端到端行为用例,覆盖当前空白:
      1. 库里真实持久化了非授权 Session 的未完成 Run、以及越权旧 Run 时,Host 启动恢复的真实行为(能直接抓出问题 1)。
      2. 真实文件/网络工具或带 additional_permissions 的调用在 Host 边界撞上 grant ceiling 被拒。
      3. 不手动调 stop_execution(),单纯 revoke / expire 后 Host::watch_execution 自动排空并关闭 Host。

代码整洁度与杂项

  • SQL 与状态匹配:platform/event-log/src/checkpoints/execution/files.rs 里的 INSERT INTO 建议补全显式列名,同时把 claim/renew 里挤成超长单行的代码拆开;platform/event-log/src/root_store/checkpoints/authority.rs 里直接对 (&current, expected) 做 match 解构,可以消掉 same/empty_local/activating 三个布尔和 .expect("activating has a configuration")。
  • 所有权与超时:ExecutionAuthority 的方法全部按值接收 Grant,导致每 250 ms 轮询和每次 check() 都在深拷贝整份 sandbox 规则与工具列表,建议改为 &Grant 或 Arc<Grant>;另外 Execution::stop 里最后一步 self.authority.stopped(stopped) 建议也加上和 observe/revoke 一致的超时保护。

Add a durable execution authority, generation-fenced checkpoint commits,
Host admission and permission ceilings, and confirmed stop-before-handoff.
Cover concurrent claims, lost replies, expiry, cleanup failures and real Host
model/tool/checkpoint behavior.

Refs apache#2370

Generated-by: Codex
Reject captured sandbox and approval ceilings that exceed the active execution grant. Cover the restore boundary in the real Host fixture and document it in both guides.

Generated-by: Codex
@M4n5ter
M4n5ter force-pushed the codex/rust-session-execution-authority branch from c3bd8dd to fe79528 Compare November 21, 2026 15:19

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/XL Under 2500 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants