Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -721,6 +721,20 @@ Every experiment report keeps these layers separate:
An internal control score may diagnose behavior. It must not be added to the
native task score or used to turn a failed task into a success.

### 5.5 Four-tier verification funnel

Full-horizon (`18h–20h`) replicated runs are too slow and expensive to act as the
first filter for runtime regressions, uncached KV-prefill overhead, or local
proxy-to-native inversions. Every new control mechanism, runtime optimization,
and benchmark study advances sequentially through a fail-closed four-tier funnel:

| Tier | Typical scope & horizon | Required evidence & gating metrics | Gate & non-substitution boundary |
|---|---|---|---|
| **Tier 0 · Offline invariants & calibration** | Zero live model spend: unit/regression suites, replay fixtures, `WarmStart` / evaluator `/context` preflight, container/auth/network boundary audits, and offline [`benchmark/proxy_order.py`](../../../benchmark/proxy_order.py) common-panel checks | Deterministic pass/fail contracts, exact artifact/source digest binding, zero judge leakage, and common-panel Kendall $\tau_b$ + tail-cliff characterization | Fails closed before any live token spend; passing static/replay checks proves only E0–E2 readiness |
| **Tier 1 · Fast efficiency-parity gate** | Short matched runs (for example `--timeout 18000` `5h` controls on bounded scalar/multi-stage tracks) contrasting `Raw-Artifact Reset Baseline`, quiet single-session execution, and `Controller–Worker` + `EpochReset` against an anchored baseline under §5.1–§5.3 | Native `score` parity, `time-to-threshold` parity, compression of inter-turn protocol latency gaps (such as training-to-next-submission gaps), `cached_prefix_tokens` vs. `uncached_input_tokens` stability, and bounded total token/time ratio | Prunes protocol overhead or execution regressions early; short-horizon parity never certifies full-horizon (`18h–20h`) net uplift or absence of late-horizon context drift |
| **Tier 2 · Bounded counterfactual calibration** | Short matched forks (`~3h` remaining cap) from one quiescent sealed `research-checkpoint` ($s_0$) with identical pinned source, evaluator conditions (`/context`), and submission cooldowns | Decision-neighborhood Kendall $\tau_b(s; a, a_0, \pi)$, false-improvement rate over the native incumbent, proxy-cliff (`Yardstick`) detection, and conditional $\Delta J(B \mid s_0)$ charging complete marginal control/evaluation cost | Isolates late-horizon decision and proxy fidelity before launching multi-arm `18h` suites; conditioned on $s_0$ and does not replace cold-start to terminal full-horizon comparisons |
| **Tier 3 · Full-horizon replicated promotion** | Full-budget replicated studies (`18h–20h`, `N >= 5` where promoted), including the matched `Raw-Artifact Reset Baseline` on episodic RSI / scalar optimization tracks (§5.2), full cost accounting (§6.2), and independent integrity audits (§6.5) | Benchmark-native outcome uplift or cost-normalized Pareto superiority with uncertainty bounds and cross-family replication | Only tier eligible for C2/C3 capability claims and C4 product-promotion consideration; regressions at Tier 0–2 trigger diagnosis rather than full-matrix padding |

## 6. Measurement Model

### 6.1 Native outcomes
Expand Down Expand Up @@ -1884,6 +1898,7 @@ This is a living research RFC, not a frozen benchmark snapshot.
| 2026-08-25 | Add StartupBench, LoopsBench, and WideSearch to the benchmark primer and integration plan as a second, complementary trio; require native metrics per benchmark and keep benchmark material out of reusable LoopX state. |
| 2026-10-16 | Archive [feedback-first observations](https://github.com/loopx-project/loopx/blob/1d60e7edb2a05fe048767444f872934c3adef992/deprecate/benchmark-legacy/edgebench/feedback-first-observations-20261016.md) separately from active practice. Runner source-subset and configuration-projection corrections remain E0/E1 work; mixed-profile, interrupted and running observations do not close E2/E3 or isolate a planning effect. General guard re-entry relaxation is deferred pending affected-caller validation. |
| 2026-11-15 | Codify the Bitter-Lesson boundary and five governed control-plane failure regimes (§2.2), require a matched, fully costed Raw-Artifact Reset Baseline (newly run or qualified historical reuse) for C2/C3 claims on episodic RSI / scalar optimization tracks (§5.2), and adopt the 2026-11 benchmark portfolio ranking (§3.8): **P0** (`EdgeBench`, `FrontierCS-2`, `LHTB / LHTB 2.0`, `LoopsBench`), **P1** (`SWE-Marathon`, `PaperBenchX` over `PaperBench`, `ALE` distinct from `ALE-Bench`, `WideSearch`, `KernelBench` / `AlgoTune`), **P2** (`ALE-Bench`, `StartupBench`, `MLE-bench`, `tb-science` episodic transfer samples only—excluded from RSI due to CI-locked `28800s` budget and 100% binary `reward.txt`), and **Control-Only / Deprioritized** (`DeepSWE`, `SWE-bench Pro`, `CORE-Bench`, `ScienceAgentBench`, `BixBench`, `FrontierScience`, deprecated `RE-Bench`), gating multi-arm expansion on unified `benchmark/runtime/` + Stop-hook E0–E2 readiness and Linux x86\_64 / GPU hosts. |
| 2026-11-22 | Codify the fail-closed **Four-Tier Verification Funnel** (§5.5: Tier 0 offline invariants & `benchmark/proxy_order.py` calibration -> Tier 1 `~5h` fast efficiency-parity gate -> Tier 2 `~3h` quiescent checkpoint counterfactual calibration -> Tier 3 `18h–20h` replicated promotion) so runtime/protocol optimizations prune regressions cheaply without weakening §4 C2–C4 claim thresholds or full-horizon non-substitution rules. |

## 18. References

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -568,6 +568,17 @@ goal-mode baseline。benchmark-reference scaffold reproduction 是额外 fidelit

内部 control score 可以诊断行为,但不得加到原生 task score 上,也不能把失败任务变成成功。

### 5.5 四层漏斗验证协议

完整长跑(`18h–20h`)重复单元的时钟与算力成本过高,不应作为排查基础设施缺陷、未命中 KV-cache 开销或局部代理指标反序的第一道过滤器。每项新控制机制、运行时优化及 benchmark 实验均按顺序通过闭环阻塞的**四层漏斗验证协议**推进:

| 层级 | 典型范围与时长 | 必需证据与核心指标 | 门禁与不可替代边界 |
|---|---|---|---|
| **Tier 0 · 离线不变量与标尺校准** | 零在线模型开销:单测与回归套件、确定性重放与 checkpoint 恢复夹具、`WarmStart` 与评测器 `/context` 预检、容器/认证/网络隔离审计,以及 [`benchmark/proxy_order.py`](../../../benchmark/proxy_order.py) 离线共同面板诊断 | 确定性通过/失败契约、精确产物与源码 digest 绑定、零隐藏评测泄漏,以及共同面板 Kendall $\tau_b$ 与尾部断崖刻画 | 任一项失败即阻止实跑消耗 token;静态与重放验证仅证明 E0–E2 工程就绪度 |
| **Tier 1 · 快速效率对齐门** | 短周期匹配实验(如标量/多阶段赛道 `--timeout 18000` 的 `5h` 对照),按 §5.1–§5.3 对比 `Raw-Artifact Reset Baseline`、静默单会话执行与 `Controller–Worker` + `EpochReset` 相对于锚定基线的表现 | 原生 `score` 对齐度、`time-to-threshold`、轮间协议延迟缺口压缩(如训练结束至下次提交的等待窗口)、`cached_prefix_tokens` 与 `uncached_input_tokens` 稳定性,以及总 token/耗时比上限 | 尽早拦截协议税膨胀或执行退化;短周期效率对齐不能替代全周期(`18h–20h`)净收益或证明长程无熵增漂移 |
| **Tier 2 · 有界反事实标定** | 从同一静止封存的 `research-checkpoint`($s_0$)发起短窗口匹配分叉(如剩余上限 `~3h`),严格固定源码、评测条件(`/context`)与提交冷却 | 决策邻域 Kendall $\tau_b(s; a, a_0, \pi)$、相对原生 incumbent 的代理假提升率、前沿断崖(`Yardstick`)检出,以及计入完整边际控制/评测成本后的条件净收益 $\Delta J(B \mid s_0)$ | 在启动全周期多臂组前隔离后期决策与局部标尺保真度;结果以起点 $s_0$ 为条件,不替代从冷启动到终局的全周期对照 |
| **Tier 3 · 全周期复现与晋升实验** | 完整预算复现实验(`18h–20h`,准备晋升时满足 `N >= 5`),含 episode 型 RSI / 标量赛道必配的 `Raw-Artifact Reset Baseline`(§5.2)、完整成本核算(§6.2)及独立完整性审计(§6.5) | 计入不确定性与跨族复现后的 benchmark 原生收益提升或成本归一化帕累托占优 | 唯一可支撑 C2/C3 能力主张与 C4 产品 promotion 评估的层级;Tier 0–2 出现退化时须停止归因与重设计,严禁靠堆砌全矩阵掩盖 |

## 6. 测量模型

### 6.1 原生结果
Expand Down Expand Up @@ -1503,6 +1514,7 @@ promotion 或修改 LoopX default。
| 2026-08-25 | 把 StartupBench、LoopsBench 与 WideSearch 加入 benchmark primer 与集成计划,作为第二组互补 trio;每个 benchmark 必须在原生 metric space 报告,且 benchmark 材料不得进入可复用的 LoopX state。 |
| 2026-10-16 | 将[反馈优先观察](https://github.com/loopx-project/loopx/blob/1d60e7edb2a05fe048767444f872934c3adef992/deprecate/benchmark-legacy/edgebench/feedback-first-observations-20261016.md)归入历史报告;运行器源码子集与配置投影修复仍属 E0/E1,混合处理、中断和进行中的观察不关闭 E2/E3,也不分离单一规划效应。通用 guard 重入放宽待受影响调用方独立验证。 |
| 2026-11-15 | 确立 Bitter-Lesson 边界与五类受治理控制面失败机制(§2.2),要求 episode 型 RSI / 标量优化赛道的 C2/C3 主张纳入匹配且保留完整成本记账的 Raw-Artifact Reset Baseline(新实跑或合格历史复用,§5.2),并采纳 2026-11 Benchmark 组合优先级排序(§3.8):**P0**(`EdgeBench`、`FrontierCS-2`、`LHTB / LHTB 2.0`、`LoopsBench`)、**P1**(`SWE-Marathon`、`PaperBenchX` 优先于 `PaperBench`、`ALE` 区别于 `ALE-Bench`、`WideSearch`、`KernelBench` / `AlgoTune`)、**P2**(`ALE-Bench`、`StartupBench`、`MLE-bench`、`tb-science` 仅保留单次迁移样本——因 CI 锁定 `28800s` 预算与 100% 二值 `reward.txt` 排除出 RSI),以及 **仅作对照 / 降级**(`DeepSWE`、`SWE-bench Pro`、`CORE-Bench`、`ScienceAgentBench`、`BixBench`、`FrontierScience`、已废弃 `RE-Bench`);多臂扩展以统一 `benchmark/runtime/` + Stop-hook E0–E2 就绪及 Linux x86\_64 / GPU 宿主机为前置门槛。 |
| 2026-11-22 | 确立闭环阻塞的**四层漏斗验证协议**(§5.5:Tier 0 离线不变量与 `benchmark/proxy_order.py` 校准 -> Tier 1 `~5h` 快速效率对齐门 -> Tier 2 `~3h` 静止检查点反事实标定 -> Tier 3 `18h–20h` 复现晋升),在不降低 §4 C2–C4 主张门槛与全周期不可替代约束的前提下提前剪枝运行时与协议退化。 |

## 18. 参考资料

Expand Down
38 changes: 38 additions & 0 deletions docs/architecture/rfcs/loopx-overall-roadmap-v0.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,44 @@ moats** outside a single model forward pass:
delivery) so sandbox resets or failures never corrupt control authority or
expose host credentials.

### Harness-neutral kernel and three-level harness co-design

LoopX keeps its core authority and decision substrate (`AuthorityStore`, `CAS`,
`claim/lease`, `Turn` fences, once-only `receipt` settlement, and
`Target/Yardstick/Stride` records) **harness-neutral**: the kernel must never
embed a single vendor CLI's prompt syntax, private loop driver, or transcript
layout. Because attached and managed hosts expose very different control
seams, LoopX pairs that single kernel with a **three-level harness co-design
surface** rather than forcing one generic wrapper across every environment:

| Co-design level | Target host surface | Primary optimization & boundary mechanism | Source ownership & convergence direction |
|---|---|---|---|
| **Level 1 · `BoundaryCLI`** | Boundary-isolated external CLI hosts (`codex-cli`, `claude-code`, `opencode`, `pi`, `zcode`, `kiro-cli`, `kunluncode`, skill-facade CLIs) where LoopX observes only turn boundaries | Keeps routine uncontested steps zero-ceremony (**`Quiet-Turn`**, zero per-Turn schema tax) and preserves native `KeepRaw` prefix KV-cache stability within a phase, deferring prefix rebasing and compact delta/snapshot resets to explicit quiescent **`EpochReset`** boundaries | [`loopx/host_loop_activation.py`](../../../loopx/host_loop_activation.py), [`loopx/hosts/skill_facade.py`](../../../loopx/hosts/skill_facade.py), `context_delivery.ts` ([History working sets](history-working-set-context-policy-v0.md#51-one-projection-owner)) |
| **Level 2 · `OrchestratedApp`** | Multi-unit app, IDE, IM, and `Controller–Worker` orchestrators coordinating concurrent lanes and handoffs | Enforces **`Subagent Scope`** isolation (`collaboration_brief_v0`, bounded `remaining_work`, per-worker workspace/lease fences, and async wake predicates) so long worker execution neither blocks controller/user interaction nor leaks parent prompt gravity across slices | [`controller-executor-unified-contract-v0.md`](controller-executor-unified-contract-v0.md#router-controller-worker), [`capable-manager-semantic-handoff-v0.md`](capable-manager-semantic-handoff-v0.md) |
| **Level 3 · `InTree`** | Runtime-embedded harnesses running inside or directly wired to LoopX's process transport (such as `loopx turn run-once --host dsh` for DeepSeek Harness) | Pairs receipt-gated `FoldToReceipt` / `EpochReset` queue compaction with deterministic turn-start context/artifact prefetch (**`FoldAndPrefetch`**) and typed failure-class recovery (`loopx_turn_host_failure_v0`) inside one process boundary | [`loopx/dsh_goal_mode/turn_host_adapter.py`](../../../loopx/dsh_goal_mode/turn_host_adapter.py), [`harness-selection-dsh-pi-v0.md`](harness-selection-dsh-pi-v0.md) |

To prevent host divergence from duplicating control-plane logic across
top-level packages:

- **Unified `HostHarnessCapabilities` descriptor:** every host binding declares
a single typed `HostHarnessCapabilities` contract covering loop-driver
ownership, native goal/automation/cron/subagent-scope primitives, `EpochReset`
versus `FoldAndPrefetch` support, stdio/encoding requirements, and stop/pause
fence fidelity—never claiming autonomous execution or budget enforcement a host
cannot verify.
- **Consolidating `loopx/*_goal_mode/` into `loopx/hosts/`:** migrate the `9`
remaining top-level `loopx/*_goal_mode/` packages (`agy_goal_mode`,
`claude_goal_mode`, `dsh_goal_mode`, `kiro_cli_goal_mode`,
`kunluncode_goal_mode`, `opencode2_goal_mode`, `opencode_goal_mode`,
`pi_goal_mode`, `zcode_goal_mode`) into `loopx/hosts/` under shared
`HostHarnessCapabilities` projections per
[the unified contract's retirement gate](controller-executor-unified-contract-v0.md#3-normative-retirement-backlog-and-ratchet-rules)
and [`top_level_module_budget.json`](../../../tests/architecture/top_level_module_budget.json),
keeping CLI compatibility aliases strictly until caller cutover completes.
- **Funnel-gated validation:** co-design optimizations across `BoundaryCLI`,
`OrchestratedApp`, and `InTree` remain architecture targets until verified
through the [Four-Tier Verification Funnel](long-horizon-harness-benchmark-research-program-v0.md#55-four-tier-verification-funnel).

### App-first conversation delivery

The installed App is the primary interaction and qualification surface; Lark
Expand Down
30 changes: 30 additions & 0 deletions docs/architecture/rfcs/loopx-overall-roadmap-v0.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,36 @@ LoopX 内部保持严格、唯一 owner 的类型化内核(`AuthorityStore`、
一次性 worker 沙盒及 worktree(`--worker turn-resume`、隔离容器、归一化补丁交付)严格分离,
确保沙盒重置或故障不会破坏控制权威或暴露宿主凭据。

### 与 Harness 无关的内核与三层协同设计

LoopX 的权威与决策基座(`AuthorityStore`、`CAS`、`claim/lease`、`Turn` 围栏、
仅一次幂等 `receipt` 结算与 `Target/Yardstick/Stride` 记录)保持**与 Harness 无关**:
内核绝不把单一厂商 CLI 的提示词语法、私有循环驱动或会话文件布局写死在核心规则里。
同时,不同宿主向 LoopX 暴露的控制接缝差异很大,因此 LoopX 以**三层 Harness 协同设计**
加上统一能力描述符承接宿主差异,而不是强行套用同一种重型包装器:

| 协同层级 | 适用宿主表面 | 核心优化与边界机制 | 源码归属与收敛方向 |
|---|---|---|---|
| **Level 1 · `BoundaryCLI`** | 进程边界隔离的外部 CLI 宿主(`codex-cli`、`claude-code`、`opencode`、`pi`、`zcode`、`kiro-cli`、`kunluncode` 及 skill-facade CLI),LoopX 仅在 Turn 边界接入 | 保持无竞争日常轮次零仪式感(**`Quiet-Turn`**,零每 Turn 必填表单税),在阶段内保持原生 `KeepRaw` 前缀 KV-cache 稳定,并把前缀重定基与精简增量/快照重置推迟到显式静止的 **`EpochReset`** 边界 | [`loopx/host_loop_activation.py`](../../../loopx/host_loop_activation.py)、[`loopx/hosts/skill_facade.py`](../../../loopx/hosts/skill_facade.py)、`context_delivery.ts`([历史工作集策略](history-working-set-context-policy-v0.zh-CN.md#51-单一投影-owner)) |
| **Level 2 · `OrchestratedApp`** | 协调并发 lane 与交接的多单元 App、IDE、IM 及 `Controller–Worker` 编排器 | 强化 **`Subagent Scope`** 隔离(`collaboration_brief_v0`、有界 `remaining_work` 投影、按 worker 隔离的 workspace/lease 围栏与异步唤醒谓词),使长程 worker 执行不阻塞 controller 与用户交互,也不跨切片泄漏父上下文重力 | [`controller-executor-unified-contract-v0.zh-CN.md`](controller-executor-unified-contract-v0.zh-CN.md#router-controller-worker)、[`capable-manager-semantic-handoff-v0.zh-CN.md`](capable-manager-semantic-handoff-v0.zh-CN.md) |
| **Level 3 · `InTree`** | 运行在 LoopX 进程内或直连进程传输的原生协同宿主(如 DeepSeek Harness 的 `loopx turn run-once --host dsh`) | 把受回执门控的 `FoldToReceipt` / `EpochReset` 队列压缩与回合启动时的确定性上下文/产物预取(**`FoldAndPrefetch`**)及类型化故障恢复(`loopx_turn_host_failure_v0`)融合在同一受控生命周期内 | [`loopx/dsh_goal_mode/turn_host_adapter.py`](../../../loopx/dsh_goal_mode/turn_host_adapter.py)、[`harness-selection-dsh-pi-v0.zh-CN.md`](harness-selection-dsh-pi-v0.zh-CN.md) |

为避免各宿主适配器在顶层包中重复复制控制面逻辑:

- **统一 `HostHarnessCapabilities` 描述符:** 每个宿主绑定通过单一类型化的
`HostHarnessCapabilities` 契约声明事实能力(循环驱动归属、原生 goal/automation/cron/subagent-scope
原语、支持 `EpochReset` 还是 `FoldAndPrefetch`、stdio/编码边界以及 stop/pause
围栏保真度),绝不伪造宿主无法执行的自治承诺或预算强约束。
- **把 `loopx/*_goal_mode/` 收敛至 `loopx/hosts/`:** 按
[统一契约的退役门禁](controller-executor-unified-contract-v0.zh-CN.md#3-具名退役清单与棘轮规则)与
[`top_level_module_budget.json`](../../../tests/architecture/top_level_module_budget.json),将现存
`9` 个顶层 `loopx/*_goal_mode/` 包(`agy_goal_mode`、`claude_goal_mode`、`dsh_goal_mode`、
`kiro_cli_goal_mode`、`kunluncode_goal_mode`、`opencode2_goal_mode`、`opencode_goal_mode`、
`pi_goal_mode`、`zcode_goal_mode`)迁入 `loopx/hosts/` 并收敛至共享 `HostHarnessCapabilities`
投影,仅在调用方切换完成前保留 CLI 兼容别名。
- **漏斗验证门禁:** `BoundaryCLI`、`OrchestratedApp` 与 `InTree` 的协同优化(`Quiet-Turn`、
`EpochReset`、`Subagent Scope`、`FoldAndPrefetch`)在通过[四层漏斗验证协议](long-horizon-harness-benchmark-research-program-v0.zh-CN.md#55-四层漏斗验证协议)前均为架构与设计目标,不直接等同已测收益或默认切换。

### 总纲与现有文档的关系

- 本页是跨产品、工程、研究和采用的路线总纲;第 2 节定义 S1–S13 工作流,第 3 节定义 G0–G5 组合里程碑,第 4 节映射全部 RFC,第 5–6 节细化多 Agent 核心路径,第 8 节保存可复核审计。
Expand Down