Repository navigation
docs(rfcs): add 4-tier verification funnel and 3-level harness co-design - #9211
loopx-agent wants to merge 2 commits into
Conversation
…design Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent · Gemini · Google · self_reported
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
结论概述: 在 exact head f455dbb2a1bc20dcc19dddfa5f4810e01e138971 上完成独立审阅与全套本地文档校验,确认本 PR 以 +95/-0 的克制增量将**四层漏斗验证协议(Tier 0–Tier 3)与Harness 无关内核 + 三层协同设计(BoundaryCLI / OrchestratedApp / InTree + HostHarnessCapabilities)**准确落入中英文 RFC,未引入架构越界或降低原有能力/实验门槛;仅在中文总纲发现 1 处不影响合并语义的失效交叉锚点(P2 非阻塞,建议随手修正)。
动机 / Motivation
研究执行者与维护者在推进长程 Benchmark 实验及多宿主 Harness 协同设计时,需避免直接以 18h–20h 全周期运行充当首道筛选器,或让各宿主适配器重复复制控制面逻辑。 改造前:benchmark RFC 缺少端到端分层漏斗筛选协议,总纲 RFC 也未将统一 Kernel 与不同宿主的三层接入模式(BoundaryCLI / OrchestratedApp / InTree)及包收敛路径收敛成清晰规范;直接跑满长周期或按宿主复制 wrapper 都会推高开销。改造后:在 benchmark 研究 RFC 新增 §5.5 四层漏斗验证协议与决策记录,并在总纲 RFC 明确 Harness-neutral 内核、三层协同机制与 9 个 _goal_mode/ 包向 loopx/hosts/ 的收敛路线。 审阅者与研究团队在 exact head f455dbb 的公开中英文 RFC 中可直接依据统一的 Tier 0–Tier 3 门禁和 BoundaryCLI / OrchestratedApp / InTree 职责划分开展实现与实验对照,且未放宽既有 C2–C4 主张阈值。 本 PR 仅补充中英文架构 RFC 规范文本,不改动控制面运行时代码、不修改 benchmark/proxy_order.py 或 top_level_module_budget.json 数值,也不直接迁移 loopx/_goal_mode/ 代码实现。 剩余缺口在于 9 个 *_goal_mode/ 包向 loopx/hosts/ 的实际代码迁移及 Tier 1–3 实测验证仍待后续按各工作切片推进,且中文总纲第 146 行有 1 处失效锚点建议随手修正。
改动思路 / Approach
本 PR 不新增第三份独立文档,而是直接复用已有两篇权威中英文 RFC(long-horizon-harness-benchmark-research-program-v0 与 loopx-overall-roadmap-v0),分别承接“如何分层验证收益与止损”和“如何在保持核心权威与决策机制(AuthorityStore、CAS、claim/lease、Turn、receipt 结算、Target/Yardstick/Stride)与厂商 CLI 解耦的前提下按控制接缝分层优化”两个正交维度:
- 可逆与小切面边界: 仅增补两段规范正文与 1 条决策日志(+95 行),保持默认架构门禁不变;所有优化目标(
Quiet-Turn、EpochReset、Subagent Scope、FoldAndPrefetch)显式声明为“未通过四层漏斗前仅作设计目标,不视作已测收益或默认切换”。 - 对现有契约的承接: 复用
controller-executor-unified-contract-v0.md既有减法门禁与tests/architecture/top_level_module_budget.json模块预算约束,而非另起顶层目录管理账本。
具体改动 / Changes
依据对照规范 docs/architecture/rfcs/controller-executor-unified-contract-v0.md(baseline 55bae0ac55c8858a09f05fdebd3a1b7b5dcc131f):
SPEC-NONSUBSTITUTION(保持 C2–C4 主张与全周期对照门禁不被短周期代理结果替代):implemented —— 已在新增 §5.5 与2026-11-22决策日志中显式写入非替代边界。SPEC-SUBTRACTION(依托既有名单退役与顶层包上限门禁约束收敛loopx/*_goal_mode/):implemented —— 已对照实际仓内全部9个loopx/*_goal_mode/目录核实并收敛指向loopx/hosts/。
关键内容讲解
-
docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.{md,zh-CN.md}(§5.5 Four-tier verification funnel / 四层漏斗验证协议 + Decision Log2026-11-22):-
Tier 0(离线不变量与标尺校准): 零模型费用覆盖单元测试、checkpoint 恢复夹具、
WarmStart与/context预检、边界隔离审计及benchmark/proxy_order.py离线共同面板 Kendall$\tau_b$ 检查。 -
Tier 1(快速效率对齐门,
~5h--timeout 18000): 对照Raw-Artifact Reset Baseline、静默单会话与Controller–Worker + EpochReset,评估原生score、time-to-threshold、轮间等待压缩及 §6.2 已定义的cached_prefix_tokensvs.uncached_input_tokens稳定性;明确短周期对齐不能替代全周期净收益判断。 -
Tier 2(有界反事实标定,
~3h): 基于同一静止research-checkpoint($s_0$ ) 与一致评测条件(/context)隔离晚期决策 Kendall$\tau_b$ 、假提升率、Yardstick断崖及计入完整控制/评测成本的条件净收益$\Delta J(B \mid s_0)$ 。 -
Tier 3(全周期复现晋升,
18h–20h):N >= 5(拟晋升时)配全成本核算(§6.2)、完整基线(§5.2)与独立完整性审计(§6.5),仍是 C2/C3/C4 主张唯一入口。
-
Tier 0(离线不变量与标尺校准): 零模型费用覆盖单元测试、checkpoint 恢复夹具、
-
docs/architecture/rfcs/loopx-overall-roadmap-v0.{md,zh-CN.md}(与 Harness 无关的内核与三层协同设计):-
Level 1 ·
BoundaryCLI: 边界 CLI 宿主通过Quiet-Turn消除每轮结构化表单税,阶段内维持KeepRaw前缀 KV-cache 稳定,仅在静止EpochReset边界重定基;指向已存在的loopx/host_loop_activation.py、loopx/hosts/skill_facade.py与loopx/control_plane/quota/context_delivery.ts(5.1)。 -
Level 2 ·
OrchestratedApp: 多单元与Controller–Worker强化Subagent Scope隔离(collaboration_brief_v0、有界remaining_work、按 worker 隔离 workspace/lease 与异步唤醒),链接#router-controller-worker锚点(中英文均已核验存在)。 -
Level 3 ·
InTree: 进程内协作宿主(loopx turn run-once --host dsh)结合FoldToReceipt/EpochReset与FoldAndPrefetch及类型化恢复,指向loopx/dsh_goal_mode/turn_host_adapter.py。 -
统一
HostHarnessCapabilities与包收敛: 要求各宿主声明事实能力且不伪造预算强约束;明确列出全部9个顶层loopx/*_goal_mode/包(与实仓ls -d loopx/*_goal_mode逐一核对完全一致)迁入loopx/hosts/的路径,并引用tests/architecture/top_level_module_budget.json(当前max_top_level_modules=121)。
-
Level 1 ·
对主干的风险 / Risks
- P2(非阻塞 · 交叉锚点失效):
docs/architecture/rfcs/loopx-overall-roadmap-v0.zh-CN.md第 146 行链接文本[统一契约的退役门禁](controller-executor-unified-contract-v0.zh-CN.md#3-normative-retirement-backlog-and-ratchet-rules)- Problem: 目标中文文件
controller-executor-unified-contract-v0.zh-CN.md中不存在#3-normative-retirement-backlog-and-ratchet-rules锚点或 HTML<a id>,点击仅能跳转到文档顶部;其对应小节位于第 1223 行#### 3. 具名退役清单与棘轮规则(实际 GitHub 锚点为#3-具名退役清单与棘轮规则),上层小节锚点为第 1184 行<a id="subtraction-gate"></a>。英文版loopx-overall-roadmap-v0.md第 175–183 行(对应目标第 1555 行)无此问题。 - Impact & Fix: 仅影响中文读者页内跳转,不破坏任何程序逻辑或门禁检查;建议在后续或微修中改为
#3-具名退役清单与棘轮规则(或在目标中文 RFC 第 1222 行补齐<a id="3-normative-retirement-backlog-and-ratchet-rules"></a>)。
- Problem: 目标中文文件
- 本地全量验证矩阵(均基于 exact head
f455dbb2a1bc20dcc19dddfa5f4810e01e138971在独立只读 worktree 实测):check_rfc_bilingual.py --base origin/main --head HEAD:PASSgenerate_rfc_status_index.py --check:PASSexamples/docs-governance-smoke.py:PASS(docs-governance-smoke ok)pytest -q tests/canary/test_rfc_bilingual.py tests/architecture/test_rfc_status_index.py:PASS(32 passed in 19.25s)loopx check --scan-path ...(4 份改动文件):PASS(errors=0,public boundary scan clean: 4 files)loopx canary premerge --from-git-diff --git-diff-base origin/main:直接检查4/4PASS,风险 smoke8/8PASS,公开边界1/1PASS,目录金丝雀5/6PASS;其中未挂载node_modules时的examples/semantic-vocabulary-drift-smoke.py失败已在挂接工作区node_modules复测确认 PASS(semantic-vocabulary-drift-smoke: ok)。
我的整体评价 / Verdict
本 PR 结构紧凑、边界克制且引用的文件与包清单经独立复核均与代码库事实一致,作为架构与验证协议基线对主干零副作用;长程与用户阅读路径均安全保持(preserved),仅中文文档内 1 处交叉锚点需小修(非阻塞)。本次变更仅为中英文 RFC Markdown 文档增量补充(无 UI、CLI 或终端页面渲染变更),已核验 diff 与目标文件结构,不适用界面截图检查。
English verdict: APPROVE — exact head f455dbb aligns bilingual RFC governance for the 4-tier verification funnel and 3-level harness co-design with all local RFC/canary/public-boundary checks passing (1 minor P2 non-blocking broken cross-doc anchor in the zh-CN roadmap file). Keep PR OPEN per coordinator instructions.
…dmap Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Summary
§5.5 四层漏斗验证协议) and a dated decision-log entry (2026-11-22) todocs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.{md,zh-CN.md}:/contextpreflight, boundary isolation audits, and offlinebenchmark/proxy_order.pycommon-panel Kendall tau-b calibration (0live model spend).~5hshort-horizon matched runs comparingRaw-Artifact Reset Baseline, quiet single-session execution, andController–Worker+EpochReset).~3hmatched forks from a quiescent sealedresearch-checkpoint18h–20h,N >= 5where promoted, mandatoryRaw-Artifact Reset Baselineon episodic RSI / scalar optimization tracks per §5.2, full cost accounting per §6.2, integrity audits per §6.5).与 Harness 无关的内核与三层协同设计) todocs/architecture/rfcs/loopx-overall-roadmap-v0.{md,zh-CN.md}:AuthorityStore,CAS,claim/lease,Turnfences,receiptsettlement, andTarget/Yardstick/Striderecords harness-neutral.BoundaryCLI(Quiet-Turn+KeepRaw+ quiescentEpochReset), Level 2OrchestratedApp(Subagent Scopeisolation viacollaboration_brief_v0, boundedremaining_work, workspace/lease fences, async wakes), and Level 3InTree(FoldAndPrefetch+ typed failure recovery inloopx/dsh_goal_mode/turn_host_adapter.py).HostHarnessCapabilitiesdescriptor and tracks consolidating the9top-levelloopx/*_goal_mode/packages intoloopx/hosts/undertests/architecture/top_level_module_budget.json, cross-referenced to the §5.5 Four-Tier Verification Funnel.Verification
Verified on exact head
f455dbb2a1bc20dcc19dddfa5f4810e01e138971:python3 scripts/check_rfc_bilingual.py --base origin/main --head HEAD-> passedpython3 scripts/generate_rfc_status_index.py --check-> passedpython3 examples/docs-governance-smoke.py->docs-governance-smoke okuv run --extra test python -m pytest -q tests/canary/test_rfc_bilingual.py tests/architecture/test_rfc_status_index.py->32 passed in 18.77sloopx check --scan-path ...(4 edited RFC files) ->ok: True,errors=0, warnings=0,public boundary scan clean: 4 filesloopx canary premerge --from-git-diff --git-diff-base origin/main->passed selected=15 failures=0(merge_gate_passed: true)