Skip to content

docs(rfcs): add 4-tier verification funnel and 3-level harness co-design - #9211

Open
loopx-agent wants to merge 2 commits into
mainfrom
codex/rfc-harness-codesign-funnel-20261122
Open

loopx-agent wants to merge 2 commits into
mainfrom
codex/rfc-harness-codesign-funnel-20261122

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Summary

  • Add §5.5 Four-tier verification funnel (§5.5 四层漏斗验证协议) and a dated decision-log entry (2026-11-22) to docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.{md,zh-CN.md}:
    • Tier 0: Offline invariants, replay/restore fixtures, /context preflight, boundary isolation audits, and offline benchmark/proxy_order.py common-panel Kendall tau-b calibration (0 live model spend).
    • Tier 1: Fast efficiency-parity gate (~5h short-horizon matched runs comparing Raw-Artifact Reset Baseline, quiet single-session execution, and Controller–Worker + EpochReset).
    • Tier 2: Bounded counterfactual calibration (~3h matched forks from a quiescent sealed research-checkpoint $s_0$, tracking Kendall tau-b, proxy false-improvement rates, and conditional $\Delta J(B \mid s_0)$).
    • Tier 3: Full-horizon replicated promotion (18h–20h, N >= 5 where promoted, mandatory Raw-Artifact Reset Baseline on episodic RSI / scalar optimization tracks per §5.2, full cost accounting per §6.2, integrity audits per §6.5).
    • Explicit fail-closed gating and non-substitution rules: lower tiers prune regressions cheaply without relaxing §4 C2–C4 claim thresholds.
  • Add Harness-neutral kernel and three-level harness co-design (与 Harness 无关的内核与三层协同设计) to docs/architecture/rfcs/loopx-overall-roadmap-v0.{md,zh-CN.md}:
    • Keeps AuthorityStore, CAS, claim/lease, Turn fences, receipt settlement, and Target/Yardstick/Stride records harness-neutral.
    • Defines three co-design levels: Level 1 BoundaryCLI (Quiet-Turn + KeepRaw + quiescent EpochReset), Level 2 OrchestratedApp (Subagent Scope isolation via collaboration_brief_v0, bounded remaining_work, workspace/lease fences, async wakes), and Level 3 InTree (FoldAndPrefetch + typed failure recovery in loopx/dsh_goal_mode/turn_host_adapter.py).
    • Introduces the unified HostHarnessCapabilities descriptor and tracks consolidating the 9 top-level loopx/*_goal_mode/ packages into loopx/hosts/ under tests/architecture/top_level_module_budget.json, cross-referenced to the §5.5 Four-Tier Verification Funnel.

Verification

Verified on exact head f455dbb2a1bc20dcc19dddfa5f4810e01e138971:

  • python3 scripts/check_rfc_bilingual.py --base origin/main --head HEAD -> passed
  • python3 scripts/generate_rfc_status_index.py --check -> passed
  • python3 examples/docs-governance-smoke.py -> docs-governance-smoke ok
  • uv run --extra test python -m pytest -q tests/canary/test_rfc_bilingual.py tests/architecture/test_rfc_status_index.py -> 32 passed in 18.77s
  • loopx check --scan-path ... (4 edited RFC files) -> ok: True, errors=0, warnings=0, public boundary scan clean: 4 files
  • loopx canary premerge --from-git-diff --git-diff-base origin/main -> passed selected=15 failures=0 (merge_gate_passed: true)

…design

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · Gemini · Google · self_reported

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

结论概述: 在 exact head f455dbb2a1bc20dcc19dddfa5f4810e01e138971 上完成独立审阅与全套本地文档校验,确认本 PR 以 +95/-0 的克制增量将**四层漏斗验证协议(Tier 0–Tier 3)与Harness 无关内核 + 三层协同设计(BoundaryCLI / OrchestratedApp / InTree + HostHarnessCapabilities)**准确落入中英文 RFC,未引入架构越界或降低原有能力/实验门槛;仅在中文总纲发现 1 处不影响合并语义的失效交叉锚点(P2 非阻塞,建议随手修正)。


动机 / Motivation

研究执行者与维护者在推进长程 Benchmark 实验及多宿主 Harness 协同设计时,需避免直接以 18h–20h 全周期运行充当首道筛选器,或让各宿主适配器重复复制控制面逻辑。 改造前:benchmark RFC 缺少端到端分层漏斗筛选协议,总纲 RFC 也未将统一 Kernel 与不同宿主的三层接入模式(BoundaryCLI / OrchestratedApp / InTree)及包收敛路径收敛成清晰规范;直接跑满长周期或按宿主复制 wrapper 都会推高开销。改造后:在 benchmark 研究 RFC 新增 §5.5 四层漏斗验证协议与决策记录,并在总纲 RFC 明确 Harness-neutral 内核、三层协同机制与 9 个 _goal_mode/ 包向 loopx/hosts/ 的收敛路线。 审阅者与研究团队在 exact head f455dbb 的公开中英文 RFC 中可直接依据统一的 Tier 0–Tier 3 门禁和 BoundaryCLI / OrchestratedApp / InTree 职责划分开展实现与实验对照,且未放宽既有 C2–C4 主张阈值。 本 PR 仅补充中英文架构 RFC 规范文本,不改动控制面运行时代码、不修改 benchmark/proxy_order.py 或 top_level_module_budget.json 数值,也不直接迁移 loopx/_goal_mode/ 代码实现。 剩余缺口在于 9 个 *_goal_mode/ 包向 loopx/hosts/ 的实际代码迁移及 Tier 1–3 实测验证仍待后续按各工作切片推进,且中文总纲第 146 行有 1 处失效锚点建议随手修正。

改动思路 / Approach

本 PR 不新增第三份独立文档,而是直接复用已有两篇权威中英文 RFC(long-horizon-harness-benchmark-research-program-v0 与 loopx-overall-roadmap-v0),分别承接“如何分层验证收益与止损”和“如何在保持核心权威与决策机制(AuthorityStore、CAS、claim/lease、Turn、receipt 结算、Target/Yardstick/Stride)与厂商 CLI 解耦的前提下按控制接缝分层优化”两个正交维度:

  • 可逆与小切面边界: 仅增补两段规范正文与 1 条决策日志(+95 行),保持默认架构门禁不变;所有优化目标(Quiet-Turn、EpochReset、Subagent Scope、FoldAndPrefetch)显式声明为“未通过四层漏斗前仅作设计目标,不视作已测收益或默认切换”。
  • 对现有契约的承接: 复用 controller-executor-unified-contract-v0.md 既有减法门禁与 tests/architecture/top_level_module_budget.json 模块预算约束,而非另起顶层目录管理账本。

具体改动 / Changes

依据对照规范 docs/architecture/rfcs/controller-executor-unified-contract-v0.md(baseline 55bae0ac55c8858a09f05fdebd3a1b7b5dcc131f):

  • SPEC-NONSUBSTITUTION(保持 C2–C4 主张与全周期对照门禁不被短周期代理结果替代):implemented —— 已在新增 §5.5 与 2026-11-22 决策日志中显式写入非替代边界。
  • SPEC-SUBTRACTION(依托既有名单退役与顶层包上限门禁约束收敛 loopx/*_goal_mode/):implemented —— 已对照实际仓内全部 9 个 loopx/*_goal_mode/ 目录核实并收敛指向 loopx/hosts/。

关键内容讲解

  1. docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.{md,zh-CN.md}(§5.5 Four-tier verification funnel / 四层漏斗验证协议 + Decision Log 2026-11-22):

    • Tier 0(离线不变量与标尺校准): 零模型费用覆盖单元测试、checkpoint 恢复夹具、WarmStart 与 /context 预检、边界隔离审计及 benchmark/proxy_order.py 离线共同面板 Kendall $\tau_b$ 检查。
    • Tier 1(快速效率对齐门,~5h --timeout 18000): 对照 Raw-Artifact Reset Baseline、静默单会话与 Controller–Worker + EpochReset,评估原生 score、time-to-threshold、轮间等待压缩及 §6.2 已定义的 cached_prefix_tokens vs. uncached_input_tokens 稳定性;明确短周期对齐不能替代全周期净收益判断。
    • Tier 2(有界反事实标定,~3h): 基于同一静止 research-checkpoint ($s_0$) 与一致评测条件(/context)隔离晚期决策 Kendall $\tau_b$、假提升率、Yardstick 断崖及计入完整控制/评测成本的条件净收益 $\Delta J(B \mid s_0)$。
    • Tier 3(全周期复现晋升,18h–20h): N >= 5(拟晋升时)配全成本核算(§6.2)、完整基线(§5.2)与独立完整性审计(§6.5),仍是 C2/C3/C4 主张唯一入口。
  2. docs/architecture/rfcs/loopx-overall-roadmap-v0.{md,zh-CN.md}(与 Harness 无关的内核与三层协同设计):

    • Level 1 · BoundaryCLI: 边界 CLI 宿主通过 Quiet-Turn 消除每轮结构化表单税,阶段内维持 KeepRaw 前缀 KV-cache 稳定,仅在静止 EpochReset 边界重定基;指向已存在的 loopx/host_loop_activation.py、loopx/hosts/skill_facade.py 与 loopx/control_plane/quota/context_delivery.ts(5.1)。
    • Level 2 · OrchestratedApp: 多单元与 Controller–Worker 强化 Subagent Scope 隔离(collaboration_brief_v0、有界 remaining_work、按 worker 隔离 workspace/lease 与异步唤醒),链接 #router-controller-worker 锚点(中英文均已核验存在)。
    • Level 3 · InTree: 进程内协作宿主(loopx turn run-once --host dsh)结合 FoldToReceipt/EpochReset 与 FoldAndPrefetch 及类型化恢复,指向 loopx/dsh_goal_mode/turn_host_adapter.py。
    • 统一 HostHarnessCapabilities 与包收敛: 要求各宿主声明事实能力且不伪造预算强约束;明确列出全部 9 个顶层 loopx/*_goal_mode/ 包(与实仓 ls -d loopx/*_goal_mode 逐一核对完全一致)迁入 loopx/hosts/ 的路径,并引用 tests/architecture/top_level_module_budget.json(当前 max_top_level_modules=121)。

对主干的风险 / Risks

  • P2(非阻塞 · 交叉锚点失效):docs/architecture/rfcs/loopx-overall-roadmap-v0.zh-CN.md 第 146 行链接文本 [统一契约的退役门禁](controller-executor-unified-contract-v0.zh-CN.md#3-normative-retirement-backlog-and-ratchet-rules)
    • Problem: 目标中文文件 controller-executor-unified-contract-v0.zh-CN.md 中不存在 #3-normative-retirement-backlog-and-ratchet-rules 锚点或 HTML <a id>,点击仅能跳转到文档顶部;其对应小节位于第 1223 行 #### 3. 具名退役清单与棘轮规则(实际 GitHub 锚点为 #3-具名退役清单与棘轮规则),上层小节锚点为第 1184 行 <a id="subtraction-gate"></a>。英文版 loopx-overall-roadmap-v0.md 第 175–183 行(对应目标第 1555 行)无此问题。
    • Impact & Fix: 仅影响中文读者页内跳转,不破坏任何程序逻辑或门禁检查;建议在后续或微修中改为 #3-具名退役清单与棘轮规则(或在目标中文 RFC 第 1222 行补齐 <a id="3-normative-retirement-backlog-and-ratchet-rules"></a>)。
  • 本地全量验证矩阵(均基于 exact head f455dbb2a1bc20dcc19dddfa5f4810e01e138971 在独立只读 worktree 实测):
    • check_rfc_bilingual.py --base origin/main --head HEAD:PASS
    • generate_rfc_status_index.py --check:PASS
    • examples/docs-governance-smoke.py:PASS(docs-governance-smoke ok)
    • pytest -q tests/canary/test_rfc_bilingual.py tests/architecture/test_rfc_status_index.py:PASS(32 passed in 19.25s)
    • loopx check --scan-path ...(4 份改动文件):PASS(errors=0, public boundary scan clean: 4 files)
    • loopx canary premerge --from-git-diff --git-diff-base origin/main:直接检查 4/4 PASS,风险 smoke 8/8 PASS,公开边界 1/1 PASS,目录金丝雀 5/6 PASS;其中未挂载 node_modules 时的 examples/semantic-vocabulary-drift-smoke.py 失败已在挂接工作区 node_modules 复测确认 PASS(semantic-vocabulary-drift-smoke: ok)。

我的整体评价 / Verdict

本 PR 结构紧凑、边界克制且引用的文件与包清单经独立复核均与代码库事实一致,作为架构与验证协议基线对主干零副作用;长程与用户阅读路径均安全保持(preserved),仅中文文档内 1 处交叉锚点需小修(非阻塞)。本次变更仅为中英文 RFC Markdown 文档增量补充(无 UI、CLI 或终端页面渲染变更),已核验 diff 与目标文件结构,不适用界面截图检查。

English verdict: APPROVE — exact head f455dbb aligns bilingual RFC governance for the 4-tier verification funnel and 3-level harness co-design with all local RFC/canary/public-boundary checks passing (1 minor P2 non-blocking broken cross-doc anchor in the zh-CN roadmap file). Keep PR OPEN per coordinator instructions.

…dmap

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant