[leftibot]

The regression corpus: real repositories, pinned, validated as they are, compared between branches, and audited against what is true in them.

What this adds

• corpus/: projects.toml lists 233 GitHub repositories (one or more for each of the 44 ecosystem profiles, plus mixed-language ones), each pinned to a commit, with the ecosystems it stands for and the open issues about what the validator gets wrong on it. Nothing is recorded in the repository: python -m corpus record DIR fetches the checkouts shallowly (about 5 GB, cached under ~/.cache/project_validator/corpus), validates them in parallel and writes, per project, the project map, the results judged with --allow-unknown and what strict judging adds (each check's status and the distinct "not read" reasons); python -m corpus diff BASE HEAD says what changed between two recordings, per project and per check. record fails when a project crashes the extraction or is not detected as the ecosystem it stands for. The checkouts are git checkouts, not archives: an archive drops what .gitattributes export-ignores, often .github/ itself.
• CI job corpus: records the pull request's head and, in a worktree, its base over the same checkouts, and puts the diff in the job summary; the head recording is an artifact. It fails only on a crash or an undetected ecosystem; what changed is for the review, since a change in what is recognised is meant to show there. Every run also exercises every parser over real inputs.
• tests/test_corpus.py (offline): every project is pinned, every ecosystem has a project that stands for it, the issue lists are numbers; shortfalls and diff on a recording made by hand.
• cli.report_dicts: the --format json structure as a function, used by the corpus.
• The audit: every project's results were compared against its repository. 556 discrepancies, clustered into 111 upstream issues (A dangling symlink in the checkout crashes the extraction (FileNotFoundError in read_head) #180 to data-wellformed's message for a 100000-deep JSON file depends on the Python build ("malformed JSON" vs "Expecting value") #290, 97 bugs and 14 enhancements), each with the evidence quoted from the checkouts. The manifest's issues lists the open issues per project (226 of 233 projects have at least one).
• tests/regressions/: 149 expected-failure tests covering 93 of the issues, each asserting the right behaviour on a small project and marked @unittest.expectedFailure # upstream #N; the fix makes it an unexpected success, which fails the suite until the decorator goes. Three more sit in the modules of the code they test (dangling symlink crash, [[ ]] as a command, quadratic covers()).

Nothing in project_validator/ changes behaviour: the bugs are logged and pinned, not fixed.

Decisions

• Integrated, not a separate repository: the manifest and the runner belong with the code, and the audit's tests with the checks they pin.
• Nothing recorded in the repository: a recording for 233 projects is 9 MB and would churn with every change in what is recognised; CI compares the pull request with its base instead, which is also a run of every parser over real inputs on every change.
• haskell/cabal and rust-lang/cargo are left out of the manifest until ToolRun.covers()/owner() is quadratic in the units: monorepos take minutes (cabal, cargo > 10 min) #181 (quadratic unit ownership; they take more than ten minutes each).

The largest classes found

• A language present only in auxiliary files (fixtures, tooling, docs assets, editor dotfiles) becomes a product ecosystem whose FAILs decide the project (A language present only in auxiliary files (fixtures, tooling, docs assets, editor dotfiles) becomes a product ecosystem and its FAILs decide the project #182, in every ecosystem group).
• Commands the readers cannot resolve are dropped silently instead of recorded as unknown: Make variables, [[ ]] tests and heredocs, custom shell: wrappers, ${PREFIX}tool, unknown composite actions, uses: ./ ([[ … ]] conditional expressions (also across lines) and array assignments are read as commands with a variable program #190 to Make variables are neither expanded nor reported unknown: $(GO) test, GO ?= go defaults, command-line overrides make test FLAGS=… #197, uses: ./ (the repository's own action) is never read: _local("./") yields "" and the path becomes "/action.yml" #210).
• GitHub Actions reading: $/ same-repository references "not pinned", caller inputs of reusable workflows, &&/|| returning booleans, continue-on-error, multi-line setup versions (action-pinning: $/.github/actions/x and $/.github/workflows/x.yml (same-repository references) are reported "not pinned" #209 to platform-matrix counts "default" as a version and the same version spelled two ways as two (1.22 vs 1.22.x, 3 vs 3.x, JDK default vs default, current vs current CS); 3.14t is folded into 3.14 #222).
• Per-ecosystem gaps in what counts as a test, a license, a coverage report or a formatter gate (coverage never reaches the "reported" rung through an upload: codecov/codecov-action, coverallsapp/github-action, codecov.yml thresholds and vitest thresholds are not read #226, license: EPL-1.0, LPPL-1.3c, Artistic-2.0 / "same terms as Perl", ODbL-1.0, CC-BY-4.0, CC-BY-NC-4.0, PostgreSQL, MIT in a Markdown blockquote, CRAN's MIT + file LICENSE stub, -or-later, LICENSE-CODE/LICENSE-TEXT pairs, Data Package licenses, dist.ini #238, JVM test discovery is hard-wired to src/test/{java,kotlin} + @Test: JUnit 3 (guava-tests/test), Kotlin multiplatform jvmTest, @TestFactory, Groovy/Spock are missed #252 to XML schemas: DTD, NVDL and RELAX NG validation through a Gradle plugin are not recognised (docbook) #289).

Part of the testing roadmap; the next step after this is fixing the issues, each fix removing a decorator and changing an expectation.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.77%. Comparing base (1d902a0) to head (304e949).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #291   +/-   ##
=======================================
  Coverage   99.77%   99.77%           
=======================================
  Files         273      273           
  Lines       14063    14065    +2     
  Branches     1544     1544           
=======================================
+ Hits        14031    14033    +2     
  Misses         32       32           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.