Skip to content

PR triage: compact bullet comment; judge again on every ready_for_review - #5308

Merged
dsmiley merged 2 commits into
apache:mainfrom
dsmiley:pr-triage-format
Nov 9, 2026
Merged

dsmiley merged 2 commits into
apache:mainfrom
dsmiley:pr-triage-format

Conversation

@dsmiley

@dsmiley dsmiley commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

🤖 Follow-up to #5281, from the review comments there.

Comment format. A bullet per judgement replaces the table.

  • A row that agrees says what the PR has (included, not needed, or type: added for the changelog). Its explanation is kept in an HTML comment, visible only in the Markdown source.
  • A row that disagrees shows what the judgement suggests and what the PR has (suggest type: added; PR has type: fixed), then the explanation. Confidence is shown only when it is below high.
  • An unanswered row reads "Unclear; please assess."
  • The assessed commit is now a hidden comment, and the footer key is gone.

No "already triaged" check. The workflow no longer reads the PR's comments before judging, so a PR that goes draft → ready → draft → ready is judged again and gets a second comment. Authors rarely toggle draft, and a fresh look is fine. The <!-- llm-pr-triage --> marker and the issues: read permission are gone with it.

Config. A new choice_label key names the changelog type in the comment, described in the _doc key.

Tested with unit tests (47 pass) and by running the script on #5306, #5305, #5303, #5296 and the merged #5026 with qwen3.8-27b.

🤖 Generated with Claude Code

- dev-tools/scripts/pr-triage-llm.py: render a bullet per judgement instead of a table. An agreeing row states what the PR has (included, not needed, or the changelog type) and keeps the reason in an HTML comment. A mismatch row shows the suggestion and what the PR has, the reason, and the confidence only when it is below high. Unanswered rows say "Unclear; please assess.". The assessed commit and the key legend move into or out of the comment.
- .github/workflows/pr-triage-llm.yml: drop the "already triaged" check and its permission; a PR that returns to ready is judged again.
- .github/pr-triage/judgements.json: choice_label names the changelog type in the comment (documented in _doc).
- dev-tools/scripts/tests/test_pr_triage.py: cover the new rendering.

Generated-by: Claude Sonnet 5.5
@dsmiley

dsmiley commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Example of the new format: a real run with qwen3.8-27b on the merged PR #5026, which has two mismatches (the changelog type, and a tests row that is a false positive because the test lives in AbstractInstallShardTest, outside the test globs).


Automated PR readiness judgements (LLM-generated; may be wrong; not updated on later pushes).

Consider enabling Allow edits from maintainers so Solr maintainers can contribute to this branch (optional).

2 possible mismatch(es) between what this change warrants and what the PR includes.

  • ✅ Jira issue in title: already present
  • ⚠️ Changelog — differs from the entry (suggest type: fixed; PR has type: changed): The change fixes a user-visible issue where INSTALLSHARD operations would log noise or fail gracefully due to the presence of 'write.​lock' files in offline index directories, which is a common scenario for users copying live indices.​
  • ✅ Reference Guide update: not needed
  • ✅ Upgrade notes: not needed
  • ⚠️ Tests — missing: The PR modifies production logic in RestoreCore.​java to skip write.​lock files, and the diff includes new tests in AbstractInstallShardTest.​java to verify this behavior.​

Markdown source
**Automated PR readiness judgements** (LLM-generated; may be wrong; not updated on later pushes).

Consider enabling [Allow edits from maintainers](https://docs.github.com/en/pull-requests/how-tos/work-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork) so Solr maintainers can contribute to this branch (optional).

**2 possible mismatch(es)** between what this change warrants and what the PR includes.

- ✅ [Jira issue in title](https://github.com/apache/solr/blob/main/CONTRIBUTING.md): already present
- ⚠️ [Changelog](https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc#_changelog_types_reference) **— differs from the entry** (suggest type: fixed; PR has type: changed): The change fixes a user&#45;visible issue where INSTALLSHARD operations would log noise or fail gracefully due to the presence of &#39;write&#46;​lock&#39; files in offline index directories&#44; which is a common scenario for users copying live indices&#46;​
- ✅ [Reference Guide update](https://github.com/apache/solr/tree/main/solr/solr-ref-guide): not needed <!-- This is an internal robustness fix that handles a specific file type gracefully&#59; it does not introduce new configuration&#44; parameters&#44; or user&#45;facing behaviors that require documentation in the Reference Guide&#46;​ -->
- ✅ [Upgrade notes](https://github.com/apache/solr/tree/main/solr/solr-ref-guide/modules/upgrade-notes/pages): not needed <!-- The change is additive and backward&#45;compatible&#44; simply skipping a file that was previously causing issues&#44; without removing or renaming any APIs or configuration&#46;​ -->
- ⚠️ Tests **— missing**: The PR modifies production logic in RestoreCore&#46;​java to skip write&#46;​lock files&#44; and the diff includes new tests in AbstractInstallShardTest&#46;​java to verify this behavior&#46;​

<!-- Assessed commit f220b849b9008f46a5c659703f58e08cae13b0b3 -->

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

Automated PR triage (LLM-generated; may be wrong). Checked once, when the PR was opened or marked ready; not updated on later pushes.

1 possible mismatch(es) between what this change warrants and what the PR includes.

Warranted? In PR? Confidence Why
✅ Jira issue in title no no high The changes are confined to the GitHub Actions workflow (.​github/​workflows/​pr-triage-llm.​yml) and developer tooling (dev-tools/​scripts/​pr-triage-llm.​py), which are internal CI/​developer tools not visible to Solr users or operators.​
✅ Changelog (not-worthy) no entry high This is a build/​CI and developer tooling change that modifies the automated PR triage bot's behavior and comment format, which does not warrant a release note for Solr users.​
✅ Reference Guide update no no high The PR modifies internal CI scripts and workflow files, not any user-facing configuration, API, or behavior documented in the Solr Reference Guide.​
✅ Upgrade notes no no high The change affects only the development workflow and triage bot, not the Solr server or its APIs, so there are no backwards-incompatible changes for users upgrading Solr.​
⚠️ Tests — probably not needed no yes high The changes are to build/​CI and developer tooling (workflow and scripts), which do not require new or changed tests according to the guidelines.​

⚠️ mismatch, ❔ uncertain or unanswered, ✅ agrees. Matching changelog, docs and test files were hidden from the model; the PR description may still mention them. File presence does not establish adequacy. Assessed commit c27025d48a976f669afae9229997c15ff2929030.

@dsmiley
dsmiley requested a review from janhoy November 6, 2026 21:09
@dsmiley dsmiley mentioned this pull request Nov 6, 2026
@dsmiley

dsmiley commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

I did quite a few iterations perfecting the language / presentation, and I'm now really happy with it.

Perhaps the pre-bullet header sentence can be dropped, e.g. the "2 possible mismatch(es) between what this change warrants and what the PR includes." since IMO the true first line header is fine and the bullet information reads well enough.

@dsmiley
dsmiley requested a review from epugh November 6, 2026 21:14

@janhoy janhoy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I too thought the table was a bit of a monster, this is a nice improvement. Over in the solr-operator port I also did a collapse to one single row if all checks were green...

Generated-by: Claude Sonnet 5.5
janhoy added a commit to janhoy/solr-operator that referenced this pull request Nov 8, 2026
The comment is now a bullet per judgement instead of a table. A row that
agrees with the PR says what the PR has (included, not needed, the
changelog kind, or "no concern" for the flag judgements that have nothing
to compare with) and keeps the model's reason in an HTML comment, visible
only in the Markdown source. A row that disagrees shows what the model
suggests and what the PR has ("suggest added; PR has fixed"), the
confidence when it is below high, and the reason. Unanswered rows read
"Unclear; please assess." The headline, unanswered count and footer
legend are gone, the header reads "Automated PR readiness judgements",
and the assessed commit is a hidden comment. The one-line collapse when
every judgement agrees is kept.

The workflow no longer looks for an earlier triage comment before
judging, so a PR toggled draft -> ready more than once is judged again
and gets a fresh comment. The comment marker and the issues:read
permission go with it; the API key check stays.

Solr's choice_label key is not ported; the changelog kind is shown bare.
janhoy added a commit to janhoy/solr-operator that referenced this pull request Nov 9, 2026
A choice judgement can name the value it is about, so the changelog row
reads "kind: added" and "suggest kind: fixed; PR has kind: changed"
instead of a bare word that means nothing on its own. Both changelog
judgements set it to "kind", the field name in Chart.yaml. The label is
optional and is left off an agreeing row when the PR has no entry at
all, where "not needed" already reads as a sentence.
@dsmiley
dsmiley merged commit b8f0c27 into apache:main Nov 9, 2026
4 checks passed
@dsmiley
dsmiley deleted the pr-triage-format branch November 9, 2026 14:35
@dsmiley dsmiley modified the milestones: 10.x, 11.x Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants