Skip to content

Commit a664502

Browse files
committed
ci(review): use OpenAI Responses with artifact-only cutover
1 parent c033a7c commit a664502

7 files changed

Lines changed: 522 additions & 279 deletions

File tree

‎.github/review-api.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# AI review API
2+
3+
`npm run review -- A..B` reviews the frozen head B through OpenAI Responses, using the same read-only broker locally and in GitHub Actions. It preserves the review JSON contract, source checks, coverage evidence and text check's `not checked` findings. No API key means exit 1 before any review work.
4+
5+
Set `OPENAI_API_KEY` locally. CI uses the repository secret `OPENAI_API_KEY`, passed explicitly from `tests.yml` to the reusable `review.yml`. Keys belong in the credential store or GitHub secrets, never in tracked files or command arguments. This tool does not read or change Codex subscription authentication.
6+
7+
GitHub publishes reviews only when the repository variable `REVIEW_POST_ENABLED` is exactly `true`. Leave it unset or `false` during cutover: code reviews still run and upload artifacts, with the posting job skipped.
8+
9+
The default review role is `gpt-6.1-sol` at `medium` reasoning. `REVIEW_MODEL` overrides `OPENAI_MODEL`; `REVIEW_REASONING_EFFORT` overrides `OPENAI_REASONING_EFFORT`. In CI configure repository variables `REVIEW_MODEL` and `REVIEW_REASONING_EFFORT`; empty variables retain the defaults. Use `high` for a difficult review. The separate small judging role may use `gpt-6-luna` at `low` or `medium`; changing that role does not require changing the review role.
10+
11+
The broker exposes only file reads inside an exported head and its diff. A text check also permits bounded read-only git/gh commands without a shell. Broker subprocesses inherit no OpenAI key or unrelated secret. A GitHub token reaches only gh for its read operations. The model has no arbitrary shell, web, write or subprocess tool. The loop allows 8 tool calls for code or 16 for text, 20 response turns, and 12 minutes total. Tool output is bounded and credentials are redacted before the next request. Incomplete responses, HTTP errors and credential echoes fail the review. `answer.json` records usage; `session.jsonl` records redacted API output.
12+
13+
Offline checks: `node --test test/test-openai-review.js` plus the repository's existing CI review test suite. Tests use injected fetch or a loopback HTTP server and synthetic credentials. `REVIEW_OPENAI_URL` is restricted to the official endpoint or a loopback test server.
14+
15+
After provisioning a key, run `npm run review -- HEAD~1..HEAD --out test-results/openai-canary` on a small, known change. Check the reported head, model, usage, source evidence and absence of credentials in artifacts. Then run `npm run review -- HEAD~1..HEAD --text .github/review-api.md --out test-results/openai-text-canary`. Offline checks do not establish live account or model access.
16+
17+
Rollback: revert the migration commit in the candidate branch, restore the previous Gemini workflow variable and secret mapping, and rerun the previous review command. Keep the old Gemini secret until the OpenAI canary and first CI review succeed. This change does not delete or rewrite historical Gemini results or evaluations.
18+
19+
API compatibility: [GPT-6.1 Sol](https://developers.openai.com/api/docs/models/gpt-6.1-sol) requires Responses for tool calling; the [function calling guide](https://developers.openai.com/api/docs/guides/function-calling) documents function outputs linked by `call_id`.

‎.github/workflows/review.yml‎

Lines changed: 13 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
# (workflow_call) once its test jobs are done, for a push to a PR with the label
33
# `review`, adding that label, or a maintainer's `/review` comment, with the run's
44
# results and coverage. On demand it runs alone, with no tests (gh workflow run
5-
# review.yml -f commit=<sha> -f base=<sha or branch>). Gemini CLI reads the PR's
5+
# review.yml -f commit=<sha> -f base=<sha or branch>). OpenAI Responses reads the PR's
66
# code with read-only tools and answers with findings, by the rules in
77
# .github/review-prompt.md, through test/ci-review.js run: the same script as
88
# `npm run review -- A..B` on a machine. A last job posts the findings as one review
@@ -26,7 +26,7 @@ on:
2626
type: string
2727
required: true
2828
secrets:
29-
GEMINI_API_KEY:
29+
OPENAI_API_KEY:
3030
required: false
3131
# On demand, for anyone with write access; the maintainer check still applies.
3232
# The open PR whose head the commit is gets the review; with none, the job summary.
@@ -100,12 +100,12 @@ jobs:
100100
if: needs.request.outputs.review == 'true'
101101
runs-on: ubuntu-latest
102102
timeout-minutes: 15
103-
# The Gemini key is used here, on the PR's code: so the token only reads, no
103+
# The OpenAI key is used here, on the PR's code: so the token only reads, no
104104
# other secret is named, and the model has read-only file tools, no shell, no
105105
# web and no GitHub tools (test/ci-review.js run). Our script and prompt are
106106
# main's on a comment's or on-demand run; on a push or a label they're the PR's
107107
# own, as for the tests (a fork PR gets no secret). Writing is the post job's.
108-
# The script stops Gemini CLI after 12 minutes, inside this job's 15.
108+
# The script stops OpenAI Responses after 12 minutes, inside this job's 15.
109109
# Reads only: actions: read is for a text check's gh run view
110110
permissions:
111111
contents: read
@@ -124,7 +124,8 @@ jobs:
124124
/.github/text-check-prompt.md
125125
/.github/writing-style.md
126126
/test/ci-review.js
127-
/test/ci-review-budget.js
127+
/test/helpers/openai-review.js
128+
/test/helpers/review-tools.js
128129
/test/check-coverage-report.js
129130
/test/helpers/node-test-runner.js
130131
/test/helpers/ci-platforms.js
@@ -160,10 +161,11 @@ jobs:
160161
pattern: coverage-*
161162
path: ci
162163
# Without the secret it stops at once and fails, so the review never passes unseen
163-
- name: Review with Gemini
164+
- name: Review with OpenAI
164165
env:
165-
GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
166-
GEMINI_MODEL: ${{ vars.GEMINI_MODEL }}
166+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
167+
REVIEW_MODEL: ${{ vars.REVIEW_MODEL }}
168+
REVIEW_REASONING_EFFORT: ${{ vars.REVIEW_REASONING_EFFORT }}
167169
# For the PR's title and description; the model never gets it
168170
GH_TOKEN: ${{ github.token }}
169171
PR: ${{ needs.request.outputs.pr }}
@@ -175,7 +177,7 @@ jobs:
175177
text=()
176178
if [[ "$CHECK" == text ]]; then text=(--text pr); fi
177179
node test/ci-review.js run "$BASE_SHA..$HEAD_SHA" --repo pr --out review --tests ci "${text[@]}"
178-
# review.json, review.md, the diff and Gemini CLI's own JSON; also when the review
180+
# review.json, review.md, the diff and OpenAI Responses's own JSON; also when the review
179181
# failed, so a stopped run's answer.json and session.jsonl show what it spent
180182
- name: Upload the review
181183
if: always()
@@ -189,8 +191,8 @@ jobs:
189191
post:
190192
name: post
191193
needs: [request, review]
192-
# A text check posts nothing: its job summary has it
193-
if: inputs.check != 'text'
194+
# Posting starts only after the operator enables it; text checks always stay in artifacts.
195+
if: inputs.check != 'text' && vars.REVIEW_POST_ENABLED == 'true'
194196
runs-on: ubuntu-latest
195197
timeout-minutes: 5
196198
# Writes the review: our script (main's on a comment's or on-demand run, the

‎.github/workflows/tests.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -280,7 +280,7 @@ jobs:
280280
sha: ${{ needs.platforms.outputs.sha }}
281281
base: ${{ needs.platforms.outputs.base }}
282282
secrets:
283-
GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
283+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
284284

285285
comment:
286286
name: comment

‎test/README.md‎

Lines changed: 4 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -139,27 +139,10 @@ A change to a shared module reaches most tests, because many tests start the who
139139

140140
## AI review
141141

142-
`npm run review -- <commit A>..<commit B>` reviews the change from A to B with Gemini CLI, by the rules in `.github/review-prompt.md` and the writing style in `.github/writing-style.md`, on your machine. It needs the key in `DC_GEMINI_API_KEY`; without it, it stops at once and fails. It writes `test-results/review/`:
143-
- `review.md`: one line with the result and the summary, then a table `| Kind | Where | What |`, one line per finding;
144-
- `review.json`: the same findings, each with its kind (real bug, nitpick, handled elsewhere, not checked), file, line and text, and the run's tokens;
145-
- `diff.patch`, and `answer.json`, Gemini CLI's own JSON.
142+
`npm run review -- A..B` reviews the frozen head B through OpenAI Responses, using `OPENAI_API_KEY` locally and the repository secret of the same name in CI. The model is `gpt-6.1-sol` at medium reasoning; `REVIEW_MODEL` and `REVIEW_REASONING_EFFORT` select another explicit configuration. See [API configuration and safeguards](../.github/review-api.md).
146143

147-
The review is given the run's tests: the files that failed, and the changed lines no test ran, from `lcov.info` with every platform together. Each uncovered range becomes a `not covered` finding unless the PR says why. On your machine it uses `coverage/` and `test-results/` from the last `npm run test:coverage` only if that ran at B on a clean tree (its stamp, `coverage/run.json`); `--tests <folder>` names another run. Without results it says "with no test results".
144+
The output folder is `test-results/review/` unless `--out` names another one. It contains `review.md`, `review.json`, `diff.patch`, `answer.json` and a redacted `session.jsonl`. Missing credentials or an incomplete API response fail the review. The broker reads only the exported head and diff; code review has no shell or writes. Text checks retain bounded read-only git/gh evidence reads, `--results` inputs and explicit `not checked` findings.
148145

149-
**The text check** reads a text about the change, usually a PR's description, the same way: `npm run review -- A..B --text <file> [--results <file>]...`. Gemini checks each claim in it against the diff, the code at B and the results files, and checks the writing by `.github/writing-style.md` (the rules: `.github/text-check-prompt.md`). It writes `text-check.md`: a `**Result:**` line (`clean`, or counts such as `1 wrong claim, 2 nitpicks`), one `- **Wrong claim**` or `- **Nitpick**` line a finding with the text's line, and `### Checked` with the claims that hold and where. Also `text-check.json`. On GitHub: `gh workflow run review.yml -f commit=<sha> -f base=<sha or branch> -f check=text` checks the PR's description and puts it in the job summary; it posts nothing.
146+
The review receives the run's tests and coverage through `--tests`, or a local clean coverage stamp at B. Uncovered changed lines remain findings. `npm run review -- A..B --text <file> [--results <file>]...` checks a text against that evidence and writes `text-check.md` and `text-check.json`.
150147

151-
In CI the review comes after the tests: `.github/workflows/tests.yml` runs the PR's labelled platforms, then calls `.github/workflows/review.yml` (the same script, with the `GEMINI_API_KEY` secret), passed or failed, with the run's results and coverage. The same maintainers start it, in the same ways as the tests:
152-
- **Add the `review` label** to review the PR at once, alone: its platforms run on their own labels' events, so that run has no test results. A push to a PR with the label tests the labelled platforms, then reviews.
153-
- **From a PR comment:** a line `/review`, with the same labelled platforms first. It only works once the workflows are on main, as for `/test`.
154-
- **With no platform label,** the review runs alone and says it had no test results.
155-
- **On demand:** `gh workflow run review.yml -f commit=<sha> -f base=<sha or branch>`, a review alone, also once it's on main.
156-
157-
What CI posts: one review with event COMMENT, never an approval. Its body is one line with the result and the summary, then the table of every finding; a finding on a line the diff changes is also a one-line comment on that line. The run's job summary and its `review` artifact always have every finding, so a commit no PR has as its head still gets them there. Without the secret, the review job fails.
158-
159-
How it stays safe (`test/ci-review.js`, `test/helpers/ci-review.js`):
160-
- `tests.yml`'s first job decides with `reviewFor` (`test/helpers/ci-platforms.js`, through `test/ci-review.js request`) and reads the PR's head and base; `review.yml`'s `request` job takes them as given, or decides itself on demand.
161-
- `review` holds the only secret, `GEMINI_API_KEY`, with a read-only token. Gemini CLI (`@google/gemini-cli`, a fixed version) runs headless in a folder of its own: our settings, its own Gemini home, B's files written by git (a link as a plain file), and the diff. It can only read files (`read_file`, `list_directory`, `glob`, `grep_search`): no shell, no web, no GitHub tools, no GitHub token. An answer that holds the key writes no review.
162-
- `post` has the write token and no secret. It reads `review.json` as data. If GitHub refuses a line comment, every finding goes in the summary. @-mentions in the model's text are broken, so a review notifies no one.
163-
- **Whose script runs:** on a comment's or on-demand run, main's (GitHub starts those from main), and the PR's head is only read, by SHA. On a push or a label, the PR's own, as for the tests: a fork PR gets no secret, and a branch in the repo could change the workflow itself anyway.
164-
165-
Set up once: the repository secret `GEMINI_API_KEY`, the label `review`, and optionally the variable `GEMINI_MODEL`. Without it the review uses `gemini-3.8-flash`, one model for every request (Gemini CLI's own routing between models is off); locally, set `GEMINI_MODEL` the same way. The model gets the diff in its prompt, and 8 tool calls to read files, the diff's files first; a hook (`test/ci-review-budget.js`) refuses later calls, and it answers. A review that runs over 12 minutes is stopped, with everything it started.
148+
`tests.yml` runs the labelled platforms and then calls `review.yml` with the named OpenAI secret. A maintainer may dispatch `review.yml` with `--ref <workflow-branch>`, `commit`, `base` and `check=code` or `check=text`. During cutover leave `REVIEW_POST_ENABLED` unset or `false`: artifacts are uploaded, and the job that posts review comments is skipped. Set it to exactly `true` after accepting the artifacts. Text checks always stay in the job summary and artifacts.

0 commit comments

Comments
 (0)