Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 75 additions & 9 deletions .github/actions/telegram-alert/action.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
name: Telegram failure alert
name: Telegram status alert
description: >
Sends a Telegram message for a failed run. Skips with a notice when the token or chat ID is
empty, and only warns if sending fails, so it never changes the job result.
Sends a Telegram message for a failed run, and with previous-workflow set only when the result
changed since the previous run: one message when it starts failing, one when it recovers, none
while the status stays the same. Skips with a notice when the token or chat ID is empty, and only
warns if deciding or sending fails, so it never changes the job result. Needs Node.js on the
runner (preinstalled on ubuntu-latest).
inputs:
bot-token:
description: Telegram bot token (pass secrets.TELEGRAM_BOT_TOKEN).
Expand All @@ -14,38 +17,101 @@ inputs:
summary:
description: What failed and where, e.g. "smoke test failed (Production, main)".
required: true
recovered-summary:
description: >
What recovered and where, e.g. "smoke test recovered (Production, main)". Used when the
result changes from failure to success; a generic text is used when empty.
required: false
default: ""
failed-checks-file:
description: >
Optional path to a file with one line naming the failing checks (written by pnpm smoke / e2e
via FAILED_CHECKS_FILE). Ignored when missing.
required: false
default: ""
result:
description: >
Result being reported: "success" or "failure" (pass job.status or steps.<id>.outcome).
Any other value, such as "cancelled" or "skipped", sends nothing. The default suits callers
that run this action only after a failure.
required: false
default: failure
previous-workflow:
description: >
File name of the workflow whose previous completed run to compare with, e.g. smoke.yml. Only
the previous run of the same target counts (see previous-match). When empty, nothing is
compared: a failure always alerts and a success never does. When the previous result cannot
be read or does not exist, the same applies.
required: false
default: ""
previous-match:
description: >
How to recognise a run of the same target: "title" (same run name, for workflows whose
run-name carries the environment) or "branch" (same branch, for workflows where the branch
implies the environment).
required: false
default: title
previous-step:
description: >
Optional name of a step. When set, a previous run's result is that step's conclusion instead
of the run's, and previous runs where the step did not finish are skipped.
required: false
default: ""
github-token:
description: >
Token for reading previous runs (pass github.token); the job needs "actions: read".
required: false
default: ""
runs:
using: composite
steps:
- id: decide
shell: bash
env:
RESULT: ${{ inputs.result }}
PREVIOUS_WORKFLOW: ${{ inputs.previous-workflow }}
PREVIOUS_MATCH: ${{ inputs.previous-match }}
PREVIOUS_STEP: ${{ inputs.previous-step }}
GITHUB_TOKEN: ${{ inputs.github-token }}
run: |
# Writes send=failed|recovered|none to $GITHUB_OUTPUT. If the script breaks, or exits without
# writing a decision (for example it was not run as the entry point), fall back to the old
# behaviour (alert on failure) rather than lose a failure alert.
if ! node "$GITHUB_ACTION_PATH/status-change.mjs" || ! grep -q '^send=' "$GITHUB_OUTPUT"; then
echo "::warning::Could not decide whether the status changed; alerting on failures only."
if [ "$RESULT" = failure ]; then echo "send=failed" >> "$GITHUB_OUTPUT"; fi
fi

# The token goes to curl through stdin (-K -), so it never appears in argv or the log.
- shell: bash
- if: steps.decide.outputs.send == 'failed' || steps.decide.outputs.send == 'recovered'
shell: bash
env:
TELEGRAM_BOT_TOKEN: ${{ inputs.bot-token }}
TELEGRAM_CHAT_ID: ${{ inputs.chat-id }}
SEND: ${{ steps.decide.outputs.send }}
SUMMARY: ${{ inputs.summary }}
RECOVERED_SUMMARY: ${{ inputs.recovered-summary }}
FAILED_CHECKS_FILE: ${{ inputs.failed-checks-file }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
if [ -z "$TELEGRAM_BOT_TOKEN" ] || [ -z "$TELEGRAM_CHAT_ID" ]; then
echo "::notice::Telegram alert skipped: TELEGRAM_BOT_TOKEN or TELEGRAM_CHAT_ID is not set."
exit 0
fi
# Short on purpose: what failed, which checks (already capped), and the run link.
text="❌ $SUMMARY"
if [ -n "$FAILED_CHECKS_FILE" ] && [ -s "$FAILED_CHECKS_FILE" ]; then
text="$text
# Short on purpose: what changed, which checks failed (already capped), and the run link.
if [ "$SEND" = recovered ]; then
text="✅ ${RECOVERED_SUMMARY:-recovered: $SUMMARY}"
else
text="❌ $SUMMARY"
if [ -n "$FAILED_CHECKS_FILE" ] && [ -s "$FAILED_CHECKS_FILE" ]; then
text="$text
Failing: $(head -c 800 "$FAILED_CHECKS_FILE" | head -n 1)"
fi
fi
text="$text
$RUN_URL"
printf 'url = "https://api.telegram.org/bot%s/sendMessage"\n' "$TELEGRAM_BOT_TOKEN" |
curl --silent --show-error --fail --max-time 15 --output /dev/null --config - \
curl --silent --show-error --fail --retry 2 --max-time 15 --output /dev/null --config - \
--data-urlencode "chat_id=$TELEGRAM_CHAT_ID" \
--data-urlencode "text=$text" \
|| echo "::warning::Telegram alert could not be sent."
169 changes: 169 additions & 0 deletions .github/actions/telegram-alert/status-change.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,169 @@
/**
* Decides whether a finished run should send a Telegram message, by comparing its result with the
* previous completed run of the same workflow and target (read from the GitHub API).
*
* failing now, not failing before (or unknown) -> "failed"
* passing now, failing before -> "recovered"
* anything else -> "none"
*
* "Unknown previous" (first run, API error, nothing comparable) alerts on a failure and stays
* silent on a pass, so a broken lookup can only produce an extra alert, never hide a failure.
* Lookups never throw out of `run()`, so the alert step cannot change the job result.
*
* Lives in the action directory, not in happyview-modules/tooling, so deploy.yml can fetch it from
* the workflow's own commit when it rolls back to an older commit that predates it.
*/
import { appendFileSync } from "node:fs";
import { pathToFileURL } from "node:url";

/** Most recent completed runs read per lookup. */
const PAGE_SIZE = 50;
/** Comparable runs examined before giving up; bounds the per-run job lookups in step mode. */
const MAX_CANDIDATES = 10;
const API_TIMEOUT_MS = 15_000;
/** @type {Record<Result, string>} */
const STATE = { pass: "passing", fail: "failing" };

/** @typedef {"pass" | "fail"} Result */
/** @typedef {"failed" | "recovered" | "none"} Alert */
/** @typedef {(path: string) => Promise<any>} GithubApi */

/**
* Maps a GitHub conclusion, or a job/step outcome, to a result. Cancelled, skipped and anything
* else is not a result.
* @param {string | null | undefined} conclusion
* @returns {Result | undefined}
*/
export function toResult(conclusion) {
if (conclusion === "success") return "pass";
if (conclusion === "failure" || conclusion === "timed_out") return "fail";
return undefined;
}

/**
* @param {Result | undefined} current
* @param {Result | undefined} previous
* @returns {Alert}
*/
export function decideAlert(current, previous) {
if (current === "fail") return previous === "fail" ? "none" : "failed";
if (current === "pass") return previous === "fail" ? "recovered" : "none";
return "none";
}

/**
* Result of the step called `stepName` in `run`, or undefined when that step did not run or did not
* finish (for example the deploy failed before the smoke test started).
* @param {GithubApi} api @param {string} repo @param {{ id: number }} run @param {string} stepName
* @returns {Promise<Result | undefined>}
*/
async function stepResult(api, repo, run, stepName) {
const { jobs } = await api(`/repos/${repo}/actions/runs/${run.id}/jobs?per_page=100`);
for (const job of jobs) {
const step = (job.steps ?? []).find((/** @type {{ name: string }} */ s) => s.name === stepName);
if (step) return toResult(step.conclusion);
}
return undefined;
}

/**
* Latest completed run of `workflow`, before the current one, for the same target, that produced a
* pass or a fail. "Same target" is the current run's `display_title` (match "title", used where
* run-name carries the environment) or its branch (match "branch", used where the environment is
* implied by the branch). With `step`, a run's result is that step's conclusion rather than the
* run's, and runs where the step did not finish are skipped.
* @param {{ api: GithubApi, repo: string, runId: string, workflow: string,
* match: string, step?: string }} options
* @returns {Promise<{ result: Result, url: string } | undefined>}
*/
export async function findPreviousResult({ api, repo, runId, workflow, match, step }) {

Check failure on line 79 in .github/actions/telegram-alert/status-change.mjs

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Refactor this function to reduce its Cognitive Complexity from 17 to the 15 allowed.

See more on https://sonarcloud.io/project/issues?id=hypercerts-org_api&issues=AaHjpxQsO-1q6mRsgVH3&open=AaHjpxQsO-1q6mRsgVH3&pullRequest=77
if (match !== "title" && match !== "branch")
throw new Error(`previous-match must be "title" or "branch", got "${match}".`);
const current = await api(`/repos/${repo}/actions/runs/${runId}`);
const sameTarget =
match === "branch"
? (/** @type {any} */ run) => run.head_branch === current.head_branch
: (/** @type {any} */ run) => run.display_title === current.display_title;

const query = new URLSearchParams({ status: "completed", per_page: String(PAGE_SIZE) });
if (match === "branch") query.set("branch", current.head_branch);
const { workflow_runs: runs } = await api(
`/repos/${repo}/actions/workflows/${workflow}/runs?${query}`,
);

const candidates = runs
.filter((/** @type {any} */ run) => String(run.id) !== String(runId) && sameTarget(run))
.slice(0, MAX_CANDIDATES);
for (const run of candidates) {
const result = step ? await stepResult(api, repo, run, step) : toResult(run.conclusion);
if (result) return { result, url: run.html_url };
}
return undefined;
}

/**
* Reads its inputs from `env`, as the composite action passes them.
* @param {Record<string, string | undefined>} env @param {GithubApi} api
* @returns {Promise<{ send: Alert, notice: string, warning?: string }>}
*/
export async function run(env, api) {
const current = toResult(env.RESULT);
const workflow = env.PREVIOUS_WORKFLOW;
/** @type {Awaited<ReturnType<typeof findPreviousResult>>} */
let previous;
let warning;
if (current && workflow) {
try {
previous = await findPreviousResult({
api,
repo: env.GITHUB_REPOSITORY ?? "",
runId: env.GITHUB_RUN_ID ?? "",
workflow,
match: env.PREVIOUS_MATCH || "title",
step: env.PREVIOUS_STEP || undefined,
});
} catch (error) {
const reason = error instanceof Error ? error.message : String(error);
warning = `Could not read previous ${workflow} runs (${reason}); treating the previous status as unknown.`;
}
}
const send = decideAlert(current, previous?.result);
const now = current ? STATE[current] : `"${env.RESULT}"`;
const before = previous
? `${STATE[previous.result]} in the previous run (${previous.url})`
: "of unknown status before";
return { send, warning, notice: `Telegram alert: ${send}. Now ${now}, ${before}.` };
}

/**
* @param {{ token?: string, baseUrl?: string, fetchImpl?: typeof fetch }} options
* @returns {GithubApi}
*/
export function createApi({ token, baseUrl = "https://api.github.com", fetchImpl = fetch }) {
return async (path) => {
// The token only ever travels in this header: not in argv, the URL or any log line.
const response = await fetchImpl(`${baseUrl}${path}`, {
headers: {
Accept: "application/vnd.github+json",
"X-GitHub-Api-Version": "2022-11-28",
...(token ? { Authorization: `Bearer ${token}` } : {}),
},
signal: AbortSignal.timeout(API_TIMEOUT_MS),
});
if (!response.ok) throw new Error(`GET ${path} returned HTTP ${response.status}`);
return response.json();
};
}

async function main() {
const env = process.env;
const api = createApi({ token: env.GITHUB_TOKEN, baseUrl: env.GITHUB_API_URL || undefined });
const { send, notice, warning } = await run(env, api);
if (warning) process.stdout.write(`::warning::${warning}\n`);
process.stdout.write(`::notice::${notice}\n`);
const line = `send=${send}\n`;
if (env.GITHUB_OUTPUT) appendFileSync(env.GITHUB_OUTPUT, line);
else process.stdout.write(line);
}

if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) await main();
52 changes: 40 additions & 12 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,10 @@ jobs:
needs: resolve
runs-on: ubuntu-latest
environment: ${{ needs.resolve.outputs.environment }}
# actions: read lets the alert step read this workflow's previous runs.
permissions:
contents: read
actions: read
# Deploys to one environment (install + smoke) run one at a time and are never cancelled midway.
concurrency:
group: deploy-${{ needs.resolve.outputs.environment }}
Expand Down Expand Up @@ -145,36 +149,60 @@ jobs:

# Smoke test runs in this job so the deploy lock above covers it: another deploy to the same
# environment cannot start until this check finishes. Anonymous GETs only, no secrets.
# A rollback to a commit that predates smoke.js skips it.
# A rollback to a commit that predates smoke.js skips it (the step shows as skipped, so it
# never counts as a smoke result). The step name is also read by the alert step below.
- name: Note that the deployed commit has no smoke test
if: hashFiles('happyview-modules/tooling/smoke.js') == ''
run: |
echo "::notice::Skipping smoke test: the deployed commit predates it."

- name: Smoke test the deployed API
id: smoke
if: hashFiles('happyview-modules/tooling/smoke.js') != ''
timeout-minutes: 5
env:
API_BASE_URL: ${{ vars.HAPPYVIEW_BASE_URL }}
FAILED_CHECKS_FILE: ${{ runner.temp }}/failed-checks.txt
run: |
if [ ! -f happyview-modules/tooling/smoke.js ]; then
echo "::notice::Skipping smoke test: the deployed commit predates it."
exit 0
fi
pnpm smoke
run: pnpm smoke

# A rollback checks out an older commit that may lack the alert action, so fetch it from the
# workflow's own commit.
# workflow's own commit. This now runs after a good deploy too, so a failed fetch must not turn
# that deploy red (continue-on-error); the alert steps below run only if it succeeded.
- name: Check out alert action
if: failure()
id: alert-action
if: success() || failure()
continue-on-error: true
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ github.sha }}
path: .ci-alert
sparse-checkout: .github/actions
persist-credentials: false

- name: Send Telegram alert on failure
if: failure()
# A failed deploy (anything other than the smoke test) alerts every time: each deploy is a
# one-off event, so there is no repeating alert to suppress.
- name: Send Telegram alert on deploy failure
if: failure() && steps.smoke.outcome != 'failure' && steps.alert-action.outcome == 'success'
uses: ./.ci-alert/.github/actions/telegram-alert
with:
bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }}
chat-id: ${{ secrets.TELEGRAM_CHAT_ID }}
summary: deploy failed (${{ needs.resolve.outputs.environment }}, ${{ needs.resolve.outputs.branch }})

# The post-deploy smoke test alerts only when its result differs from the last deploy to this
# environment (same branch) whose smoke test finished: one alert when it starts failing, one
# when it recovers. Deploys that failed before the smoke test do not count.
- name: Send Telegram alert when the smoke status changes
if: (success() || failure()) && steps.alert-action.outcome == 'success' && (steps.smoke.outcome == 'success' || steps.smoke.outcome == 'failure')
uses: ./.ci-alert/.github/actions/telegram-alert
with:
bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }}
chat-id: ${{ secrets.TELEGRAM_CHAT_ID }}
github-token: ${{ github.token }}
result: ${{ steps.smoke.outcome }}
previous-workflow: deploy.yml
previous-match: branch
previous-step: Smoke test the deployed API
failed-checks-file: ${{ runner.temp }}/failed-checks.txt
summary: ${{ steps.smoke.outcome == 'failure' && 'post-deploy smoke test failed' || 'deploy failed' }} (${{ needs.resolve.outputs.environment }}, ${{ needs.resolve.outputs.branch }})
summary: post-deploy smoke test failed (${{ needs.resolve.outputs.environment }}, ${{ needs.resolve.outputs.branch }})
recovered-summary: post-deploy smoke test recovered (${{ needs.resolve.outputs.environment }}, ${{ needs.resolve.outputs.branch }})
Loading
Loading