Skip to content

Retry gh-pages pushes rejected by concurrent writers - #3146

Merged
antonwolfy merged 5 commits into
masterfrom
gh-pages-retry-publish
Nov 11, 2026
Merged

antonwolfy merged 5 commits into
masterfrom
gh-pages-retry-publish

Conversation

@antonwolfy

Copy link
Copy Markdown
Contributor

Three jobs write to gh-pages: the master docs deploy, the PR preview publish, and the clean job that removes a closed PR's preview. Each pushed once with no retry (peaceiris/actions-gh-pages makes a single git push, and so did clean), so a run that raced another writer failed with ! [rejected] gh-pages -> gh-pages (fetch first). This happened in 3 of the ~100 Sphinx runs since 2026-10-26, two of them master deploys, which left the published docs stale until the next merge.

This PR routes every gh-pages write through a new .github/scripts/update_gh_pages.sh:

  • Each attempt fetches the current branch tip, rebuilds this writer's change on top of it and pushes without force. A rejected push is retried on the new tip, up to 5 attempts with a growing random delay.
  • It works on a shallow, blobless clone and only touches the git index, so the 6 GB, 328k-file site is never checked out. Only commits and trees are fetched (~110 MB); no old file contents are downloaded.
  • Modes: publish adds files to the site root (keeping other files, as before) or replaces pull/<N>; remove deletes directories; prune deletes pull/<N> for every closed or merged PR.
  • The token is passed through environment-only git config, not a command line or a file.

build-sphinx.yml now calls the script for all three writes, which drops peaceiris/actions-gh-pages and the token-in-URL push in clean. clean no longer needs a full-history checkout.

The new prune-pr-docs.yml runs prune weekly and on manual dispatch. Until #3144, clean only ran when the docs build on close succeeded, so 118 previews of closed PRs (4.7 GB of current files) are left on gh-pages. The first prune run will remove them; only numeric pull/<N> directories are considered, and a failed PR lookup keeps the preview.

Two behaviour differences from the old action:

  • A republished preview replaces pull/<N>, so pages removed from the docs disappear (the action kept old files).
  • A rebuild with no changes makes no commit (the action made an empty one).

Tested locally against a scratch remote that allows blobless clones like GitHub:

  • every mode works, the no-change cases push nothing, and bad arguments and unsafe destination paths are rejected;
  • 8 parallel publishes had 8-10 rejected pushes across runs, and all 8 previews landed every time;
  • when every push is refused, the script fails after the configured number of attempts;
  • no mode fetches old file contents.

Also tested against a local mirror of the real gh-pages: republishing a 2,010-file preview with one file changed committed exactly that change, and prune with live PR states removed exactly the 118 previews of closed PRs and kept the 17 open ones.

The master deploy and prune will first run after merge; this PR's own Sphinx run exercises the preview publish.

  • Have you provided a meaningful PR description?
  • Have you added a test, reproducer or referred to an issue with a reproducer?
  • Have you tested your changes locally for CPU and GPU devices?
  • Have you made sure that new changes do not introduce compiler warnings?
  • Have you checked performance impact of proposed changes?
  • Have you added documentation for your changes, if necessary?
  • Have you added your changes to the changelog?

The master docs deploy, the PR preview publish and the clean job each
pushed to gh-pages once, so a run that raced another writer failed with
"rejected (fetch first)" (3 of ~100 Sphinx runs, two of them master
deploys). Route every gh-pages write through
.github/scripts/update_gh_pages.sh, which rebuilds its change on the
latest branch tip and retries a rejected push. It works on a blobless,
shallow clone and only touches the index, so the 6 GB site is never
checked out.

The clean job only ran when the docs build on close succeeded, so 118
previews of closed PRs (4.7 GB) were left behind. Add a weekly workflow
that removes previews of closed and merged PRs.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The review identifies no active issues in the documentation publishing and preview-cleanup changes.

No open issues 🎉

What changed in this PR

This PR makes documentation publishing more resilient to concurrent gh-pages updates and adds routine cleanup of previews for closed pull requests.

Changes:

  • Centralizes publishing and removal in a shared script that retries pushes on the latest branch tip without checking out the site.
  • Routes existing docs deployment and cleanup steps through the script; republished previews replace their previous contents.
  • Adds weekly and manually triggered pruning of closed-PR previews, retaining previews when PR lookups fail.
File Description
.github/​workflows/​prune-pr-docs.yml Adds scheduled and manual cleanup of closed-PR previews.
.github/​workflows/​build-sphinx.yml Uses the shared script for all docs writes and removes the full-history cleanup checkout.
.github/​scripts/​update_gh_pages.sh Implements index-only publishing, removal, and pruning with push retries.
.github/​AGENTS.md Documents the requirement to route all gh-pages writes through the script.

Review effort: Balanced

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

View rendered docs @ https://intelpython.github.io/dpnp/index.html

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Array API standard conformance tests for dpnp=0.22.0dev1=np2py314h8d9cdd5_16 ran successfully.
Passed: 1491
Failed: 0
Skipped: 6

@coveralls

coveralls commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Coverage Status

coverage: 84.424%. remained the same — gh-pages-retry-publish into master

@ndgrigorian ndgrigorian left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments, mostly on how the script behaves when something fails

Comment thread .github/AGENTS.md Resolved
# Unlike read-tree --prefix, this doesn't fetch blobs of existing entries
GIT_INDEX_FILE="${src_index}" git ls-files -s -z |
sed -z "s|\t|\t${prefix}|" |
git update-index -z --index-info

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

peaceiris/actions-gh-pages adds a .nojekyll by default, and nothing does that now. It's on gh-pages only because the action left it there, so if it's ever removed or the branch gets recreated, Pages runs Jekyll, drops _static/ and _sources/, and the docs lose their CSS and JS. Could we add an empty one at the root when it's missing? Something like:

        if [[ -z "$(git ls-files -- .nojekyll)" ]]; then
            git update-index --add --cacheinfo \
                "100644,$(git hash-object -w --stdin </dev/null),.nojekyll"
        fi

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs build already writes it: doc/conf.py enables sphinx.ext.githubpages, which puts an empty .nojekyll into doc/_build/html/, and the script publishes dotfiles too. This PR's own preview has it: pull/3146/.nojekyll. Every master deploy publishes that output to the site root, so the root .nojekyll is added back on each one, with no change while it's there.

The only gap is a recreated branch that gets PR previews before the next master deploy, so I'd rather not add it to the script as well. Does that work for you?

Comment thread .github/scripts/update_gh_pages.sh Resolved
Comment thread .github/scripts/update_gh_pages.sh Resolved
Comment thread .github/scripts/update_gh_pages.sh Resolved
antonwolfy and others added 2 commits November 6, 2026 12:43
Retry a failed push only when the remote tip has moved, so a failure
other than a concurrent writer stops at once instead of after all
attempts. Check the arguments before the clone. In prune, check that
pull/ exists, let set -e catch a failed listing, warn about each failed
pull request lookup and fail when all of them fail. Fix the script path
in .github/AGENTS.md.

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Thank you @antonwolfy

@antonwolfy
antonwolfy merged commit 79a74cb into master Nov 11, 2026
44 checks passed
@antonwolfy
antonwolfy deleted the gh-pages-retry-publish branch November 11, 2026 14:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants