Skip to content

Apply patches: to packages built from a git source - #148

Merged
pbuncic merged 7 commits into
bitsorg:mainfrom
m-fol:stage-patches-after-clone
Nov 18, 2026
Merged

pbuncic merged 7 commits into
bitsorg:mainfrom
m-fol:stage-patches-after-clone

Conversation

@m-fol

@m-fol m-fol commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

patches: are not used for packages built from a git source:. Since a recipe can get its source in two ways: a tarball (sources:) or a git repository (source: + tag:), this feature can be used if a package needs a fix on top of an upstream version and is building from git. Examples:

  • the version is a branch or a tag with no published tarball, for example master in a nightly or a patch branch,
  • a defaults file overrides a package to build from git, for example dev3’s sources: [] overrides,
  • an experiment needs its own patched version of a package

Without this change, the only workarounds are to find or create a tarball, or to apply the patch by hand in the recipe body.

Changes:

  • The patch copy moves into _stage_patches() and runs once the source is in place: before the download for tarballs (unchanged) and after the checkout for git sources.
  • After git checkout -f on a reused checkout, .bits_patched and .bits_applied_patches are removed, so the patches are applied again.

Reproducer (builds a test package from a git source with one patch to a txt file):

(
T=$(mktemp -d); cd $T || exit 1

run() {
  W=$T/$2; mkdir -p $W; cd $W || return
  git clone -q --depth 1 -b $2 https://github.com/$1.git bits

  git init -q upstream; echo "NOT PATCHED" > upstream/status.txt
  git -C upstream add status.txt
  git -C upstream -c user.name=t -c user.email=t@t commit -q -m init
  git -C upstream tag v1.0

  mkdir -p recipes/patches
  printf 'package: defaults-release\nversion: v1\n---\n' > recipes/defaults-release.sh
  printf 'package: demo\nversion: "1.0"\ntag: v1.0\nsource: %s/upstream\npatches:\n  - fix.patch\n---\ncp $SOURCEDIR/status.txt $INSTALLROOT/\nfalse\n' $W > recipes/demo.sh
  printf -- '--- a/status.txt\n+++ b/status.txt\n@@ -1 +1 @@\n-NOT PATCHED\n+PATCHED\n' > recipes/patches/fix.patch
  git -C recipes init -q; git -C recipes add -A
  git -C recipes -c user.name=t -c user.email=t@t commit -q -m recipes

  echo "== $2"
  bits/bitsBuild build demo -c recipes --defaults release --no-system -w sw > run1.log 2>&1
  grep -m1 "Can't open patch file" run1.log
  echo "run 1 (fresh clone):    $(cat sw/SOURCES/demo/1.0/v1.0/status.txt)"

  sed -i '/^false/d' recipes/demo.sh
  git -C recipes -c user.name=t -c user.email=t@t commit -q -am fix
  bits/bitsBuild build demo -c recipes --defaults release --no-system -w sw > run2.log 2>&1
  echo "run 2 (reused checkout): $(cat sw/*/demo/*/status.txt 2>/dev/null || echo 'build failed')"
}

run bitsorg/bits main
run m-fol/bits stage-patches-after-clone
echo "logs: $T"
)

@pbuncic

pbuncic commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Thanks, the staging order is the right fix. However, there is one problem on a reused checkout:

git checkout -f doesn’t remove untracked files, so a patch that creates a file gets “Assuming -R” from patch --batch on the next build. The file is deleted, exit status 0, and on the build after that it is back. A file added by a patch later dropped from the recipe also lingers.

Running clean -fdxq after the checkout, instead of deleting the two sentinels, fixes both. It should go through an scm method so Sapling checkouts work too.

@m-fol

m-fol commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Thank you for the review. I’ve updated the code as suggested: on a reused checkout, scm.cleanCmd() now runs after the checkout (git clean -fdxq for Git, sl clean --all for Sapling), so any files left over from earlier builds are removed before the patches are applied.

@pbuncic

pbuncic commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Two things to settle before merge.

  1. Scope of the clean.
    scm_exec(scm.cleanCmd(), source_dir) is unconditional — every git package gets git clean -fdxq on re-checkout, not only patched ones, and -x removes ignored files too (generated configure, submodules, any in-source artifacts). That's broader than the patch-staleness purpose. Guarding it with if spec.get("patches") would limit the blast radius — but note that misses the case where a recipe drops all its patches (then patches is falsy, the clean is skipped, and the old patch-created files persist). So if the unconditional reset is deliberate (to catch that case and give every git source a pristine tree), a one-line comment saying so is enough; if not, scope it. Either way worth a conscious call, since it changes rebuild behaviour for all git packages.

  2. sl clean --all semantics.
    Does it remove ignored files too, not just untracked? -x is deliberate — a patch-created file matching upstream .gitignore must still be removed. If sl clean --all only drops untracked files, the Sapling update path keeps the bug the git path fixes.

@m-fol

m-fol commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

(a) The unconditional clean is on purpose, since a reused checkout should be the same as a fresh clone, because that’s what a first build of the package gets. This also covers the scenario when a recipe drops its patches. The files those patches created are removed too, not only their changes to existing files. I also added a comment line explaining this.
I have also added some extra tests that show what the clean removes and what is kept:

  • packages without patches are cleaned,
  • ignored files are removed,
  • nested repositories (submodules) are kept

Also, none of the lcg.bits recipes write into $SOURCEDIR. I ran lint-out-of-source.sh from bits-recipe-tools on all lcg.bits recipes and it reports no recipe that builds from or writes to SOURCES. So for lcg.bits, the clean only removes bits’ own leftover files: the .patch files, .bits_patched and files from old patches.

(b) Yes, I checked and sl clean --all does remove ignored files. But also according to this line within clean.py, --all is deprecated now, so I have switched to ["clean", "--ignored"], which gives the same result.

@m-fol
m-fol requested a review from pbuncic November 17, 2026 13:24
@pbuncic
pbuncic merged commit e058af0 into bitsorg:main Nov 18, 2026
10 of 14 checks passed
@pbuncic

pbuncic commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Thanks, merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants