Skip to content

Support package recipes under recipes/ - #151

Merged
pbuncic merged 4 commits into
bitsorg:mainfrom
jmcarcell:feat/packages-layout
Nov 19, 2026
Merged

pbuncic merged 4 commits into
bitsorg:mainfrom
jmcarcell:feat/packages-layout

Conversation

@jmcarcell

@jmcarcell jmcarcell commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Search each checkout root first, then recipes/, before trying the next checkout. Both locations can coexist. A copied recipe remains shadowed until its root copy is removed.
  • Apply the same precedence to pinned Git refs and commands that discover, scan or edit recipes. Defaults, patches, checksums and CVMFS configuration stay anchored to the checkout root.

The move separates recipes from repository configuration and support files. It does not change GitHub's 1,000-entry directory listing limit.

Validation

  • The clean-checkout unittest suite ran 2,651 tests (42 skipped). Ruff and git diff --check passed.
  • Regression tests cover mixed checkouts, including root-only and nested-only recipes, collisions and fallback after removal, and the same precedence at pinned Git refs.
  • On AlmaLinux 9, a mixed checkout built a nested-only recipe and used the root copy on a clash. Removing the root copy made the next build use the nested recipe. The migrated LCG checkout still resolved zlib through the flat Stacks and registry checkouts; the compliance scan found 1,108 recipes.

Rollout

Merge and deploy this change before lcg.bits #6, the only repository migration planned for now. Other repositories and unmigrated branches can stay flat.

Keep repository assets at the checkout root and retain flat-layout support.
@pbuncic

pbuncic commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

With lcg.bits (1108 *.sh at the root and already over GitHub's 1,000-entry listing cap) plus the migration patches ,the problem is real .

Moving 1108 recipes into a subdirectory does not fix the web-UI listing cap, recipes/ still holds 1108 entries and truncates at 1000 just as it does now. The real benefit is a clean separation of concerns: recipes stop being interleaved with defaults-*.sh, patches/, cvmfs.yaml, CI and docs, so the root is navigable and tooling can target one known place.

Half of the PR that consolidates recipe discovery into recipe_files()/package_recipe_dir() which is a clear maintainability win.

Since bits' own vocabulary is recipe throughout (recipe_files, getRecipeReader, parseRecipe, "recipe discovery"), while packages is already overloaded (the built artifacts, getPackageList, and the live packages.py generator in cms.bits) so, please replace packages/ with recipes/.

For now I'd migrate only lcg.bits, the code should continue to work with other repositories in present form.

@jmcarcell jmcarcell changed the title Support package recipes under packages/ Support package recipes under recipes/ Nov 19, 2026
Keep asset-only directories compatible with flat lookups, and run the layout tests under unittest in CI.
@jmcarcell

Copy link
Copy Markdown
Contributor Author

Thanks @pbuncic. Updated Bits to look under recipes/ and kept flat checkouts and historical refs working. lcg.bits #6 is now the only migration; I closed the other drafts. Both PR descriptions explain that the move separates recipes from repo configuration and does not change GitHub’s directory listing cap. On AlmaLinux 9, flat and nested fixture builds produced the same build hash, and zlib resolved from the migrated LCG checkout through the unchanged flat stacks.bits and provider registry. The local unittest suite passed (2,651 tests run, 42 skipped).

Do not mistake a nested versioned Latest recipe for a flat-layout marker.
@pbuncic

pbuncic commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

As it stands the PR makes recipes/ exclusive — package_recipe_dir() returns repo/recipes when present, and root *.sh package recipes are then ignored (only root defaults- are read). That has to change, because backward compatibility must hold:

  • alidist/ALICE cms.bits/CMS stays flat forever; lcg.bits may migrate only some branches.
  • we must be able to work with same version of bits on different branches, some will still have old layout
  • a checkout can legitimately have package recipes at the root and in recipes/ at the same time.

Required behaviour:

  • Root recipes are always searched; recipes/ is searched in addition, never instead.
  • That way defaults-*.sh can be kept and be found in root directory, for example
  • On a name clash, the root recipe wins (so adding a recipes/ dir never changes which recipe is used for anything already at the root — the maximal-compat guarantee).

Note on semantics so it's a conscious choice: root precedence means a recipe copied into recipes/ stays shadowed by a stale root copy until that root copy is deleted — migration is "copy to recipes/, then remove from root," old-wins during the overlap. That's the conservative, zero-surprise reading, which is what we want.

Everything else is ready; this precedence change (and its two tests) is the one remaining item.

@jmcarcell

Copy link
Copy Markdown
Contributor Author

Thanks @pbuncic. Updated in 7a6245c: Bits now searches each checkout root first, then recipes/. Mixed checkouts work, defaults stay at the root, and the root copy wins until it is removed. The regression tests cover root-only and nested-only recipes, clashes, and the same precedence at pinned Git refs. On physical-alma9, I built a nested-only recipe, verified that a clash installed the root recipe’s output, then removed the root copy and verified that the build installed the nested recipe’s output. The clean-checkout suite ran 2,651 tests (42 skipped). The rollout is still limited to lcg.bits #6.

Allow flat and nested recipes to coexist without replacing root recipes, including in pinned refs.
@pbuncic
pbuncic merged commit bd183c8 into bitsorg:main Nov 19, 2026
10 of 14 checks passed
@pbuncic

pbuncic commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

OK, merged. Thanks!

pbuncic pushed a commit that referenced this pull request Nov 19, 2026
* Find package recipes under packages/

Keep repository assets at the checkout root and retain flat-layout support.

* Use recipes/ for the nested recipe layout

Keep asset-only directories compatible with flat lookups, and run the layout tests under unittest in CI.

* Keep recipe discovery errors visible in strict scans

Do not mistake a nested versioned Latest recipe for a flat-layout marker.

* Search the checkout root before recipes/

Allow flat and nested recipes to coexist without replacing root recipes, including in pinned refs.

---------

Co-authored-by: Juan Miguel Carceller <jmcarcell@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants