Skip to content

About

BITS artifact store

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

bits-store

BITS artifact store: one service per site in front of the store's volumes (S3 buckets, disk), serving every group. It checks capability tokens, redirects reads and writes to presigned URLs, indexes what the store holds and mirrors sources. Design: bits-services ADR 0004. One static Go binary: make build, make test.

  • docs/TOKEN.md: the store token, with test vectors in testdata/tokens/ (python3 testdata/tokens/vectors.py check).
  • bits-store -config config.json: see config.example.json. A binding maps a logical prefix to a volume and key prefix; GET and HEAD are redirected to presigned URLs (plain URLs for public bindings, which use an anonymous volume, sources aside).
  • bits-store -check -config config.json checks a configuration as a start would (the volumes' key variables set), and that each volume with keys can list its bucket: keys S3 does not take, a missing bucket or no s3:ListBucket are refused with the volume's name. The running service reloads its configuration when the file changes (-reload-every, 5s); one that fails the checks is logged and the running one kept. Changed keys in the environment, or listen, need a restart. In a container, mount the configuration's directory: a mounted single file does not see a file replaced by rename.
  • A binding's format (internal/format) names the objects under its prefix; blob, the default, uses the path as the key. Formats are compiled in and register themselves; each has conformance cases in format_test.go.
  • sources is bits' source cache, <md5[:2]>/<md5>/<file> (key prefix SOURCES/cache as bits writes it): the MD5 of the URL as the recipe spells it, the file its last segment. A read with ?url=<that URL> and a token that may GET the path fetches a missing archive: one fetch per name, streamed into staging and promoted, its SHA-256, URL and asker (fetched-by) in the metadata; meanwhile 202 with Retry-After, then 302. A failure answers 502 (fetch-failed, with a reason), remembered for 10 minutes or until a reload. Clients percent-encode the URL (a raw + reads as a space). The binding's max_size bounds a fetch (bytes, default 32 GiB).
  • tarballs is bits' package store: <arch>/<hash> names the build of that hash, kept as bits writes it (key prefix TARS: <arch>/store/<hash[:2]>/<hash>/<pkg>- <version>[-<revision>].<arch>.tar.gz). GET or HEAD <prefix>/<arch>/<hash> redirects to it, its file name in Bits-Object-Name (bits-store's 404 if none; of several, in old stores, the lowest revision by number, as bits picks). An upload of a hash names its file in the start ("name", of the path's architecture), its declared SHA-256 kept as sha256 (sha256:<hex>), as bits' own writers keep it; it answers 409 exists when the hash holds a file under any name, which a HEAD then names. bits-store finds the file by listing that hash's folder (the volume's keys need s3:ListBucket), the index later; reads remember a hit for 10 minutes, a miss for 30 seconds. Writers with keys of their own can still add a second file under a hash.
  • A tarballs binding with "gate": "group" (not public; for packages that may not be redistributed, in a bucket of their own) keeps each group's builds apart: a write is filed under the token's group (<key prefix>/<group>/<arch>/store/...), a read finds the build in the token's group, then in those that list it among their readers ({"sft": ["lhcb", "atlas"]}, by name); else 404. Groups are plain names ([a-z0-9][a-z0-9._-]*, up to 64). Builds not under a group are not read: a gate starts on an empty prefix. The collection plan takes these builds by group and architecture (<group>/<arch>), without manifests: no root keeps one, nor certification, only age, reads and the newest builds of each version (with -gc-keep-builds 0, a version's only build may go: it is rebuilt when needed); refused while its bucket was not listed lately, reads are counted for less than -gc-read, or one group's architecture has more than -gc-max to go. Files not under a group name are never planned. Each is signed when its upload starts, by a security-proxy sign route (restricted_signer: agent_socket, route, host, prefix, found as bits-console finds its own; required with a gate), its key never on this host: over bits-restricted-build v1\n and {"arch","group","hash","name","sha256"} (canonical JSON, the declared sha256 in hex), kept in its metadata bits-signature (<key id>:<base64 signature>); no signature, no upload (503 unsigned). A read names the group whose build it is in Bits-Object-Group (its folder, not its metadata): bits reuses the build only when the signature verifies over these and the bytes it downloaded; one without, from before, never. deny (["sha256:<hex>", ...]) withdraws builds by the sha256 they were stored with, from the next reload of config.json: a read finds none (404: bits rebuilds it, or reads another group's) and an upload of those bytes is refused (403 denied), even one started before (one already being stored may still land, not read), so they are never signed again. The object stays until deleted, and with it the hash's folder: delete it so that a rebuild (other bytes) is stored. Build hosts keep what they downloaded. The admin API withdraws builds the same way, at once (PUT /_admin/withdraw); as with deny, a URL handed out before stays good until it expires (url_lifetime).
  • pypi mirrors a Python package index (index, PEP 691 JSON; https://pypi.org/simple when unset) for pip: public, read by anyone (PIP_INDEX_URL=https://host/pypi/simple), whose first read of a file also has it fetched; else with the token as Basic auth (https://bits:<token>@host/pypi/simple). GET <prefix>/simple/<project>/ (the name as PEP 503 normalises it; Accept: application/vnd.pypi.simple.v1+json, else 406) answers the index's page, asked again after 10 minutes and served as last read while the index cannot answer; its files with a SHA-256 link to <prefix>/files/<project>/<sha256>/<file>. A file the store holds is read from it; one it does not is sent to the index's URL (302) while the store fetches it, once, keeping it only when its SHA-256 is the one listed (max_size, the fetch budgets as for sources: make the files' host, pythonhosted.org, trusted, else beyond the day's budget a file is not kept, only logged; with too many fetches running, as when pip reads a burst, a file waits up to 10 minutes for room). A kept file stays readable at its link when the index drops it (what bits' pins install); the page lists only what the index lists. Its files' links are https (or the index's scheme); without their core-metadata pip downloads each file it considers. Nothing is written to it.
  • crates mirrors a Cargo registry's sparse index (index; https://index.crates.io when unset) for cargo, public as pypi; its url is its own address as cargo reaches it (https://<host><prefix>), and cargo replaces crates.io with sparse+<url>/index/ (bits does so in its containers). GET <prefix>/index/config.json points downloads here (<url>/crates); <prefix>/index/<path> answers the index's file of a crate (1/<n>, 2/<n>, 3/<c>/<n>, <ab>/<cd>/<n>), asked again after 5 minutes and served as last read while the index cannot answer. GET <prefix>/crates/<crate>/<version>/download: a crate the store holds (files/<crate>/<crate>-<version>.crate) is read from it; one it does not is sent to the registry's download (its config.json's dl, 302) while the store fetches it, once, keeping it only when its SHA-256 is the cksum the index lists, as pypi's files (max_size, fetch budgets: make static.crates.io trusted). A kept crate stays readable when the registry drops it. Nothing is written to it.
  • git mirrors git repositories for git's smart HTTP, protocol v2: <prefix>/<host>/<path> is https://<host>/<path> (.git or not), read with git clone --filter=tree:0 (or blob:none) and git fetch; bits points sources there with insteadOf. Each repository is a partial mirror under -data's git/<key_prefix>/, served by git http-backend (git in the image). Before git answers, the store fetches what the request needs from upstream: on ls-refs, a tag the client names that it lacks, at once, and every ref after a minute (commits and trees, no files); on fetch, the objects wanted that it lacks: files only when a client names them, as a checkout does, its tree then kept under refs/bits/<tree>, so a branch moved or deleted upstream never loses it. A tag held never moves (a moved one is logged, once); upstream down, what is held is served. One fetch per repository at a time, another request waiting for it (up to 15 minutes); git's own fetches reach only what the fetch policy allows, and a new repository of an untrusted host counts against the day's budgets. A client without protocol v2 is sent upstream (302); a full clone (no --filter) is not served; pushes are refused; a repository upstream does not have is not made. Needs -data; the volume, with keys, is for the repositories' bundles (later).
  • manifests is bits' MANIFESTS area: the path is the key, as blob (the signed common manifests), and GET <prefix>/rev-index/<arch>/<pkg>/<version>?revisions answers {revision: hash} from that version's revision markers (rev-index/<arch>/<pkg>/<version>-<revision>, each holding a hash): bits-store lists them and reads each (at most 1000; beyond, 502). A marker is written with the usual upload, once (409 when taken). Common manifests replaced in place and markers deleted (bits sign, bits compliance) still need the writer's own keys.
  • A public binding on a volume with keys (sources, tarballs, manifests) reads by plain URLs (its key prefix public-read) and takes writes from a token allowed to PUT; one on an anonymous volume only reads. A token with a broad PUT pattern can therefore write to public areas.
  • A public sources binding reads without a token, by plain URLs (make its key prefix public-read; the volume needs keys, to write). A miss is the bucket's 404, and bits then downloads from the URL itself. Only a token with GET on the path fetches; any other token reads as anyone does, a bad one gets 401. What a fetch brings is then public, from site_hosts too. A public tarballs binding likewise reads by plain URLs, its volume's keys listing for it.
  • Where a fetch may go (fetch in the configuration): http(s) only, no proxy; never loopback, link-local or the host's own addresses; the site network (private addresses and site_ranges) only for the hosts in site_hosts; anywhere public otherwise. bits-store resolves each host itself and checks every address it connects to, redirects included. Domains outside trusted have a daily budget per token subject and per domain (untrusted_per_subject, untrusted_per_domain, default 100 fetches); a spent one answers 429 until midnight UTC. 16 fetches at once, 4 per subject; each buffers one part (16 MiB at the default max_size). Every fetch is logged with its asker, bytes and time.
  • The index (-data <dir>, its index.db; none: no index): what each bucket of a volume with keys holds, STAGING/ aside: key, size, ETag, time, and the declared SHA-256 and the writer's group (its token's, kept in the object's metadata bits-group) of what the service wrote. Each promote adds its object. Each bucket is listed against it when never listed, then every index_verify (default 24h, at least 1h), one at a time, a failed listing again after 10 minutes: what changed outside the service is repaired and logged as missing, changed and extra. It is a cache: losing it costs one listing, and a write it fails to record still succeeds; an index of an older release is dropped and refilled (a listing does not show the writers: those are lost until read from the objects). One instance only. It also reads the .json files under each manifests binding (not rev-index/; up to 16 MiB, 500 a minute per bucket, never on a request), new or changed: a BOM (a build_id and packages) gives its build and the (architecture, hash) of each package, a common-manifest-*.json its sources and packages; other files are noted as read, those that cannot be read tried again after an hour. A common manifest is verified when its .sig (bits' Ed25519 envelope) verifies with one of manifest_keys (PEM public keys, as bits' keys/*.pem), checked again when either file or the keys change; without manifest_keys none is.
  • The admin API, for bits-console's admin pages: on its own address (-admin-listen, with -data; off without it), never on the store's; plain HTTP, so bind it to an internal network the console alone shares. A request needs an Authorization: Bearer token of a key whose signer is admin (sub admin:<user>), living 15 minutes at most and allowed the method and path (GET /_admin/**; an action PUT /_admin/<action>); such a token opens no store path. Each request is logged. GET /_admin/summary: for each bucket (named by its first volume) when it was last listed, its drift and totals, and each binding's objects and bytes, those read in the last 30 days, a tarballs binding's by architecture (a gated one's by group/architecture) and those a certified build lists (as /metrics); computed as /metrics is, at most every 15 minutes. GET /_admin/objects?binding=<prefix> searches that binding's objects in the index, in key order: under a folder (an architecture, <group>/<arch>, rev-index), q text in the key (any case), kind (bom, common, or none: read and not a manifest; a manifests binding's), written since or before (a date or RFC 3339), kept (a tarballs binding's), after a key, limit rows (100; at most 500). Each row has the key, size, ETag, write time, reads, last read, writer group; a package its architecture, hash, file (and group behind a gate) and kept, the collection's verdict with its default options, refusals aside (recomputed at most every 5 minutes): listed (a manifest or BOM lists its hash), uncertified-arch, young, read, newest, other (not a package) or goes; planned for the bucket searched, by a plan that goes on when the request is given up. next continues it (after), "" at the end; a search stops after looking at 100,000 rows and answers with next. GET /_admin/object?binding=<prefix>&key=<key>: one object as a search shows it, with its SHA-256 (when the service wrote it), whether the bucket holds it and its user metadata there (a HEAD; in_bucket null when the bucket does not answer); a package's listed_by (each BOM and common manifest listing its hash under its architecture, the builds it names, whether one of manifest_keys verifies it); a manifest's kind and builds. GET /_admin/access: the configuration in force (as reloaded), without secrets: issuer, audience, lifetimes; the token keys' ids and signers; each volume's endpoint and bucket and whether the service holds keys for it (their values stay in the environment); each binding with its gate, readers and deny; the manifest keys' ids; the restricted builds' signer route; the fetch budgets and trusted domains (not the site's addresses). PUT /_admin/withdraw {"binding", "sha256", "why", "key"} withdraws a gated binding's build as its deny does, from then on (key, optional: the object it was seen as); a build withdrawn already keeps its first record (added false). PUT /_admin/lift {"binding", "sha256"} lifts one (denied true when deny in config.json still withdraws it). GET /_admin/withdrawn lists them, each with who (by, the token's sub), when and why. They are kept in withdrawn.json beside the index (-data), not in it: an index dropped and listed again gives none back; a file that cannot be read stops the start. PUT /_admin/delete {"binding", "key", "etag", "why"} deletes one package file of a tarballs binding as seen (etag, the index's), when the bucket has it alike (size and ETag) and no upload to its folder is under way; outside a gate, only when a collection could plan its bucket now and no manifest keeps its hash (as -gc has them: 409 otherwise, with why). The index forgets it at once; the answer is {"deleted", "size"}. PUT /_admin/gc/plan {"grace", "read", "reads", "keep_builds", "max"} (Go durations; -gc's defaults for those left out, all with {}) plans a collection in the background as -gc does: 202 {"id"}, 409 while another job runs (one at a time). Each plan is kept in gc/ beside the index (the newest 20): <id>.json as -gc-out writes it (-gc-apply takes it) and <id>.summary.json, by bucket what goes or why it is refused; one without its summary is not done. GET /_admin/gc: the job running (or the last: who, when, its error) and the last 10 plans; GET /_admin/gc/files?id=<id>: a plan's files to go (offset, limit 100, at most 500; total). PUT /_admin/gc/apply {"id", "files"} (files: the plan's count to go, as a confirmation) applies it in the background as -gc-apply does (planned again now, each file checked again): 202; 400 for another count, 404 for no such plan; 409 while a job runs, once applied to its end, or planned more than 7 days ago. One a restart stopped, or that could not start (said in its log), may be applied again. Its lines go to <id>.log (GET /_admin/gc/log?id=<id>, the last MiB); its summary records who, when and -gc-apply's code. A file deleted by the service (an apply, PUT /_admin/delete) leaves its index at once; -gc-apply on the host leaves that to the next listing.
  • Problems: the service's warnings and errors are logged as before and, with the index, kept there 30 days, the same one (level, kind, subject, message) within an hour counted on one row (first, last, count). GET /_admin/problems?since=<ms>&limit=<n> lists those last seen at or after since, oldest first (up to 1000); a reader asks again from the last one's last. Kinds: volume_unreachable, volume_request_failed, presign_failed, promote_failed, promote_conflict, staging_left, index_read_failed, index_write_failed, index_drift (after a first listing), listing_failed, manifests_unread, reads_dropped, fill_failed, fill_refused, fill_throttled, pypi_page_failed, crates_index_failed, git_failed, git_backend, git_upstream_down, tag_moved, git_no_data, git_missing, collection_failed, metrics_failed, config_rejected, disk_low (less than a tenth of -data's file system free), internal.
  • /metrics (Prometheus text, no token: block it in front, for the site's scraper only): what the service does since it started, by binding ((none): none) and format: bits_store_requests_total (by method and status), bits_store_request_seconds (a histogram), bits_store_response_bytes_total; fetches from upstream: bits_store_fills_total (by result ok or failed), bits_store_fill_bytes_total (not git's), bits_store_fills_refused_total (by reason full or throttled), bits_store_fetches_running; bits_store_goroutines, bits_store_heap_bytes, bits_store_start_seconds. With the index, for each bucket listed at least once, named by its first volume, bits_store_objects and bits_store_bytes by architecture (the first segment under each tarballs binding's key prefix), bits_store_objects_total, bits_store_bytes_total, and what each content type adds to them: bits_store_content_objects and _bytes by key_prefix (of the bucket's bindings), binding and format, without what a longer key prefix inside it holds, the rest as binding (none), so that they add up to the totals; bits_store_manifest_objects and _bytes by manifest, arch and signed: a package goes to the first build (by id) whose BOM lists it among the sources of verified common manifests (of all, without manifest_keys), signed when a verified one names it, else to (uncertified) ((restricted) behind a gate); what its last listing found (bits_store_index_drift, kind missing, changed or extra) and when it began (bits_store_index_verified_seconds); bits_store_index_errors_total, the writes the index failed to record; with -data, each git binding's repositories there by host (bits_store_git_repositories, bits_store_git_bytes), and the size and free bytes of -data's file system (bits_store_data_size_bytes, bits_store_data_free_bytes). It is computed at most every 15 minutes (a scan of the index; scrapes in between share its answer), in the background, not bound to the scrape that asked (a large index can take longer than a scraper waits): a scrape waits up to 5 seconds for a newer answer, else gets the last (at most 30 minutes old); one with none waits as long as it may; one that waited for a failed computation gets 500.
  • Reads: each GET the service redirects counts for its object in the index (how many and the last), written once a minute; reads straight from the bucket (public plain URLs, b3://) are not seen, nor those of more than 100,000 objects in a minute (logged), so the counts are a floor.
  • bits-store -gc -config config.json -data DIR plans a collection from the service's index, read only, and exits (0; 2 when a volume is refused; 1 on an error): per architecture under the tarballs bindings, the packages (<arch>/store/<h2>/<hash>/<file>) that no manifest keeps, older than -gc-grace (30 days), not read lately, and not among the -gc-keep-builds (3) newest builds of their package version (by write time, per binding; the revision a number or local<number>, a file named otherwise its own version), in architectures a verified common manifest lists packages of; -gc-list names them, with their reads. Kept by a manifest: the hashes any BOM or common manifest lists, and common manifests outside the manifests bindings (the orchestrator's CANARY/ ones) are read at plan time. Read lately: within -gc-read (30 days), or at least -gc-reads (10) times within four times that (-gc-read 0 -gc-reads 0: no such rule, and no wait for the reads to be counted). Files that are not packages always stay, and nothing outside the tarballs bindings is ever planned: sources are kept for reference. Refused: without manifest_keys, a listing older than index_verify and two hours, manifests unread or with packages skipped, a common manifest outside the bindings that cannot be read, one inside or its .sig in the store not as the index read it, reads counted for less than -gc-read, or more than -gc-max (0.25) of an architecture's packages to go. It deletes nothing: -gc-out FILE writes the files that may go now (volume, bucket, key, size, ETag, reads; none of a refused volume) and the options, for review; -gc-apply FILE (with the same options, within 7 days) plans again and deletes those of them it still sweeps, as planned in the index and in the bucket (size and ETag, a HEAD each), not read since nor listed by a manifest since (checked again every 30 seconds as it goes, the volume stopped when manifests are left to read); the others stay, each said. Exit 2: a volume refused or stopped, what is left of it stays; 1: the plan file not taken, or a file not deleted (after 10 in a row, it stops). The index is the service's, read only: the next listing drops what went (extra), and the service may still name a deleted build for up to 10 minutes (a download then fails: rebuilt). -gc-* without -gc is refused.
  • access_log: errors (the default) logs errors and writes, not hits; all every request; none no request lines. Fetches and failures are logged in any case.
  • Writes go through staging; a client never gets a URL for a final key: POST <path>?uploads with {"size", "md5" (base64), "sha256" (hex)} returns a presigned PUT into STAGING/<id> and the headers to send; after the upload, POST <path>?complete=<id> promotes it with a conditional server-side copy (201, also on a retry), or answers 409 if the name exists (first writer wins), 422 if size or MD5 differ. Size and MD5 are checked; the SHA-256 is kept as the uploader's declared-sha256 (a tarball's as sha256). One presigned PUT takes up to 5 GiB; a promote above 5 GiB copies in parts, the upload's id in the object's metadata marking it as its own.
  • Uploads in parts (up to 1 TiB): POST <path>?uploads with {"size", "sha256", "part_size", "part_md5"} (each part's MD5, base64; parts of 5 MiB to 5 GiB, the last shorter, at most 10000) returns "parts", a presigned PUT per part signing its Content-MD5; their URLs live a second per 8 MiB longer than url_lifetime (not beyond the token). ?complete=<id> assembles the parts, 409 not-uploaded while one is missing or short (put it, complete again), 422 mismatch when one differs (start over), and promotes as above. A complete keeps its upload while it may run. Parts of about 1 GiB keep the answer small. Part ETags must be MD5s: no SSE-KMS or SSE-C on the bucket. First-writer-wins needs a single instance (the lease is in memory) and bits-store as the bucket's only writer. Give the bucket lifecycle rules expiring STAGING/ and aborting incomplete multipart uploads after a day.
  • Container: make image builds bits-store:<version>, the static binary alone in an empty image, as user 65532, listening on 8098 (the default when the configuration sets no listen). The index's directory, /var/lib/bits-store, is that user's: a new Docker named volume there takes its owner (-data /var/lib/bits-store). Mount the configuration at /etc/bits-store/config.json and give the S3 keys in the environment its volumes name; for a site CA, mount it and set SSL_CERT_FILE. bits-store -health URL is its health check; the image checks port 8098, so a listen on another port needs the health check changed too. /health answers 503 naming the volumes whose bucket its keys could not list at the last probe (at most a minute old, never waited for; a start or reload probes at once, and answers ok until that probe ends); why is logged, once per change, and /health is never in the access log.
  • go test -run TestS3 ./internal/server runs reads and writes against a real S3 endpoint when BITS_STORE_S3IT_* is set (see internal/server/s3_test.go).
  • Clients send paths unencoded: the allowed punctuation (~+!@=,:) as is, since a % is refused. A TLS proxy in front must pass the request target unchanged (nginx proxy_pass without a URI part); the __Host- cookie needs HTTPS.

About

BITS artifact store

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages