BITS artifact store: one service per site in front of the store's volumes (S3
buckets, disk), serving every group. It checks capability tokens, redirects reads
and writes to presigned URLs, indexes what the store holds and mirrors sources.
Design: bits-services ADR 0004. One static Go binary: make build, make test.
- docs/TOKEN.md: the store token, with test vectors in
testdata/tokens/(python3 testdata/tokens/vectors.py check). bits-store -config config.json: seeconfig.example.json. A binding maps a logical prefix to a volume and key prefix; GET and HEAD are redirected to presigned URLs (plain URLs for public bindings, which use an anonymous volume, sources aside).bits-store -check -config config.jsonchecks a configuration as a start would (the volumes' key variables set), and that each volume with keys can list its bucket: keys S3 does not take, a missing bucket or nos3:ListBucketare refused with the volume's name. The running service reloads its configuration when the file changes (-reload-every, 5s); one that fails the checks is logged and the running one kept. Changed keys in the environment, orlisten, need a restart. In a container, mount the configuration's directory: a mounted single file does not see a file replaced by rename.- A binding's
format(internal/format) names the objects under its prefix;blob, the default, uses the path as the key. Formats are compiled in and register themselves; each has conformance cases informat_test.go. sourcesis bits' source cache,<md5[:2]>/<md5>/<file>(key prefixSOURCES/cacheas bits writes it): the MD5 of the URL as the recipe spells it, the file its last segment. A read with?url=<that URL>and a token that may GET the path fetches a missing archive: one fetch per name, streamed into staging and promoted, its SHA-256, URL and asker (fetched-by) in the metadata; meanwhile202withRetry-After, then302. A failure answers502(fetch-failed, with a reason), remembered for 10 minutes or until a reload. Clients percent-encode the URL (a raw+reads as a space). The binding'smax_sizebounds a fetch (bytes, default 32 GiB).tarballsis bits' package store:<arch>/<hash>names the build of that hash, kept as bits writes it (key prefixTARS:<arch>/store/<hash[:2]>/<hash>/<pkg>- <version>[-<revision>].<arch>.tar.gz).GETorHEAD <prefix>/<arch>/<hash>redirects to it, its file name inBits-Object-Name(bits-store's404if none; of several, in old stores, the lowest revision by number, as bits picks). An upload of a hash names its file in the start ("name", of the path's architecture), its declared SHA-256 kept assha256(sha256:<hex>), as bits' own writers keep it; it answers409 existswhen the hash holds a file under any name, which aHEADthen names. bits-store finds the file by listing that hash's folder (the volume's keys needs3:ListBucket), the index later; reads remember a hit for 10 minutes, a miss for 30 seconds. Writers with keys of their own can still add a second file under a hash.- A
tarballsbinding with"gate": "group"(not public; for packages that may not be redistributed, in a bucket of their own) keeps each group's builds apart: a write is filed under the token's group (<key prefix>/<group>/<arch>/store/...), a read finds the build in the token's group, then in those that list it among theirreaders({"sft": ["lhcb", "atlas"]}, by name); else404. Groups are plain names ([a-z0-9][a-z0-9._-]*, up to 64). Builds not under a group are not read: a gate starts on an empty prefix. The collection plan takes these builds by group and architecture (<group>/<arch>), without manifests: no root keeps one, nor certification, only age, reads and the newest builds of each version (with-gc-keep-builds 0, a version's only build may go: it is rebuilt when needed); refused while its bucket was not listed lately, reads are counted for less than-gc-read, or one group's architecture has more than-gc-maxto go. Files not under a group name are never planned. Each is signed when its upload starts, by a security-proxy sign route (restricted_signer:agent_socket,route,host,prefix, found as bits-console finds its own; required with a gate), its key never on this host: overbits-restricted-build v1\nand{"arch","group","hash","name","sha256"}(canonical JSON, the declared sha256 in hex), kept in its metadatabits-signature(<key id>:<base64 signature>); no signature, no upload (503 unsigned). A read names the group whose build it is inBits-Object-Group(its folder, not its metadata): bits reuses the build only when the signature verifies over these and the bytes it downloaded; one without, from before, never.deny(["sha256:<hex>", ...]) withdraws builds by the sha256 they were stored with, from the next reload of config.json: a read finds none (404: bits rebuilds it, or reads another group's) and an upload of those bytes is refused (403 denied), even one started before (one already being stored may still land, not read), so they are never signed again. The object stays until deleted, and with it the hash's folder: delete it so that a rebuild (other bytes) is stored. Build hosts keep what they downloaded. The admin API withdraws builds the same way, at once (PUT /_admin/withdraw); as withdeny, a URL handed out before stays good until it expires (url_lifetime). pypimirrors a Python package index (index, PEP 691 JSON;https://pypi.org/simplewhen unset) for pip:public, read by anyone (PIP_INDEX_URL=https://host/pypi/simple), whose first read of a file also has it fetched; else with the token as Basic auth (https://bits:<token>@host/pypi/simple).GET <prefix>/simple/<project>/(the name as PEP 503 normalises it;Accept: application/vnd.pypi.simple.v1+json, else406) answers the index's page, asked again after 10 minutes and served as last read while the index cannot answer; its files with a SHA-256 link to<prefix>/files/<project>/<sha256>/<file>. A file the store holds is read from it; one it does not is sent to the index's URL (302) while the store fetches it, once, keeping it only when its SHA-256 is the one listed (max_size, the fetch budgets as forsources: make the files' host,pythonhosted.org,trusted, else beyond the day's budget a file is not kept, only logged; with too many fetches running, as when pip reads a burst, a file waits up to 10 minutes for room). A kept file stays readable at its link when the index drops it (what bits' pins install); the page lists only what the index lists. Its files' links are https (or the index's scheme); without theircore-metadatapip downloads each file it considers. Nothing is written to it.cratesmirrors a Cargo registry's sparse index (index;https://index.crates.iowhen unset) for cargo,publicaspypi; itsurlis its own address as cargo reaches it (https://<host><prefix>), and cargo replaces crates.io withsparse+<url>/index/(bits does so in its containers).GET <prefix>/index/config.jsonpoints downloads here (<url>/crates);<prefix>/index/<path>answers the index's file of a crate (1/<n>,2/<n>,3/<c>/<n>,<ab>/<cd>/<n>), asked again after 5 minutes and served as last read while the index cannot answer.GET <prefix>/crates/<crate>/<version>/download: a crate the store holds (files/<crate>/<crate>-<version>.crate) is read from it; one it does not is sent to the registry's download (itsconfig.json'sdl,302) while the store fetches it, once, keeping it only when its SHA-256 is thecksumthe index lists, aspypi's files (max_size, fetch budgets: makestatic.crates.iotrusted). A kept crate stays readable when the registry drops it. Nothing is written to it.gitmirrors git repositories for git's smart HTTP, protocol v2:<prefix>/<host>/<path>ishttps://<host>/<path>(.gitor not), read withgit clone --filter=tree:0(orblob:none) andgit fetch; bits points sources there withinsteadOf. Each repository is a partial mirror under-data'sgit/<key_prefix>/, served bygit http-backend(git in the image). Before git answers, the store fetches what the request needs from upstream: onls-refs, a tag the client names that it lacks, at once, and every ref after a minute (commits and trees, no files); onfetch, the objects wanted that it lacks: files only when a client names them, as a checkout does, its tree then kept underrefs/bits/<tree>, so a branch moved or deleted upstream never loses it. A tag held never moves (a moved one is logged, once); upstream down, what is held is served. One fetch per repository at a time, another request waiting for it (up to 15 minutes); git's own fetches reach only what the fetch policy allows, and a new repository of an untrusted host counts against the day's budgets. A client without protocol v2 is sent upstream (302); a full clone (no--filter) is not served; pushes are refused; a repository upstream does not have is not made. Needs-data; the volume, with keys, is for the repositories' bundles (later).manifestsis bits'MANIFESTSarea: the path is the key, asblob(the signed common manifests), andGET <prefix>/rev-index/<arch>/<pkg>/<version>?revisionsanswers{revision: hash}from that version's revision markers (rev-index/<arch>/<pkg>/<version>-<revision>, each holding a hash): bits-store lists them and reads each (at most 1000; beyond,502). A marker is written with the usual upload, once (409when taken). Common manifests replaced in place and markers deleted (bits sign,bits compliance) still need the writer's own keys.- A public binding on a volume with keys (
sources,tarballs,manifests) reads by plain URLs (its key prefix public-read) and takes writes from a token allowed toPUT; one on an anonymous volume only reads. A token with a broadPUTpattern can therefore write to public areas. - A
publicsources binding reads without a token, by plain URLs (make its key prefix public-read; the volume needs keys, to write). A miss is the bucket's404, and bits then downloads from the URL itself. Only a token with GET on the path fetches; any other token reads as anyone does, a bad one gets401. What a fetch brings is then public, fromsite_hoststoo. A publictarballsbinding likewise reads by plain URLs, its volume's keys listing for it. - Where a fetch may go (
fetchin the configuration): http(s) only, no proxy; never loopback, link-local or the host's own addresses; the site network (private addresses andsite_ranges) only for the hosts insite_hosts; anywhere public otherwise. bits-store resolves each host itself and checks every address it connects to, redirects included. Domains outsidetrustedhave a daily budget per token subject and per domain (untrusted_per_subject,untrusted_per_domain, default 100 fetches); a spent one answers429until midnight UTC. 16 fetches at once, 4 per subject; each buffers one part (16 MiB at the defaultmax_size). Every fetch is logged with its asker, bytes and time. - The index (
-data <dir>, itsindex.db; none: no index): what each bucket of a volume with keys holds,STAGING/aside: key, size, ETag, time, and the declared SHA-256 and the writer's group (its token's, kept in the object's metadatabits-group) of what the service wrote. Each promote adds its object. Each bucket is listed against it when never listed, then everyindex_verify(default24h, at least1h), one at a time, a failed listing again after 10 minutes: what changed outside the service is repaired and logged asmissing,changedandextra. It is a cache: losing it costs one listing, and a write it fails to record still succeeds; an index of an older release is dropped and refilled (a listing does not show the writers: those are lost until read from the objects). One instance only. It also reads the.jsonfiles under eachmanifestsbinding (notrev-index/; up to 16 MiB, 500 a minute per bucket, never on a request), new or changed: a BOM (abuild_idandpackages) gives its build and the (architecture, hash) of each package, acommon-manifest-*.jsonitssourcesand packages; other files are noted as read, those that cannot be read tried again after an hour. A common manifest is verified when its.sig(bits' Ed25519 envelope) verifies with one ofmanifest_keys(PEM public keys, as bits'keys/*.pem), checked again when either file or the keys change; withoutmanifest_keysnone is. - The admin API, for bits-console's admin pages: on its own address (
-admin-listen, with-data; off without it), never on the store's; plain HTTP, so bind it to an internal network the console alone shares. A request needs anAuthorization: Bearertoken of a key whosesignerisadmin(subadmin:<user>), living 15 minutes at most and allowed the method and path (GET /_admin/**; an actionPUT /_admin/<action>); such a token opens no store path. Each request is logged.GET /_admin/summary: for each bucket (named by its first volume) when it was last listed, its drift and totals, and each binding's objects and bytes, those read in the last 30 days, atarballsbinding's by architecture (a gated one's by group/architecture) and those a certified build lists (as/metrics); computed as/metricsis, at most every 15 minutes.GET /_admin/objects?binding=<prefix>searches that binding's objects in the index, in key order:undera folder (an architecture,<group>/<arch>,rev-index),qtext in the key (any case),kind(bom,common, ornone: read and not a manifest; a manifests binding's), writtensinceorbefore(a date or RFC 3339),kept(a tarballs binding's),aftera key,limitrows (100; at most 500). Each row has the key, size, ETag, write time, reads, last read, writer group; a package its architecture, hash, file (and group behind a gate) andkept, the collection's verdict with its default options, refusals aside (recomputed at most every 5 minutes):listed(a manifest or BOM lists its hash),uncertified-arch,young,read,newest,other(not a package) orgoes; planned for the bucket searched, by a plan that goes on when the request is given up.nextcontinues it (after), "" at the end; a search stops after looking at 100,000 rows and answers withnext.GET /_admin/object?binding=<prefix>&key=<key>: one object as a search shows it, with its SHA-256 (when the service wrote it), whether the bucket holds it and its user metadata there (a HEAD;in_bucketnull when the bucket does not answer); a package'slisted_by(each BOM and common manifest listing its hash under its architecture, the builds it names, whether one ofmanifest_keysverifies it); a manifest'skindandbuilds.GET /_admin/access: the configuration in force (as reloaded), without secrets: issuer, audience, lifetimes; the token keys' ids and signers; each volume's endpoint and bucket and whether the service holds keys for it (their values stay in the environment); each binding with its gate,readersanddeny; the manifest keys' ids; the restricted builds' signer route; the fetch budgets and trusted domains (not the site's addresses).PUT /_admin/withdraw{"binding", "sha256", "why", "key"}withdraws a gated binding's build as itsdenydoes, from then on (key, optional: the object it was seen as); a build withdrawn already keeps its first record (addedfalse).PUT /_admin/lift{"binding", "sha256"}lifts one (deniedtrue whendenyin config.json still withdraws it).GET /_admin/withdrawnlists them, each with who (by, the token'ssub), when and why. They are kept inwithdrawn.jsonbeside the index (-data), not in it: an index dropped and listed again gives none back; a file that cannot be read stops the start.PUT /_admin/delete{"binding", "key", "etag", "why"}deletes one package file of atarballsbinding as seen (etag, the index's), when the bucket has it alike (size and ETag) and no upload to its folder is under way; outside a gate, only when a collection could plan its bucket now and no manifest keeps its hash (as-gchas them:409otherwise, with why). The index forgets it at once; the answer is{"deleted", "size"}.PUT /_admin/gc/plan{"grace", "read", "reads", "keep_builds", "max"}(Go durations;-gc's defaults for those left out, all with{}) plans a collection in the background as-gcdoes:202 {"id"},409while another job runs (one at a time). Each plan is kept ingc/beside the index (the newest 20):<id>.jsonas-gc-outwrites it (-gc-applytakes it) and<id>.summary.json, by bucket what goes or why it is refused; one without its summary is not done.GET /_admin/gc: the job running (or the last: who, when, its error) and the last 10 plans;GET /_admin/gc/files?id=<id>: a plan's files to go (offset,limit100, at most 500;total).PUT /_admin/gc/apply{"id", "files"}(files: the plan's count to go, as a confirmation) applies it in the background as-gc-applydoes (planned again now, each file checked again):202;400for another count,404for no such plan;409while a job runs, once applied to its end, or planned more than 7 days ago. One a restart stopped, or that could not start (said in its log), may be applied again. Its lines go to<id>.log(GET /_admin/gc/log?id=<id>, the last MiB); its summary records who, when and-gc-apply's code. A file deleted by the service (an apply,PUT /_admin/delete) leaves its index at once;-gc-applyon the host leaves that to the next listing. - Problems: the service's warnings and errors are logged as before and, with the
index, kept there 30 days, the same one (
level,kind,subject,message) within an hour counted on one row (first,last,count).GET /_admin/problems?since=<ms>&limit=<n>lists those last seen at or aftersince, oldest first (up to 1000); a reader asks again from the last one'slast. Kinds:volume_unreachable,volume_request_failed,presign_failed,promote_failed,promote_conflict,staging_left,index_read_failed,index_write_failed,index_drift(after a first listing),listing_failed,manifests_unread,reads_dropped,fill_failed,fill_refused,fill_throttled,pypi_page_failed,crates_index_failed,git_failed,git_backend,git_upstream_down,tag_moved,git_no_data,git_missing,collection_failed,metrics_failed,config_rejected,disk_low(less than a tenth of-data's file system free),internal. /metrics(Prometheus text, no token: block it in front, for the site's scraper only): what the service does since it started, by binding ((none): none) and format:bits_store_requests_total(by method and status),bits_store_request_seconds(a histogram),bits_store_response_bytes_total; fetches from upstream:bits_store_fills_total(byresultok or failed),bits_store_fill_bytes_total(not git's),bits_store_fills_refused_total(byreasonfull or throttled),bits_store_fetches_running;bits_store_goroutines,bits_store_heap_bytes,bits_store_start_seconds. With the index, for each bucket listed at least once, named by its first volume,bits_store_objectsandbits_store_bytesby architecture (the first segment under eachtarballsbinding's key prefix),bits_store_objects_total,bits_store_bytes_total, and what each content type adds to them:bits_store_content_objectsand_bytesbykey_prefix(of the bucket's bindings),bindingandformat, without what a longer key prefix inside it holds, the rest as binding(none), so that they add up to the totals;bits_store_manifest_objectsand_bytesbymanifest,archandsigned: a package goes to the first build (by id) whose BOM lists it among the sources of verified common manifests (of all, withoutmanifest_keys),signedwhen a verified one names it, else to(uncertified)((restricted)behind a gate); what its last listing found (bits_store_index_drift,kindmissing, changed or extra) and when it began (bits_store_index_verified_seconds);bits_store_index_errors_total, the writes the index failed to record; with-data, eachgitbinding's repositories there by host (bits_store_git_repositories,bits_store_git_bytes), and the size and free bytes of-data's file system (bits_store_data_size_bytes,bits_store_data_free_bytes). It is computed at most every 15 minutes (a scan of the index; scrapes in between share its answer), in the background, not bound to the scrape that asked (a large index can take longer than a scraper waits): a scrape waits up to 5 seconds for a newer answer, else gets the last (at most 30 minutes old); one with none waits as long as it may; one that waited for a failed computation gets500.- Reads: each
GETthe service redirects counts for its object in the index (how many and the last), written once a minute; reads straight from the bucket (public plain URLs,b3://) are not seen, nor those of more than 100,000 objects in a minute (logged), so the counts are a floor. bits-store -gc -config config.json -data DIRplans a collection from the service's index, read only, and exits (0; 2 when a volume is refused; 1 on an error): per architecture under thetarballsbindings, the packages (<arch>/store/<h2>/<hash>/<file>) that no manifest keeps, older than-gc-grace(30 days), not read lately, and not among the-gc-keep-builds(3) newest builds of their package version (by write time, per binding; the revision a number orlocal<number>, a file named otherwise its own version), in architectures a verified common manifest lists packages of;-gc-listnames them, with their reads. Kept by a manifest: the hashes any BOM or common manifest lists, and common manifests outside themanifestsbindings (the orchestrator'sCANARY/ones) are read at plan time. Read lately: within-gc-read(30 days), or at least-gc-reads(10) times within four times that (-gc-read 0 -gc-reads 0: no such rule, and no wait for the reads to be counted). Files that are not packages always stay, and nothing outside thetarballsbindings is ever planned: sources are kept for reference. Refused: withoutmanifest_keys, a listing older thanindex_verifyand two hours, manifests unread or with packages skipped, a common manifest outside the bindings that cannot be read, one inside or its.sigin the store not as the index read it, reads counted for less than-gc-read, or more than-gc-max(0.25) of an architecture's packages to go. It deletes nothing:-gc-out FILEwrites the files that may go now (volume, bucket, key, size, ETag, reads; none of a refused volume) and the options, for review;-gc-apply FILE(with the same options, within 7 days) plans again and deletes those of them it still sweeps, as planned in the index and in the bucket (size and ETag, a HEAD each), not read since nor listed by a manifest since (checked again every 30 seconds as it goes, the volume stopped when manifests are left to read); the others stay, each said. Exit 2: a volume refused or stopped, what is left of it stays; 1: the plan file not taken, or a file not deleted (after 10 in a row, it stops). The index is the service's, read only: the next listing drops what went (extra), and the service may still name a deleted build for up to 10 minutes (a download then fails: rebuilt).-gc-*without-gcis refused.access_log:errors(the default) logs errors and writes, not hits;allevery request;noneno request lines. Fetches and failures are logged in any case.- Writes go through staging; a client never gets a URL for a final key:
POST <path>?uploadswith{"size", "md5" (base64), "sha256" (hex)}returns a presignedPUTintoSTAGING/<id>and the headers to send; after the upload,POST <path>?complete=<id>promotes it with a conditional server-side copy (201, also on a retry), or answers 409 if the name exists (first writer wins), 422 if size or MD5 differ. Size and MD5 are checked; the SHA-256 is kept as the uploader'sdeclared-sha256(a tarball's assha256). One presigned PUT takes up to 5 GiB; a promote above 5 GiB copies in parts, the upload's id in the object's metadata marking it as its own. - Uploads in parts (up to 1 TiB):
POST <path>?uploadswith{"size", "sha256", "part_size", "part_md5"}(each part's MD5, base64; parts of 5 MiB to 5 GiB, the last shorter, at most 10000) returns"parts", a presignedPUTper part signing itsContent-MD5; their URLs live a second per 8 MiB longer thanurl_lifetime(not beyond the token).?complete=<id>assembles the parts,409 not-uploadedwhile one is missing or short (put it, complete again),422 mismatchwhen one differs (start over), and promotes as above. A complete keeps its upload while it may run. Parts of about 1 GiB keep the answer small. Part ETags must be MD5s: no SSE-KMS or SSE-C on the bucket. First-writer-wins needs a single instance (the lease is in memory) and bits-store as the bucket's only writer. Give the bucket lifecycle rules expiringSTAGING/and aborting incomplete multipart uploads after a day. - Container:
make imagebuildsbits-store:<version>, the static binary alone in an empty image, as user 65532, listening on 8098 (the default when the configuration sets nolisten). The index's directory,/var/lib/bits-store, is that user's: a new Docker named volume there takes its owner (-data /var/lib/bits-store). Mount the configuration at/etc/bits-store/config.jsonand give the S3 keys in the environment its volumes name; for a site CA, mount it and setSSL_CERT_FILE.bits-store -health URLis its health check; the image checks port 8098, so alistenon another port needs the health check changed too./healthanswers503naming the volumes whose bucket its keys could not list at the last probe (at most a minute old, never waited for; a start or reload probes at once, and answersokuntil that probe ends); why is logged, once per change, and/healthis never in the access log. go test -run TestS3 ./internal/serverruns reads and writes against a real S3 endpoint whenBITS_STORE_S3IT_*is set (seeinternal/server/s3_test.go).- Clients send paths unencoded: the allowed punctuation (
~+!@=,:) as is, since a%is refused. A TLS proxy in front must pass the request target unchanged (nginxproxy_passwithout a URI part); the__Host-cookie needs HTTPS.