[marcleblanc2]

One pull request for the CI work spread across #1468, #1471, #1472, #1474, #1475, and the AGENTS.md from #1396. It replaces all six. Nothing here changes what customers deploy, except the two Compose file fixes listed under "Compose files".

Smoke test VM

• gcloud instead of Vagrant: .buildkite/smoke-test-vm.sh creates the VM with gcloud, and copies the repository and runs the tests over ssh. vagrant-run.sh, test/Vagrantfile, and test/servers.yaml are gone.
• 20 GB boot disk, which the Ubuntu 26.04 image needs (Docker 29's containerd image store keeps compressed and unpacked layers).
• 1 hour TTL: each VM is created with --max-run-duration=1h --instance-termination-action=DELETE, so GCP deletes the VM and its boot disk even if the job dies. GCP has no TTL for a disk whose VM was never created, so each job also deletes docker-test-job-* VMs and unattached disks older than an hour. The step times out after 30 minutes.
• Retry on a lost VM: if the VM stops answering SSH, the script deletes it and runs the tests once more on a new VM. If the VM still answers, the tests failed, and the script copies the logs and does not retry. SSH keepalives end the connection to a dead VM after a minute, instead of hanging until the step times out.
• Newer image: the script and tests work on both the current Ubuntu 20.04 image and the Ubuntu Minimal 26.04 image from sourcegraph/infrastructure#7627. Both have the docker compose plugin; the script installs jq only where it is missing. Switching is a one-line change to IMAGE once that image is built.
• VM and disk are named after the Buildkite job, deleted without waiting, and get a key pair made for that job only. The Docker Hub password goes to the VM over stdin.

Smoke test

• Jaeger and the executor are deployed and tested: the test starts the base file, jaeger/, executors/, and test/docker-compose.override.yaml, checks that Jaeger responds, then runs test/executor-smoke-test.sh, which checks that the executor registers.
• Faster: images are pulled in parallel, and the test polls the health checks instead of sleeping 90 seconds. The job went from 458–594s to 228–353s in ci: Speed up the smoke test #1468.
• Stricter: fails if any container restarted, and counts the services from the Compose files instead of a hard-coded 25.
• Image pull retry: the test tries the image pull up to three times. Build 16854 failed when Docker Hub timed out during the pull.
• Logs: on failure, the step uploads every container's logs. Before, it saved nothing for an exited or unhealthy container.
• Branches which only change Markdown files skip the smoke test. main always runs it.

Other checks

• validate-docker-compose.sh validates every Compose file, in each combination the docs say to run.
• New shellcheck step (0.11.0 in .tool-versions).
• batch-change/release.yaml: the tag step is safe to retry.

Compose files

• db-only-migrate.docker-compose.yaml now extends pgsql and codeintel-db from the base file, so the two can no longer drift. sg ops update-images skips services without an image:, so the release tooling is unaffected.
• executors/executor.docker-compose.yaml declares the sourcegraph network, so it is valid on its own.

Decisions

• The retry is in the script, not a Buildkite retry: rule, so a real test failure is never rerun.
• gcloud beta compute instances create, because the agents' gcloud 417 only has --max-run-duration in beta (GA in 483).
• A pd-balanced disk was tried in ci: Speed up the smoke test #1468 and was no faster, so the disk stays pd-standard.
• No YAML formatter (chore: Upgrade prettier to 3.9.10 and format all files #1441 was closed).
• docker compose up --wait does not replace the polling loop: in build 16853 it failed with "container searcher-0 is unhealthy" on a stack which was healthy seconds later. searcher has no health check start period, so it reports unhealthy until the frontend is up.
• The queue is still named vagrant, and the agents keep Vagrant: release branches up to 8.1.x still use it.

Not in this pull request

• The Buildkite pipeline's default branch is still master. It is a pipeline setting, not in any repository, so it has to be changed in Buildkite. pipeline.yaml names main and works with either value.
• docker-compose: Run services with a read-only root filesystem #1470 (read-only root filesystem) changes what customers deploy, and stays separate.

Checklist

• All images have a valid tag and SHA256 sum

Test plan

• The Buildkite build of this branch: validation, shellcheck, and the smoke test. In build 16848 the smoke test job took 4 minutes 14 seconds: 27 of 27 containers ran, none restarted, Jaeger responded, and the executor registered. The VM had maxRunDuration 3600 seconds, instanceTerminationAction: DELETE, and a 20 GB boot disk.
• A failed test on a real VM (build 16853): no second VM, and the step uploaded the logs of all 28 containers.
• The retry was tested with stubbed gcloud and ssh: tests pass, tests fail (no retry, logs copied), VM dies during the tests (retry passes), VM creation fails (retry passes), both VMs die (fails), VM dies then tests fail (fails, logs copied).
• The log saving was tested with a stubbed docker.

[github-actions]

Open the Diff Tour for this PR in Sourcegraph

[marcleblanc2]

Closing in favor of the stack at https://github.com/sourcegraph/deploy-sourcegraph-docker/stack/1491, which splits this into seven PRs: #1484, #1485, #1486, #1487, #1488, #1489, #1490. Build 16857 on this branch showed the pieces work together. One thing differs: this branch retried the smoke test on a new VM, and #1488 only retries the SSH connection on the same VM. The branch stays.