[marcleblanc2]

Problem

An audit of every config and example file in this repository against the v8.2.0 source and a running deployment found:

• Settings nothing reads. worker sets SRC_GIT_SERVERS, SEARCHER_URL, and INDEXED_SEARCH_SERVERS, which only the frontend reads. zoekt-indexserver-0 sets OPENTELEMETRY_DISABLED and OTEL_EXPORTER_OTLP_ENDPOINT, but only zoekt-webserver starts a tracer. The executor sets UID and GID, which it does not read.
• Resources nothing uses. The /mnt/cache volumes of sourcegraph-frontend-0, sourcegraph-frontend-internal, and worker stay empty: only searcher reads CACHE_DIR. Prometheus scrapes redis-cache:9121 and redis-store:9121, but the stack has no Redis exporter, so both targets are always down. expose: on cadvisor and node-exporter does nothing on a user-defined network. The Jaeger overlay publishes 4317, 4318, and 5778 on the host, but only otel-collector sends to Jaeger, over the Compose network.
• Tracing does not work out of the box. The default collector command uses the image's logging.yaml, which binds OTLP and the metrics port to loopback, so no service can send spans and the otel-collector:8888 target is down. The repository's own otel-collector/config.yaml, which the docs call the default, is mounted but unused, and the collector rejects it (invalid keys: address).
• Broken or misleading config. The Grafana Jaeger datasource has the wrong type (Jaeger) and a hostname that does not exist (jaeger-query). The GF_SERVER_ROOT_URL line in the Grafana comment breaks the /-/debug/grafana proxy, and two GF_* lines there have stray quotes. The @http redirect block in https.custom-cert.Caddyfile never matches. The Linux steps in prometheus/README.md scrape localhost:9323, which is the Prometheus container itself.
• Examples which fail as written.
  • external-databases: uncommenting the two IAM lines puts them at the top level of the file, which Compose rejects.
  • migrator-operations: uncommenting a command: line next to the active one is a duplicate key.
  • aws-rds-databases: sets PG_CONNECTION_UPDATER on five services which never read it.
  • aws-alb: fails with port is already allocated unless applied with --remove-orphans.
• Effective but undocumented. The pgsql Grafana datasource is configured by six GRAFANA_PGSQL_* variables which appear nowhere. syntactic-code-intel-worker:6060 serves metrics which Prometheus does not scrape. Several "Ports exposed" comments list ports with no listener (3443, 9121, 9187) or omit real ones.

Solution

Remove what is unused, fix what is broken, and document what works.

Removed from the base file (customer override files may refer to these):

Kind	Removed
Environment variables	worker: SRC_GIT_SERVERS, SEARCHER_URL, INDEXED_SEARCH_SERVERS zoekt-indexserver-0: OPENTELEMETRY_DISABLED, OTEL_EXPORTER_OTLP_ENDPOINT executor: UID, GID
Volumes	sourcegraph-frontend-0, sourcegraph-frontend-internal-0, worker, and their /mnt/cache mounts
Published ports (Jaeger overlay)	4317, 4318, 5778; 16686 stays
expose:	cadvisor 48080, node-exporter 9100
Prometheus targets	redis-cache:9121, redis-store:9121
Prometheus recording rule	container:process_cpu_seconds_total:ratio_rate5m: its two sides have different labels, so it never recorded a series

An override file which sets one of the removed variables still works, since Compose adds it back. An override file which mounts something else at one of the removed named volumes must now declare that volume itself. Existing volumes are left on disk; docker volume rm removes them.

Changed behaviour:

• otel-collector now starts with --config /etc/otel-collector/config.yaml, and that file is rewritten for the collector version in the image: OTLP on 0.0.0.0:4317 and 0.0.0.0:4318, a debug exporter, and metrics on 0.0.0.0:8888. An override which already sets command or mounts its own file over that path is unaffected. The Jaeger overlay still selects the image's jaeger.yaml.
• Prometheus scrapes syntactic-code-intel-worker:6060, and the frontend's /-/debug/ page lists both code intel workers (SRC_PROF_SERVICES).
• The Grafana Jaeger datasource points at http://jaeger:16686/-/debug/jaeger with type jaeger.
• https.custom-cert.Caddyfile loses the redirect block which never matched; behaviour is unchanged.

Documentation and examples:

• Documented the six GRAFANA_PGSQL_* variables, and that the four replica variables on the frontends take a list or a replica count.
• Corrected the "Ports exposed" comments, the migrator docs link, the dev overlay comment, and the sourcegraph-uploads.env pointer.
• prometheus/README.md: Linux steps which work (metrics-addr on 0.0.0.0:9323, host.docker.internal:host-gateway).
• Examples: external-databases points at aws-rds-databases for IAM; aws-rds-databases sets PG_CONNECTION_UPDATER only on the migrator and both frontends, the services which read it; migrator-operations says to replace the command line; aws-alb says to use --remove-orphans; both database examples say that Prometheus shows the three removed exporters as down; outbound-http-proxy says that Git needs lowercase http_proxy for http:// code hosts.

Not in this pull request:

• Checking every Compose file in CI, and the executor file failing to parse on its own: ci: Check every Compose file and shell script, and test Jaeger and the executor #1489.
• docker-compose: Run services with a read-only root filesystem #1470 adds volume lines next to the ones removed here, so whichever merges second needs a rebase.
• Redis exporters, so that Redis has metrics at all.
• The docs repository fixes: self-hosted: Correct Docker Compose settings which Sourcegraph does not read docs#2162.

Checklist

• All images have a valid tag and SHA256 sum (no image: line changed)

Test plan

• npm ci && npm run prettier-check passes, and every changed YAML file parses.
• Buildkite build 16978 passed at bffc6b7.
• Each removed setting was checked against the v8.2.0 source of sourcegraph/sourcegraph and the pinned sourcegraph/zoekt revision for a reader, and found none.
• On main in an Amp orb running the full stack: the three cache volumes were empty; both Redis targets were down with connection refused; the collector with logging.yaml listened on loopback only and rejected the old config.yaml; the Grafana Jaeger datasource failed until its type and URL were changed to the values in this pull request; the custom-cert Caddyfile behaved the same without the redirect block; the README's localhost:9323 target was down, and the new steps brought the docker target up; each example failure above was reproduced, and the corrected instructions ran. Evidence: base stack, examples.
• This branch in a second orb, started fresh at 301672a (bffc6b7 differs by three comment lines in the examples) and compared with main at d671015 (verification thread):
  • docker compose config -q exits 0 for the base file, alone and with each of the jaeger, executors, and dev files and each of the six examples, and for db-only-migrate; .buildkite/validate-docker-compose.sh exits 0.
  • 25 containers running, 0 restarts, 0 unhealthy after 10 minutes, the same as main. The frontend services and worker log nothing about /mnt/cache or missing service addresses, and worker still connects to gitserver, searcher, and zoekt. A repository clones and indexes; literal and symbol searches return matches.
  • Prometheus: 19 of 19 targets up. On main, otel-collector:8888 and both Redis targets are down.
  • The collector runs config.yaml, listens on all interfaces on 4317, 4318, and 8888, and accepted and sent 707 spans.
  • With the jaeger file: a trace of 69 spans is returned by Jaeger, by the frontend's /-/debug/jaeger proxy, and by the Grafana Jaeger datasource (main's datasource file returns 500 "Unable to find datasource plugin").
  • The custom-cert Caddyfile serves HTTPS and redirects HTTP with 308, the same as on main.
  • prometheus/README.md Linux steps: the docker target is up; with 127.0.0.1:9323 it is down, as the README says.
  • migrator-operations: the add-log command exits 0 and adds the migration_logs row.
  • The six GRAFANA_PGSQL_* lines make the datasource's SELECT 1 return a row; without them it returns 500.
  • /-/debug/ links both code intel workers and both links return 200; docker_rules.yml loads 3 rules with health ok.
• Not tested: the aws-rds-databases example against real RDS with IAM (source-backed only); upgrading a deployment whose three /mnt/cache volumes already exist.

[github-actions]

Open the Diff Tour for this PR in Sourcegraph