From 4906b57b8c2c59f147bc4b847de5cc502d7cc91d Mon Sep 17 00:00:00 2001 From: Marc LeBlanc <7050295+marcleblanc2@users.noreply.github.com> Date: Tue, 10 Nov 2026 18:27:30 -0600 Subject: [PATCH 1/3] prometheus: Run with a read-only root filesystem The prometheus image now keeps Alertmanager's configuration and state on the /prometheus data volume, so nothing writes to the root filesystem. Needs https://github.com/sourcegraph/sourcegraph/pull/18737 in the pinned image. Amp-Thread-ID: https://ampcode.com/threads/T-01a13cfb-9756-7111-8b29-381a841065e1 Co-authored-by: Amp --- charts/sourcegraph/CHANGELOG.md | 1 + charts/sourcegraph/README.md | 2 +- charts/sourcegraph/values.yaml | 4 +--- 3 files changed, 3 insertions(+), 4 deletions(-) diff --git a/charts/sourcegraph/CHANGELOG.md b/charts/sourcegraph/CHANGELOG.md index 9f8ad6c40..5ad7c97af 100644 --- a/charts/sourcegraph/CHANGELOG.md +++ b/charts/sourcegraph/CHANGELOG.md @@ -8,6 +8,7 @@ Use `**BREAKING**:` to denote a breaking change ## Unreleased +- `prometheus` now runs with a read-only root filesystem (`prometheus.containerSecurityContext.readOnlyRootFilesystem: true`), like every other workload. This needs a `prometheus` image which keeps Alertmanager's files on the data volume ([sourcegraph/sourcegraph#18737](https://github.com/sourcegraph/sourcegraph/pull/18737)); set it back to `false` when running an older image with this chart. - Added `.extraInitContainers` to every workload, rendered before any init containers the chart defines, so e.g. a secrets-fetching init container on `frontend` runs before `migrator`. Use `restartPolicy: Always` for a native sidecar that must be started before the main container, such as one that writes a `grafana` datasource credential file read by `datasources.yml`. - Added `.extraServicePorts` to append ports to each component's primary Service (`sourcegraph-frontend`, `indexed-search`, and `worker`, not `sourcegraph-frontend-internal`, `indexed-search-indexer`, or `worker-executors`), e.g. to expose a sidecar's port - Removed the hardcoded `PRECISE_CODE_INTEL_UPLOAD_BACKEND=blobstore` and `PRECISE_CODE_INTEL_UPLOAD_AWS_ENDPOINT=http://blobstore:9000` environment variables from `frontend`, `worker`, `precise-code-intel-worker`, and `syntactic-code-intel-worker`. They matched Sourcegraph's built-in defaults, so default deployments are unchanged. `SOURCEGRAPH_UPLOAD_*` in `*.env` is now the only object storage configuration in the chart, as in the `external-object-storage` examples. Previously, with `blobstore.enabled: true`, the chart's values were rendered after `*.env` and overrode any `PRECISE_CODE_INTEL_UPLOAD_*` set there. Helm's three-way merge removes every duplicate of those names on this upgrade, including the `*.env` override, and the override is applied on the next `helm upgrade` after this one, when code-intel uploads switch to its bucket. Before that, make sure the bucket exists, or move the settings to `SOURCEGRAPH_UPLOAD_*`, or remove them. Run a second `helm upgrade` right away to make the switch at a known time. diff --git a/charts/sourcegraph/README.md b/charts/sourcegraph/README.md index 31e0d10fb..f244e975b 100644 --- a/charts/sourcegraph/README.md +++ b/charts/sourcegraph/README.md @@ -346,7 +346,7 @@ In addition to the documented values, all services also support the following va | preciseCodeIntel.storageSize | string | `""` | Size of the `/tmp` scratch volume. Required when storageType is `pvc`. When storageType is `emptyDir` this sets `sizeLimit` (optional, leave empty for unlimited). | | preciseCodeIntel.storageType | string | `"emptyDir"` | Backing store for the `/tmp` scratch volume, used for large SCIP upload processing. One of: emptyDir, pvc. emptyDir: node ephemeral storage, i.e. the node boot disk. pvc: per-pod PVC, discarded with the pod. The pod `fsGroup` defaults to the container `runAsGroup` so the non-root worker can write to the volume; set `podSecurityContext.fsGroup` to override it. | | priorityClasses | list | `[]` | Additional priorityClasses minimize re-scheduling downtime for StatefulSets. Each StatefulSets might use different priority class. learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass) Sample class definition: - name: gitserver-class value: 100 preemptionPolicy: Never description: "gitserver priority class" | -| prometheus.containerSecurityContext | object | `{"allowPrivilegeEscalation":false,"readOnlyRootFilesystem":false,"runAsGroup":100,"runAsUser":100}` | Security context for the `prometheus` container, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container) | +| prometheus.containerSecurityContext | object | `{"allowPrivilegeEscalation":false,"readOnlyRootFilesystem":true,"runAsGroup":100,"runAsUser":100}` | Security context for the `prometheus` container, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container) | | prometheus.createRoleBinding | bool | `true` | Disable the creation of a RoleBinding object, for customers who block all RBAC resource creation | | prometheus.enabled | bool | `true` | Enable `prometheus` (recommended) | | prometheus.existingConfig | string | `""` | Name of existing ConfigMap for `pgsql`. It must contain a `prometheus.yml` key | diff --git a/charts/sourcegraph/values.yaml b/charts/sourcegraph/values.yaml index dc02477c1..5664784e5 100644 --- a/charts/sourcegraph/values.yaml +++ b/charts/sourcegraph/values.yaml @@ -1062,9 +1062,7 @@ prometheus: allowPrivilegeEscalation: false runAsUser: 100 runAsGroup: 100 - # Read-only filesystem not supported for the prometheus container, - # see [sourcegraph/issues/34012](https://github.com/sourcegraph/sourcegraph/issues/34012) for more information - readOnlyRootFilesystem: false + readOnlyRootFilesystem: true # -- Name used by resources. Does not affect service names or PVCs. name: "prometheus" # -- Enable RBAC for `prometheus` From 66ec71b2c8722cd7066f6fa5b52063294d236127 Mon Sep 17 00:00:00 2001 From: Marc LeBlanc <7050295+marcleblanc2@users.noreply.github.com> Date: Tue, 10 Nov 2026 18:28:10 -0600 Subject: [PATCH 2/3] changelog: Drop inaccurate comparison to other workloads --- charts/sourcegraph/CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/charts/sourcegraph/CHANGELOG.md b/charts/sourcegraph/CHANGELOG.md index 5ad7c97af..b35aeeaef 100644 --- a/charts/sourcegraph/CHANGELOG.md +++ b/charts/sourcegraph/CHANGELOG.md @@ -8,7 +8,7 @@ Use `**BREAKING**:` to denote a breaking change ## Unreleased -- `prometheus` now runs with a read-only root filesystem (`prometheus.containerSecurityContext.readOnlyRootFilesystem: true`), like every other workload. This needs a `prometheus` image which keeps Alertmanager's files on the data volume ([sourcegraph/sourcegraph#18737](https://github.com/sourcegraph/sourcegraph/pull/18737)); set it back to `false` when running an older image with this chart. +- `prometheus` now runs with a read-only root filesystem (`prometheus.containerSecurityContext.readOnlyRootFilesystem: true`). This needs a `prometheus` image which keeps Alertmanager's files on the data volume ([sourcegraph/sourcegraph#18737](https://github.com/sourcegraph/sourcegraph/pull/18737)); set it back to `false` when running an older image with this chart. - Added `.extraInitContainers` to every workload, rendered before any init containers the chart defines, so e.g. a secrets-fetching init container on `frontend` runs before `migrator`. Use `restartPolicy: Always` for a native sidecar that must be started before the main container, such as one that writes a `grafana` datasource credential file read by `datasources.yml`. - Added `.extraServicePorts` to append ports to each component's primary Service (`sourcegraph-frontend`, `indexed-search`, and `worker`, not `sourcegraph-frontend-internal`, `indexed-search-indexer`, or `worker-executors`), e.g. to expose a sidecar's port - Removed the hardcoded `PRECISE_CODE_INTEL_UPLOAD_BACKEND=blobstore` and `PRECISE_CODE_INTEL_UPLOAD_AWS_ENDPOINT=http://blobstore:9000` environment variables from `frontend`, `worker`, `precise-code-intel-worker`, and `syntactic-code-intel-worker`. They matched Sourcegraph's built-in defaults, so default deployments are unchanged. `SOURCEGRAPH_UPLOAD_*` in `*.env` is now the only object storage configuration in the chart, as in the `external-object-storage` examples. Previously, with `blobstore.enabled: true`, the chart's values were rendered after `*.env` and overrode any `PRECISE_CODE_INTEL_UPLOAD_*` set there. Helm's three-way merge removes every duplicate of those names on this upgrade, including the `*.env` override, and the override is applied on the next `helm upgrade` after this one, when code-intel uploads switch to its bucket. Before that, make sure the bucket exists, or move the settings to `SOURCEGRAPH_UPLOAD_*`, or remove them. Run a second `helm upgrade` right away to make the switch at a known time. From 6d11873a1664bd6d10a2cfe697599b237b2e6535 Mon Sep 17 00:00:00 2001 From: Marc LeBlanc <7050295+marcleblanc2@users.noreply.github.com> Date: Tue, 10 Nov 2026 21:56:19 -0600 Subject: [PATCH 3/3] changelog: Drop the note about older prometheus images --- charts/sourcegraph/CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/charts/sourcegraph/CHANGELOG.md b/charts/sourcegraph/CHANGELOG.md index b35aeeaef..12a6aa5ab 100644 --- a/charts/sourcegraph/CHANGELOG.md +++ b/charts/sourcegraph/CHANGELOG.md @@ -8,7 +8,7 @@ Use `**BREAKING**:` to denote a breaking change ## Unreleased -- `prometheus` now runs with a read-only root filesystem (`prometheus.containerSecurityContext.readOnlyRootFilesystem: true`). This needs a `prometheus` image which keeps Alertmanager's files on the data volume ([sourcegraph/sourcegraph#18737](https://github.com/sourcegraph/sourcegraph/pull/18737)); set it back to `false` when running an older image with this chart. +- `prometheus` now runs with a read-only root filesystem (`prometheus.containerSecurityContext.readOnlyRootFilesystem: true`). This needs a `prometheus` image which keeps Alertmanager's files on the data volume ([sourcegraph/sourcegraph#18737](https://github.com/sourcegraph/sourcegraph/pull/18737)). - Added `.extraInitContainers` to every workload, rendered before any init containers the chart defines, so e.g. a secrets-fetching init container on `frontend` runs before `migrator`. Use `restartPolicy: Always` for a native sidecar that must be started before the main container, such as one that writes a `grafana` datasource credential file read by `datasources.yml`. - Added `.extraServicePorts` to append ports to each component's primary Service (`sourcegraph-frontend`, `indexed-search`, and `worker`, not `sourcegraph-frontend-internal`, `indexed-search-indexer`, or `worker-executors`), e.g. to expose a sidecar's port - Removed the hardcoded `PRECISE_CODE_INTEL_UPLOAD_BACKEND=blobstore` and `PRECISE_CODE_INTEL_UPLOAD_AWS_ENDPOINT=http://blobstore:9000` environment variables from `frontend`, `worker`, `precise-code-intel-worker`, and `syntactic-code-intel-worker`. They matched Sourcegraph's built-in defaults, so default deployments are unchanged. `SOURCEGRAPH_UPLOAD_*` in `*.env` is now the only object storage configuration in the chart, as in the `external-object-storage` examples. Previously, with `blobstore.enabled: true`, the chart's values were rendered after `*.env` and overrode any `PRECISE_CODE_INTEL_UPLOAD_*` set there. Helm's three-way merge removes every duplicate of those names on this upgrade, including the `*.env` override, and the override is applied on the next `helm upgrade` after this one, when code-intel uploads switch to its bucket. Before that, make sure the bucket exists, or move the settings to `SOURCEGRAPH_UPLOAD_*`, or remove them. Run a second `helm upgrade` right away to make the switch at a known time.