fix/observability: Prometheus read only root volume

charts/sourcegraph/CHANGELOG.md

@@ -8,6 +8,7 @@ Use `**BREAKING**:` to denote a breaking change
 
 ## Unreleased
 
+- `prometheus` now runs with a read-only root filesystem (`prometheus.containerSecurityContext.readOnlyRootFilesystem: true`). This needs a `prometheus` image which keeps Alertmanager's files on the data volume ([sourcegraph/sourcegraph#18737](https://github.com/sourcegraph/sourcegraph/pull/18737)).
 - Added `<serviceName>.extraInitContainers` to every workload, rendered before any init containers the chart defines, so e.g. a secrets-fetching init container on `frontend` runs before `migrator`. Use `restartPolicy: Always` for a native sidecar that must be started before the main container, such as one that writes a `grafana` datasource credential file read by `datasources.yml`.
 - Added `<serviceName>.extraServicePorts` to append ports to each component's primary Service (`sourcegraph-frontend`, `indexed-search`, and `worker`, not `sourcegraph-frontend-internal`, `indexed-search-indexer`, or `worker-executors`), e.g. to expose a sidecar's port
 - Removed the hardcoded `PRECISE_CODE_INTEL_UPLOAD_BACKEND=blobstore` and `PRECISE_CODE_INTEL_UPLOAD_AWS_ENDPOINT=http://blobstore:9000` environment variables from `frontend`, `worker`, `precise-code-intel-worker`, and `syntactic-code-intel-worker`. They matched Sourcegraph's built-in defaults, so default deployments are unchanged. `SOURCEGRAPH_UPLOAD_*` in `*.env` is now the only object storage configuration in the chart, as in the `external-object-storage` examples. Previously, with `blobstore.enabled: true`, the chart's values were rendered after `*.env` and overrode any `PRECISE_CODE_INTEL_UPLOAD_*` set there. Helm's three-way merge removes every duplicate of those names on this upgrade, including the `*.env` override, and the override is applied on the next `helm upgrade` after this one, when code-intel uploads switch to its bucket. Before that, make sure the bucket exists, or move the settings to `SOURCEGRAPH_UPLOAD_*`, or remove them. Run a second `helm upgrade` right away to make the switch at a known time.

charts/sourcegraph/README.md

@@ -346,7 +346,7 @@ In addition to the documented values, all services also support the following va
 | preciseCodeIntel.storageSize | string | `""` | Size of the `/tmp` scratch volume. Required when storageType is `pvc`. When storageType is `emptyDir` this sets `sizeLimit` (optional, leave empty for unlimited). |
 | preciseCodeIntel.storageType | string | `"emptyDir"` | Backing store for the `/tmp` scratch volume, used for large SCIP upload processing. One of: emptyDir, pvc. emptyDir: node ephemeral storage, i.e. the node boot disk. pvc: per-pod PVC, discarded with the pod. The pod `fsGroup` defaults to the container `runAsGroup` so the non-root worker can write to the volume; set `podSecurityContext.fsGroup` to override it. |
 | priorityClasses | list | `[]` | Additional priorityClasses minimize re-scheduling downtime for StatefulSets. Each StatefulSets might use different priority class. learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass) Sample class definition: - name: gitserver-class   value: 100   preemptionPolicy: Never   description: "gitserver priority class" |
-| prometheus.containerSecurityContext | object | `{"allowPrivilegeEscalation":false,"readOnlyRootFilesystem":false,"runAsGroup":100,"runAsUser":100}` | Security context for the `prometheus` container, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container) |
+| prometheus.containerSecurityContext | object | `{"allowPrivilegeEscalation":false,"readOnlyRootFilesystem":true,"runAsGroup":100,"runAsUser":100}` | Security context for the `prometheus` container, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container) |
 | prometheus.createRoleBinding | bool | `true` | Disable the creation of a RoleBinding object, for customers who block all RBAC resource creation |
 | prometheus.enabled | bool | `true` | Enable `prometheus` (recommended) |
 | prometheus.existingConfig | string | `""` | Name of existing ConfigMap for `pgsql`. It must contain a `prometheus.yml` key |

charts/sourcegraph/values.yaml

@@ -1062,9 +1062,7 @@ prometheus:
     allowPrivilegeEscalation: false
     runAsUser: 100
     runAsGroup: 100
-    # Read-only filesystem not supported for the prometheus container,
-    # see [sourcegraph/issues/34012](https://github.com/sourcegraph/sourcegraph/issues/34012) for more information
-    readOnlyRootFilesystem: false
+    readOnlyRootFilesystem: true
   # -- Name used by resources. Does not affect service names or PVCs.
   name: "prometheus"
   # -- Enable RBAC for `prometheus`