diff --git a/charts/sourcegraph-executor/CHANGELOG.md b/charts/sourcegraph-executor/CHANGELOG.md index 5e2f4fb18..e28d0791c 100644 --- a/charts/sourcegraph-executor/CHANGELOG.md +++ b/charts/sourcegraph-executor/CHANGELOG.md @@ -3,6 +3,13 @@ ## Unreleased +- Fixed `executor.maximumNumJobs` being ignored by the `sourcegraph-executor-dind` chart, which also refused `EXECUTOR_MAXIMUM_NUM_JOBS` in `executor.env`, so the job limit could not be changed. Executors used their own default, 1 job at a time, while the chart documented 10. The chart now passes `executor.maximumNumJobs` to every executor, with the documented default of 10. **On upgrade, executors that do not set it go from 1 job at a time to 10**, and their pods restart once. Each job runs in the pod's dind container, so before upgrading, set `executor.resources` and `executor.storage` for 10 concurrent jobs, or set `executor.maximumNumJobs: 1` to keep the current behavior. The chart refuses to render when `executor.maximumNumJobs` is null or empty, because the executor cannot start without a number. `helm upgrade --reuse-values` from a chart older than 7.4.0, which had no such value, fails this way; add `--set executor.maximumNumJobs=`. `EXECUTOR_DOCKER_ADD_HOST_GATEWAY` and `EXECUTOR_KEEP_WORKSPACES`, which had no value, can now be set in `executor.env`. +- Added `queues[].maximumNumJobs` to the `sourcegraph-executor-dind` chart, so each queue can set its own job limit. A queue that does not set it uses `executor.maximumNumJobs`. Before, the chart ignored the field, and refused `EXECUTOR_MAXIMUM_NUM_JOBS` in a queue's `env`, so every queue had the same limit. +- Fixed `executor.nodeSelector`, `executor.affinity`, `executor.tolerations`, and `privateDockerRegistry.nodeSelector`, `affinity`, `tolerations`, and `podSecurityContext` being ignored by the `sourcegraph-executor-dind` chart, which listed them in its README. They now replace `sourcegraph.nodeSelector`, `affinity`, and `tolerations` for that pod, and are documented in `values.yaml`. +- Fixed `sourcegraph.nodeSelector`, `sourcegraph.affinity`, and `sourcegraph.tolerations` being ignored by the `sourcegraph-executor-k8s` chart. They are now used when `executor.nodeSelector`, `executor.affinity`, or `executor.tolerations` is empty, so a release that sets them gets its executor pods rescheduled on upgrade. Job pods are not affected: they are still scheduled only by `executor.kubernetesJob.node` and `executor.kubernetesJob.pod`. Removed `executor.securityContext.privileged`, which the chart never read, and the README's list of `privateDockerRegistry` values, which this chart does not have. +- Removed `storageClass.create`, `provisioner`, `type`, `parameters`, and `allowedTopologies` from the `sourcegraph-executor-dind` chart's values. The chart has no StorageClass template, so they never had an effect. `storageClass.name` stays: it is the StorageClass of the `private-docker-registry` volume. +- Fixed the documentation of `nameOverride` in the `sourcegraph-executor-k8s` and `sourcegraph-executor-dind` charts. It was documented as `sourcegraph.nameOverride`, which the charts never read, so setting it had no effect. The charts read the top-level `nameOverride`, which replaces the chart name in the `app.kubernetes.io/name` selector label. Set it only on a new install, because Kubernetes does not allow changing a selector. If you set `sourcegraph.nameOverride`, remove it; moving its value to `nameOverride` changes your selectors. + ## 8.0.0 * Added a `network-policy` example to the `sourcegraph-executor-k8s` and `sourcegraph-executor-dind` charts, with `CiliumNetworkPolicy` resources which limit Executor and job pods to the frontend API. diff --git a/charts/sourcegraph-executor/dind/README.md b/charts/sourcegraph-executor/dind/README.md index 14bff60a9..c21e9e0c1 100644 --- a/charts/sourcegraph-executor/dind/README.md +++ b/charts/sourcegraph-executor/dind/README.md @@ -42,20 +42,6 @@ Manifests rendered using the `helm template` command can be used for direct depl Reference the table below for available configuration parameters and consult [executor] documentation. -In addition to the documented values, the `executor` and `private-docker-registry` services also supports the following values - -- `executor.affinity` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) -- `executor.nodeSelector` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) -- `executor.tolerations` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) -- `executor.podSecurityContext` - [learn more](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) -- `executor.env` - consult `values.yaml` - -- `privateDockerRegistry.affinity` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) -- `privateDockerRegistry.nodeSelector` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) -- `privateDockerRegistry.tolerations` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) -- `privateDockerRegistry.podSecurityContext` - [learn more](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) -- `privateDockerRegistry.env` - consult `values.yaml` file - | Key | Type | Default | Description | |-----|------|---------|-------------| | dind.command | list | `["dockerd"]` | Command for the dind container. | @@ -67,6 +53,7 @@ In addition to the documented values, the `executor` and `private-docker-registr | dind.image.registry | string | `"index.docker.io"` | | | dind.image.repository | string | `"docker"` | | | dind.image.tag | string | `"29.5.3-dind"` | | +| executor.affinity | object | `{}` | Affinity for executor pods. Replaces `sourcegraph.affinity` when set, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) | | executor.env | object | `{}` | Extra environment variables to set on the executor container. | | executor.frontendExistingSecret | string | `""` | Name of existing k8s Secret to use for frontend password. The k8s Secret must contain the key EXECUTOR_FRONTEND_PASSWORD matching the site config executors.accessToken value. frontendPassword is ignored if this is set. | | executor.frontendPassword | string | `""` | The shared secret configured in the Sourcegraph instance site config under executors.accessToken. Required if frontendExistingSecret is not configured. | @@ -75,8 +62,9 @@ In addition to the documented values, the `executor` and `private-docker-registr | executor.image.name | string | `"executor"` | | | executor.log.format | string | `"json"` | | | executor.log.level | string | `"warn"` | Possible values are dbug, info, warn, eror, crit. | -| executor.maximumNumJobs | int | `10` | The maximum amount of jobs that can be executed concurrently. | +| executor.maximumNumJobs | int | `10` | The maximum number of jobs each executor pod runs at once. Each job runs in the pod's dind container, so size `resources` and `storage` for this many jobs. Each queue can override this with its own maximumNumJobs field. | | executor.maximumRuntimePerJob | string | `"30m"` | The maximum wall time that can be spent on a single job. | +| executor.nodeSelector | object | `{}` | Node selector for executor pods. Replaces `sourcegraph.nodeSelector` when set, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) | | executor.podSecurityContext | object | `{"fsGroup":101}` | Pod-level security context applied to executor pods. fsGroup: 101 ensures the scratch volume is writable by the sourcegraph user (uid=100, gid=101) when running in Docker mode (dind). Set to {} to disable. | | executor.queueName | string | `""` | The name of the queue to pull jobs from. Possible values: batches and codeintel. Either this or queueNames is required (when not using queues). | | executor.queueNames | list | `[]` | The names of multiple queues to pull jobs from. Possible values: batches and codeintel. Either this or queueName is required (when not using queues). | @@ -85,32 +73,32 @@ In addition to the documented values, the `executor` and `private-docker-registr | executor.storage.class | string | `""` | StorageClass for the ephemeral volume. Only used when type is ephemeral. Defaults to the cluster default StorageClass when empty. | | executor.storage.size | string | `""` | Size of the scratch volume. emptyDir: sets sizeLimit (optional, leave empty for unlimited). ephemeral: sets the PVC storage request (required). | | executor.storage.type | string | `"emptyDir"` | Type of scratch volume for job workspaces. One of: emptyDir, ephemeral. emptyDir: plain emptyDir, no storage class required. ephemeral: per-pod PVC via the cluster default storage class; size is required. | +| executor.tolerations | list | `[]` | Tolerations for executor pods. Replaces `sourcegraph.tolerations` when set, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) | +| nameOverride | string | `""` | Replaces the chart name in the `app.kubernetes.io/name` label, which the Deployment and StatefulSet selectors use. Set it only on a new install: Kubernetes does not allow changing a selector. As a subchart with an `alias`, the alias replaces the chart name; set `.nameOverride: sourcegraph-executor-dind` in the parent chart to keep the selectors of a standalone install. | +| privateDockerRegistry.affinity | object | `{}` | Affinity for the registry pod. Replaces `sourcegraph.affinity` when set, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) | | privateDockerRegistry.enabled | bool | `true` | Whether to deploy the private registry. Only one registry is needed when deploying multiple executors. More information: https://sourcegraph.com/docs/self-hosted/executors/deploy-executors#using-private-registries | | privateDockerRegistry.env | object | `{"REGISTRY_PROXY_REMOTEURL":{"value":"https://registry-1.docker.io"}}` | Extra environment variables to set on the registry container. Registry can be configured with environment variables using the pattern REGISTRY_
_ (uppercase). Full reference: https://distribution.github.io/distribution/about/configuration To proxy an internal private registry with credentials, override REGISTRY_PROXY_REMOTEURL and supply REGISTRY_PROXY_USERNAME / REGISTRY_PROXY_PASSWORD, e.g.: env: REGISTRY_PROXY_REMOTEURL: value: https://my-internal-registry.example.com REGISTRY_PROXY_USERNAME: value: myuser REGISTRY_PROXY_PASSWORD: valueFrom: secretKeyRef: name: registry-proxy-credentials key: password | | privateDockerRegistry.image.registry | string | `"index.docker.io"` | | | privateDockerRegistry.image.repository | string | `"registry"` | | | privateDockerRegistry.image.tag | int | `3` | | +| privateDockerRegistry.nodeSelector | object | `{}` | Node selector for the registry pod. Replaces `sourcegraph.nodeSelector` when set, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) | +| privateDockerRegistry.podSecurityContext | object | `{}` | Pod-level security context for the registry pod, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) | | privateDockerRegistry.storageSize | string | `"10Gi"` | | -| queues | list | `[]` | Optional list of queues to deploy as standalone Deployments. When set, the single executor Deployment is not rendered. Each entry supports: name (required) — used as the deployment name suffix (executor-) queueName — sets EXECUTOR_QUEUE_NAME; defaults to name if omitted queueNames — sets EXECUTOR_QUEUE_NAMES (comma-joined); takes precedence over queueName when set replicaCount, resources, env (merged with executor.env, queue overrides) | -| sourcegraph.affinity | object | `{}` | Affinity, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) | +| privateDockerRegistry.tolerations | list | `[]` | Tolerations for the registry pod. Replaces `sourcegraph.tolerations` when set, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) | +| queues | list | `[]` | Optional list of queues to deploy as standalone Deployments. When set, the single executor Deployment is not rendered. Each entry supports: name (required) — used as the deployment name suffix (executor-) queueName — sets EXECUTOR_QUEUE_NAME; defaults to name if omitted queueNames — sets EXECUTOR_QUEUE_NAMES (comma-joined); takes precedence over queueName when set replicaCount, resources, maximumNumJobs — default to the executor.* value of the same name env — merged with executor.env, queue overrides | +| sourcegraph.affinity | object | `{}` | Default affinity for every pod. `executor.affinity`, `privateDockerRegistry.affinity` replace it when set, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) | | sourcegraph.image.defaultTag | string | `"{{ .Chart.AppVersion }}"` | Global docker image tag | | sourcegraph.image.pullPolicy | string | `"IfNotPresent"` | Global docker image pull policy | | sourcegraph.image.repository | string | Sourcegraph's public Docker Hub registry | Global docker image registry or prefix | | sourcegraph.image.useGlobalTagAsDefault | bool | `false` | When set to true, sourcegraph.image.defaultTag is used as the default defaultTag for all services, instead of service-specific default defaultTags | | sourcegraph.imagePullSecrets | list | `[]` | Mount named secrets containing docker credentials | | sourcegraph.labels | object | `{}` | Add a global label to all resources | -| sourcegraph.nameOverride | string | `""` | Set a custom name for the app.kubernetes.io/name annotation | -| sourcegraph.nodeSelector | object | `{}` | NodeSelector, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) | +| sourcegraph.nodeSelector | object | `{}` | Default nodeSelector for every pod. `executor.nodeSelector`, `privateDockerRegistry.nodeSelector` replace it when set, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) | | sourcegraph.podAnnotations | object | `{}` | Add extra annotations to attach to all pods | | sourcegraph.podLabels | object | `{}` | Add extra labels to attach to all pods | | sourcegraph.priorityClassName | string | `""` | Assign a priorityClass to all pods (daemonSets, deployments, and statefulSets) | -| sourcegraph.tolerations | list | `[]` | Tolerations, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) | -| storageClass.allowedTopologies | object | `{}` | Persistent volumes topology configuration, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/storage-classes/#allowed-topologies) | -| storageClass.create | bool | `false` | Enable creation of storageClass. Defaults to Google Cloud Platform. Disable if you have your own existing storage class | -| storageClass.name | string | `"sourcegraph"` | Name of the storageClass. Use to customize to the existing storage class name | -| storageClass.parameters | object | `{}` | Extra parameters of storageClass, consult your cloud provider persistent storage documentation | -| storageClass.provisioner | string | `"kubernetes.io/gce-pd"` | Name of the storageClass provisioner, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/storage-classes/#provisioner) and consult your cloud provider persistent storage documentation | -| storageClass.type | string | `"pd-ssd"` | Value of `type` key in storageClass `parameters`, consult your cloud provider persistent storage documentation | +| sourcegraph.tolerations | list | `[]` | Default tolerations for every pod. `executor.tolerations`, `privateDockerRegistry.tolerations` replace it when set, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) | +| storageClass.name | string | `"sourcegraph"` | Name of an existing StorageClass for the `private-docker-registry` volume. This chart does not create a StorageClass. | ## Troubleshooting diff --git a/charts/sourcegraph-executor/dind/README.md.gotmpl b/charts/sourcegraph-executor/dind/README.md.gotmpl index fbc7da966..62565101e 100644 --- a/charts/sourcegraph-executor/dind/README.md.gotmpl +++ b/charts/sourcegraph-executor/dind/README.md.gotmpl @@ -42,20 +42,6 @@ Manifests rendered using the `helm template` command can be used for direct depl Reference the table below for available configuration parameters and consult [executor] documentation. -In addition to the documented values, the `executor` and `private-docker-registry` services also supports the following values - -- `executor.affinity` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) -- `executor.nodeSelector` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) -- `executor.tolerations` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) -- `executor.podSecurityContext` - [learn more](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) -- `executor.env` - consult `values.yaml` - -- `privateDockerRegistry.affinity` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) -- `privateDockerRegistry.nodeSelector` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) -- `privateDockerRegistry.tolerations` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) -- `privateDockerRegistry.podSecurityContext` - [learn more](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) -- `privateDockerRegistry.env` - consult `values.yaml` file - {{ template "chart.valuesTable" . }} ## Troubleshooting diff --git a/charts/sourcegraph-executor/dind/examples/gcp/values.yaml b/charts/sourcegraph-executor/dind/examples/gcp/values.yaml index feee1a29c..7bac3dd49 100644 --- a/charts/sourcegraph-executor/dind/examples/gcp/values.yaml +++ b/charts/sourcegraph-executor/dind/examples/gcp/values.yaml @@ -1,5 +1,4 @@ storageClass: - create: false name: standard-rwo executor: diff --git a/charts/sourcegraph-executor/dind/templates/_helpers.tpl b/charts/sourcegraph-executor/dind/templates/_helpers.tpl index 8b87b5d24..3b9910a1d 100644 --- a/charts/sourcegraph-executor/dind/templates/_helpers.tpl +++ b/charts/sourcegraph-executor/dind/templates/_helpers.tpl @@ -59,45 +59,6 @@ useGlobalTagAsDefault configuration {{- $top.Values.sourcegraph.image.repository }}/{{ $imageName }}:{{ default $defaultTag (index $top.Values $service "image" "tag") }} {{- end }} -{{- define "sourcegraph.nodeSelector" -}} -{{- $top := index . 0 }} -{{- $service := index . 1 }} -{{- $globalNodeSelector := (index $top.Values "sourcegraph" "nodeSelector") }} -{{- $serviceNodeSelector := (index $top.Values $service "nodeSelector") }} -nodeSelector: -{{- if $serviceNodeSelector }} -{{- $serviceNodeSelector | toYaml | trim | nindent 2 }} -{{- else if $globalNodeSelector }} -{{- $globalNodeSelector | toYaml | trim | nindent 2 }} -{{- end }} -{{- end }} - -{{- define "sourcegraph.affinity" -}} -{{- $top := index . 0 }} -{{- $service := index . 1 }} -{{- $globalAffinity := (index $top.Values "sourcegraph" "affinity") }} -{{- $serviceAffinity := (index $top.Values $service "affinity") }} -affinity: -{{- if $serviceAffinity }} -{{- tpl ($serviceAffinity | toYaml) $top | trim | nindent 2 }} -{{- else if $globalAffinity }} -{{- tpl ($globalAffinity | toYaml) $top | trim | nindent 2 }} -{{- end }} -{{- end }} - -{{- define "sourcegraph.tolerations" -}} -{{- $top := index . 0 }} -{{- $service := index . 1 }} -{{- $globalTolerations := (index $top.Values "sourcegraph" "tolerations") }} -{{- $serviceTolerations := (index $top.Values $service "tolerations") }} -tolerations: -{{- if $serviceTolerations }} -{{- $serviceTolerations | toYaml | trim | nindent 2 }} -{{- else if $globalTolerations }} -{{- $globalTolerations | toYaml | trim | nindent 2 }} -{{- end }} -{{- end }} - {{- define "executor.name" -}} {{- if .Values.executor.queueName -}} @@ -124,7 +85,7 @@ app.kubernetes.io/component: executor {{/* Render a single executor Deployment. -Usage: include "executor.deployment" (dict "root" $ "name" "executor-foo" "queueName" "foo" "queueNames" (list) "replicaCount" 1 "resources" $res "env" $env) +Usage: include "executor.deployment" (dict "root" $ "name" "executor-foo" "queueName" "foo" "queueNames" (list) "replicaCount" 1 "resources" $res "maximumNumJobs" 10 "env" $env) */}} {{- define "executor.deployment" -}} {{- $r := .root -}} @@ -230,6 +191,8 @@ spec: value: {{ $r.Values.executor.log.level | quote }} - name: SRC_LOG_FORMAT value: {{ $r.Values.executor.log.format | quote }} + - name: EXECUTOR_MAXIMUM_NUM_JOBS + value: {{ required "executor.maximumNumJobs is required, for example 10" .maximumNumJobs | quote }} - name: EXECUTOR_MAXIMUM_RUNTIME_PER_JOB value: {{ $r.Values.executor.maximumRuntimePerJob | quote }} - name: EXECUTOR_USE_FIRECRACKER @@ -313,16 +276,16 @@ spec: {{- if $r.Values.dind.gVisor.enabled }} runtimeClassName: gvisor {{- end }} - {{- with $r.Values.sourcegraph.nodeSelector }} + {{- with $r.Values.executor.nodeSelector | default $r.Values.sourcegraph.nodeSelector }} nodeSelector: {{- toYaml . | nindent 8 }} {{- end }} - {{- with $r.Values.sourcegraph.affinity }} + {{- with $r.Values.executor.affinity | default $r.Values.sourcegraph.affinity }} affinity: {{- toYaml . | nindent 8 }} {{- end }} {{- with include "sourcegraph.priorityClassName" (list $r "executor") | trim }}{{ . | nindent 6 }}{{- end }} - {{- with $r.Values.sourcegraph.tolerations }} + {{- with $r.Values.executor.tolerations | default $r.Values.sourcegraph.tolerations }} tolerations: {{- toYaml . | nindent 8 }} {{- end }} @@ -374,9 +337,7 @@ Usage: include "executor.validateEnv" (list $envDict "label") "SRC_LOG_LEVEL" "SRC_LOG_FORMAT" "EXECUTOR_MAXIMUM_NUM_JOBS" - "EXECUTOR_MAXIMUM_RUNTIME_PER_JOB" - "EXECUTOR_DOCKER_ADD_HOST_GATEWAY" - "EXECUTOR_KEEP_WORKSPACES" -}} + "EXECUTOR_MAXIMUM_RUNTIME_PER_JOB" -}} {{- range $managed -}} {{- if hasKey $envDict . -}} {{- fail (printf "%s: env must not contain managed variable %s; use the structured executor fields instead" $label .) -}} diff --git a/charts/sourcegraph-executor/dind/templates/executor/executor.Deployment.yaml b/charts/sourcegraph-executor/dind/templates/executor/executor.Deployment.yaml index e251e60b2..b28246893 100644 --- a/charts/sourcegraph-executor/dind/templates/executor/executor.Deployment.yaml +++ b/charts/sourcegraph-executor/dind/templates/executor/executor.Deployment.yaml @@ -2,6 +2,11 @@ {{- range .Values.queues }} {{- include "executor.validateEnv" (list $.Values.executor.env "executor.env") }} {{- include "executor.validateEnv" (list (.env | default dict) (printf "queues[%s].env" .name)) }} +{{- /* A queue's maximumNumJobs of 0 pauses that queue, so only a missing, null, or empty one falls back. */}} +{{- $maximumNumJobs := .maximumNumJobs }} +{{- if or (kindIs "invalid" $maximumNumJobs) (eq (toString $maximumNumJobs) "") }} +{{- $maximumNumJobs = $.Values.executor.maximumNumJobs }} +{{- end }} --- {{ include "executor.deployment" (dict "root" $ @@ -10,6 +15,7 @@ "queueNames" (.queueNames | default (list)) "replicaCount" (.replicaCount | default $.Values.executor.replicaCount) "resources" (.resources | default $.Values.executor.resources) + "maximumNumJobs" $maximumNumJobs "env" (mergeOverwrite (deepCopy $.Values.executor.env) (.env | default dict)) ) }} {{- end }} @@ -23,6 +29,7 @@ "queueNames" .Values.executor.queueNames "replicaCount" .Values.executor.replicaCount "resources" .Values.executor.resources + "maximumNumJobs" .Values.executor.maximumNumJobs "env" .Values.executor.env ) }} {{- end }} diff --git a/charts/sourcegraph-executor/dind/templates/private-docker-registry/private-docker-registry.Statefulset.yaml b/charts/sourcegraph-executor/dind/templates/private-docker-registry/private-docker-registry.Statefulset.yaml index 71f3194e8..dcb578cfc 100644 --- a/charts/sourcegraph-executor/dind/templates/private-docker-registry/private-docker-registry.Statefulset.yaml +++ b/charts/sourcegraph-executor/dind/templates/private-docker-registry/private-docker-registry.Statefulset.yaml @@ -40,6 +40,10 @@ spec: deploy: sourcegraph app: private-docker-registry spec: + {{- with .Values.privateDockerRegistry.podSecurityContext }} + securityContext: + {{- toYaml . | nindent 8 }} + {{- end }} containers: - name: private-docker-registry image: {{ .Values.privateDockerRegistry.image.registry}}/{{ .Values.privateDockerRegistry.image.repository}}:{{ .Values.privateDockerRegistry.image.tag}} @@ -69,16 +73,16 @@ spec: volumeMounts: - mountPath: /var/lib/registry name: cache - {{- with .Values.sourcegraph.nodeSelector }} + {{- with .Values.privateDockerRegistry.nodeSelector | default .Values.sourcegraph.nodeSelector }} nodeSelector: {{- toYaml . | nindent 8 }} {{- end }} - {{- with .Values.sourcegraph.affinity }} + {{- with .Values.privateDockerRegistry.affinity | default .Values.sourcegraph.affinity }} affinity: {{- toYaml . | nindent 8 }} {{- end }} {{- with include "sourcegraph.priorityClassName" (list . "privateDockerRegistry") | trim }}{{ . | nindent 6 }}{{- end }} - {{- with .Values.sourcegraph.tolerations }} + {{- with .Values.privateDockerRegistry.tolerations | default .Values.sourcegraph.tolerations }} tolerations: {{- toYaml . | nindent 8 }} {{- end }} diff --git a/charts/sourcegraph-executor/dind/tests/executor_test.yaml b/charts/sourcegraph-executor/dind/tests/executor_test.yaml index ebc890027..f2093360e 100644 --- a/charts/sourcegraph-executor/dind/tests/executor_test.yaml +++ b/charts/sourcegraph-executor/dind/tests/executor_test.yaml @@ -45,3 +45,232 @@ tests: apiVersion: apps/v1 name: executor-batches documentIndex: 1 + + - it: should pass maximumNumJobs to every queue's executor + template: executor/executor.Deployment.yaml + set: + executor: + maximumNumJobs: 4 + queues: + - name: codeintel + - name: batches + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: EXECUTOR_MAXIMUM_NUM_JOBS + value: "4" + + - it: should use a queue's own maximumNumJobs, and executor.maximumNumJobs when the queue sets none + template: executor/executor.Deployment.yaml + set: + executor: + maximumNumJobs: 4 + queues: + - name: codeintel + maximumNumJobs: 2 + - name: batches + - name: paused + maximumNumJobs: 0 + - name: unset + maximumNumJobs: null + - name: empty + maximumNumJobs: "" + asserts: + - hasDocuments: + count: 5 + - contains: + path: spec.template.spec.containers[0].env + content: + name: EXECUTOR_MAXIMUM_NUM_JOBS + value: "2" + documentIndex: 0 + - contains: + path: spec.template.spec.containers[0].env + content: + name: EXECUTOR_MAXIMUM_NUM_JOBS + value: "4" + documentIndex: 1 + - contains: + path: spec.template.spec.containers[0].env + content: + name: EXECUTOR_MAXIMUM_NUM_JOBS + value: "0" + documentIndex: 2 + - contains: + path: spec.template.spec.containers[0].env + content: + name: EXECUTOR_MAXIMUM_NUM_JOBS + value: "4" + documentIndex: 3 + - contains: + path: spec.template.spec.containers[0].env + content: + name: EXECUTOR_MAXIMUM_NUM_JOBS + value: "4" + documentIndex: 4 + + - it: should refuse a queue without maximumNumJobs when executor.maximumNumJobs is null + template: executor/executor.Deployment.yaml + set: + executor: + maximumNumJobs: null + queues: + - name: codeintel + maximumNumJobs: 2 + - name: batches + asserts: + - failedTemplate: + errorMessage: executor.maximumNumJobs is required, for example 10 + + - it: should default to 10 jobs at once on a single executor + template: executor/executor.Deployment.yaml + set: + executor: + queueName: batches + asserts: + - hasDocuments: + count: 1 + - contains: + path: spec.template.spec.containers[0].env + content: + name: EXECUTOR_MAXIMUM_NUM_JOBS + value: "10" + + - it: should refuse a null executor.maximumNumJobs, which the executor cannot start with + template: executor/executor.Deployment.yaml + set: + executor: + queueName: batches + maximumNumJobs: null + asserts: + - failedTemplate: + errorMessage: executor.maximumNumJobs is required, for example 10 + + - it: should pass a maximumNumJobs of 0, which pauses the executor + template: executor/executor.Deployment.yaml + set: + executor: + queueName: batches + maximumNumJobs: 0 + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: EXECUTOR_MAXIMUM_NUM_JOBS + value: "0" + + - it: should use executor.nodeSelector, affinity, and tolerations instead of the sourcegraph ones + template: executor/executor.Deployment.yaml + set: + sourcegraph: + nodeSelector: + global-pool: "true" + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: kubernetes.io/hostname + tolerations: + - key: global + operator: Exists + executor: + queueName: batches + nodeSelector: + pool: executors + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - {key: pool, operator: In, values: [executors]} + tolerations: + - key: executors + operator: Exists + effect: NoSchedule + asserts: + - equal: + path: spec.template.spec.nodeSelector + value: + pool: executors + - equal: + path: spec.template.spec.affinity + value: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - {key: pool, operator: In, values: [executors]} + - equal: + path: spec.template.spec.tolerations + value: + - key: executors + operator: Exists + effect: NoSchedule + + - it: should fall back to sourcegraph.nodeSelector, affinity, and tolerations when executor sets none + template: executor/executor.Deployment.yaml + set: + sourcegraph: + nodeSelector: + global-pool: "true" + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: kubernetes.io/hostname + tolerations: + - key: global + operator: Exists + executor: + queueName: batches + asserts: + - equal: + path: spec.template.spec.nodeSelector + value: + global-pool: "true" + - equal: + path: spec.template.spec.affinity + value: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: kubernetes.io/hostname + - equal: + path: spec.template.spec.tolerations + value: + - key: global + operator: Exists + + - it: should accept EXECUTOR_DOCKER_ADD_HOST_GATEWAY and EXECUTOR_KEEP_WORKSPACES in executor.env and a queue's env + template: executor/executor.Deployment.yaml + set: + executor: + env: + EXECUTOR_DOCKER_ADD_HOST_GATEWAY: + value: "true" + queues: + - name: batches + env: + EXECUTOR_KEEP_WORKSPACES: + value: "true" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: EXECUTOR_DOCKER_ADD_HOST_GATEWAY + value: "true" + - contains: + path: spec.template.spec.containers[0].env + content: + name: EXECUTOR_KEEP_WORKSPACES + value: "true" + + - it: should refuse EXECUTOR_MAXIMUM_NUM_JOBS in executor.env, which executor.maximumNumJobs sets + template: executor/executor.Deployment.yaml + set: + executor: + queueName: batches + env: + EXECUTOR_MAXIMUM_NUM_JOBS: + value: "2" + asserts: + - failedTemplate: + errorMessage: "executor.env: env must not contain managed variable EXECUTOR_MAXIMUM_NUM_JOBS; use the structured executor fields instead" diff --git a/charts/sourcegraph-executor/dind/tests/privateDockerRegistry_test.yaml b/charts/sourcegraph-executor/dind/tests/privateDockerRegistry_test.yaml index cbff736c7..e5b327439 100644 --- a/charts/sourcegraph-executor/dind/tests/privateDockerRegistry_test.yaml +++ b/charts/sourcegraph-executor/dind/tests/privateDockerRegistry_test.yaml @@ -26,3 +26,84 @@ tests: asserts: - hasDocuments: count: 0 + + - it: should use the registry's own nodeSelector, affinity, tolerations, and podSecurityContext + template: private-docker-registry/private-docker-registry.Statefulset.yaml + set: + sourcegraph: + nodeSelector: + global-pool: "true" + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: kubernetes.io/hostname + tolerations: + - key: global + operator: Exists + privateDockerRegistry: + nodeSelector: + pool: registry + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - {key: pool, operator: In, values: [registry]} + tolerations: + - key: registry + operator: Exists + effect: NoSchedule + podSecurityContext: + fsGroup: 1000 + asserts: + - equal: + path: spec.template.spec.nodeSelector + value: + pool: registry + - equal: + path: spec.template.spec.affinity + value: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - {key: pool, operator: In, values: [registry]} + - equal: + path: spec.template.spec.tolerations + value: + - key: registry + operator: Exists + effect: NoSchedule + - equal: + path: spec.template.spec.securityContext.fsGroup + value: 1000 + + - it: should fall back to sourcegraph.nodeSelector, affinity, and tolerations when the registry sets none + template: private-docker-registry/private-docker-registry.Statefulset.yaml + set: + sourcegraph: + nodeSelector: + global-pool: "true" + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: kubernetes.io/hostname + tolerations: + - key: global + operator: Exists + asserts: + - equal: + path: spec.template.spec.nodeSelector + value: + global-pool: "true" + - equal: + path: spec.template.spec.affinity + value: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: kubernetes.io/hostname + - equal: + path: spec.template.spec.tolerations + value: + - key: global + operator: Exists diff --git a/charts/sourcegraph-executor/dind/values.yaml b/charts/sourcegraph-executor/dind/values.yaml index 571ebe624..23615a321 100644 --- a/charts/sourcegraph-executor/dind/values.yaml +++ b/charts/sourcegraph-executor/dind/values.yaml @@ -1,5 +1,11 @@ # These values are derived from https://github.com/sourcegraph/deploy-sourcegraph-helm/blob/main/charts/sourcegraph/values.yaml +# -- Replaces the chart name in the `app.kubernetes.io/name` label, which the Deployment and StatefulSet selectors use. +# Set it only on a new install: Kubernetes does not allow changing a selector. +# As a subchart with an `alias`, the alias replaces the chart name; set `.nameOverride: sourcegraph-executor-dind` +# in the parent chart to keep the selectors of a standalone install. +nameOverride: "" + ### Common Configuration ### sourcegraph: image: @@ -16,15 +22,13 @@ sourcegraph: imagePullSecrets: [] # -- Add a global label to all resources labels: {} - # -- Set a custom name for the app.kubernetes.io/name annotation - nameOverride: "" - # -- Affinity, + # -- Default affinity for every pod. `executor.affinity`, `privateDockerRegistry.affinity` replace it when set, # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) affinity: {} - # -- NodeSelector, + # -- Default nodeSelector for every pod. `executor.nodeSelector`, `privateDockerRegistry.nodeSelector` replace it when set, # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) nodeSelector: {} - # -- Tolerations, + # -- Default tolerations for every pod. `executor.tolerations`, `privateDockerRegistry.tolerations` replace it when set, # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) tolerations: [] # -- Add extra annotations to attach to all pods @@ -36,24 +40,9 @@ sourcegraph: storageClass: - # -- Enable creation of storageClass. Defaults to Google Cloud Platform. Disable if you have your own existing storage class - create: false - # -- Name of the storageClass. - # Use to customize to the existing storage class name + # -- Name of an existing StorageClass for the `private-docker-registry` volume. + # This chart does not create a StorageClass. name: sourcegraph - # -- Name of the storageClass provisioner, - # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/storage-classes/#provisioner) - # and consult your cloud provider persistent storage documentation - provisioner: kubernetes.io/gce-pd - # -- Value of `type` key in storageClass `parameters`, - # consult your cloud provider persistent storage documentation - type: pd-ssd - # -- Extra parameters of storageClass, - # consult your cloud provider persistent storage documentation - parameters: {} - # -- Persistent volumes topology configuration, - # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/storage-classes/#allowed-topologies) - allowedTopologies: {} # -- Optional list of queues to deploy as standalone Deployments. # When set, the single executor Deployment is not rendered. @@ -61,11 +50,13 @@ storageClass: # name (required) — used as the deployment name suffix (executor-) # queueName — sets EXECUTOR_QUEUE_NAME; defaults to name if omitted # queueNames — sets EXECUTOR_QUEUE_NAMES (comma-joined); takes precedence over queueName when set -# replicaCount, resources, env (merged with executor.env, queue overrides) +# replicaCount, resources, maximumNumJobs — default to the executor.* value of the same name +# env — merged with executor.env, queue overrides queues: [] # - name: codeintel # queueName: codeintel # replicaCount: 2 +# maximumNumJobs: 4 # resources: # requests: # cpu: "2" @@ -100,7 +91,9 @@ executor: queueName: "" # -- The names of multiple queues to pull jobs from. Possible values: batches and codeintel. Either this or queueName is required (when not using queues). queueNames: [] - # -- The maximum amount of jobs that can be executed concurrently. + # -- The maximum number of jobs each executor pod runs at once. Each job runs in the pod's dind container, + # so size `resources` and `storage` for this many jobs. + # Each queue can override this with its own maximumNumJobs field. maximumNumJobs: 10 # -- The maximum wall time that can be spent on a single job. maximumRuntimePerJob: "30m" @@ -136,6 +129,15 @@ executor: # when running in Docker mode (dind). Set to {} to disable. podSecurityContext: fsGroup: 101 + # -- Node selector for executor pods. Replaces `sourcegraph.nodeSelector` when set, + # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) + nodeSelector: {} + # -- Affinity for executor pods. Replaces `sourcegraph.affinity` when set, + # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) + affinity: {} + # -- Tolerations for executor pods. Replaces `sourcegraph.tolerations` when set, + # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) + tolerations: [] dind: image: @@ -237,3 +239,15 @@ privateDockerRegistry: env: REGISTRY_PROXY_REMOTEURL: value: https://registry-1.docker.io + # -- Pod-level security context for the registry pod, + # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) + podSecurityContext: {} + # -- Node selector for the registry pod. Replaces `sourcegraph.nodeSelector` when set, + # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) + nodeSelector: {} + # -- Affinity for the registry pod. Replaces `sourcegraph.affinity` when set, + # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) + affinity: {} + # -- Tolerations for the registry pod. Replaces `sourcegraph.tolerations` when set, + # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) + tolerations: [] diff --git a/charts/sourcegraph-executor/k8s/README.md b/charts/sourcegraph-executor/k8s/README.md index 67bae6cc3..f3ff0f3a7 100644 --- a/charts/sourcegraph-executor/k8s/README.md +++ b/charts/sourcegraph-executor/k8s/README.md @@ -40,23 +40,9 @@ Manifests rendered using the `helm template` command can be used for direct depl Reference the table below for available configuration parameters and consult [executor] documentation. -In addition to the documented values, the `executor` and `private-docker-registry` services also supports the following values - -- `executor.affinity` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) -- `executor.nodeSelector` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) -- `executor.tolerations` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) -- `executor.podSecurityContext` - [learn more](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) -- `executor.extraEnv` - consult `values.yaml` - -- `privateDockerRegistry.affinity` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) -- `privateDockerRegistry.nodeSelector` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) -- `privateDockerRegistry.tolerations` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) -- `privateDockerRegistry.podSecurityContext` - [learn more](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) -- `privateDockerRegistry.env` - consult `values.yaml` file - | Key | Type | Default | Description | |-----|------|---------|-------------| -| executor.affinity | object | `{}` | Affinity, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) | +| executor.affinity | object | `{}` | Affinity for the executor pod, not for job pods, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) | | executor.configureRbac | bool | `true` | Whether to configure the necessary RBAC resources. Required only once for all executor deployments. | | executor.containerSecurityContext | object | `{"privileged":false}` | Security context for the container, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container) | | executor.debug.keepJobs | string | `"false"` | If true, Kubernetes jobs will not be deleted after they complete. Not recommended for production use as it can hit cluster limits. | @@ -90,7 +76,7 @@ In addition to the documented values, the `executor` and `private-docker-registr | executor.maximumNumJobs | int | `10` | The maximum amount of jobs that can be executed concurrently. | | executor.maximumRuntimePerJob | string | `"30m"` | | | executor.namespace | string | `"default"` | The namespace in which jobs are generated by the executor. | -| executor.nodeSelector | object | `{}` | NodeSelector, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) | +| executor.nodeSelector | object | `{}` | NodeSelector for the executor pod, not for job pods, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) | | executor.podSecurityContext | object | `{}` | Security context for the pod, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) | | executor.queueName | string | `""` | The name of the queue to pull jobs from to. Possible values: batches and codeintel. **Either this or queueNames is required.** | | executor.queueNames | list | `[]` | The names of multiple queues to pull jobs from to. Possible values: batches and codeintel. **Either this or queueName is required.** | @@ -98,13 +84,13 @@ In addition to the documented values, the `executor` and `private-docker-registr | executor.resources.limits.memory | string | `"4Gi"` | | | executor.resources.requests.cpu | string | `"500m"` | | | executor.resources.requests.memory | string | `"200Mi"` | | -| executor.securityContext | object | `{"fsGroup":null,"privileged":false,"runAsGroup":null,"runAsUser":null}` | DEPRECATED: Use `executor.containerSecurityContext` or `executor.podSecurityContext` instead. | +| executor.securityContext | object | `{"fsGroup":null,"runAsGroup":null,"runAsUser":null}` | DEPRECATED: Use `executor.containerSecurityContext` or `executor.podSecurityContext` instead. | | executor.securityContext.fsGroup | string | `nil` | DEPRECATED: Use `executor.containerSecurityContext` or `executor.podSecurityContext` instead. | -| executor.securityContext.privileged | bool | `false` | DEPRECATED: Use `executor.containerSecurityContext` or `executor.podSecurityContext` instead. | | executor.securityContext.runAsGroup | string | `nil` | DEPRECATED: Use `executor.containerSecurityContext` or `executor.podSecurityContext` instead. | | executor.securityContext.runAsUser | string | `nil` | DEPRECATED: Use `executor.containerSecurityContext` or `executor.podSecurityContext` instead. | -| executor.tolerations | list | `[]` | Tolerations, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) | -| sourcegraph.affinity | object | `{}` | Affinity, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) | +| executor.tolerations | list | `[]` | Tolerations for the executor pod, not for job pods, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) | +| nameOverride | string | `""` | Replaces the chart name in the `app.kubernetes.io/name` label, which every selector uses. Set it only on a new install: Kubernetes does not allow changing a selector. As a subchart with an `alias`, the alias replaces the chart name; set `.nameOverride: sourcegraph-executor-k8s` in the parent chart to keep the selectors of a standalone install. | +| sourcegraph.affinity | object | `{}` | Default affinity for the executor pod; `executor.affinity` replaces it when set. Job pods do not use it: set `executor.kubernetesJob.node.requiredAffinityMatchExpressions`, `executor.kubernetesJob.node.requiredAffinityMatchFields`, or `executor.kubernetesJob.pod.affinity` for them. Learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) | | sourcegraph.image.defaultTag | string | `"{{ .Chart.AppVersion }}"` | Global docker image tag | | sourcegraph.image.pullPolicy | string | `"IfNotPresent"` | Global docker image pull policy | | sourcegraph.image.repository | string | Sourcegraph's public Docker Hub registry | Global docker image registry or prefix | @@ -112,9 +98,8 @@ In addition to the documented values, the `executor` and `private-docker-registr | sourcegraph.imagePullSecrets | list | `[]` | Mount named secrets containing docker credentials | | sourcegraph.labels | object | `{}` | Add a global label to all resources | | sourcegraph.localDevMode | bool | `false` | When true, remove all resource stanzas, allowing the scheduler to best-fit pods. Intended for local development with limited resources. | -| sourcegraph.nameOverride | string | `""` | Set a custom name for the app.kubernetes.io/name annotation | -| sourcegraph.nodeSelector | object | `{}` | NodeSelector, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) | +| sourcegraph.nodeSelector | object | `{}` | Default nodeSelector for the executor pod; `executor.nodeSelector` replaces it when set. Job pods do not use it: set `executor.kubernetesJob.node.selector` for them. Learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) | | sourcegraph.podAnnotations | object | `{}` | Add extra annotations to attach to all pods | | sourcegraph.podLabels | object | `{}` | Add extra labels to attach to all pods | | sourcegraph.priorityClassName | string | `""` | Assign a priorityClass to all pods (daemonSets, deployments, and statefulSets) | -| sourcegraph.tolerations | list | `[]` | Tolerations, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) | +| sourcegraph.tolerations | list | `[]` | Default tolerations for the executor pod; `executor.tolerations` replaces it when set. Job pods do not use it: set `executor.kubernetesJob.node.tolerations` for them. Learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) | diff --git a/charts/sourcegraph-executor/k8s/README.md.gotmpl b/charts/sourcegraph-executor/k8s/README.md.gotmpl index 02c346832..62898364a 100644 --- a/charts/sourcegraph-executor/k8s/README.md.gotmpl +++ b/charts/sourcegraph-executor/k8s/README.md.gotmpl @@ -40,18 +40,4 @@ Manifests rendered using the `helm template` command can be used for direct depl Reference the table below for available configuration parameters and consult [executor] documentation. -In addition to the documented values, the `executor` and `private-docker-registry` services also supports the following values - -- `executor.affinity` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) -- `executor.nodeSelector` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) -- `executor.tolerations` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) -- `executor.podSecurityContext` - [learn more](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) -- `executor.extraEnv` - consult `values.yaml` - -- `privateDockerRegistry.affinity` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) -- `privateDockerRegistry.nodeSelector` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) -- `privateDockerRegistry.tolerations` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) -- `privateDockerRegistry.podSecurityContext` - [learn more](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) -- `privateDockerRegistry.env` - consult `values.yaml` file - {{ template "chart.valuesTable" . }} diff --git a/charts/sourcegraph-executor/k8s/templates/_helpers.tpl b/charts/sourcegraph-executor/k8s/templates/_helpers.tpl index 70f75b28c..f96063b75 100644 --- a/charts/sourcegraph-executor/k8s/templates/_helpers.tpl +++ b/charts/sourcegraph-executor/k8s/templates/_helpers.tpl @@ -59,45 +59,6 @@ useGlobalTagAsDefault configuration {{- $top.Values.sourcegraph.image.repository }}/{{ $imageName }}:{{ default $defaultTag (index $top.Values $service "image" "tag") }} {{- end }} -{{- define "sourcegraph.nodeSelector" -}} -{{- $top := index . 0 }} -{{- $service := index . 1 }} -{{- $globalNodeSelector := (index $top.Values "sourcegraph" "nodeSelector") }} -{{- $serviceNodeSelector := (index $top.Values $service "nodeSelector") }} -nodeSelector: -{{- if $serviceNodeSelector }} -{{- $serviceNodeSelector | toYaml | trim | nindent 2 }} -{{- else if $globalNodeSelector }} -{{- $globalNodeSelector | toYaml | trim | nindent 2 }} -{{- end }} -{{- end }} - -{{- define "sourcegraph.affinity" -}} -{{- $top := index . 0 }} -{{- $service := index . 1 }} -{{- $globalAffinity := (index $top.Values "sourcegraph" "affinity") }} -{{- $serviceAffinity := (index $top.Values $service "affinity") }} -affinity: -{{- if $serviceAffinity }} -{{- tpl ($serviceAffinity | toYaml) $top | trim | nindent 2 }} -{{- else if $globalAffinity }} -{{- tpl ($globalAffinity | toYaml) $top | trim | nindent 2 }} -{{- end }} -{{- end }} - -{{- define "sourcegraph.tolerations" -}} -{{- $top := index . 0 }} -{{- $service := index . 1 }} -{{- $globalTolerations := (index $top.Values "sourcegraph" "tolerations") }} -{{- $serviceTolerations := (index $top.Values $service "tolerations") }} -tolerations: -{{- if $serviceTolerations }} -{{- $serviceTolerations | toYaml | trim | nindent 2 }} -{{- else if $globalTolerations }} -{{- $globalTolerations | toYaml | trim | nindent 2 }} -{{- end }} -{{- end }} - {{- define "executor.name" -}} {{- if .Values.executor.queueName -}} executor-{{.Values.executor.queueName}} diff --git a/charts/sourcegraph-executor/k8s/templates/executor.Deployment.yaml b/charts/sourcegraph-executor/k8s/templates/executor.Deployment.yaml index 1cb35d2c4..7202aa4ca 100644 --- a/charts/sourcegraph-executor/k8s/templates/executor.Deployment.yaml +++ b/charts/sourcegraph-executor/k8s/templates/executor.Deployment.yaml @@ -101,16 +101,16 @@ spec: - /usr/local/bin/executor periodSeconds: 5 terminationMessagePolicy: FallbackToLogsOnError - {{- with .Values.executor.nodeSelector }} + {{- with .Values.executor.nodeSelector | default .Values.sourcegraph.nodeSelector }} nodeSelector: {{- toYaml . | nindent 8 }} {{- end }} - {{- with .Values.executor.affinity }} + {{- with .Values.executor.affinity | default .Values.sourcegraph.affinity }} affinity: {{- toYaml . | nindent 8 }} {{- end }} {{- with include "sourcegraph.priorityClassName" (list . "executor") | trim }}{{ . | nindent 6 }}{{- end }} - {{- with .Values.executor.tolerations }} + {{- with .Values.executor.tolerations | default .Values.sourcegraph.tolerations }} tolerations: {{- toYaml . | nindent 8 }} {{- end }} diff --git a/charts/sourcegraph-executor/k8s/tests/executor_test.yaml b/charts/sourcegraph-executor/k8s/tests/executor_test.yaml index 256638deb..963078920 100644 --- a/charts/sourcegraph-executor/k8s/tests/executor_test.yaml +++ b/charts/sourcegraph-executor/k8s/tests/executor_test.yaml @@ -125,3 +125,82 @@ tests: path: spec.template.spec.securityContext.runAsUser - isNull: path: spec.template.spec.securityContext.runAsGroup + + - it: should fall back to sourcegraph.nodeSelector, affinity, and tolerations when executor sets none + template: executor.Deployment.yaml + set: + sourcegraph: + nodeSelector: + global-pool: "true" + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: kubernetes.io/hostname + tolerations: + - key: global + operator: Exists + executor: + queueName: "test" + asserts: + - equal: + path: spec.template.spec.nodeSelector + value: + global-pool: "true" + - equal: + path: spec.template.spec.affinity + value: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: kubernetes.io/hostname + - equal: + path: spec.template.spec.tolerations + value: + - key: global + operator: Exists + + - it: should use executor.nodeSelector, affinity, and tolerations instead of the sourcegraph ones + template: executor.Deployment.yaml + set: + sourcegraph: + nodeSelector: + global-pool: "true" + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: kubernetes.io/hostname + tolerations: + - key: global + operator: Exists + executor: + queueName: "test" + nodeSelector: + pool: executors + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - {key: pool, operator: In, values: [executors]} + tolerations: + - key: executors + operator: Exists + effect: NoSchedule + asserts: + - equal: + path: spec.template.spec.nodeSelector + value: + pool: executors + - equal: + path: spec.template.spec.affinity + value: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - {key: pool, operator: In, values: [executors]} + - equal: + path: spec.template.spec.tolerations + value: + - key: executors + operator: Exists + effect: NoSchedule diff --git a/charts/sourcegraph-executor/k8s/values.yaml b/charts/sourcegraph-executor/k8s/values.yaml index b55767725..0defa0bc6 100644 --- a/charts/sourcegraph-executor/k8s/values.yaml +++ b/charts/sourcegraph-executor/k8s/values.yaml @@ -1,5 +1,11 @@ # These values are derived from https://github.com/sourcegraph/deploy-sourcegraph-helm/blob/main/charts/sourcegraph/values.yaml +# -- Replaces the chart name in the `app.kubernetes.io/name` label, which every selector uses. +# Set it only on a new install: Kubernetes does not allow changing a selector. +# As a subchart with an `alias`, the alias replaces the chart name; set `.nameOverride: sourcegraph-executor-k8s` +# in the parent chart to keep the selectors of a standalone install. +nameOverride: "" + ### Common Configuration ### sourcegraph: image: @@ -18,16 +24,18 @@ sourcegraph: labels: { } # -- When true, remove all resource stanzas, allowing the scheduler to best-fit pods. Intended for local development with limited resources. localDevMode: false - # -- Set a custom name for the app.kubernetes.io/name annotation - nameOverride: "" - # -- Affinity, - # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) + # -- Default affinity for the executor pod; `executor.affinity` replaces it when set. + # Job pods do not use it: set `executor.kubernetesJob.node.requiredAffinityMatchExpressions`, + # `executor.kubernetesJob.node.requiredAffinityMatchFields`, or `executor.kubernetesJob.pod.affinity` for them. + # Learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) affinity: { } - # -- NodeSelector, - # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) + # -- Default nodeSelector for the executor pod; `executor.nodeSelector` replaces it when set. + # Job pods do not use it: set `executor.kubernetesJob.node.selector` for them. + # Learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) nodeSelector: { } - # -- Tolerations, - # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) + # -- Default tolerations for the executor pod; `executor.tolerations` replaces it when set. + # Job pods do not use it: set `executor.kubernetesJob.node.tolerations` for them. + # Learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) tolerations: [ ] # -- Add extra annotations to attach to all pods podAnnotations: { } @@ -89,8 +97,6 @@ executor: runAsGroup: # -- DEPRECATED: Use `executor.containerSecurityContext` or `executor.podSecurityContext` instead. fsGroup: - # -- DEPRECATED: Use `executor.containerSecurityContext` or `executor.podSecurityContext` instead. - privileged: false kubernetesJob: # -- The number of seconds after which a Kubernetes job will be terminated. @@ -138,15 +144,15 @@ executor: keepJobs: "false" keepWorkspaces: "false" - # -- Affinity, + # -- Affinity for the executor pod, not for job pods, # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) affinity: { } - # -- NodeSelector, + # -- NodeSelector for the executor pod, not for job pods, # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) nodeSelector: { } - # -- Tolerations, + # -- Tolerations for the executor pod, not for job pods, # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) tolerations: [ ] diff --git a/charts/sourcegraph-migrator/README.md b/charts/sourcegraph-migrator/README.md index eba7f0ecd..3baf0c085 100644 --- a/charts/sourcegraph-migrator/README.md +++ b/charts/sourcegraph-migrator/README.md @@ -63,41 +63,44 @@ Manifests rendered using the `helm template` command can be used for direct depl Reference the table below for available configuration parameters and consult [migrator] documentation. -In addition to the documented values, the `migrator` service also supports the following values - -- `migrator.affinity` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) -- `migrator.nodeSelector` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) -- `migrator.tolerations` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) -- `migrator.podSecurityContext` - [learn more](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) -- `migrator.env` - consult `values.yaml` file - | Key | Type | Default | Description | |-----|------|---------|-------------| | codeInsightsDB.auth.existingSecret | string | `""` | Name of existing secret to use for codeinsights-db credentials This should match the setting in the sourcegraph chart values | | codeInsightsDB.name | string | `"codeinsights-db"` | | | codeIntelDB.auth.existingSecret | string | `""` | Name of existing secret to use for codeintel-db credentials This should match the setting in the sourcegraph chart values | | codeIntelDB.name | string | `"codeintel-db"` | | +| migrator.affinity | object | `{}` | Affinity for the migrator pod. Replaces `sourcegraph.affinity` when set, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) | | migrator.args | list | `["up","-db=all"]` | Override default `migrator` container args Available commands can be found at https://docs.sourcegraph.com/admin/how-to/manual_database_migrations | | migrator.containerSecurityContext | object | `{"allowPrivilegeEscalation":false,"readOnlyRootFilesystem":true,"runAsGroup":101,"runAsUser":100}` | Security context for the `migrator` container, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container) | | migrator.env | object | `{}` | Environment variables for the `migrator` container | +| migrator.extraContainers | list | `[]` | Extra containers in the migrator pod | +| migrator.extraVolumeMounts | list | `[]` | Extra volume mounts for the `migrator` container | +| migrator.extraVolumes | list | `[]` | Extra volumes for the migrator pod | | migrator.image.defaultTag | string | `@sha256:`, pinned by each release | Docker image tag for the `migrator` image | | migrator.image.name | string | `"migrator"` | Docker image name for the `migrator` image | +| migrator.labels | object | `{}` | Labels for the migrator Job | +| migrator.name | string | `""` | Name prefix of the migrator Job. A random suffix is added to every release's Job. | +| migrator.nodeSelector | object | `{}` | Node selector for the migrator pod. Replaces `sourcegraph.nodeSelector` when set, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) | +| migrator.podAnnotations | object | `{}` | Annotations for the migrator pod | +| migrator.podLabels | object | `{}` | Labels for the migrator pod | +| migrator.podSecurityContext | object | `{}` | Pod-level security context for the migrator pod, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) | | migrator.resources | object | `{"limits":{"cpu":"500m","memory":"100M"},"requests":{"cpu":"100m","memory":"50M"}}` | Resource requests & limits for the `migrator` container, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) | +| migrator.tolerations | list | `[]` | Tolerations for the migrator pod. Replaces `sourcegraph.tolerations` when set, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) | +| nameOverride | string | `""` | Replaces the chart name in the `app.kubernetes.io/name` label of the migrator Job and its pod. The chart has no selector, so it can be changed on an installed release. As a subchart with an `alias`, the alias replaces the chart name; set `.nameOverride: sourcegraph-migrator` in the parent chart to keep the label of a standalone install. | | pgsql.auth.existingSecret | string | `""` | Name of existing secret to use for pgsql credentials This should match the setting in the sourcegraph chart values | | pgsql.name | string | `"pgsql"` | | -| sourcegraph.affinity | object | `{}` | Affinity, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) | +| sourcegraph.affinity | object | `{}` | Default affinity for every pod. `migrator.affinity` replaces it when set, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) | | sourcegraph.image.defaultTag | string | `"{{ .Chart.AppVersion }}"` | Global docker image tag | | sourcegraph.image.pullPolicy | string | `"IfNotPresent"` | Global docker image pull policy | | sourcegraph.image.repository | string | Sourcegraph's public Docker Hub registry | Global docker image registry or prefix | | sourcegraph.image.useGlobalTagAsDefault | bool | `false` | When set to true, sourcegraph.image.defaultTag is used as the default defaultTag for all services, instead of service-specific default defaultTags | | sourcegraph.imagePullSecrets | list | `[]` | Mount named secrets containing docker credentials | | sourcegraph.labels | object | `{}` | Add a global label to all resources | -| sourcegraph.nameOverride | string | `""` | Set a custom name for the app.kubernetes.io/name annotation | -| sourcegraph.nodeSelector | object | `{}` | NodeSelector, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) | +| sourcegraph.nodeSelector | object | `{}` | Default nodeSelector for every pod. `migrator.nodeSelector` replaces it when set, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) | | sourcegraph.podAnnotations | object | `{}` | Add extra annotations to attach to all pods | | sourcegraph.podLabels | object | `{}` | Add extra labels to attach to all pods | | sourcegraph.priorityClassName | string | `""` | Assign a priorityClass to all pods (daemonSets, deployments, and statefulSets) | -| sourcegraph.tolerations | list | `[]` | Tolerations, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) | +| sourcegraph.tolerations | list | `[]` | Default tolerations for every pod. `migrator.tolerations` replaces it when set, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) | ## Troubleshooting diff --git a/charts/sourcegraph-migrator/README.md.gotmpl b/charts/sourcegraph-migrator/README.md.gotmpl index ee46175c0..0889a7ae5 100644 --- a/charts/sourcegraph-migrator/README.md.gotmpl +++ b/charts/sourcegraph-migrator/README.md.gotmpl @@ -63,14 +63,6 @@ Manifests rendered using the `helm template` command can be used for direct depl Reference the table below for available configuration parameters and consult [migrator] documentation. -In addition to the documented values, the `migrator` service also supports the following values - -- `migrator.affinity` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) -- `migrator.nodeSelector` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) -- `migrator.tolerations` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) -- `migrator.podSecurityContext` - [learn more](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) -- `migrator.env` - consult `values.yaml` file - {{ template "chart.valuesTable" . }} ## Troubleshooting diff --git a/charts/sourcegraph-migrator/templates/migrator/sourcegraph-migrator.Job.yaml b/charts/sourcegraph-migrator/templates/migrator/sourcegraph-migrator.Job.yaml index c77ec3b33..d09a394bd 100644 --- a/charts/sourcegraph-migrator/templates/migrator/sourcegraph-migrator.Job.yaml +++ b/charts/sourcegraph-migrator/templates/migrator/sourcegraph-migrator.Job.yaml @@ -58,18 +58,20 @@ spec: {{- if .Values.migrator.extraContainers }} {{- toYaml .Values.migrator.extraContainers | nindent 6 }} {{- end }} + {{- with .Values.migrator.podSecurityContext }} securityContext: - {{- toYaml .Values.migrator.podSecurityContext | nindent 8 }} - {{- with .Values.sourcegraph.nodeSelector }} + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.migrator.nodeSelector | default .Values.sourcegraph.nodeSelector }} nodeSelector: {{- toYaml . | nindent 8 }} {{- end }} - {{- with .Values.sourcegraph.affinity }} + {{- with .Values.migrator.affinity | default .Values.sourcegraph.affinity }} affinity: {{- toYaml . | nindent 8 }} {{- end }} {{- with include "sourcegraph.priorityClassName" (list . "migrator") | trim }}{{ . | nindent 6 }}{{- end }} - {{- with .Values.sourcegraph.tolerations }} + {{- with .Values.migrator.tolerations | default .Values.sourcegraph.tolerations }} tolerations: {{- toYaml . | nindent 8 }} {{- end }} diff --git a/charts/sourcegraph-migrator/tests/migrator_test.yaml b/charts/sourcegraph-migrator/tests/migrator_test.yaml index abadb2e92..b0edc3502 100644 --- a/charts/sourcegraph-migrator/tests/migrator_test.yaml +++ b/charts/sourcegraph-migrator/tests/migrator_test.yaml @@ -139,3 +139,95 @@ tests: asserts: - isNull: path: spec.template.spec.priorityClassName + + - it: should use migrator.nodeSelector, affinity, and tolerations instead of the sourcegraph ones + set: + sourcegraph: + nodeSelector: + global-pool: "true" + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: kubernetes.io/hostname + tolerations: + - key: global + operator: Exists + migrator: + nodeSelector: + pool: migrator + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - {key: pool, operator: In, values: [migrator]} + tolerations: + - key: migrator + operator: Exists + effect: NoSchedule + asserts: + - equal: + path: spec.template.spec.nodeSelector + value: + pool: migrator + - equal: + path: spec.template.spec.affinity + value: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - {key: pool, operator: In, values: [migrator]} + - equal: + path: spec.template.spec.tolerations + value: + - key: migrator + operator: Exists + effect: NoSchedule + + - it: should fall back to sourcegraph.nodeSelector, affinity, and tolerations when migrator sets none + set: + sourcegraph: + nodeSelector: + global-pool: "true" + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: kubernetes.io/hostname + tolerations: + - key: global + operator: Exists + asserts: + - equal: + path: spec.template.spec.nodeSelector + value: + global-pool: "true" + - equal: + path: spec.template.spec.affinity + value: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: kubernetes.io/hostname + - equal: + path: spec.template.spec.tolerations + value: + - key: global + operator: Exists + + - it: should not render a pod securityContext by default + asserts: + - isNull: + path: spec.template.spec.securityContext + + - it: should render migrator.podSecurityContext when set + set: + migrator: + podSecurityContext: + fsGroup: 101 + runAsNonRoot: true + asserts: + - equal: + path: spec.template.spec.securityContext + value: + fsGroup: 101 + runAsNonRoot: true diff --git a/charts/sourcegraph-migrator/values.yaml b/charts/sourcegraph-migrator/values.yaml index 0274af9b3..b9179bcf1 100644 --- a/charts/sourcegraph-migrator/values.yaml +++ b/charts/sourcegraph-migrator/values.yaml @@ -1,5 +1,11 @@ # These values are derived from https://github.com/sourcegraph/deploy-sourcegraph-helm/blob/main/charts/sourcegraph/values.yaml +# -- Replaces the chart name in the `app.kubernetes.io/name` label of the migrator Job and its pod. +# The chart has no selector, so it can be changed on an installed release. +# As a subchart with an `alias`, the alias replaces the chart name; set `.nameOverride: sourcegraph-migrator` +# in the parent chart to keep the label of a standalone install. +nameOverride: "" + ### Common Configuration ### sourcegraph: image: @@ -16,15 +22,13 @@ sourcegraph: imagePullSecrets: [] # -- Add a global label to all resources labels: {} - # -- Set a custom name for the app.kubernetes.io/name annotation - nameOverride: "" - # -- Affinity, + # -- Default affinity for every pod. `migrator.affinity` replaces it when set, # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) affinity: {} - # -- NodeSelector, + # -- Default nodeSelector for every pod. `migrator.nodeSelector` replaces it when set, # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) nodeSelector: {} - # -- Tolerations, + # -- Default tolerations for every pod. `migrator.tolerations` replaces it when set, # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) tolerations: [] # -- Add extra annotations to attach to all pods @@ -35,52 +39,6 @@ sourcegraph: priorityClassName: "" -# Generic application configuration options, used by most applications below -# app: # Generally matches directory name -# replicaCount: 1 -# -# image: -# # Application-specific default tag. Overrides the global tag. Do not modify - use tag instead -# defaultTag: -# # Override the default tag on a service-specific basis -# tag: -# # Override the default docker image name -# name: -# -# # Add additional labels and annotations to various resources -# labels: {} -# podLabels: {} -# podAnnotations: {} -# serviceLabels: {} -# serviceAnnotations: {} -# # Override default service/deployment names -# name: "" -# # Provide custom environment variables -# env: {} -# # Set resource requests / limits -# resources: {} -# podSecurityContext: {} -# securityContext: {} -# # Add additional volumes and mounts to deployments -# extraVolumeMounts: {} -# extraVolumes: {} -# # Add sidecars -# extraContainers: {} -# # Toggle serviceAccount creation -# serviceAccount.create: true -# # Customize service account name -# serviceAccount.name: "" -# # Override service type (defaults to ClusterIP) -# serviceType: "" -# # Set disk size on PVC's for stateful deployments -# storageSize: "" -# # Optionally bind PVC's to an existing PV -# volumeName: "" -# # Use an existing configmap (specified by name) instead of creating one -# existingConfig: "" -# # Toggle deployment of applications on/off. Applies to databases and third-party applications -# enabled: true - codeInsightsDB: name: "codeinsights-db" auth: @@ -127,6 +85,32 @@ migrator: runAsUser: 100 runAsGroup: 101 readOnlyRootFilesystem: true + # -- Name prefix of the migrator Job. A random suffix is added to every release's Job. + name: "" + # -- Labels for the migrator Job + labels: {} + # -- Annotations for the migrator pod + podAnnotations: {} + # -- Labels for the migrator pod + podLabels: {} + # -- Pod-level security context for the migrator pod, + # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) + podSecurityContext: {} + # -- Node selector for the migrator pod. Replaces `sourcegraph.nodeSelector` when set, + # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) + nodeSelector: {} + # -- Affinity for the migrator pod. Replaces `sourcegraph.affinity` when set, + # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) + affinity: {} + # -- Tolerations for the migrator pod. Replaces `sourcegraph.tolerations` when set, + # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) + tolerations: [] + # -- Extra volumes for the migrator pod + extraVolumes: [] + # -- Extra volume mounts for the `migrator` container + extraVolumeMounts: [] + # -- Extra containers in the migrator pod + extraContainers: [] # -- Override default `migrator` container args # Available commands can be found at # https://docs.sourcegraph.com/admin/how-to/manual_database_migrations diff --git a/charts/sourcegraph/CHANGELOG.md b/charts/sourcegraph/CHANGELOG.md index a8b444320..e076432a4 100644 --- a/charts/sourcegraph/CHANGELOG.md +++ b/charts/sourcegraph/CHANGELOG.md @@ -8,6 +8,10 @@ Use `**BREAKING**:` to denote a breaking change ## Unreleased +- Fixed `storageClass.allowVolumeExpansion: false` being ignored: the StorageClass always allowed volume expansion. Documented `storageClass.allowVolumeExpansion`, `reclaimPolicy`, and `volumeBindingMode`, `frontend.serviceType` and `serviceAnnotations`, `openTelemetry.gateway.env`, and `redisExporter.env`, which the chart already read. Documented that `openTelemetry.gateway.config.traces.processors` applies only when `exporters` is set and `jaeger.enabled` is false, and fixed the value path in the error shown when both `exporters` and `jaeger.enabled` are set. +- Fixed example override files that set values the chart does not read. `basic` used `gitServer`, `replicas`, and `indexedSearch.webserver` and `indexserver`; it now uses `gitserver`, `replicaCount`, `indexedSearch.resources`, and `indexedSearchIndexer.resources`. `custom-security-context` set container fields under `podSecurityContext`; they are now under `containerSecurityContext`. `openshift` no longer sets `serviceAccount` on `indexedSearchIndexer` and `migrator`, or `podSecurityContext` on `indexedSearchIndexer`. `opentelemetry-exporter` used the `jaeger` exporter, which the OpenTelemetry Collector removed, and `$JAEGER_HOST`; it now uses the `otlp` exporter and `${env:JAEGER_HOST}`, and `override-processor.yaml` says it needs your own exporters. +- Fixed `migrator.nodeSelector`, `migrator.affinity`, and `migrator.tolerations` being ignored by the `sourcegraph-migrator` chart, which listed them in its README. They now replace `sourcegraph.nodeSelector`, `affinity`, and `tolerations` for the migrator Job. The chart's `values.yaml` now documents the migrator values the Job reads (`name`, `labels`, `podAnnotations`, `podLabels`, `podSecurityContext`, `extraVolumes`, `extraVolumeMounts`, `extraContainers`), and no longer lists ones it never read (`replicaCount`, `serviceLabels`, `serviceAnnotations`, `serviceAccount`, `serviceType`, `storageSize`, `volumeName`, `existingConfig`, `enabled`). The Job no longer renders `securityContext: null` when `migrator.podSecurityContext` is empty. +- Fixed the documentation of `nameOverride` in the `sourcegraph` and `sourcegraph-migrator` charts. It was documented as `sourcegraph.nameOverride`, which the charts never read, so setting it had no effect. The charts read the top-level `nameOverride`, which replaces the chart name in the `app.kubernetes.io/name` selector label. Set it only on a new install of the `sourcegraph` chart, because Kubernetes does not allow changing a selector; the migrator chart has no selector. If you set `sourcegraph.nameOverride`, remove it; moving its value to `nameOverride` changes your selectors. As a subchart with an `alias`, the alias replaces the chart name; the `subchart` example now sets `sg.nameOverride: sourcegraph` to keep the selectors of a standalone install. This is for new installs only: do not add it to a release already installed with an alias, whose selectors contain the alias. - The `correct-data-dir-permissions` init containers of `pgsql`, `codeintel-db`, and `codeinsights-db` now run the database's own image instead of `sourcegraph/alpine-3.14`, which was based on an end-of-life Alpine release. The chart no longer pulls `alpine-3.14`, so private registries no longer need to mirror it. `alpine.image` has been removed and is ignored if set; `alpine.containerSecurityContext`, `alpine.resources`, and `codeInsightsDB.init.containerSecurityContext` still apply. - Added `sourcegraph.selectorLabelsOverride`, `.extraSelectorLabels`, and `storageLabels` on `gitserver`, `grafana`, `indexedSearch`, and `searcher`, so the chart can render the selectors and volume claim templates of workloads created outside it, such as with Kustomize, and a Helm release can take them over without deleting them. All default to empty, so existing releases render the same selectors. Never change them on an installed release: Kubernetes does not allow changing these fields. See [Adopting existing workloads](README.md#adopting-existing-workloads). - Fixed `frontend.command` and `migrator.command` rendering a list as a single element, e.g. `["sh", "-c", "