Skip to content

executor-k8s: render job scheduling values as JSON - #1020

Closed
shanurwan wants to merge 1 commit into
sourcegraph:mainfrom
shanurwan:fix/executor-k8s-node-json
Closed

shanurwan wants to merge 1 commit into
sourcegraph:mainfrom
shanurwan:fix/executor-k8s-node-json

Conversation

@shanurwan

@shanurwan shanurwan commented Nov 15, 2026 •

Copy link
Copy Markdown

Fixes #349.

The executor reads EXECUTOR_KUBERNETES_NODE_TOLERATIONS, EXECUTOR_KUBERNETES_NODE_REQUIRED_AFFINITY_MATCH_EXPRESSIONS, EXECUTOR_KUBERNETES_NODE_REQUIRED_AFFINITY_MATCH_FIELDS, EXECUTOR_KUBERNETES_POD_AFFINITY, and EXECUTOR_KUBERNETES_POD_ANTI_AFFINITY with json.Unmarshal. The sourcegraph-executor-k8s ConfigMap rendered them as "{{ .Values... }}", which broke both ways of setting them:

Value set as Rendered Result
A JSON encoded string, the documented form "[{"key":"foo",...}]", unescaped quotes inside a double-quoted scalar helm template fails: yaml: line 26: did not find expected key
A YAML list, as in #349 "[map[effect:NoSchedule key:foo operator:Equal value:bar]]" The executor exits: invalid EXECUTOR_KUBERNETES_NODE_TOLERATIONS, failed to parse: invalid character 'm' looking for beginning of value

Changes

  • New executor.jsonEnv helper: a string is quoted unchanged, a list or map is encoded with toJson and quoted, and an empty or unset value renders "".
  • The five ConfigMap entries use it. Default values render the same ConfigMap as main.
  • The values.yaml examples escaped their quotes inside single-quoted YAML ('[{\"key\":...}]'), so copied as written they were not valid JSON. They are now, and the descriptions say a YAML list also works. README regenerated with ./scripts/helm-docs.sh.

Checklist

Test plan

  • 3 new tests in charts/sourcegraph-executor/k8s/tests/executor_test.yaml: default, JSON string, and YAML list, each across all five values. On main, 2 fail and 1 errors; all pass here.
  • helm unittest on all four charts, as CI runs it (helm-unittest v1.0.2): 284 tests pass. helm lint charts/sourcegraph-executor/k8s passes.
  • Decoded each rendered value with json.Unmarshal into the k8s.io/api/core/v1 types the executor declares ([]Toleration, []NodeSelectorRequirement, []PodAffinityTerm): both forms decode here; on main the YAML list fails with the error above and the JSON string does not render.
  • Ran sourcegraph/executor-kubernetes:8.1.4873 (the digest pinned in values.yaml) with the rendered ConfigMap as its environment and an unreachable frontend URL:
Render Executor
main, YAML list exits 1 at config validation with 5 invalid ... failed to parse errors, one per value
This PR, YAML list passes config validation and retries connecting to the frontend until stopped
This PR, JSON string same as above

Tested on a cluster

k3s v1.35.8, Helm v3.19.0, executor-kubernetes:8.1.4873, in a throwaway namespace with an unreachable frontend URL, so a working executor passes config validation and then retries connecting to the frontend.

Step Chart tolerations Result
1 main default Pod Running and Ready
2 this PR, helm upgrade default ConfigMap data identical (29 keys); same pod, not restarted
3 this PR YAML list ConfigMap holds [{"effect":"NoSchedule","key":"dedicated",...}]; pod Running, no parse errors
4 main YAML list ConfigMap holds [map[effect:NoSchedule ...]]; pod in CrashLoopBackOff with invalid EXECUTOR_KUBERNETES_NODE_TOLERATIONS, failed to parse: invalid character 'm' looking for beginning of value
5 main JSON string helm upgrade fails: YAML parse error ... did not find expected key
6 this PR JSON string upgrade succeeds; string passed through unchanged; pod Running, no parse errors

Steps 3, 4 and 6 used kubectl rollout restart, because the Deployment has no checksum annotation for its ConfigMap, so a values change alone does not restart the executor.

Not tested: job pods being scheduled with these tolerations. The executor creates job pods only after dequeuing a job from a Sourcegraph frontend, and none was deployed.

The executor parses EXECUTOR_KUBERNETES_NODE_TOLERATIONS, the node
affinity match expressions and fields, and the pod affinity and
anti-affinity with json.Unmarshal. The ConfigMap rendered them as
"{{ .Values... }}", which broke both ways of setting them:

- A JSON encoded string, the documented form, put unescaped double
  quotes inside a double-quoted YAML scalar, so the render failed
  with "did not find expected key".
- A YAML list was printed with Go formatting, e.g.
  "[map[effect:NoSchedule key:foo operator:Equal value:bar]]", so the
  executor failed with "invalid character 'm' looking for beginning
  of value".

Add an executor.jsonEnv helper that quotes a string unchanged and
encodes a list or map with toJson, and use it for all five values.
Empty values still render as "". The values.yaml examples escaped
their quotes inside single-quoted YAML, so copied as written they
were not valid JSON; they now are.

Fixes sourcegraph#349
@shanurwan

Copy link
Copy Markdown
Author

Closing in favour of #902, which fixed #349 first and which I missed before opening this. #902 is also the safer fix: it keeps the documented pre-escaped form ('[{\"key\":...}]') working, while this PR's quote escaped it a second time (invalid character '\' looking for beginning of object key string). Sorry for the noise.

@shanurwan shanurwan closed this Nov 15, 2026
@shanurwan
shanurwan deleted the fix/executor-k8s-node-json branch November 15, 2026 19:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Executor Helm Chart - kubernetesJob.node.tolerations doesn't render as expected

1 participant