Skip to content

EMF metrics: same metric name with different label values clobber each other #10

Closed

Description

@jjbaumert

Summary: crates/common/src/metrics/emf.rs::emit_once buckets metrics by dimension keys only (e.g. ["route"]), not dimension key+value pairs. So req{route="/a"}=5 and req{route="/b"}=3 land in the same bucket and get passed together into build_emf_doc, which does root.insert(label_key, value) and root.insert(m.name, value) into a single shared JSON map per doc. The last metric in the bucket wins — e.g. the emitted line ends up {"route":"/b","req":3} and the /a data point for that flush interval is silently dropped, never reaching CloudWatch.

Severity: S3 — the metrics feature "works" (data is emitted every interval) but is systematically wrong whenever a labeled metric has 2+ active label values in the same flush window, which is the common case for route- or doc-type-labeled counters/histograms under real traffic. No end user is directly affected, but this degrades an internal-user-facing (on-call/ops) flow and can mask real incidents (e.g. a spike in errors on one route gets overwritten by a healthy count on another route).

Environment:

  • Deploy: any (code-level bug in the EMF emitter, most visible under real production traffic with multiple concurrent label values)
  • Commit: 6411ddb

Repro steps:

  1. Record two counter increments with the same metric name but different label values in the same flush interval, e.g. counter::inc(MetricKey::new("req", &[("route", "/a")])) and counter::inc(MetricKey::new("req", &[("route", "/b")])).
  2. Call emit_once (or wait for spawn_emf_emitter's interval).
  3. Inspect the emitted EMF JSON line(s).

Expected: Per the EMF spec, one JSON document per distinct dimension-value combination — two documents, one with "route":"/a","req":5 and one with "route":"/b","req":3.

Observed: Both metrics fall into the same buckets entry (keyed only by ["route"]), get formatted into one build_emf_doc call, and root.insert overwrites both the dimension value and the metric value — only the last-inserted series survives in the emitted line.

Evidence:

  • Code: crates/common/src/metrics/emf.rs:66-70 — let dim_keys: Vec<&'static str> = key.labels.iter().map(|(k, _)| *k).collect(); buckets.entry(dim_keys.clone()).or_default().push(...) (buckets by key names, not key+value)
  • Code: crates/common/src/metrics/emf.rs:151-200 (build_emf_doc) — root.insert(k.clone(), json!(v)) for label values and root.insert(m.name.clone(), ...) for metric values, both into one shared root map per chunk of metrics that share only dimension keys.

Suspected area: crates/common/src/metrics/emf.rs::emit_once — the bucketing key needs to include the label values, not just the label keys, so each distinct dimension-value combination gets its own EMF document (or build_emf_doc needs to emit one JSON line per distinct label_values set within a dimension-key bucket rather than merging them into one map). Relevant to the pending observability initiative (design/observability.md).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingverification-findingFinding from a verification run

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions