Repository navigation
Description
Summary: crates/common/src/metrics/emf.rs::emit_once buckets metrics by dimension keys only (e.g. ["route"]), not dimension key+value pairs. So req{route="/a"}=5 and req{route="/b"}=3 land in the same bucket and get passed together into build_emf_doc, which does root.insert(label_key, value) and root.insert(m.name, value) into a single shared JSON map per doc. The last metric in the bucket wins — e.g. the emitted line ends up {"route":"/b","req":3} and the /a data point for that flush interval is silently dropped, never reaching CloudWatch.
Severity: S3 — the metrics feature "works" (data is emitted every interval) but is systematically wrong whenever a labeled metric has 2+ active label values in the same flush window, which is the common case for route- or doc-type-labeled counters/histograms under real traffic. No end user is directly affected, but this degrades an internal-user-facing (on-call/ops) flow and can mask real incidents (e.g. a spike in errors on one route gets overwritten by a healthy count on another route).
Environment:
- Deploy: any (code-level bug in the EMF emitter, most visible under real production traffic with multiple concurrent label values)
- Commit: 6411ddb
Repro steps:
- Record two counter increments with the same metric name but different label values in the same flush interval, e.g.
counter::inc(MetricKey::new("req", &[("route", "/a")]))andcounter::inc(MetricKey::new("req", &[("route", "/b")])). - Call
emit_once(or wait forspawn_emf_emitter's interval). - Inspect the emitted EMF JSON line(s).
Expected: Per the EMF spec, one JSON document per distinct dimension-value combination — two documents, one with "route":"/a","req":5 and one with "route":"/b","req":3.
Observed: Both metrics fall into the same buckets entry (keyed only by ["route"]), get formatted into one build_emf_doc call, and root.insert overwrites both the dimension value and the metric value — only the last-inserted series survives in the emitted line.
Evidence:
- Code:
crates/common/src/metrics/emf.rs:66-70—let dim_keys: Vec<&'static str> = key.labels.iter().map(|(k, _)| *k).collect(); buckets.entry(dim_keys.clone()).or_default().push(...)(buckets by key names, not key+value) - Code:
crates/common/src/metrics/emf.rs:151-200(build_emf_doc) —root.insert(k.clone(), json!(v))for label values androot.insert(m.name.clone(), ...)for metric values, both into one sharedrootmap per chunk of metrics that share only dimension keys.
Suspected area: crates/common/src/metrics/emf.rs::emit_once — the bucketing key needs to include the label values, not just the label keys, so each distinct dimension-value combination gets its own EMF document (or build_emf_doc needs to emit one JSON line per distinct label_values set within a dimension-key bucket rather than merging them into one map). Relevant to the pending observability initiative (design/observability.md).
Activity
fix(quip-import): chunk UNRESOLVED# rows and reserve per-thread doc ids