Repository navigation
Description
Problem
Alloy keeps its log read positions inside the container, not on the alloy-data volume. A container recreate then loses them, and Alloy may resend old container logs into Loki in a burst.
The compose command: replaces the image's default command. The image default is run /etc/alloy/config.alloy --storage.path=/var/lib/alloy/data. Ours is run --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/config.alloy, with no --storage.path. So Alloy falls back to data-alloy under its working directory, /.
Read-only check on production, 2026-11-13:
| Read | Value |
|---|---|
docker inspect docsplus-alloy args |
run --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/config.alloy |
/data-alloy (container layer) |
exists, 92K |
/var/lib/alloy/data (the alloy-data volume) |
mounted, empty |
A docker compose restart keeps the container layer, so today's deploys keep the positions. A recreate does not: an image bump, or any change to Alloy's compose block. Found in the #712 review.
Benchmark
| Field | Value |
|---|---|
| Metric | Where Alloy's positions live, and whether they survive a recreate |
| Before | /data-alloy in the container layer. Lost on a recreate |
| After | /var/lib/alloy/data on the alloy-data volume. Kept on a recreate. Local, n = 1 (Measured): lines Loki received across the recreate, 271,824 before and 176 after. Batches rejected with 400, 16 before and 0 after |
| How to run | Locally: run Alloy from the compose block with one test container that logs a numbered line each second. Recreate Alloy with docker compose up -d --force-recreate alloy. Count the lines Loki receives across the recreate, and the Alloy batches rejected with 400 |
| Pass rule | Far fewer lines received across the recreate, and 0 batches rejected with 400. A per-line count in Loki cannot see a resend, because Loki drops an exact copy without an error |
What to do
- Agent in the repo. In
docker-compose.observability.yml, add--storage.path=/var/lib/alloy/datato the Alloycommand:. - Agent in the repo. Run the local test above on the change, and once on today's command, so the before is measured too.
- Agent in the repo. Decide the first deploy. The new path starts empty, so the first start may read old logs again. Test locally what Alloy's
loki.source.dockerreads with no positions. If it resends, copy/data-alloyinto the volume before the recreate, and write that step into the deploy note. - Maintainer. Deploy with
(build): observability. Done:6efccb7cd, deployed 2026-11-16 13:43Z. - Maintainer, read-only. The one step left. On the host, confirm that
positions.ymlexists under/var/lib/alloy/data(for exampledocker exec docsplus-alloy ls /var/lib/alloy/data/loki.source.docker.logs/). Then close.
Out of scope: the steady Alloy 400s, about 18 an hour. They predate this fix, and this fix was never meant to stop them. File them in a later milestone only if Loki load or a missing line shows they matter.
Checks
docker inspect docsplus-alloyshows--storage.path=/var/lib/alloy/data.- After the deploy,
/var/lib/alloy/dataholds the positions file. - No burst of old lines in Loki: compare
sum(count_over_time({service=~".+"}[5m]))around the deploy with the hour before.
Rollback
Remove the flag and deploy (build): observability. Alloy then goes back to /data-alloy.
Activity
HMarzban commentedon Nov 13, 2026
Measured, 2026-11-13: Alloy already resends old lines, and Loki rejects them
Read-only, from
docker logson production, 2026-11-13 at about 17:57Z.Alloy logs
final error sending batch, no retries left, dropping datawithstatus=400. Loki's reason istimestamp too old. The rejected entries are from 2026-10-21, for quiet containers such asredis-exporterandglitchtip-redis. Loki accepts entries no older than 7 days (reject_old_samples_max_age: 168h).So Alloy re-reads old lines today, with no recreate. One likely cause: a quiet container writes no new line, so its position is never saved. Each reconnect then reads from the start. The proxy cuts each log stream after 10 minutes (
timeout server 10m), so reconnects are frequent. This is a guess, not traced. Step 3 of this issue should test it, and count these 400s before and after the fix.No current line is lost: Loki rejects only the old copies. The cost is wasted work, and the rejections hide in Alloy's error log.
fix(observability): keep Alloy log positions on its volume
HMarzban commentedon Nov 16, 2026
Measured, 2026-11-16
What changed
Committed in
6efccb7cd.docker-compose.observability.yml, Alloy block only:The block also has a 2-line comment that says why the flag is there. With the flag,
positions.ymlis written to/var/lib/alloy/data/loki.source.docker.logs/, on thealloy-datavolume./data-alloystays empty.How it was measured
This was a local Docker test on OrbStack (Docker 29.8.2, Compose v5.5.0,
grafana/alloy:v1.17.0,grafana/loki:3.7.3). It ran Alloy, Loki and the socket proxy from the compose file, with a throwaway project name. An override removed the fixed names and made the network local. It also added atickercontainer that logstick Neach second, and aquietcontainer that logs 5 lines and then sleeps. About 18 other local containers were also running, and their logs went back to 2026-10-23. The steps were: start, wait 60 s, restart the proxy, wait 30 s, thendocker compose up -d --force-recreate alloy, then wait 60 s. n = 1 run per side. Every test container, network and volume was removed afterwards.Across a forced recreate
--storage.path)timestamp too old, 9too far behind)The last row passes on both sides. This is because Loki drops an exact copy (same timestamp, same line) without an error. So the pass rule in the issue cannot see a resend. The real signal is the count of lines received, and the resent lines older than 7 days, which Loki rejects with 400.
Step 3: no positions, and the first deploy
With no positions,
loki.source.dockeropens each container withsince=0and reads its whole log.The new volume path starts empty, so a plain deploy would resend everything once. I tested the copy: start on today's command, then
docker exec <alloy> cp -a /data-alloy/. /var/lib/alloy/data/, then recreate on the new command.The two rows were measured over different windows: the before run had a proxy restart, the copy run had a copy step. Read the pair as a direction, not a precise percent.
| With the copy | 405 | 174 | 0 |
The 2026-11-13 guess: the proxy's 10-minute cut
With positions in place, the agent waited 13 minutes. The proxy cut 10 quiet log streams at 600 s (haproxy
sD--). Across 12 minutes after the cut: 0 new Alloy 400s, 0 new Loki discards, and 3,616 entries read (405 to 4,021). Thesince=value of the reconnects was not captured, because haproxy logs a request only when it closes. A probe line written to the quiet container after the cut reached Loki once. So the guess did not reproduce here. The steady 400s on production are still not explained.Deploy note
The deploy's
up -drecreates Alloy, because its command changed. So run this on the server just before you trigger(build): observability:docker exec docsplus-alloy cp -a /data-alloy/. /var/lib/alloy/data/Lines read between the copy and the recreate are read again. They are minutes old, and Loki drops exact copies, so the risk is low.
Gates
bunx prettier --check docker-compose.observability.ymldocker compose -f docker-compose.observability.yml config --quietLeft
(build): observability.docker inspectargs, the positions file in/var/lib/alloy/data, and the line count around the deploy.status=400timestamp too oldper hour for a day after the deploy, and compare with the 2026-11-13 table. If the 400s go on, open a follow-up. The cause is not the 10-minute cut, at least not locally.After the deploy
Deployed with
(build): observability(288d4d260, run 43878530215, 2026-11-16 13:43Z). Read-only, production Loki and Prometheus, 2026-11-16:status=400lines, 10 min after the deploy (to 13:49Z)So the
/data-alloycopy step probably did not run before this deploy (the host was not read), and Alloy re-read old logs once, as the local test predicted. Loki rejected the old lines with 400, so no current line was lost, and the burst ended within about 3 minutes. From now on the positions live on thealloy-datavolume, so the next recreate keeps them.Left: confirm on the host that
/var/lib/alloy/data/loki.source.docker.logs/positions.ymlexists (maintainer, SSH). Count the hourly 400s for a day and compare with the 2026-11-13 table.HMarzban commentedon Nov 16, 2026
Measured, 2026-11-16: Alloy 400 lines per hour, before and after the deploy (partial)
Read-only, production Loki, one hour-sized instant query per hour:
sum(count_over_time({container="docsplus-alloy"} |= "status=400" [1h])), read at 16:06Z.status=400linesLeft: the full day read on 2026-11-17 (24 hourly values against the 18-per-hour before), then name the cause of the steady rejections or open an issue for it. Owner: agent.
docs(changelog): record the v2.3.7 work of 2026-11-16
HMarzban commentedon Nov 16, 2026
Review, 2026-11-16 (maintainer approved).
positions.ymlexists under/var/lib/alloy/dataon the host. Then close.HMarzban commentedon Nov 18, 2026
Measured, 2026-11-18: step 5 and the full-day read
Read-only reads on the production host, 2026-11-18 about 02:15Z.
Step 5: positions on the volume
/var/lib/alloy/data/loki.source.docker.logs/positions.yml/data-alloy(container layer)docker inspect docsplus-alloyargsrun --server.http.listen-addr=0.0.0.0:12345 --storage.path=/var/lib/alloy/data /etc/alloy/config.alloy(build): observabilitydeploy, run 44027220672, a restart, not a recreate)So the positions live on the
alloy-datavolume, as step 1 intended. The next recreate is the first one on production with positions in place.Full day, 2026-11-17 (UTC): Alloy batches dropped with 400
Count of
status=400lines indocker logs docsplus-alloy, one line per dropped batch, per hour:Day total: 313. Hour 05 holds 26 batches in the minute after the 05:53:57Z restart. Of the 38 batches that hour, 14 name
timestamp too oldand 24 nametoo far behind. Without that minute, the day averages 12.0 an hour (287 in 24 hours). The 2026-11-13 estimate was about 18 an hour. That is 12.0 against 18, 33% fewer, but the 18 was an estimate with no n or range, so this is a read, not a claim.What the steady 400s are
In hour 05, before the restart, the drops come in pairs at minute 1 of every 10 minutes (05:01, 05:11, 05:21, 05:31, 05:41, then 05:51 and 05:52). Each batch holds old lines from quiet containers, for example
docsplus-node-exporterat 2026-10-09,docsplus-redis-exporterat 2026-10-21 anddocsplus-glitchtip-redisat 2026-07-02. Loki refuses them as older than 7 days.That fits the 2026-11-13 guess: the socket proxy cuts quiet log streams at 600 s, and Alloy re-reads those containers from an old point. The local test on 2026-11-16 did not reproduce it. No line is lost: these lines are already in Loki or past retention. As ruled in the review, the steady 400s stay out of scope for v2.3.7. If they need an owner later, this is the cause to start from.
Close
Steps 1 to 5 are done. Closing.