Skip to content

Keep Alloy's log positions on its volume, so a recreate does not resend old logs #783

Closed

Description

@HMarzban

Problem

Alloy keeps its log read positions inside the container, not on the alloy-data volume. A container recreate then loses them, and Alloy may resend old container logs into Loki in a burst.

The compose command: replaces the image's default command. The image default is run /etc/alloy/config.alloy --storage.path=/var/lib/alloy/data. Ours is run --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/config.alloy, with no --storage.path. So Alloy falls back to data-alloy under its working directory, /.

Read-only check on production, 2026-11-13:

Read Value
docker inspect docsplus-alloy args run --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/config.alloy
/data-alloy (container layer) exists, 92K
/var/lib/alloy/data (the alloy-data volume) mounted, empty

A docker compose restart keeps the container layer, so today's deploys keep the positions. A recreate does not: an image bump, or any change to Alloy's compose block. Found in the #712 review.

Benchmark

Field Value
Metric Where Alloy's positions live, and whether they survive a recreate
Before /data-alloy in the container layer. Lost on a recreate
After /var/lib/alloy/data on the alloy-data volume. Kept on a recreate. Local, n = 1 (Measured): lines Loki received across the recreate, 271,824 before and 176 after. Batches rejected with 400, 16 before and 0 after
How to run Locally: run Alloy from the compose block with one test container that logs a numbered line each second. Recreate Alloy with docker compose up -d --force-recreate alloy. Count the lines Loki receives across the recreate, and the Alloy batches rejected with 400
Pass rule Far fewer lines received across the recreate, and 0 batches rejected with 400. A per-line count in Loki cannot see a resend, because Loki drops an exact copy without an error

What to do

  1. Agent in the repo. In docker-compose.observability.yml, add --storage.path=/var/lib/alloy/data to the Alloy command:.
  2. Agent in the repo. Run the local test above on the change, and once on today's command, so the before is measured too.
  3. Agent in the repo. Decide the first deploy. The new path starts empty, so the first start may read old logs again. Test locally what Alloy's loki.source.docker reads with no positions. If it resends, copy /data-alloy into the volume before the recreate, and write that step into the deploy note.
  4. Maintainer. Deploy with (build): observability. Done: 6efccb7cd, deployed 2026-11-16 13:43Z.
  5. Maintainer, read-only. The one step left. On the host, confirm that positions.yml exists under /var/lib/alloy/data (for example docker exec docsplus-alloy ls /var/lib/alloy/data/loki.source.docker.logs/). Then close.

Out of scope: the steady Alloy 400s, about 18 an hour. They predate this fix, and this fix was never meant to stop them. File them in a later milestone only if Loki load or a missing line shows they matter.

Checks

  • docker inspect docsplus-alloy shows --storage.path=/var/lib/alloy/data.
  • After the deploy, /var/lib/alloy/data holds the positions file.
  • No burst of old lines in Loki: compare sum(count_over_time({service=~".+"}[5m])) around the deploy with the hour before.

Rollback

Remove the flag and deploy (build): observability. Alloy then goes back to /data-alloy.

Activity

  1. added this to the v2.3.7 milestone on Nov 13, 2026
  2. HMarzban commented on Nov 13, 2026

    @HMarzban
    CollaboratorAuthor

    Measured, 2026-11-13: Alloy already resends old lines, and Loki rejects them

    Read-only, from docker logs on production, 2026-11-13 at about 17:57Z.

    Alloy logs final error sending batch, no retries left, dropping data with status=400. Loki's reason is timestamp too old. The rejected entries are from 2026-10-21, for quiet containers such as redis-exporter and glitchtip-redis. Loki accepts entries no older than 7 days (reject_old_samples_max_age: 168h).

    Hour (Z) Rejected batches
    11 2
    12 8
    13 12
    14 14
    15 23
    16 10
    17 14

    So Alloy re-reads old lines today, with no recreate. One likely cause: a quiet container writes no new line, so its position is never saved. Each reconnect then reads from the start. The proxy cuts each log stream after 10 minutes (timeout server 10m), so reconnects are frequent. This is a guess, not traced. Step 3 of this issue should test it, and count these 400s before and after the fix.

    No current line is lost: Loki rejects only the old copies. The cost is wasted work, and the rejections hide in Alloy's error log.

  3. HMarzban commented on Nov 16, 2026

    @HMarzban
    CollaboratorAuthor

    Measured, 2026-11-16

    What changed

    Committed in 6efccb7cd. docker-compose.observability.yml, Alloy block only:

    command: run --server.http.listen-addr=0.0.0.0:12345 --storage.path=/var/lib/alloy/data /etc/alloy/config.alloy

    The block also has a 2-line comment that says why the flag is there. With the flag, positions.yml is written to /var/lib/alloy/data/loki.source.docker.logs/, on the alloy-data volume. /data-alloy stays empty.

    How it was measured

    This was a local Docker test on OrbStack (Docker 29.8.2, Compose v5.5.0, grafana/alloy:v1.17.0, grafana/loki:3.7.3). It ran Alloy, Loki and the socket proxy from the compose file, with a throwaway project name. An override removed the fixed names and made the network local. It also added a ticker container that logs tick N each second, and a quiet container that logs 5 lines and then sleeps. About 18 other local containers were also running, and their logs went back to 2026-10-23. The steps were: start, wait 60 s, restart the proxy, wait 30 s, then docker compose up -d --force-recreate alloy, then wait 60 s. n = 1 run per side. Every test container, network and volume was removed afterwards.

    Across a forced recreate

    Metric Before (today's command) After (--storage.path) Change
    Lines Loki received across the recreate 271,824 176 99% fewer
    Entries the new Alloy read in its first 60 s 247,434 440 99% fewer
    Alloy batches dropped with 400 16 (7 timestamp too old, 9 too far behind) 0 100% fewer
    Loki discards: too old / too far behind / rate limited 67,177 / 105,421 / 25,021 0 / 0 / 0 100% fewer
    Ticker lines in Loki (172 produced): extra copies / missing 0 / 0 0 / 0 none

    The last row passes on both sides. This is because Loki drops an exact copy (same timestamp, same line) without an error. So the pass rule in the issue cannot see a resend. The real signal is the count of lines received, and the resent lines older than 7 days, which Loki rejects with 400.

    Step 3: no positions, and the first deploy

    With no positions, loki.source.docker opens each container with since=0 and reads its whole log.

    First start, no positions 400 batches, too old Loki too-old discards
    Today's command 12 67,138
    New command, empty volume 11 67,223

    The new volume path starts empty, so a plain deploy would resend everything once. I tested the copy: start on today's command, then docker exec <alloy> cp -a /data-alloy/. /var/lib/alloy/data/, then recreate on the new command.

    Recreate onto the new path Entries read Loki lines 400s
    No copy: recreate on today's command (before run) 247,434 271,824 16

    The two rows were measured over different windows: the before run had a proxy restart, the copy run had a copy step. Read the pair as a direction, not a precise percent.
    | With the copy | 405 | 174 | 0 |

    The 2026-11-13 guess: the proxy's 10-minute cut

    With positions in place, the agent waited 13 minutes. The proxy cut 10 quiet log streams at 600 s (haproxy sD--). Across 12 minutes after the cut: 0 new Alloy 400s, 0 new Loki discards, and 3,616 entries read (405 to 4,021). The since= value of the reconnects was not captured, because haproxy logs a request only when it closes. A probe line written to the quiet container after the cut reached Loki once. So the guess did not reproduce here. The steady 400s on production are still not explained.

    Deploy note

    The deploy's up -d recreates Alloy, because its command changed. So run this on the server just before you trigger (build): observability:

    docker exec docsplus-alloy cp -a /data-alloy/. /var/lib/alloy/data/

    Lines read between the copy and the recreate are read again. They are minutes old, and Loki drops exact copies, so the risk is low.

    Gates

    Command Result
    bunx prettier --check docker-compose.observability.yml pass
    docker compose -f docker-compose.observability.yml config --quiet pass

    Left

    • Maintainer: run the copy above on the server, then deploy (build): observability.
    • Maintainer: run the Checks in the issue: the docker inspect args, the positions file in /var/lib/alloy/data, and the line count around the deploy.
    • Maintainer: count Alloy status=400 timestamp too old per hour for a day after the deploy, and compare with the 2026-11-13 table. If the 400s go on, open a follow-up. The cause is not the 10-minute cut, at least not locally.

    After the deploy

    Deployed with (build): observability (288d4d260, run 43878530215, 2026-11-16 13:43Z). Read-only, production Loki and Prometheus, 2026-11-16:

    Read Value
    Alloy status=400 lines, 10 min after the deploy (to 13:49Z) 169
    Same, the 10 min one hour earlier 3
    Same, 13:46–13:49Z 0
    Loki working set, peak in the hour 189 MiB (limit 768 MiB)

    So the /data-alloy copy step probably did not run before this deploy (the host was not read), and Alloy re-read old logs once, as the local test predicted. Loki rejected the old lines with 400, so no current line was lost, and the burst ended within about 3 minutes. From now on the positions live on the alloy-data volume, so the next recreate keeps them.

    Left: confirm on the host that /var/lib/alloy/data/loki.source.docker.logs/positions.yml exists (maintainer, SSH). Count the hourly 400s for a day and compare with the 2026-11-13 table.

  4. HMarzban commented on Nov 16, 2026

    @HMarzban
    CollaboratorAuthor

    Measured, 2026-11-16: Alloy 400 lines per hour, before and after the deploy (partial)

    Read-only, production Loki, one hour-sized instant query per hour: sum(count_over_time({container="docsplus-alloy"} |= "status=400" [1h])), read at 16:06Z.

    Hour (UTC) Alloy status=400 lines
    08:00 18
    09:00 18
    10:00 18
    11:00 18
    12:00 18
    13:00 (deploy at 13:43Z) 182
    14:00 8
    15:00 12
    • Before the deploy, Alloy logged a steady 18 rejection lines an hour, the same steady rejections the 2026-11-13 comment found.
    • The deploy hour holds the one-time burst (the copy step was skipped, see the comment above).
    • After the deploy: 8 and 12. The steady rejections continue, so the flag does not stop them. They may be lower, but 2 hours is a hint, not a result.

    Left: the full day read on 2026-11-17 (24 hourly values against the 18-per-hour before), then name the cause of the steady rejections or open an issue for it. Owner: agent.

  5. HMarzban commented on Nov 16, 2026

    @HMarzban
    CollaboratorAuthor

    Review, 2026-11-16 (maintainer approved).

    • The pass rule now uses lines Loki receives across the recreate, and batches rejected with 400. Loki drops an exact copy without an error, so the old per-line rule cannot see a resend.
    • The After row now holds the local result: 271,824 lines before and 176 after; 16 rejected batches before and 0 after.
    • One step is left, now step 5: the maintainer confirms that positions.yml exists under /var/lib/alloy/data on the host. Then close.
    • The steady Alloy 400s, about 18 an hour, are out of scope. They predate the fix and lose no line.
  6. HMarzban commented on Nov 18, 2026

    @HMarzban
    CollaboratorAuthor

    Measured, 2026-11-18: step 5 and the full-day read

    Read-only reads on the production host, 2026-11-18 about 02:15Z.

    Step 5: positions on the volume

    Read Value
    /var/lib/alloy/data/loki.source.docker.logs/positions.yml exists, 81,088 B, written 2026-11-18 02:10
    /data-alloy (container layer) does not exist
    docker inspect docsplus-alloy args run --server.http.listen-addr=0.0.0.0:12345 --storage.path=/var/lib/alloy/data /etc/alloy/config.alloy
    Container created / last started 2026-11-16 13:44Z (the #783 deploy) / 2026-11-17 05:53:57Z (the (build): observability deploy, run 44027220672, a restart, not a recreate)

    So the positions live on the alloy-data volume, as step 1 intended. The next recreate is the first one on production with positions in place.

    Full day, 2026-11-17 (UTC): Alloy batches dropped with 400

    Count of status=400 lines in docker logs docsplus-alloy, one line per dropped batch, per hour:

    Hour 00 01 02 03 04 05 06 07 08 09 10 11
    Batches 7 6 9 12 12 38 6 7 12 12 8 8
    Hour 12 13 14 15 16 17 18 19 20 21 22 23
    Batches 13 14 14 18 17 15 15 12 14 13 15 16

    Day total: 313. Hour 05 holds 26 batches in the minute after the 05:53:57Z restart. Of the 38 batches that hour, 14 name timestamp too old and 24 name too far behind. Without that minute, the day averages 12.0 an hour (287 in 24 hours). The 2026-11-13 estimate was about 18 an hour. That is 12.0 against 18, 33% fewer, but the 18 was an estimate with no n or range, so this is a read, not a claim.

    What the steady 400s are

    In hour 05, before the restart, the drops come in pairs at minute 1 of every 10 minutes (05:01, 05:11, 05:21, 05:31, 05:41, then 05:51 and 05:52). Each batch holds old lines from quiet containers, for example docsplus-node-exporter at 2026-10-09, docsplus-redis-exporter at 2026-10-21 and docsplus-glitchtip-redis at 2026-07-02. Loki refuses them as older than 7 days.

    That fits the 2026-11-13 guess: the socket proxy cuts quiet log streams at 600 s, and Alloy re-reads those containers from an old point. The local test on 2026-11-16 did not reproduce it. No line is lost: these lines are already in Loki or past retention. As ruled in the review, the steady 400s stay out of scope for v2.3.7. If they need an owner later, this is the cause to start from.

    Close

    Steps 1 to 5 are done. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions