redis-store commands rejected because it is out of memory
+ +**Descriptions** + +- critical redis: 0reqps+ redis-store commands rejected because it is out of memory for 5m0s + +**Next steps** + +- redis-store reached its `maxmemory` and rejects writes, because it does not evict keys. Sessions, locks, and background jobs that write to it fail until memory is freed. +- Increase the redis-store memory limit, which raises `maxmemory`, to recover. +- Find what fills redis-store with `redis-cli --bigkeys` or `MEMORY USAGE <key>` in the redis-store container. +- More help interpreting this metric is available in the [dashboards reference](/self-hosted/observability/dashboards#redis-redis-store_oom_rejected_commands). +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "critical_redis_redis-store_oom_rejected_commands" +] +``` + +*Managed by the Sourcegraph Platform team.* + +redis-store memory used, as a percentage of maxmemory
+ +**Descriptions** + +- warning redis: 90%+ redis-store memory used, as a percentage of maxmemory for 15m0s + +**Next steps** + +- redis-store rejects writes once it reaches `maxmemory`. Increase the redis-store memory limit, which raises `maxmemory`, before it does. +- Find what fills redis-store with `redis-cli --bigkeys` or `MEMORY USAGE <key>` in the redis-store container. +- More help interpreting this metric is available in the [dashboards reference](/self-hosted/observability/dashboards#redis-redis-store_memory_used_percentage). +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "warning_redis_redis-store_memory_used_percentage" +] +``` + +*Managed by the Sourcegraph Platform team.* + +redis-cache availability
@@ -3600,17 +3665,18 @@ Generated query for warning alert: `min((max(rate(src_repoupdater_sched_auto_fet **Descriptions** -- warning worker: less than 0 scheduler loops for 9h0m0s +- critical worker: less than 0 scheduler loops for 1h0m0s **Next steps** -- Check worker logs for errors. This is expected to fire if there are no user added code hosts -- Learn more about the related dashboard panel in the [dashboards reference](/self-hosted/observability/dashboards#worker-sched_loops). +- Check worker logs for errors from the `repo-updater.update-scheduler` routine. +- The scheduler takes a lock in redis-store before each run, so check that redis-store is up and not out of memory. +- More help interpreting this metric is available in the [dashboards reference](/self-hosted/observability/dashboards#worker-sched_loops). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ - "warning_worker_sched_loops" + "critical_worker_sched_loops" ] ``` @@ -3619,7 +3685,7 @@ Generated query for warning alert: `min((max(rate(src_repoupdater_sched_auto_fetrepos that haven't been fetched in more than 8 hours
+repos without a fetch attempt in more than 12 hours
**Descriptions** -- warning worker: 1+ repos that haven't been fetched in more than 8 hours for 25m0s +- warning worker: 1+ repos without a fetch attempt in more than 12 hours for 1h0m0s **Next steps** -- Check worker logs for errors. - Check for rows in gitserver_repos where LastError is not an empty string. -- Learn more about the related dashboard panel in the [dashboards reference](/self-hosted/observability/dashboards#worker-src_repoupdater_stale_repos). +- Check whether the `sched_loops` alert is also firing. +- Check worker logs for errors from the repo update workers, and the size of the repo_update_jobs queue. +- More help interpreting this metric is available in the [dashboards reference](/self-hosted/observability/dashboards#worker-src_repoupdater_stale_repos). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json diff --git a/docs/self-hosted/observability/dashboards.mdx b/docs/self-hosted/observability/dashboards.mdx index c8bd51059..a6d5ecd37 100644 --- a/docs/self-hosted/observability/dashboards.mdx +++ b/docs/self-hosted/observability/dashboards.mdx @@ -13975,6 +13975,54 @@ redis_up{app="redis-store"}Redis-store commands rejected because it is out of memory
+ +redis-store uses the noeviction policy, so once it reaches maxmemory every command that would use more memory fails with an OOM error. + +Refer to the [alerts reference](/self-hosted/observability/alerts#redis-redis-store_oom_rejected_commands) for 1 alert related to this panel. + +To see this panel, visit `/-/debug/grafana/d/redis/redis?viewPanel=100010` on your Sourcegraph instance. + +*Managed by the Sourcegraph Platform team.* + +Redis-store memory used, as a percentage of maxmemory
+ +redis-store does not evict keys, so its memory only shrinks when keys expire or are deleted. Instances without a maxmemory setting show no data. + +Refer to the [alerts reference](/self-hosted/observability/alerts#redis-redis-store_memory_used_percentage) for 1 alert related to this panel. + +To see this panel, visit `/-/debug/grafana/d/redis/redis?viewPanel=100011` on your Sourcegraph instance. + +*Managed by the Sourcegraph Platform team.* + +Scheduler loops
+The update scheduler runs every 30 seconds, whether or not any code hosts are configured. +When it stops, no repository is scheduled for fetching, so every repository goes stale. + Refer to the [alerts reference](/self-hosted/observability/alerts#worker-sched_loops) for 1 alert related to this panel. To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101350` on your Sourcegraph instance. @@ -16231,7 +16282,11 @@ max(rate(src_repoupdater_sched_loops[1m])) #### worker: src_repoupdater_stale_repos -Repos that haven't been fetched in more than 8 hours
+Repos without a fetch attempt in more than 12 hours
+ +The scheduler attempts to fetch every repository at least every 8 hours, including repositories whose fetches fail. +Repositories without an attempt in 12 hours are not being scheduled or fetched at all. +This is always 0 when automatic git updates are disabled. Refer to the [alerts reference](/self-hosted/observability/alerts#worker-src_repoupdater_stale_repos) for 1 alert related to this panel.