diff --git a/cspell-allow-list.txt b/cspell-allow-list.txt index 5cb75f48b..ba469f385 100644 --- a/cspell-allow-list.txt +++ b/cspell-allow-list.txt @@ -405,6 +405,7 @@ pekko persistentvolumeclaims pgbouncer pgcrypto +PGDATA PGDATABASE PGDATASOURCE pgdump @@ -519,6 +520,7 @@ starcoder Starlark startedat startswith +statefulset statefulsets stepscontainer stepsenv @@ -604,6 +606,7 @@ urlencode urlsafe userprincipalname USERTRUST +vacuumdb Valsorda vegeta vercel diff --git a/docs/self-hosted/deploy/docker-compose/operations.mdx b/docs/self-hosted/deploy/docker-compose/operations.mdx index ba71790b8..5d5f54470 100644 --- a/docs/self-hosted/deploy/docker-compose/operations.mdx +++ b/docs/self-hosted/deploy/docker-compose/operations.mdx @@ -13,20 +13,14 @@ Guides for managing cloud storage and backups are available in our cloud-specifi ## Access the database -The following command allows a user to shell into the Sourcegraph database container and run `psql` to interact with the container's postgres database: +To open a `psql` session on one of the databases, run the command for its container: ```bash -docker exec -it pgsql psql -U sg #access pgsql container and run psql -docker exec -it codeintel-db psql -U sg #access codeintel-db container and run psql -docker exec -it codeinsights-db psql -U postgres #access codeinsights-db container and run psql +docker exec -it pgsql psql -U sg +docker exec -it codeintel-db psql -U sg +docker exec -it codeinsights-db psql -U postgres ``` -## Database Migrations - -The `frontend` container in the `docker-compose.yaml` file will automatically run on startup and migrate the databases if any changes are required, however administrators may wish to migrate their databases before upgrading the rest of the system when working with large databases. Sourcegraph guarantees database backward compatibility to the most recent minor point release so the database can safely be upgraded before the application code. - -To execute the database migrations independently, follow the [docker-compose instructions on how to manually run database migrations](/self-hosted/updates/migrator/migrator-operations#docker-compose). Running the `up` (default) command on the `migrator` of the _version you are upgrading to_ will apply all migrations required by the next version of Sourcegraph. - ## Backup and restore The following instructions are specific to backing up and restoring the Sourcegraph databases in a Docker Compose deployment. These do not apply to other deployment types. @@ -44,137 +38,83 @@ The following instructions are specific to backing up and restoring the Sourcegr These instructions will back up the primary `sourcegraph` database, the [codeintel](/code-navigation/) database, and the [codeinsights](/code-insights/) database. -1\. `ssh` from your local machine into the machine hosting the `sourcegraph` deployment +1. `ssh` from your local machine into the machine hosting the `sourcegraph` deployment -2\. `cd` to the `deploy-sourcegraph-docker/docker-compose` directory on the host +2. `cd` to the `deploy-sourcegraph-docker/docker-compose` directory on the host -3\. Verify the deployment is running: +3. Verify the deployment is running: ```bash -docker-compose ps - Name Command State Ports -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -caddy caddy run --config /etc/ca ... Up 2019/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:80->80/tcp -cadvisor /usr/bin/cadvisor -logtost ... Up (health: starting) 8080/tcp -codeinsights-db docker-entrypoint.sh postgres Up 5432/tcp -codeintel-db /postgres.sh Up (healthy) 5432/tcp -gitserver-0 /sbin/tini -- /usr/local/b ... Up -grafana /entry.sh Up 3000/tcp, 0.0.0.0:3370->3370/tcp -jaeger /go/bin/all-in-one-linux - ... Up 0.0.0.0:14250->14250/tcp, 14268/tcp, 0.0.0.0:16686->16686/tcp, 5775/udp, 0.0.0.0:5778->5778/tcp, - 0.0.0.0:6831->6831/tcp, 6831/udp, 0.0.0.0:6832->6832/tcp, 6832/udp -blobstore /usr/bin/docker-entrypoint ... Up (healthy) 9000/tcp -pgsql /postgres.sh Up (healthy) 5432/tcp -precise-code-intel-worker /sbin/tini -- /usr/local/b ... Up (health: starting) 3188/tcp -prometheus /bin/prom-wrapper Up 0.0.0.0:9090->9090/tcp -query-runner /sbin/tini -- /usr/local/b ... Up -redis-cache /sbin/tini -- redis-server ... Up 6379/tcp -redis-store /sbin/tini -- redis-server ... Up 6379/tcp -searcher-0 /sbin/tini -- /usr/local/b ... Up (healthy) -syntect-server sh -c /http-server-stabili ... Up (healthy) 9238/tcp -worker /sbin/tini -- /usr/local/b ... Up 3189/tcp -zoekt-indexserver-0 /sbin/tini -- zoekt-source ... Up -zoekt-webserver-0 /sbin/tini -- /bin/sh -c z ... Up (healthy) +docker compose ps --format 'table {{.Name}}\t{{.Status}}' +NAME STATUS +blobstore Up About a minute (healthy) +caddy Up About a minute +cadvisor Up About a minute +codeinsights-db Up About a minute (healthy) +codeinsights-db-exporter Up About a minute +codeintel-db Up About a minute (healthy) +codeintel-db-exporter Up About a minute +executor Up About a minute +gitserver-0 Up About a minute +grafana Up About a minute +jaeger Up About a minute +node-exporter Up About a minute +otel-collector Up About a minute +pgsql Up About a minute (healthy) +pgsql-exporter Up About a minute +precise-code-intel-worker Up About a minute (healthy) +prometheus Up About a minute +redis-cache Up About a minute +redis-store Up About a minute +searcher-0 Up About a minute (healthy) +sourcegraph-frontend-0 Up About a minute (healthy) +sourcegraph-frontend-internal Up About a minute (healthy) +syntactic-code-intel-worker Up About a minute (healthy) +syntect-server Up About a minute (healthy) +worker Up About a minute +zoekt-indexserver-0 Up About a minute +zoekt-webserver-0 Up About a minute (healthy) ``` -4\. Stop the deployment, and restart the databases service only to ensure there are no other connections during backup and restore. +4. Generate the database dumps ```bash -docker-compose down -docker-compose -f db-only-migrate.docker-compose.yaml up -d -``` +docker exec pgsql \ + pg_dump -C --clean --if-exists --username sg sg \ + > sourcegraph_db.out -5\. Generate the database dumps +docker exec codeintel-db \ + pg_dump -C --clean --if-exists --username sg sg \ + > codeintel_db.out -```bash -docker exec pgsql sh -c 'pg_dump -C --clean --if-exists --username sg sg' > sourcegraph_db.out -docker exec codeintel-db sh -c 'pg_dump -C --clean --if-exists --username sg sg' > codeintel_db.out -docker exec codeinsights-db sh -c 'pg_dump -C --clean --if-exists --username postgres postgres' > codeinsights_db.out +docker exec codeinsights-db \ + pg_dump -C --clean --if-exists --username postgres postgres \ + > codeinsights_db.out ``` -6\. Ensure the `sourcegraph_db.out`, `codeintel_db.out` and `codeinsights_db.out` files are moved to a safe and secure location. - -### Restore Sourcegraph databases into a new environment - -The following instructions apply **only if you are restoring your databases into a new deployment** of Sourcegraph ie: a new virtual machine. If you are restoring a previously running environment, see the instructions for [restoring a previously running deployment](#restoring-sourcegraph-databases-into-an-existing-environment) - -1\. Copy the database dump files into the `deploy-sourcegraph-docker/docker-compose` directory. +5. Ensure the `sourcegraph_db.out`, `codeintel_db.out` and `codeinsights_db.out` files are moved to a safe and secure location. -2\. Start the database services +### Restore Sourcegraph databases -```bash -docker-compose -f db-only-migrate.docker-compose.yaml up -d -``` +These instructions apply to both a new deployment and an existing one. You don't need to delete the database volumes: the restore drops and recreates each database. -3\. Copy the database files into the containers +If you started Sourcegraph with more Compose files, such as `jaeger/docker-compose.yaml` or `executors/executor.docker-compose.yaml`, pass the same `-f` options to `docker compose down` and `docker compose up -d`. Otherwise those containers keep running during the restore, and Sourcegraph restarts without their settings. -```bash -docker cp sourcegraph_db.out pgsql:/tmp/sourcegraph_db.out -docker cp codeintel_db.out codeintel-db:/tmp/codeintel_db.out -docker cp codeinsights_db.out codeinsights-db:/tmp/codeinsights_db.out -``` +1. Copy the database dump files into the `deploy-sourcegraph-docker/docker-compose` directory, and `cd` to it. -4\. Restore the databases +2. If you are restoring into an existing deployment, stop it, so nothing writes to the databases during the restore ```bash -docker exec pgsql sh -c 'psql -v ERROR_ON_STOP=1 --username sg -f /tmp/sourcegraph_db.out sg' -docker exec codeintel-db sh -c 'psql -v ERROR_ON_STOP=1 --username sg -f /tmp/codeintel_db.out sg' -docker exec codeinsights-db sh -c 'psql -v ERROR_ON_STOP=1 --username postgres -f /tmp/codeinsights_db.out postgres' +docker compose down ``` -5\. Start the remaining Sourcegraph services +3. Start the database services ```bash -docker-compose up -d +docker compose -f db-only-migrate.docker-compose.yaml up -d ``` -6\. Verify the deployment has started - -```bash -docker-compose ps - Name Command State Ports -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -caddy caddy run --config /etc/ca ... Up 2019/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:80->80/tcp -cadvisor /usr/bin/cadvisor -logtost ... Up (health: starting) 8080/tcp -codeinsights-db docker-entrypoint.sh postgres Up 5432/tcp -codeintel-db /postgres.sh Up (healthy) 5432/tcp -gitserver-0 /sbin/tini -- /usr/local/b ... Up -grafana /entry.sh Up 3000/tcp, 0.0.0.0:3370->3370/tcp -jaeger /go/bin/all-in-one-linux - ... Up 0.0.0.0:14250->14250/tcp, 14268/tcp, 0.0.0.0:16686->16686/tcp, 5775/udp, 0.0.0.0:5778->5778/tcp, - 0.0.0.0:6831->6831/tcp, 6831/udp, 0.0.0.0:6832->6832/tcp, 6832/udp -blobstore /usr/bin/docker-entrypoint ... Up (healthy) 9000/tcp -pgsql /postgres.sh Up (healthy) 5432/tcp -precise-code-intel-worker /sbin/tini -- /usr/local/b ... Up (health: starting) 3188/tcp -prometheus /bin/prom-wrapper Up 0.0.0.0:9090->9090/tcp -query-runner /sbin/tini -- /usr/local/b ... Up -redis-cache /sbin/tini -- redis-server ... Up 6379/tcp -redis-store /sbin/tini -- redis-server ... Up 6379/tcp -searcher-0 /sbin/tini -- /usr/local/b ... Up (healthy) -syntect-server sh -c /http-server-stabili ... Up (healthy) 9238/tcp -worker /sbin/tini -- /usr/local/b ... Up 3189/tcp -zoekt-indexserver-0 /sbin/tini -- zoekt-source ... Up -zoekt-webserver-0 /sbin/tini -- /bin/sh -c z ... Up (healthy)> docker-compose ps -``` - -7\. Browse to your Sourcegraph deployment, login and verify your existing configuration has been restored - -### Restore Sourcegraph databases into an existing environment - -1\. `cd` to the `deploy-sourcegraph-docker/docker-compose` and stop the previous deployment and remove any existing volumes - -```bash -docker-compose down -docker volume rm docker-compose_pgsql -docker volume rm docker-compose_codeintel-db -docker volume rm docker-compose_codeinsights-db -``` - -2\. Start the databases services only - -```bash -docker-compose -f db-only-migrate.docker-compose.yaml up -d -``` - -3\. Copy the database files into the containers +4. Copy the database files into the containers ```bash docker cp sourcegraph_db.out pgsql:/tmp/sourcegraph_db.out @@ -182,49 +122,35 @@ docker cp codeintel_db.out codeintel-db:/tmp/codeintel_db.out docker cp codeinsights_db.out codeinsights-db:/tmp/codeinsights_db.out ``` -4\. Restore the databases +5. Restore the databases ```bash -docker exec pgsql sh -c 'psql -v ERROR_ON_STOP=1 --username sg -f /tmp/sourcegraph_db.out sg' -docker exec codeintel-db sh -c 'psql -v ERROR_ON_STOP=1 --username sg -f /tmp/codeintel_db.out sg' -docker exec codeinsights-db sh -c 'psql -v ERROR_ON_STOP=1 --username postgres -f /tmp/codeinsights_db.out postgres' +docker exec pgsql \ + psql -v ON_ERROR_STOP=1 --username sg \ + -f /tmp/sourcegraph_db.out postgres + +docker exec codeintel-db \ + psql -v ON_ERROR_STOP=1 --username sg \ + -f /tmp/codeintel_db.out postgres + +docker exec codeinsights-db \ + psql -v ON_ERROR_STOP=1 --username postgres \ + -f /tmp/codeinsights_db.out template1 ``` -5\. Start the remaining Sourcegraph services +6. Start the remaining Sourcegraph services ```bash -docker-compose up -d +docker compose up -d ``` -6\. Verify the deployment has started +7. Verify the deployment has started ```bash -docker-compose ps - Name Command State Ports -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -caddy caddy run --config /etc/ca ... Up 2019/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:80->80/tcp -cadvisor /usr/bin/cadvisor -logtost ... Up (health: starting) 8080/tcp -codeinsights-db docker-entrypoint.sh postgres Up 5432/tcp -codeintel-db /postgres.sh Up (healthy) 5432/tcp -gitserver-0 /sbin/tini -- /usr/local/b ... Up -grafana /entry.sh Up 3000/tcp, 0.0.0.0:3370->3370/tcp -jaeger /go/bin/all-in-one-linux - ... Up 0.0.0.0:14250->14250/tcp, 14268/tcp, 0.0.0.0:16686->16686/tcp, 5775/udp, 0.0.0.0:5778->5778/tcp, - 0.0.0.0:6831->6831/tcp, 6831/udp, 0.0.0.0:6832->6832/tcp, 6832/udp -blobstore /usr/bin/docker-entrypoint ... Up (healthy) 9000/tcp -pgsql /postgres.sh Up (healthy) 5432/tcp -precise-code-intel-worker /sbin/tini -- /usr/local/b ... Up (health: starting) 3188/tcp -prometheus /bin/prom-wrapper Up 0.0.0.0:9090->9090/tcp -query-runner /sbin/tini -- /usr/local/b ... Up -redis-cache /sbin/tini -- redis-server ... Up 6379/tcp -redis-store /sbin/tini -- redis-server ... Up 6379/tcp -searcher-0 /sbin/tini -- /usr/local/b ... Up (healthy) -syntect-server sh -c /http-server-stabili ... Up (healthy) 9238/tcp -worker /sbin/tini -- /usr/local/b ... Up 3189/tcp -zoekt-indexserver-0 /sbin/tini -- zoekt-source ... Up -zoekt-webserver-0 /sbin/tini -- /bin/sh -c z ... Up (healthy)> docker-compose ps +docker compose ps ``` -7\. Browse to your Sourcegraph deployment, login and verify your existing configuration has been restored +8. Browse to your Sourcegraph deployment, login and verify your existing configuration has been restored ## Monitoring diff --git a/docs/self-hosted/deploy/kubernetes/operations.mdx b/docs/self-hosted/deploy/kubernetes/operations.mdx index a91f4667c..1f9ab6bcf 100644 --- a/docs/self-hosted/deploy/kubernetes/operations.mdx +++ b/docs/self-hosted/deploy/kubernetes/operations.mdx @@ -196,12 +196,12 @@ $ kubectl rollout restart deployment sourcegraph-frontend ## Access the database -Get the id of one `pgsql` Pod: +Check that the `pgsql` pod is running: ```bash $ kubectl get pods -l app=pgsql -NAME READY STATUS RESTARTS AGE -pgsql-76a4bfcd64-rt4cn 2/2 Running 0 19m +NAME READY STATUS RESTARTS AGE +pgsql-0 2/2 Running 0 19m ``` Make sure you are operating under the correct namespace (i.e. add `-n prod` if your pod is under the `prod` namespace). @@ -209,7 +209,7 @@ Make sure you are operating under the correct namespace (i.e. add `-n prod` if y Open a PostgreSQL interactive terminal: ```bash -$ kubectl exec -it pgsql-76a4bfcd64-rt4cn -- psql -U sg +$ kubectl exec -it pgsql-0 -- psql -U sg ``` Run your SQL query: @@ -233,31 +233,25 @@ The following instructions are specific to backing up and restoring the sourcegr > > The above may take a while if you have a lot of repositories. In the meantime, searches may be slow or return incomplete results. This process rarely takes longer than 6 hours and is usually **much** faster. -> NOTE: In some places you will see `$NAMESPACE` used. Add `-n $NAMESPACE` to commands if you are not using the default namespace +> NOTE: Add `-n $NAMESPACE` to commands if you are not using the default namespace. The database pods are `pgsql-0`, `codeintel-db-0`, and `codeinsights-db-0`. > More kubectl configuration options can be found here: [kubectl Cheat Sheet](https://kubernetes.io/docs/reference/kubectl/cheatsheet/) ### Back up Sourcegraph databases These instructions will back up the primary `sourcegraph` database, the [codeintel](/code-navigation/) database, and the [codeinsights](/code-insights/) database. -A. Verify deployment running +1. Verify your Sourcegraph pods are running / healthy, especially the 3 database pods ```bash -$ kubectl get pods -A +$ kubectl get pods ``` -B. Stop all connections to the database by removing the frontend deployment +2. Check for corrupt database indexes. The check blocks writes to each table while it checks that table's indexes. If amcheck returns errors, please reach out to [support@sourcegraph.com](mailto:support@sourcegraph.com) -```bash -$ kubectl scale --replicas=0 deployment/sourcegraph-frontend -# or -$ kubectl delete deployment sourcegraph-frontend -``` - -C. Check for corrupt database indexes. If amcheck returns errors, please reach out to [support@sourcegraph.com](mailto:support@sourcegraph.com) +Save this query in a file named `amcheck.sql`: ```sql -create extension amcheck; +create extension if not exists amcheck; select bt_index_parent_check(c.oid, true), c.relname, c.relpages from pg_index i @@ -268,140 +262,119 @@ join pg_namespace n ON c.relnamespace = n.oid where am.amname = 'btree' -- Don't check temp tables, which may be from another session: and c.relpersistence != 't' +-- Skip partitioned indexes; the indexes of their partitions are checked: +and c.relkind = 'i' -- Function may throw an error when this is omitted: and i.indisready AND i.indisvalid; ``` -D. Generate the database dumps +Run it on each database. Each command prints any errors, and exits 0 when every index is intact: ```bash -$ kubectl exec -it $pgsql_POD_NAME -- bash -c 'pg_dump -C --clean --if-exists --username sg sg' > sourcegraph_db.out -$ kubectl exec -it $codeintel-db_POD_NAME -- bash -c 'pg_dump -C --clean --if-exists --username sg sg' > codeintel_db.out -$ kubectl exec -it $codeinsights-db_POD_NAME -- bash -c 'pg_dump -C --clean --if-exists --username postgres postgres' > codeinsights_db.out -``` +$ kubectl exec -i pgsql-0 -- \ + psql -v ON_ERROR_STOP=1 --username sg sg \ + < amcheck.sql > /dev/null -Ensure the `sourcegraph_db.out`, `codeintel_db.out` and `codeinsights_db.out` files are moved to a safe and secure location. - -### Restore Sourcegraph databases +$ kubectl exec -i codeintel-db-0 -- \ + psql -v ON_ERROR_STOP=1 --username sg sg \ + < amcheck.sql > /dev/null -#### Restoring Sourcegraph databases into a new environment - -The following instructions apply only if you are restoring your databases into a new deployment of Sourcegraph ie: a new virtual machine - -If you are restoring a previously running environment, see the instructions for [restoring a previously running deployment](#restoring-sourcegraph-databases-into-an-existing-environment) - -A. Copy the database dump files (eg. `sourcegraph_db.out`, `codeintel_db.out` and `codeinsights_db.out`) into your deployment directory - -B. Start the database services by running the following command from your deployment directory: - -```bash -$ kubectl rollout restart deployment pgsql -$ kubectl rollout restart deployment codeintel-db -$ kubectl rollout restart deployment codeinsights-db +$ kubectl exec -i codeinsights-db-0 -- \ + psql -v ON_ERROR_STOP=1 --username postgres postgres \ + < amcheck.sql > /dev/null ``` -C. Copy the database files into the pods by running the following commands: +3. Generate the database dumps ```bash -$ kubectl cp sourcegraph_db.out $NAMESPACE/$pgsql_POD_NAME:/tmp/sourcegraph_db.out -$ kubectl cp codeintel_db.out $NAMESPACE/$codeintel-db_POD_NAME:/tmp/codeintel_db.out -$ kubectl cp codeinsights_db.out $NAMESPACE/$codeinsights-db_POD_NAME:/tmp/codeinsights_db.out -``` +$ kubectl exec pgsql-0 -- \ + pg_dump -C --clean --if-exists --username sg sg \ + > sourcegraph_db.out -D. Restore the databases +$ kubectl exec codeintel-db-0 -- \ + pg_dump -C --clean --if-exists --username sg sg \ + > codeintel_db.out -```bash -$ kubectl exec -it $pgsql_POD_NAME -- bash -c 'psql -v ERROR_ON_STOP=1 --username sg -f /tmp/sourcegraph_db.out sg' -$ kubectl exec -it $codeintel-db_POD_NAME -- bash -c 'psql -v ERROR_ON_STOP=1 --username sg -f /tmp/codeintel_db.out sg' -$ kubectl exec -it $codeinsights-db_POD_NAME -- bash -c 'psql -v ERROR_ON_STOP=1 --username postgres -f /tmp/codeinsights_db.out postgres' +$ kubectl exec codeinsights-db-0 -- \ + pg_dump -C --clean --if-exists --username postgres postgres \ + > codeinsights_db.out ``` -E. Check for corrupt database indexes. If amcheck returns errors, please reach out to [support@sourcegraph.com](mailto:support@sourcegraph.com) +Run these commands without `-it`: with a terminal attached, every line of the dump ends with a carriage return. -```sql -create extension amcheck; +Ensure the `sourcegraph_db.out`, `codeintel_db.out` and `codeinsights_db.out` files are moved to a safe and secure location. -select bt_index_parent_check(c.oid, true), c.relname, c.relpages -from pg_index i -join pg_opclass op ON i.indclass[0] = op.oid -join pg_am am ON op.opcmethod = am.oid -join pg_class c ON i.indexrelid = c.oid -join pg_namespace n ON c.relnamespace = n.oid -where am.amname = 'btree' --- Don't check temp tables, which may be from another session: -and c.relpersistence != 't' --- Function may throw an error when this is omitted: -and i.indisready AND i.indisvalid; -``` +### Restore Sourcegraph databases -F. Start the remaining Sourcegraph services by following the steps in [applying manifests](#applying-manifests). +These instructions apply to both a new deployment and an existing one. You don't need to delete the database volumes: the restore drops and recreates each database. -#### Restoring Sourcegraph databases into an existing environment +1. On the machine where you run `kubectl`, `cd` to the directory with the database dump files: `sourcegraph_db.out`, `codeintel_db.out` and `codeinsights_db.out` -A. Stop the existing deployment by removing the frontend deployment +2. Stop every Sourcegraph service except the databases, so nothing writes to the databases during the restore ```bash -$ kubectl scale --replicas=0 deployment/sourcegraph-frontend -# or -$ kubectl delete deployment sourcegraph-frontend +$ kubectl scale deployment,statefulset --replicas=0 -l ' + deploy=sourcegraph, + app.kubernetes.io/component notin + (pgsql,codeintel-db,codeinsights-db)' ``` -B. Remove any existing volumes for the databases in the existing deployment +Wait until `kubectl get pods` shows no Sourcegraph pods other than `pgsql-0`, `codeintel-db-0`, and `codeinsights-db-0`. + +3. Copy the database files into the pods ```bash -$ kubectl delete pvc pgsql -$ kubectl delete pvc codeintel-db -$ kubectl delete pvc codeinsights-db -$ kubectl delete pv $pgsql_PV_NAME --force -$ kubectl delete pv $codeintel-db_PV_NAME --force -$ kubectl delete pv $codeinsights-db_PV_NAME --force +$ kubectl cp sourcegraph_db.out pgsql-0:/data/sourcegraph_db.out +$ kubectl cp codeintel_db.out codeintel-db-0:/data/codeintel_db.out +$ kubectl cp codeinsights_db.out \ + codeinsights-db-0:/var/lib/postgresql/data/codeinsights_db.out ``` -C. Copy the database dump files (eg. `sourcegraph_db.out`, `codeintel_db.out` and `codeinsights_db.out`) into your deployment directory +The database containers have a read-only root filesystem, so copy the files into the database volumes. Step 5 deletes them after the restore. + +4. Restore the databases -D. Start the database services only +`DROP DATABASE ... WITH (FORCE)` disconnects the other sessions on each database, such as the `pgsql-exporter` sidecar, and drops it, so the dump can recreate it. `ON_ERROR_STOP=1` stops the restore at the first error; if a restore fails, fix the cause and run it again. ```bash -$ kubectl rollout restart deployment pgsql -$ kubectl rollout restart deployment codeintel-db -$ kubectl rollout restart deployment codeinsights-db -``` +$ kubectl exec -it pgsql-0 -- \ + psql -v ON_ERROR_STOP=1 --username sg \ + -c "DROP DATABASE IF EXISTS sg WITH (FORCE)" \ + -f /data/sourcegraph_db.out postgres -E. Copy the database files into the pods by running the following commands: +$ kubectl exec -it codeintel-db-0 -- \ + psql -v ON_ERROR_STOP=1 --username sg \ + -c "DROP DATABASE IF EXISTS sg WITH (FORCE)" \ + -f /data/codeintel_db.out postgres -```bash -$ kubectl cp sourcegraph_db.out $NAMESPACE/$pgsql_POD_NAME:/tmp/sourcegraph_db.out -$ kubectl cp codeintel_db.out $NAMESPACE/$codeintel-db_POD_NAME:/tmp/codeintel_db.out -$ kubectl cp codeinsights_db.out $NAMESPACE/$codeinsights-db_POD_NAME:/tmp/codeinsights_db.out +$ kubectl exec -it codeinsights-db-0 -- \ + psql -v ON_ERROR_STOP=1 --username postgres \ + -c "DROP DATABASE IF EXISTS postgres WITH (FORCE)" \ + -f /var/lib/postgresql/data/codeinsights_db.out template1 ``` -F. Restore the databases +5. After all three restores succeed, delete the backup files from the pods ```bash -$ kubectl exec -it $pgsql_POD_NAME -- bash -c 'psql -v ERROR_ON_STOP=1 --username sg -f /tmp/sourcegraph_db.out sg' -$ kubectl exec -it $codeintel-db_POD_NAME -- bash -c 'psql -v ERROR_ON_STOP=1 --username sg -f /tmp/codeintel_db.out sg' -$ kubectl exec -it $codeinsights-db_POD_NAME -- bash -c 'psql -v ERROR_ON_STOP=1 --username postgres -f /tmp/codeinsights_db.out postgres' +$ kubectl exec pgsql-0 -- rm /data/sourcegraph_db.out +$ kubectl exec codeintel-db-0 -- rm /data/codeintel_db.out +$ kubectl exec codeinsights-db-0 -- \ + rm /var/lib/postgresql/data/codeinsights_db.out ``` -G. Check for corrupt database indexes. If amcheck returns errors, please reach out to [support@sourcegraph.com](mailto:support@sourcegraph.com) +6. Check for corrupt database indexes, with the commands in step 2 of [Back up Sourcegraph databases](#back-up-sourcegraph-databases). If amcheck returns errors, please reach out to [support@sourcegraph.com](mailto:support@sourcegraph.com) -```sql -create extension amcheck; +7. Start the remaining Sourcegraph services, by running `helm upgrade` with the chart version and values file of your deployment (`helm list` shows the chart version). It sets every service back to the number of replicas in your values. -select bt_index_parent_check(c.oid, true), c.relname, c.relpages -from pg_index i -join pg_opclass op ON i.indclass[0] = op.oid -join pg_am am ON op.opcmethod = am.oid -join pg_class c ON i.indexrelid = c.oid -join pg_namespace n ON c.relnamespace = n.oid -where am.amname = 'btree' --- Don't check temp tables, which may be from another session: -and c.relpersistence != 't' --- Function may throw an error when this is omitted: -and i.indisready AND i.indisvalid; +```bash +$ helm upgrade --install --values ./override.yaml \ + --version \ + sourcegraph sourcegraph/sourcegraph ``` -H. Start the remaining Sourcegraph services by following the steps in [applying manifests](#applying-manifests). +If you deployed with Kustomize, follow the steps in [applying manifests](#applying-manifests) instead. + +8. Browse to your Sourcegraph deployment, log in, and verify your existing configuration has been restored ## List of ports @@ -438,15 +411,6 @@ syntect-server ClusterIP 10.72.14.49 9238/TC worker ClusterIP 10.72.7.72 3189/TCP,6060/TCP 25h ``` -## Migrate to Kustomize - -See the [migration docs for Kustomize](/self-hosted/deploy/kubernetes/kustomize/migrate) for more information. - -## Upgrade - -- See the [Updating Sourcegraph docs](/self-hosted/updates/) on how to upgrade.
-- See the [Updating a Kubernetes Sourcegraph instance docs](/self-hosted/deploy/kubernetes/upgrade) for details on changes in each version to determine if manual migration steps are necessary. - ## Troubleshoot See the [Troubleshooting docs](/self-hosted/deploy/kubernetes/troubleshoot). diff --git a/docs/self-hosted/how-to/index.mdx b/docs/self-hosted/how-to/index.mdx index 63ada9327..29917151f 100644 --- a/docs/self-hosted/how-to/index.mdx +++ b/docs/self-hosted/how-to/index.mdx @@ -22,7 +22,6 @@ - [How to address common monorepo problems](/admin/how-to/monorepo-issues) - [How to Set a password for Redis using a ConfigMap](/self-hosted/how-to/redis-configmap) - [How to import a set of internal repositories to Sourcegraph](/admin/how-to/internal-github-repos) -- [How to identify and resolve index corruption in postgres 14](/self-hosted/how-to/postgres14-index-corruption) - [How to export search results](/admin/how-to/export-search-results) - [How to debug / confirm blobstore is healthy](/self-hosted/how-to/blobstore-debugging) - [How to handle postgresql 12 to 16 drift](/self-hosted/how-to/postgres-12-to-16-drift) diff --git a/docs/self-hosted/how-to/postgres14-index-corruption.mdx b/docs/self-hosted/how-to/postgres14-index-corruption.mdx deleted file mode 100644 index 027867f81..000000000 --- a/docs/self-hosted/how-to/postgres14-index-corruption.mdx +++ /dev/null @@ -1,57 +0,0 @@ -# How to identify and resolve index corruption in Postgres v14.0 - v14.3 - -A bug has been identified in PostgreSQL 14.0-14.3 that can cause database index corruption in indexes created concurrently. Sourcegraph [utilizes](https://sourcegraph.com/search?q=context:global+repo:%5Egithub%5C.com/sourcegraph/sourcegraph%24+%22CREATE+INDEX%22+AND+CONCURRENTLY&patternType=keyword) concurrent index creation, and if you are on these versions, you may experience slow queries or database corruption. To learn more about this bug see PostgreSQL's [out-of-cycle release announcement](https://www.postgresql.org/message-id/165473835807.573551.1512237163040609764%40wrigleys.postgresql.org) or [migops article](https://www.migops.com/blog/important-postgresql-14-update-to-avoid-silent-corruption-of-indexes/) on the subject. - -> NOTE: If you are running a default Sourcegraph deployment with default Postgres image values, it is likely Postgres is running in version 12 and your instance will be unaffected by this bug. - -## Identify your database version - -To identify which version of Sourcegraph you are running in a default Sourcegraph deployment. You can access your database via the `docker` or `kubectl` cli tools and run the following command: - -``` -SELECT version(); -``` - -> NOTE: You can refer to the following instructions for accessing databases on your deployment type: [Docker Compose](/self-hosted/deploy/docker-compose/#access-the-database), [Kubernetes](/self-hosted/deploy/kubernetes/operations#access-the-database). - -You may also check for index corruption in your database using the `amcheck` by running the following query in your database - -``` -create extension amcheck; - -select bt_index_parent_check(c.oid, true), c.relname, c.relpages -from pg_index i -join pg_opclass op ON i.indclass[0] = op.oid -join pg_am am ON op.opcmethod = am.oid -join pg_class c ON i.indexrelid = c.oid -join pg_namespace n ON c.relnamespace = n.oid -where am.amname = 'btree' --- Don't check temp tables, which may be from another session: -and c.relpersistence != 't' --- Function may throw an error when this is omitted: -and i.indisready AND i.indisvalid; -``` - -This query will return error if there are corrupted indexes. - -## Resolve - -If you are impacted, you can remediate this by upgrading to a newer version of Postgres (14.4+) and running the following commands in your database - -Determine database name: - -``` -SELECT current_database(); -``` - -``` -REINDEX DATABASE ; -``` - -_You may want to use the amcheck query above to verify the reindex has resolved index corruption._ - -### Upgrading your database - -In default Sourcegraph deployments internal PostgreSQL instances are used and may be upgraded via the [pg_upgrade](https://www.postgresql.org/docs/11/pgupgrade.html). For external databases consult your service providers documentation. For a deeper look at database upgrade operations please consult our [PostgreSQL documentation](/self-hosted/postgres#upgrading-postgresql). - -If you have any questions, please reach out to support on Slack or email support@sourcegraph.com. diff --git a/docs/self-hosted/how-to/rebuild-corrupt-postgres-indexes.mdx b/docs/self-hosted/how-to/rebuild-corrupt-postgres-indexes.mdx index 521646e55..07c5554d6 100644 --- a/docs/self-hosted/how-to/rebuild-corrupt-postgres-indexes.mdx +++ b/docs/self-hosted/how-to/rebuild-corrupt-postgres-indexes.mdx @@ -57,7 +57,7 @@ After deleting duplicates, just re-run the above statement. Repeat the process u At the end of the index rebuilding process, as a last sanity check, we use the amcheck extension to verify there are no corrupt indexes — an error is raised if there are (you should expect to see some output from this command). ```sql -create extension amcheck; +create extension if not exists amcheck; select bt_index_parent_check(c.oid, true), c.relname, c.relpages from pg_index i @@ -68,6 +68,8 @@ join pg_namespace n ON c.relnamespace = n.oid where am.amname = 'btree' -- Don't check temp tables, which may be from another session: and c.relpersistence != 't' +-- Skip partitioned indexes; the indexes of their partitions are checked: +and c.relkind = 'i' -- Function may throw an error when this is omitted: and i.indisready AND i.indisvalid; ``` diff --git a/docs/self-hosted/index.mdx b/docs/self-hosted/index.mdx index abb846d17..c71afe2e0 100644 --- a/docs/self-hosted/index.mdx +++ b/docs/self-hosted/index.mdx @@ -99,7 +99,6 @@ Get started running Sourcegraph on-prem. - [Clear code intelligence data](/self-hosted/how-to/clear-codeintel-data) - [Dirty database](/self-hosted/how-to/dirty-database) - [PostgreSQL 12 to 16 drift](/self-hosted/how-to/postgres-12-to-16-drift) -- [PostgreSQL 14 index corruption](/self-hosted/how-to/postgres14-index-corruption) - [Privileged migrations](/self-hosted/how-to/privileged-migrations) - [Rebuild corrupt PostgreSQL indexes](/self-hosted/how-to/rebuild-corrupt-postgres-indexes) - [Redis ConfigMap](/self-hosted/how-to/redis-configmap) diff --git a/src/data/redirects.ts b/src/data/redirects.ts index 8fcdb8c25..4f9c316e8 100644 --- a/src/data/redirects.ts +++ b/src/data/redirects.ts @@ -3435,7 +3435,11 @@ const redirectsData = [ }, { source: '/admin/how-to/postgres14-index-corruption', - destination: '/self-hosted/how-to/postgres14-index-corruption' + destination: '/self-hosted/how-to/rebuild-corrupt-postgres-indexes' + }, + { + source: '/self-hosted/how-to/postgres14-index-corruption', + destination: '/self-hosted/how-to/rebuild-corrupt-postgres-indexes' }, { source: '/admin/how-to/postgres_12_to_16_drift',