Fix broken database backup / restore commands

cspell-allow-list.txt

@@ -405,6 +405,7 @@ pekko
 persistentvolumeclaims
 pgbouncer
 pgcrypto
+PGDATA
 PGDATABASE
 PGDATASOURCE
 pgdump
@@ -519,6 +520,7 @@ starcoder
 Starlark
 startedat
 startswith
+statefulset
 statefulsets
 stepscontainer
 stepsenv
@@ -604,6 +606,7 @@ urlencode
 urlsafe
 userprincipalname
 USERTRUST
+vacuumdb
 Valsorda
 vegeta
 vercel

docs/self-hosted/deploy/kubernetes/operations.mdx

@@ -196,20 +196,20 @@ $ kubectl rollout restart deployment sourcegraph-frontend
 
 ## Access the database
 
-Get the id of one `pgsql` Pod:
+Check that the `pgsql` pod is running:
 
 ```bash
 $ kubectl get pods -l app=pgsql
-NAME                     READY     STATUS    RESTARTS   AGE
-pgsql-76a4bfcd64-rt4cn   2/2       Running   0          19m
+NAME      READY   STATUS    RESTARTS   AGE
+pgsql-0   2/2     Running   0          19m
 ```
 
 Make sure you are operating under the correct namespace (i.e. add `-n prod` if your pod is under the `prod` namespace).
 
 Open a PostgreSQL interactive terminal:
 
 ```bash
-$ kubectl exec -it pgsql-76a4bfcd64-rt4cn -- psql -U sg
+$ kubectl exec -it pgsql-0 -- psql -U sg
 ```
 
 Run your SQL query:
@@ -233,31 +233,25 @@ The following instructions are specific to backing up and restoring the sourcegr
 >
 > The above may take a while if you have a lot of repositories. In the meantime, searches may be slow or return incomplete results. This process rarely takes longer than 6 hours and is usually **much** faster.
 
-> NOTE: In some places you will see `$NAMESPACE` used. Add `-n $NAMESPACE` to commands if you are not using the default namespace
+> NOTE: Add `-n $NAMESPACE` to commands if you are not using the default namespace. The database pods are `pgsql-0`, `codeintel-db-0`, and `codeinsights-db-0`.
 > More kubectl configuration options can be found here: [kubectl Cheat Sheet](https://kubernetes.io/docs/reference/kubectl/cheatsheet/)
 
 ### Back up Sourcegraph databases
 
 These instructions will back up the primary `sourcegraph` database, the [codeintel](/code-navigation/) database, and the [codeinsights](/code-insights/) database.
 
-A. Verify deployment running
+1. Verify your Sourcegraph pods are running / healthy, especially the 3 database pods
 
 ```bash
-$ kubectl get pods -A
+$ kubectl get pods
 ```
 
-B. Stop all connections to the database by removing the frontend deployment
+2. Check for corrupt database indexes. The check blocks writes to each table while it checks that table's indexes. If amcheck returns errors, please reach out to [support@sourcegraph.com](mailto:support@sourcegraph.com)
 
-```bash
-$ kubectl scale --replicas=0 deployment/sourcegraph-frontend
-# or
-$ kubectl delete deployment sourcegraph-frontend
-```
-
-C. Check for corrupt database indexes. If amcheck returns errors, please reach out to [support@sourcegraph.com](mailto:support@sourcegraph.com)
+Save this query in a file named `amcheck.sql`:
 
 ```sql
-create extension amcheck;
+create extension if not exists amcheck;
 
 select bt_index_parent_check(c.oid, true), c.relname, c.relpages
 from pg_index i
@@ -268,140 +262,119 @@ join pg_namespace n ON c.relnamespace = n.oid
 where am.amname = 'btree'
 -- Don't check temp tables, which may be from another session:
 and c.relpersistence != 't'
+-- Skip partitioned indexes; the indexes of their partitions are checked:
+and c.relkind = 'i'
 -- Function may throw an error when this is omitted:
 and i.indisready AND i.indisvalid;
 ```
 
-D. Generate the database dumps
+Run it on each database. Each command prints any errors, and exits 0 when every index is intact:
 
 ```bash
-$ kubectl exec -it $pgsql_POD_NAME -- bash -c 'pg_dump -C --clean --if-exists --username sg sg' > sourcegraph_db.out
-$ kubectl exec -it $codeintel-db_POD_NAME -- bash -c 'pg_dump -C --clean --if-exists --username sg sg' > codeintel_db.out
-$ kubectl exec -it $codeinsights-db_POD_NAME -- bash -c 'pg_dump -C --clean --if-exists --username postgres postgres' > codeinsights_db.out
-```
+$ kubectl exec -i pgsql-0 -- \
+    psql -v ON_ERROR_STOP=1 --username sg sg \
+    < amcheck.sql > /dev/null
 
-Ensure the `sourcegraph_db.out`, `codeintel_db.out` and `codeinsights_db.out` files are moved to a safe and secure location.
-
-### Restore Sourcegraph databases
+$ kubectl exec -i codeintel-db-0 -- \
+    psql -v ON_ERROR_STOP=1 --username sg sg \
+    < amcheck.sql > /dev/null
 
-#### Restoring Sourcegraph databases into a new environment
-
-The following instructions apply only if you are restoring your databases into a new deployment of Sourcegraph ie: a new virtual machine
-
-If you are restoring a previously running environment, see the instructions for [restoring a previously running deployment](#restoring-sourcegraph-databases-into-an-existing-environment)
-
-A. Copy the database dump files (eg. `sourcegraph_db.out`, `codeintel_db.out` and `codeinsights_db.out`) into your deployment directory
-
-B. Start the database services by running the following command from your deployment directory:
-
-```bash
-$ kubectl rollout restart deployment pgsql
-$ kubectl rollout restart deployment codeintel-db
-$ kubectl rollout restart deployment codeinsights-db
+$ kubectl exec -i codeinsights-db-0 -- \
+    psql -v ON_ERROR_STOP=1 --username postgres postgres \
+    < amcheck.sql > /dev/null
 ```
 
-C. Copy the database files into the pods by running the following commands:
+3. Generate the database dumps
 
 ```bash
-$ kubectl cp sourcegraph_db.out $NAMESPACE/$pgsql_POD_NAME:/tmp/sourcegraph_db.out
-$ kubectl cp codeintel_db.out $NAMESPACE/$codeintel-db_POD_NAME:/tmp/codeintel_db.out
-$ kubectl cp codeinsights_db.out $NAMESPACE/$codeinsights-db_POD_NAME:/tmp/codeinsights_db.out
-```
+$ kubectl exec pgsql-0 -- \
+    pg_dump -C --clean --if-exists --username sg sg \
+    > sourcegraph_db.out
 
-D. Restore the databases
+$ kubectl exec codeintel-db-0 -- \
+    pg_dump -C --clean --if-exists --username sg sg \
+    > codeintel_db.out
 
-```bash
-$ kubectl exec -it $pgsql_POD_NAME -- bash -c 'psql -v ERROR_ON_STOP=1 --username sg -f /tmp/sourcegraph_db.out sg'
-$ kubectl exec -it $codeintel-db_POD_NAME -- bash -c 'psql -v ERROR_ON_STOP=1 --username sg -f /tmp/codeintel_db.out sg'
-$ kubectl exec -it $codeinsights-db_POD_NAME -- bash -c 'psql -v ERROR_ON_STOP=1 --username postgres -f /tmp/codeinsights_db.out postgres'
+$ kubectl exec codeinsights-db-0 -- \
+    pg_dump -C --clean --if-exists --username postgres postgres \
+    > codeinsights_db.out
 ```
 
-E. Check for corrupt database indexes. If amcheck returns errors, please reach out to [support@sourcegraph.com](mailto:support@sourcegraph.com)
+Run these commands without `-it`: with a terminal attached, every line of the dump ends with a carriage return.
 
-```sql
-create extension amcheck;
+Ensure the `sourcegraph_db.out`, `codeintel_db.out` and `codeinsights_db.out` files are moved to a safe and secure location.
 
-select bt_index_parent_check(c.oid, true), c.relname, c.relpages
-from pg_index i
-join pg_opclass op ON i.indclass[0] = op.oid
-join pg_am am ON op.opcmethod = am.oid
-join pg_class c ON i.indexrelid = c.oid
-join pg_namespace n ON c.relnamespace = n.oid
-where am.amname = 'btree'
--- Don't check temp tables, which may be from another session:
-and c.relpersistence != 't'
--- Function may throw an error when this is omitted:
-and i.indisready AND i.indisvalid;
-```
+### Restore Sourcegraph databases
 
-F. Start the remaining Sourcegraph services by following the steps in [applying manifests](#applying-manifests).
+These instructions apply to both a new deployment and an existing one. You don't need to delete the database volumes: the restore drops and recreates each database.
 
-#### Restoring Sourcegraph databases into an existing environment
+1. On the machine where you run `kubectl`, `cd` to the directory with the database dump files: `sourcegraph_db.out`, `codeintel_db.out` and `codeinsights_db.out`
 
-A. Stop the existing deployment by removing the frontend deployment
+2. Stop every Sourcegraph service except the databases, so nothing writes to the databases during the restore
 
 ```bash
-$ kubectl scale --replicas=0 deployment/sourcegraph-frontend
-# or
-$ kubectl delete deployment sourcegraph-frontend
+$ kubectl scale deployment,statefulset --replicas=0 -l '
+    deploy=sourcegraph,
+    app.kubernetes.io/component notin
+        (pgsql,codeintel-db,codeinsights-db)'
 ```
 
-B. Remove any existing volumes for the databases in the existing deployment
+Wait until `kubectl get pods` shows no Sourcegraph pods other than `pgsql-0`, `codeintel-db-0`, and `codeinsights-db-0`.
+
+3. Copy the database files into the pods
 
 ```bash
-$ kubectl delete pvc pgsql
-$ kubectl delete pvc codeintel-db
-$ kubectl delete pvc codeinsights-db
-$ kubectl delete pv $pgsql_PV_NAME --force
-$ kubectl delete pv $codeintel-db_PV_NAME --force
-$ kubectl delete pv $codeinsights-db_PV_NAME --force
+$ kubectl cp sourcegraph_db.out pgsql-0:/data/sourcegraph_db.out
+$ kubectl cp codeintel_db.out codeintel-db-0:/data/codeintel_db.out
+$ kubectl cp codeinsights_db.out \
+    codeinsights-db-0:/var/lib/postgresql/data/codeinsights_db.out
 ```
 
-C. Copy the database dump files (eg. `sourcegraph_db.out`, `codeintel_db.out` and `codeinsights_db.out`) into your deployment directory
+The database containers have a read-only root filesystem, so copy the files into the database volumes. Step 5 deletes them after the restore.
+
+4. Restore the databases
 
-D. Start the database services only
+`DROP DATABASE ... WITH (FORCE)` disconnects the other sessions on each database, such as the `pgsql-exporter` sidecar, and drops it, so the dump can recreate it. `ON_ERROR_STOP=1` stops the restore at the first error; if a restore fails, fix the cause and run it again.
 
 ```bash
-$ kubectl rollout restart deployment pgsql
-$ kubectl rollout restart deployment codeintel-db
-$ kubectl rollout restart deployment codeinsights-db
-```
+$ kubectl exec -it pgsql-0 -- \
+    psql -v ON_ERROR_STOP=1 --username sg \
+    -c "DROP DATABASE IF EXISTS sg WITH (FORCE)" \
+    -f /data/sourcegraph_db.out postgres
 
-E. Copy the database files into the pods by running the following commands:
+$ kubectl exec -it codeintel-db-0 -- \
+    psql -v ON_ERROR_STOP=1 --username sg \
+    -c "DROP DATABASE IF EXISTS sg WITH (FORCE)" \
+    -f /data/codeintel_db.out postgres
 
-```bash
-$ kubectl cp sourcegraph_db.out $NAMESPACE/$pgsql_POD_NAME:/tmp/sourcegraph_db.out
-$ kubectl cp codeintel_db.out $NAMESPACE/$codeintel-db_POD_NAME:/tmp/codeintel_db.out
-$ kubectl cp codeinsights_db.out $NAMESPACE/$codeinsights-db_POD_NAME:/tmp/codeinsights_db.out
+$ kubectl exec -it codeinsights-db-0 -- \
+    psql -v ON_ERROR_STOP=1 --username postgres \
+    -c "DROP DATABASE IF EXISTS postgres WITH (FORCE)" \
+    -f /var/lib/postgresql/data/codeinsights_db.out template1
 ```
 
-F. Restore the databases
+5. After all three restores succeed, delete the backup files from the pods
 
 ```bash
-$ kubectl exec -it $pgsql_POD_NAME -- bash -c 'psql -v ERROR_ON_STOP=1 --username sg -f /tmp/sourcegraph_db.out sg'
-$ kubectl exec -it $codeintel-db_POD_NAME -- bash -c 'psql -v ERROR_ON_STOP=1 --username sg -f /tmp/codeintel_db.out sg'
-$ kubectl exec -it $codeinsights-db_POD_NAME -- bash -c 'psql -v ERROR_ON_STOP=1 --username postgres -f /tmp/codeinsights_db.out postgres'
+$ kubectl exec pgsql-0 -- rm /data/sourcegraph_db.out
+$ kubectl exec codeintel-db-0 -- rm /data/codeintel_db.out
+$ kubectl exec codeinsights-db-0 -- \
+    rm /var/lib/postgresql/data/codeinsights_db.out
 ```
 
-G. Check for corrupt database indexes. If amcheck returns errors, please reach out to [support@sourcegraph.com](mailto:support@sourcegraph.com)
+6. Check for corrupt database indexes, with the commands in step 2 of [Back up Sourcegraph databases](#back-up-sourcegraph-databases). If amcheck returns errors, please reach out to [support@sourcegraph.com](mailto:support@sourcegraph.com)
 
-```sql
-create extension amcheck;
+7. Start the remaining Sourcegraph services, by running `helm upgrade` with the chart version and values file of your deployment (`helm list` shows the chart version). It sets every service back to the number of replicas in your values.
 
-select bt_index_parent_check(c.oid, true), c.relname, c.relpages
-from pg_index i
-join pg_opclass op ON i.indclass[0] = op.oid
-join pg_am am ON op.opcmethod = am.oid
-join pg_class c ON i.indexrelid = c.oid
-join pg_namespace n ON c.relnamespace = n.oid
-where am.amname = 'btree'
--- Don't check temp tables, which may be from another session:
-and c.relpersistence != 't'
--- Function may throw an error when this is omitted:
-and i.indisready AND i.indisvalid;
+```bash
+$ helm upgrade --install --values ./override.yaml \
+    --version <chart_version> \
+    sourcegraph sourcegraph/sourcegraph
 ```
 
-H. Start the remaining Sourcegraph services by following the steps in [applying manifests](#applying-manifests).
+If you deployed with Kustomize, follow the steps in [applying manifests](#applying-manifests) instead.
+
+8. Browse to your Sourcegraph deployment, log in, and verify your existing configuration has been restored
 
 ## List of ports
 
@@ -438,15 +411,6 @@ syntect-server                  ClusterIP   10.72.14.49    <none>        9238/TC
 worker                          ClusterIP   10.72.7.72     <none>        3189/TCP,6060/TCP            25h
 ```
 
-## Migrate to Kustomize
-
-See the [migration docs for Kustomize](/self-hosted/deploy/kubernetes/kustomize/migrate) for more information.
-
-## Upgrade
-
--   See the [Updating Sourcegraph docs](/self-hosted/updates/) on how to upgrade.<br/>
--   See the [Updating a Kubernetes Sourcegraph instance docs](/self-hosted/deploy/kubernetes/upgrade) for details on changes in each version to determine if manual migration steps are necessary.
-
 ## Troubleshoot
 
 See the [Troubleshooting docs](/self-hosted/deploy/kubernetes/troubleshoot).

docs/self-hosted/how-to/index.mdx

@@ -22,7 +22,6 @@
 -   [How to address common monorepo problems](/admin/how-to/monorepo-issues)
 -   [How to Set a password for Redis using a ConfigMap](/self-hosted/how-to/redis-configmap)
 -   [How to import a set of internal repositories to Sourcegraph](/admin/how-to/internal-github-repos)
--   [How to identify and resolve index corruption in postgres 14](/self-hosted/how-to/postgres14-index-corruption)
 -   [How to export search results](/admin/how-to/export-search-results)
 -   [How to debug / confirm blobstore is healthy](/self-hosted/how-to/blobstore-debugging)
 -   [How to handle postgresql 12 to 16 drift](/self-hosted/how-to/postgres-12-to-16-drift)

docs/self-hosted/how-to/rebuild-corrupt-postgres-indexes.mdx

@@ -57,7 +57,7 @@ After deleting duplicates, just re-run the above statement. Repeat the process u
 At the end of the index rebuilding process, as a last sanity check, we use the amcheck extension to verify there are no corrupt indexes — an error is raised if there are (you should expect to see some output from this command).
 
 ```sql
-create extension amcheck;
+create extension if not exists amcheck;
 
 select bt_index_parent_check(c.oid, true), c.relname, c.relpages
 from pg_index i
@@ -68,6 +68,8 @@ join pg_namespace n ON c.relnamespace = n.oid
 where am.amname = 'btree'
 -- Don't check temp tables, which may be from another session:
 and c.relpersistence != 't'
+-- Skip partitioned indexes; the indexes of their partitions are checked:
+and c.relkind = 'i'
 -- Function may throw an error when this is omitted:
 and i.indisready AND i.indisvalid;
 ```