Skip to content

ci: Retry the SSH connection to the smoke test VM #1471

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
marcleblanc2 wants to merge 3 commits into from
Closed

ci: Retry the SSH connection to the smoke test VM #1471

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
040f145
ci: Retry the smoke test when its VM stops answering SSH
marcleblanc2 Nov 10, 2026
e611a1f
ci: Retry on a new VM in vagrant-run.sh instead of in Buildkite
marcleblanc2 Nov 10, 2026
cb84ae3
ci: Retry the SSH connection to the same VM instead of making a new VM
marcleblanc2 Nov 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions .buildkite/vagrant-run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,27 @@ export GOOGLE_JSON_KEY_LOCATION="${KEY_PATH}"
export DOCKER_USERNAME=sourcegraph
export DOCKER_PASSWORD="$(gcloud secrets versions access latest --secret=DOCKER_PASSWORD_BUILDKITE_AGENT --quiet --project=sourcegraph-ci)"

ssh_answers() {
vagrant ssh "$box" --no-tty --command true -- -o ConnectTimeout=15
}

echo --- ":vagrant: starting box $box"
vagrant up "$box" --provider=google || exit_code=$?

# vagrant up gives up when the VM stops answering SSH, which is not a test failure.
# Retry the SSH connection to the same VM, then sync the repo and run the test on it
if [ "$exit_code" != 0 ] && ! ssh_answers; then
for attempt in {1..10}; do
echo "$box does not answer SSH, retry $attempt of 10 in 15 seconds"
sleep 15
if ssh_answers; then
exit_code=0
vagrant provision "$box" || exit_code=$?
break
fi
done
fi

if [ "$exit_code" != 0 ]; then
vagrant scp "${box}:/deploy-sourcegraph-docker/*.log" ../
exit $exit_code
Expand Down
Loading