Skip to content

Make pending unpostpone observable and cancellable - #1798

Open
Mohith26 wants to merge 2 commits into
github:masterfrom
Mohith26:feat/observable-cancellable-unpostpone
Open

Mohith26 wants to merge 2 commits into
github:masterfrom
Mohith26:feat/observable-cancellable-unpostpone

Conversation

@Mohith26

@Mohith26 Mohith26 commented Nov 4, 2026 •

Copy link
Copy Markdown

A Pull Request should be associated with an Issue.

Related issue: #1685

Description

This PR makes a pending unpostpone visible in status and lets operators withdraw it.

After unpostpone, cut-over still waits (by design, #921) until the changelog heartbeat lag is below both --max-lag-millis and --cut-over-lock-timeout-seconds. Until now status showed the same postponing cut-over whether a request was pending or a previous attempt had failed and reset the flag, and nothing could cancel a request, so tooling that drives cut-over from a UI had no way to observe or give up on it.

  • Status: while a request is pending, State reads postponing cut-over, unpostpone requested, or postponing cut-over, unpostpone requested: waiting for heartbeat lag (4.20s) to drop below 1.50s when lag is what cut-over is waiting on (same state, detail shape as throttled, ...). With no request pending the output is unchanged.
  • cancel-unpostpone: new interactive command that withdraws a pending request; optional table name, like no-throttle. Requests, cancellations and consumption go through MigrationContext helpers behind one mutex, and consuming a request ends postponement in the same critical section, so a request is either consumed by cut-over or cancelled, never both, and a repeated unpostpone is refused once cut-over has begun.
  • The postpone closure in cutOver() is moved into shouldPostponeCutOver() / cutOverHeartbeatLag() with the same logic (lag > min(maxLag, lockTimeout) is equivalent to the old two comparisons), so it can be unit tested.
  • Docs: doc/interactive-commands.md and the help command.

Tests (all unit level, no MySQL needed):

  • TestServerUnpostponeCommands: unpostpone/cancel round trip, cancel with nothing pending, table-name check, help text.
  • TestMigratorShouldPostponeCutOver: a pending request survives high heartbeat lag and is consumed once lag recovers; a cancelled request keeps postponing; state strings.
  • TestMigratorUnpostponeCancelRace: 500 concurrent consume-vs-cancel rounds, exactly one side wins each time.
  • TestMigratorRepeatedUnpostponeAfterConsume / TestMigratorRepeatedUnpostponeRace: a repeated unpostpone after (or racing) consumption is refused and a later cancel reports nothing pending (300 rounds under -race).
  • TestMigratorCutOverPostponeLoop: drives the real sleepWhileTrue loop through unpostpone under lag, cancel, lag recovery (still postponed), and unpostpone again (proceeds).

Ran locally (Apple Silicon): go test -race ./go/... (all pass except the testcontainers suites TestApplier/TestMigrator/TestEventsStreamer, which need Docker and fail the same way on master here), golangci-lint run ./go/... v2.11.4: 0 issues on the first commit; go vet and gofmt -s -l go/ clean on the follow-up. I could not run the localtests replica harness without Docker, so this has not been exercised against a live MySQL.

  • contributed code is using same conventions as original code
  • script/cibuild returns with no formatting errors, build errors or unit test errors (ran the equivalent steps above; Docker-backed suites excepted)

After `unpostpone`, cut-over still waits for the changelog heartbeat lag to
drop below --max-lag-millis and --cut-over-lock-timeout-seconds. Until now
nothing in `status` showed that a request was pending, and there was no way
to withdraw it, so tooling driving cut-over could not tell "waiting on lag"
apart from "attempted, failed and back to postponing", and could not give up.

- `status`/`sup` now report `postponing cut-over, unpostpone requested`,
  plus the current heartbeat lag and limit when that is what cut-over is
  waiting on. The state is unchanged when no request is pending.
- New `cancel-unpostpone` interactive command withdraws a pending request.
  The cut-over loop and the command both use compare-and-swap on
  UserCommandedUnpostponeFlag, so a request is either consumed by cut-over
  or cancelled, never both.
- The postpone check is extracted into shouldPostponeCutOver and
  cutOverHeartbeatLag so it can be unit tested; behavior is otherwise the
  same.

Closes github#1685
Copilot AI auto review requested due to automatic review settings November 4, 2026 13:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A critical race in repeated unpostpone requests can cause cancellation to report success even though cut-over is already proceeding with the postpone file present.

1 open issue
What changed in this PR

Extends gh-ost’s interactive cut-over controls so operators can observe and withdraw pending unpostpone requests, addressing #1685 while preserving heartbeat-lag gating.

Changes:

  • Shows pending requests and heartbeat-lag waits in migration status.
  • Adds cancel-unpostpone with optional table-name validation, help text, and documentation.
  • Extracts postpone checks and adds unit tests for commands, lag handling, cancellation races, and polling behavior.
File Description
go/​logic/​server.go Adds the cancellation command and help entry.
go/​logic/​server_test.go Tests command responses, table validation, and help text.
go/​logic/​migrator.go Extracts postpone checks, atomically consumes requests, and expands status output.
go/​logic/​migrator_test.go Tests lag gating, cancellation races, and the postpone loop.
doc/​interactive-commands.md Documents pending-request status and cancellation behavior.

Review effort: Max (auto)

Comment thread go/logic/server.go Outdated
Comment on lines +451 to +452
if atomic.CompareAndSwapInt64(&srv.migrationContext.UserCommandedUnpostponeFlag, 1, 0) {
fmt.Fprintf(writer, "Unpostpone cancelled; cut-over stays postponed while the postpone flag file exists\n")

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, fixed in b37800e. Requests, cancellations and consumption now go through MigrationContext helpers behind one mutex; consuming a request clears IsPostponingCutOver in the same critical section, and leaving the postpone loop drops any late request. Added TestMigratorRepeatedUnpostponeAfterConsume (fails on the previous commit) and a 300-round TestMigratorRepeatedUnpostponeRace under -race.

A repeated 'unpostpone' could pass the IsPostponingCutOver check, then
re-set the pending flag after the cut-over loop had consumed the first
request. 'cancel-unpostpone' would then report success while cut-over
was already proceeding.

Requests, cancellations and consumption now go through MigrationContext
helpers guarded by one mutex. Consuming a request ends postponement in
the same critical section, and leaving the postpone loop clears any
late request, so a repeated 'unpostpone' is refused once cut-over has
begun and a later cancel reports nothing pending.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants