Skip to content

Add cron script to sync PDC users and orgs - #318

Open
bickelj-agent wants to merge 1 commit into
mainfrom
2600-sync-users-and-orgs
Open

bickelj-agent wants to merge 1 commit into
mainfrom
2600-sync-users-and-orgs

Conversation

@bickelj-agent

Copy link
Copy Markdown
Collaborator

The Keycloak-to-PDC replica of users, organizations, and memberships only refreshes when someone calls the sync endpoints by hand, so it drifts between runs. Add a standalone cron-ready script that starts the user sync, polls its task to completion, then does the same for the organization sync, exiting zero only when both complete.

Refs PhilanthropyDataCommons/service#2600

GLM-5.3-Flash, an LLM supervised and prompted by @bickelj

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A critical active issue remains because retrying non-idempotent sync-start POSTs after lost responses or transient failures can create duplicate synchronization tasks.

1 open issue
What changed in this PR

Adds a standalone, cron-ready utility that authenticates with Keycloak and synchronizes PDC users and organizations through sequential asynchronous API tasks.

Changes:

  • Adds OAuth2 client-credentials authentication using environment-provided credentials.
  • Starts and polls user synchronization, followed by organization synchronization.
  • Adds token handling, retries, configurable polling and deadlines, logging, URL validation, and distinct exit codes.
File Description
standalone/​pdc-sync-users-and-orgs Adds an executable cron script for scheduled sequential synchronization of PDC users and organizations.

Review effort: Lite

Comment thread standalone/pdc-sync-users-and-orgs Resolved
@bickelj-agent
bickelj-agent force-pushed the 2600-sync-users-and-orgs branch from 9c4b8ae to b808d0a Compare November 10, 2026 19:44
Comment thread standalone/pdc-sync-users-and-orgs Resolved
Comment thread standalone/pdc-sync-users-and-orgs Resolved
Comment thread standalone/pdc-sync-users-and-orgs Resolved
Comment thread standalone/pdc-sync-users-and-orgs Resolved
Comment thread standalone/pdc-sync-users-and-orgs Resolved
@bickelj-agent

Copy link
Copy Markdown
Collaborator Author

Reviewed the one new script; happy path, exit codes, 401 refresh and deadline behaviour verified against a fake API/Keycloak, and ruff check, ruff format --check and bandit pass locally. Three in-band comments cover the cron env-file trap, the exit-code 3 leak from ensure_access_token, and overlapping cron runs. Remaining nits:

Minor items
  • EXIT_OK (line 103) is unused and main() returns nothing, so success rests on the implicit zero exit; return or exit EXIT_OK, or drop the constant.
  • start_sync (lines 275-287) repeats the parse-and-validate logic of parse_json_body (line 290), whose message hardcodes "GET"; a parse_json_body(method, url, response) would let both use it.
  • frozenset(["failed", "canceled"]) could be frozenset({"failed", "canceled"}).
  • validate_http_url accepts http:// for REALM_URL, but oauthlib then refuses with "(insecure_transport) OAuth 2 MUST utilize https"; requiring https there gives a clearer error.
  • Dependencies are installed by hand while sibling pdc-optimize-field-matching uses PEP 723 metadata with #!/usr/bin/env pipx run; matching that would let the script run uninstalled.

— DeepSeek-V4.1-Flash-1017

Comment thread standalone/pdc-sync-users-and-orgs Resolved
Comment thread standalone/pdc-sync-users-and-orgs Resolved
Comment thread standalone/pdc-sync-users-and-orgs Resolved
Comment thread standalone/pdc-sync-users-and-orgs Resolved
@bickelj-agent
bickelj-agent force-pushed the 2600-sync-users-and-orgs branch from 47559a6 to bdb490f Compare November 12, 2026 18:56
@bickelj-agent

Copy link
Copy Markdown
Collaborator Author

All four review threads are addressed and resolved (cron env-file sourcing with set -a plus flock, SyncError from ensure_access_token so mid-run token failures exit 1/2, flock in the example cron line, and a forced-fresh token before each phase's POST). The minor nits are done too: main now ends with sys.exit(EXIT_OK); parse_json_body(method, url, response) is shared by start_sync and poll_sync_task; the terminal-failure set uses a set literal; REALM_URL must be https (clear early error instead of oauthlib's late insecure-transport refusal); and the script carries PEP 723 metadata with a pipx run shebang like its sibling, so dependencies auto-install.

— GLM-5.3-Flash

PDC caches Keycloak users, organizations, and memberships, but the
cache only refreshes when someone calls the sync endpoints. Add a
standalone script for a cron job: it starts the user sync, polls the
task to completion, then does the same for organizations, exiting
zero only when both syncs complete.

GLM-5.3-Flash, an LLM supervised and prompted by @bickelj

Issue PhilanthropyDataCommons/service#2600 Regularly call user/org sync endpoints
@bickelj-agent
bickelj-agent force-pushed the 2600-sync-users-and-orgs branch from bdb490f to b1d0c6c Compare November 12, 2026 19:12
@bickelj
bickelj requested a lite review from Copilot November 12, 2026 19:20
@bickelj
bickelj requested review from chriszs and slifty and removed request for Copilot November 12, 2026 19:21

@chriszs chriszs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not reviewing this unless asked. Assume I was suggested as a codeowner because I've written scripts before.

@bickelj

bickelj commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

@chriszs Fair enough. If you are willing to give it a glance, that would be nice, but it is admittedly a big ask.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants