diff --git a/docs/techsoup.md b/docs/techsoup.md new file mode 100644 index 0000000..47a5337 --- /dev/null +++ b/docs/techsoup.md @@ -0,0 +1,165 @@ +# TechSoup Integration — Technical Report + +**Source file:** [`../src/techsoup.ts`](../src/techsoup.ts) +**Generated:** 2026-09-18 (updated 2026-11-11: single-JSON-file import only; zip and markdown readers removed) +**Purpose:** Import one TechSoup organization, delivered as a single already-mapped PDC JSON file, into the Philanthropy Data Commons (PDC) as changemaker field values — either into an existing changemaker or a newly created one. + +**Modules:** [`../src/techsoup.ts`](../src/techsoup.ts) (CLI) · [`../src/pdc-json.ts`](../src/pdc-json.ts) (JSON reader) · [`../src/fields.ts`](../src/fields.ts) (`TechSoupDocument` as read, `TechSoupPackage` once sorted by category) · [`../src/changemakers.ts`](../src/changemakers.ts) (choosing the changemaker, preview, PDC writes). + +--- + +## 1. Overview + +`techsoup` is a single [yargs](https://yargs.js.org/) command registered in [`../src/index.ts`](../src/index.ts). It takes **one JSON file** as its argument and loads that file's organization fields into **one** PDC changemaker. + +```bash +npm run techsoup -- [--create-changemaker | --changemaker-id ] [--dry-run] --pdc-api-base-url --oidc-base-url --oidc-client-id --oidc-client-secret +``` + +How the target changemaker is chosen: + +| Flags | Target | +| ----------------------- | ------------------------------------------------------------------------------------------------------------------------- | +| `--create-changemaker` | Create a **new** changemaker (`taxId` = file EIN, `name` = `changemaker.name`). Refused if the EIN already exists in PDC. | +| `--changemaker-id ` | Load into the existing changemaker with that ID (the file EIN is not required; a mismatch is warned about). | +| neither | Match the file EIN (`organization_tax_id`) against PDC changemakers; exactly one must match. | + +`--create-changemaker` and `--changemaker-id` are mutually exclusive. + +`--dry-run` prints the target changemaker (ID, name, taxId — or `NEW changemaker`, or why nothing would be imported) and every field that would be imported, then stops. It reads the PDC changemaker list anonymously but does not authenticate or write. + +> The OIDC options are declared as required by the shared `oidcOptions`, so supply them (or the `DS_*` environment variables) even with `--dry-run`; they are only used for a real import. + +--- + +## 2. Input file + +One JSON document per organization — the TechSoup exporter's `_pdc.json` shape: + +```json +{ + "changemaker": { "name": "Example Org" }, + "fields": [ + { "baseFieldShortCode": "organization_name", "value": "Example Org", "batch": { "notes": "README.md#title" } }, + { "baseFieldShortCode": "organization_tax_id", "value": "12-3456789" }, + { "baseFieldShortCode": "organization_dba_name", "value": ["Alias One", "Alias Two"] }, + { "baseFieldShortCode": "proposal_name", "value": "A funding ask" } + ] +} +``` + +- **EIN** — the `organization_tax_id` field value, kept only when it passes `isValidEin` (`^\d{2}-?\d{7}$`). Non-US ids yield no EIN. +- **Name** — `changemaker.name` (needed only when creating a changemaker). +- **Values** — coerced by `toFieldValueString`: strings trimmed (blank dropped), numbers/booleans stringified, arrays joined with `"; "`, objects dropped. +- **Provenance** — `batch.notes` (or the short code) is shown in the dry-run output. +- **`goodAsOf`** — always `null` (the file carries no generation date). +- **Split** — fields are classified by the PDC's own base-field categories (`GET /baseFields`, which needs no token). Only `organization` fields are posted to the changemaker; fields of any other category (`project`, `budget`, …) are proposal fields and are listed but not imported. Short codes that are not base fields in the target PDC are skipped with a warning. +- A file without a `fields` array, or that is not valid JSON, fails with an error. + +--- + +## 3. Data Flow + +``` + + │ + ▼ +readPdcJsonFile ──► TechSoupDocument { slug, ein, name, goodAsOf, fields } + │ + ▼ +GET PDC /baseFields (anonymous) ──► applyBaseFieldCategories ──► TechSoupPackage { …, organizationFields, proposalFields } + │ + ▼ +GET PDC /changemakers (anonymous) + │ + ▼ +planImport ──► update (existing changemaker) | create (new changemaker) | skip (reason) + │ + ├─ --dry-run ──► log formatImportPreview lines and stop + ├─ skip ──────► fail with the reason + ▼ +Authenticate (OIDC client_credentials) + │ + ▼ +find PDC Source "techsoup" (oldest if several; create only if none — 403/404 → stop, nothing written) + │ + ▼ +create changemaker (if planned; 403 → warn and stop) ──► open ChangemakerFieldValueBatch + │ + ▼ +POST each organization field to /changemakerFieldValues (sequentially; 403 → warn and continue) +proposal fields: reported, NOT written (stubbed — see §4.2) +``` + +--- + +## 4. Field Mapping + +The JSON file is already mapped to PDC base-field short codes, so no mapping happens at import time. The tables below record the mapping TechSoup's exporter applies (from its OKF markdown profile), to document what the JSON is expected to contain. Which fields are written is decided by each base field's PDC category, not by these tables. + +### 4.1 Organization fields — written to PDC + +| OKF source (`file#frontmatter.path`) | PDC base field short code | +| ---------------------------------------- | ------------------------------ | +| `README.md#title` | `organization_name` | +| `README.md#description` | `organization_overview` | +| `README.md#aliases` | `organization_dba_name` | +| `README.md#status` | `organization_status` | +| `README.md#x-civic.registration_country` | `organization_country` | +| `README.md#x-civic.registration.id` | `organization_tax_id` | +| `README.md#x-civic.ntee` | `organization_ntee_code` | +| `README.md#x-civic.situation` | `organization_geographic_area` | + +> **Known data-quality caveat (gap analysis §5.1):** `README#status` is an OKF _document_ lifecycle value (`stable`/`draft`), not an organizational status. It is mapped to `organization_status` to match the existing exporter output. + +### 4.2 Proposal fields — reported, **not yet written** + +| OKF source (`file#frontmatter.path`) | PDC base field short code | +| --------------------------------------------------------- | --------------------------------- | +| `impact.md#description` | `proposal_project_outcomes` | +| `programs.md#description` | `proposal_related_programs` | +| `what_i_need_funding_for.md#title` | `proposal_name` | +| `what_i_need_funding_for.md#description` | `proposal_funding_summary` | +| `technical-volunteers/.md#description` | `proposal_learning_and_evalution` | +| `population.md#description` | `community_definition_overview` | + +`community_definition_overview` is a `project`-category base field in the PDC, so it can only be stored with a proposal even though it does not start with `proposal_`. + +PDC proposal data needs an **Opportunity + ApplicationForm + ProposalVersion** scaffold (see [`../src/postProposalVersions.ts`](../src/postProposalVersions.ts)) that a single organization file cannot supply. Until that model is decided, proposal fields are listed by `--dry-run` and warned about by `warnProposalsStubbed` (`TODO(techsoup-proposals)`), but not POSTed. + +--- + +## 5. Error Handling + +| Concern | Handling | +| ------------------------------------------------ | ------------------------------------------------------------------------------------------------- | +| File missing / not JSON / no `fields` array | Error; nothing is read from or written to the PDC | +| No valid EIN and no `--changemaker-id` | Skipped with reason (dry-run prints it; real run fails) | +| EIN not in PDC (default mode) | Skipped; reason suggests `--create-changemaker` | +| EIN matches several changemakers | Skipped; reason suggests `--changemaker-id` | +| `--changemaker-id` not found | Skipped with reason | +| `--changemaker-id` taxId differs from file EIN | Warning; import proceeds into the given ID | +| `--create-changemaker` but EIN already in PDC | Skipped (no duplicate); reason names the existing changemaker | +| `--create-changemaker` but no `changemaker.name` | Skipped with reason | +| HTTP 403 creating the changemaker | Warned; nothing imported | +| HTTP 403 on a field-value write | Warned; remaining fields continue; summary warning at end | +| PDC concurrency timeouts | Field values POSTed sequentially | +| No TechSoup source | Attempt to create (usually admin-only). 403/404 → clear "ask a pdc-admin" error, before any write | + +--- + +## 6. Testing + +Unit tests in [`../src/techsoup.unit.test.ts`](../src/techsoup.unit.test.ts) cover: + +- **`readPdcJsonFile` / `parsePdcJsonDocument`** — reads one file (slug from the file name), keeps every usable field in file order, and rejects non-PDC-shaped or invalid JSON. +- **`pdcJsonToDocument`** — EIN from `organization_tax_id` (null for non-US), name from `changemaker.name`, array flattening, `batch.notes` provenance, blank values dropped. +- **`toChangemakerTarget`** — maps `--create-changemaker` / `--changemaker-id` / neither to a target. +- **`planImport`** — every branch: EIN match/missing/ambiguous/absent; explicit ID found/missing/without EIN; create new/duplicate/ambiguous/no name/no EIN. +- **`formatImportPreview`** — the dry-run output names the target changemaker (existing or new), lists each field to import, lists proposal fields separately, and prints only the reason on skip. +- **`applyBaseFieldCategories`** — keeps only `organization`-category fields, moves other categories to the proposal fields, drops unknown short codes. +- **`selectTechSoupSource`** — one, none, or several TechSoup sources. +- **`toLoggableError`** — HTTP errors are logged without request headers or tokens. +- **`resolveChangemaker`**, **`toFieldValueString`**. + +The PDC-facing write path (`runImportWrite`) and the command handler are not unit-tested; the decision logic is extracted into the pure `planImport` / `formatImportPreview` helpers above. diff --git a/package-lock.json b/package-lock.json index 4141393..4226a59 100644 --- a/package-lock.json +++ b/package-lock.json @@ -807,6 +807,29 @@ "url": "https://github.com/sponsors/sindresorhus" } }, + "node_modules/@eslint/eslintrc/node_modules/js-yaml": { + "version": "4.3.2", + "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.3.2.tgz", + "integrity": "sha512-SFNOvSJ+Dgf/9An904Yx+CgSlIPCkIpao4qo51lpee25TIRejdH3rhR4EZMGoNx3/TP3O+wzWuiTFl4sqbltzA==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/puzrin" + }, + { + "type": "github", + "url": "https://github.com/sponsors/nodeca" + } + ], + "license": "MIT", + "dependencies": { + "argparse": "^2.0.1" + }, + "bin": { + "js-yaml": "bin/js-yaml.js" + } + }, "node_modules/@eslint/js": { "version": "9.39.5", "resolved": "https://registry.npmjs.org/@eslint/js/-/js-9.39.5.tgz", @@ -6443,29 +6466,6 @@ "dev": true, "license": "MIT" }, - "node_modules/js-yaml": { - "version": "4.3.0", - "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.3.0.tgz", - "integrity": "sha512-1td788aAnnZ5qs7V2QIRl1owjtYpbKt749Y3xauqQgwIIGF/xXWz1wMTEBx5O3LK3lXLVuqXPdPxj2BoFHaW9Q==", - "dev": true, - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/puzrin" - }, - { - "type": "github", - "url": "https://github.com/sponsors/nodeca" - } - ], - "license": "MIT", - "dependencies": { - "argparse": "^2.0.1" - }, - "bin": { - "js-yaml": "bin/js-yaml.js" - } - }, "node_modules/jsesc": { "version": "3.1.0", "resolved": "https://registry.npmjs.org/jsesc/-/jsesc-3.1.0.tgz", diff --git a/package.json b/package.json index d621a09..d6ea915 100644 --- a/package.json +++ b/package.json @@ -22,7 +22,8 @@ "postProposalVersions": "node --import ./register.mjs src/postProposalVersions.ts", "charityNavigator": "node --import ./register.mjs src/index.ts charityNavigator", "givingTuesday": "node --import ./register.mjs src/index.ts givingTuesday", - "getMetrics": "node --import ./register.mjs src/index.ts getMetrics" + "getMetrics": "node --import ./register.mjs src/index.ts getMetrics", + "techsoup": "node --import ./register.mjs src/index.ts techsoup" }, "author": "Open Tech Strategies", "license": "AGPL-3.0-or-later", diff --git a/src/changemakers.ts b/src/changemakers.ts new file mode 100644 index 0000000..9ced5fa --- /dev/null +++ b/src/changemakers.ts @@ -0,0 +1,433 @@ +import { AxiosError } from 'axios'; +import { logger } from './logger.js'; +import { type AccessTokenSet, getToken } from './oidc.js'; +import { + getSources, + postChangemaker, + postChangemakerFieldValue, + postChangemakerFieldValueBatch, + postSource, + type WritableChangemakerFieldValue, +} from './pdc-api.js'; +import type { MappedField, TechSoupDocument, TechSoupPackage } from './fields.js'; +import type { Changemaker, ChangemakerBundle, Source, SourceBundle } from '@pdc/sdk'; + +const TS_SHORT_CODE = 'techsoup'; +// When `@pdc/http-status-codes` is ready (issues 18-20 solved), use it instead. +const HTTP_STATUS_FORBIDDEN = 403; +const HTTP_STATUS_NOT_FOUND = 404; + +// --------------------------------------------------------------------------- +// EIN → changemaker matching +// --------------------------------------------------------------------------- + +/** All changemakers whose (hyphen-insensitive) taxId equals the given EIN. */ +const findChangemakersByEin = (ein: string, changemakers: ChangemakerBundle): Changemaker[] => { + const normalized = ein.replace('-', ''); + return changemakers.entries.filter((c) => c.taxId.replace('-', '') === normalized); +}; + +/** + * The outcome of resolving an EIN against the PDC changemakers: a unique match + * (`matched`), several matches (`ambiguous` — never auto-created), or none + * (`missing` — a create candidate when `--create-changemaker` is set). + */ +type ChangemakerResolution = + { kind: 'matched'; changemaker: Changemaker } | { kind: 'ambiguous' } | { kind: 'missing' }; + +const resolveChangemaker = (ein: string, changemakers: ChangemakerBundle): ChangemakerResolution => { + const matches = findChangemakersByEin(ein, changemakers); + if (matches.length > 1) { + return { kind: 'ambiguous' }; + } + const [match] = matches; + if (match !== undefined) { + return { kind: 'matched', changemaker: match }; + } + return { kind: 'missing' }; +}; + +// --------------------------------------------------------------------------- +// PDC writes (source, changemaker creation, field values) +// --------------------------------------------------------------------------- + +/** Light wrapper around `postChangemakerFieldValue` that logs a warning on HTTP 403. */ +const postChangemakerFieldValueWarnOnForbidden = async ( + baseUrl: string, + token: AccessTokenSet, + data: WritableChangemakerFieldValue, + warnedChangemakers: Set, // Mutated! This is for observation/logs, not control! +): Promise => { + try { + const fieldValue = await postChangemakerFieldValue(baseUrl, token, data); + logger.info(`Added changemaker field value: ${JSON.stringify(fieldValue)}`); + } catch (e: unknown) { + if (e instanceof AxiosError && e.status === HTTP_STATUS_FORBIDDEN) { + logger.warn(`No permission (403) to create ${JSON.stringify(data)}`); + warnedChangemakers.add(data.changemakerId); + } else { + throw e; + } + } +}; + +/** + * Pick the TechSoup source (`dataProviderShortCode === 'techsoup'`) from the PDC + * sources, or null when there is none. When several exist, the oldest (lowest + * ID) is used, with a warning, rather than creating yet another. + */ +const selectTechSoupSource = (sources: SourceBundle): Source | null => { + const techSoupSources = sources.entries + .filter((s) => s.dataProviderShortCode === TS_SHORT_CODE) + .sort((a, b) => a.id - b.id); + const [oldest] = techSoupSources; + if (oldest === undefined) { + return null; + } + if (techSoupSources.length > 1) { + logger.warn( + `Found ${techSoupSources.length} TechSoup sources (${techSoupSources.map((s) => s.id).join(', ')}); using source ${oldest.id}`, + ); + } + return oldest; +}; + +/** + * Find the TechSoup source, creating it only when none exists. Creating needs + * the `techsoup` data provider to exist and permission to add sources to it + * (typically a `pdc-admin`); a 403/404 from the PDC is reported as that setup + * problem instead of a raw HTTP error. + */ +const getOrCreateSource = async (baseUrl: string, token: AccessTokenSet): Promise => { + const sources = await getSources(baseUrl, token); + logger.debug( + { sources: sources.entries.map((s) => ({ id: s.id, dataProviderShortCode: s.dataProviderShortCode })) }, + `GET /sources returned ${sources.entries.length} source(s)`, + ); + const existing = selectTechSoupSource(sources); + if (existing !== null) { + return existing; + } + logger.warn('No TechSoup source found in the PDC; trying to create one (usually requires a `pdc-admin`).'); + try { + return await postSource(baseUrl, token, { + dataProviderShortCode: TS_SHORT_CODE, + label: 'TechSoup', + }); + } catch (e: unknown) { + if (e instanceof AxiosError && (e.status === HTTP_STATUS_NOT_FOUND || e.status === HTTP_STATUS_FORBIDDEN)) { + const problem = + e.status === HTTP_STATUS_NOT_FOUND + ? `the "${TS_SHORT_CODE}" data provider does not exist in this PDC` + : `this account may not create sources for the "${TS_SHORT_CODE}" data provider`; + throw new Error( + `Cannot create the TechSoup source (HTTP ${e.status}): ${problem}. Ask a pdc-admin to create the "${TS_SHORT_CODE}" data provider and its source, then re-run.`, + { cause: e }, + ); + } + throw e; + } +}; + +/** + * Create a new PDC changemaker for a package whose EIN is not yet in the PDC. + * Requires the package name; the EIN is stored verbatim (hyphenated) as the + * taxId. Returns null (with a warning) when the name is missing or the create is + * forbidden (403 — creating changemakers is privileged). + */ +const createChangemakerForPackage = async ( + baseUrl: string, + token: AccessTokenSet, + ein: string, + pkg: TechSoupPackage, +): Promise => { + if (pkg.name === null) { + logger.warn(`Cannot create a changemaker for ${pkg.slug}: no organization name. Skipping.`); + return null; + } + try { + const changemaker = await postChangemaker(baseUrl, token, { taxId: ein, name: pkg.name }); + logger.info({ id: changemaker.id, taxId: ein, name: pkg.name }, `Created changemaker for ${pkg.slug}`); + return changemaker; + } catch (e: unknown) { + if (e instanceof AxiosError && e.status === HTTP_STATUS_FORBIDDEN) { + logger.warn(`No permission (403) to create a changemaker for ${pkg.slug} (taxId ${ein}). Skipping.`); + return null; + } + throw e; + } +}; + +/** + * Proposal fields are collected but not yet written. PDC proposal data requires + * an Opportunity + ApplicationForm + ProposalVersion scaffold that is out of + * scope for this pass; surface what would be written so it is visible. + */ +const warnProposalsStubbed = (slug: string, proposalFields: MappedField[]): void => { + if (proposalFields.length === 0) { + return; + } + // TODO(techsoup-proposals): write these as PDC proposal data (Opportunity + + // ApplicationForm + ProposalVersion, cf. postProposalVersions.ts). Until the + // proposal model is decided, proposal fields are reported but NOT written. + logger.warn( + { slug, proposalFields: proposalFields.map((f) => f.baseFieldShortCode) }, + `Collected ${proposalFields.length} proposal field(s) — NOT written (stubbed, pending proposal model)`, + ); +}; + +/** + * Sort the document's fields by the PDC's own base-field categories. Only + * `organization` fields may be posted to `/changemakerFieldValues`; fields of + * any other category (`project`, `budget`, …) belong to a proposal and go to + * `proposalFields`. Fields with no matching base field in this PDC are dropped + * with a warning, since the PDC would reject them. + */ +const applyBaseFieldCategories = ( + doc: TechSoupDocument, + baseFields: { entries: ReadonlyArray<{ shortCode: string; category: string }> }, +): TechSoupPackage => { + const { fields, ...pkg } = doc; + const categoryByShortCode = new Map(baseFields.entries.map((b) => [b.shortCode, b.category])); + const organizationFields: MappedField[] = []; + const proposalFields: MappedField[] = []; + const unknown: string[] = []; + for (const field of fields) { + const category = categoryByShortCode.get(field.baseFieldShortCode); + if (category === undefined) { + unknown.push(field.baseFieldShortCode); + } else if (category === 'organization') { + organizationFields.push(field); + } else { + proposalFields.push(field); + } + } + if (unknown.length > 0) { + logger.warn( + `${pkg.slug}: ${unknown.length} field(s) are not base fields in this PDC; skipping ${unknown.join(', ')}`, + ); + } + return { ...pkg, organizationFields, proposalFields }; +}; + +// --------------------------------------------------------------------------- +// Import plan: which changemaker the package is loaded into +// --------------------------------------------------------------------------- + +/** + * How the target changemaker is chosen: create a new one (`create`), use an + * explicit PDC changemaker ID (`id`), or match the package EIN against the + * existing changemakers (`ein`). + */ +type ChangemakerTarget = { kind: 'create' } | { kind: 'id'; changemakerId: number } | { kind: 'ein' }; + +/** Map the CLI flags to a `ChangemakerTarget` (the CLI rejects combining the two flags). */ +const toChangemakerTarget = (createChangemaker: boolean, changemakerId: number | undefined): ChangemakerTarget => { + if (createChangemaker) { + return { kind: 'create' }; + } + if (changemakerId !== undefined) { + return { kind: 'id', changemakerId }; + } + return { kind: 'ein' }; +}; + +/** + * What an import will do: load fields into an existing changemaker (`update`), + * create a changemaker and then load fields into it (`create`), or do nothing + * (`skip`, with the reason). + */ +type ImportPlan = + | { action: 'update'; changemaker: Changemaker } + | { action: 'create'; taxId: string; name: string } + | { action: 'skip'; reason: string }; + +const planForExplicitId = ( + pkg: TechSoupPackage, + changemakers: ChangemakerBundle, + changemakerId: number, +): ImportPlan => { + const changemaker = changemakers.entries.find((c) => c.id === changemakerId); + if (changemaker === undefined) { + return { action: 'skip', reason: `No changemaker with ID ${changemakerId} exists in the PDC` }; + } + if (pkg.ein !== null && pkg.ein.replace('-', '') !== changemaker.taxId.replace('-', '')) { + logger.warn( + `${pkg.slug}: file EIN ${pkg.ein} differs from changemaker ${changemakerId} taxId ${changemaker.taxId}; using --changemaker-id as given`, + ); + } + return { action: 'update', changemaker }; +}; + +const planForCreate = (pkg: TechSoupPackage, ein: string, resolution: ChangemakerResolution): ImportPlan => { + if (resolution.kind === 'matched') { + return { + action: 'skip', + reason: `EIN ${ein} already belongs to changemaker ${resolution.changemaker.id}; rerun without --create-changemaker to load into it`, + }; + } + if (resolution.kind === 'ambiguous') { + return { action: 'skip', reason: `EIN ${ein} already matches multiple changemakers; not creating another` }; + } + if (pkg.name === null) { + return { action: 'skip', reason: `${pkg.slug} has no organization name, which a new changemaker requires` }; + } + return { action: 'create', taxId: ein, name: pkg.name }; +}; + +const planForEin = (ein: string, resolution: ChangemakerResolution): ImportPlan => { + if (resolution.kind === 'matched') { + return { action: 'update', changemaker: resolution.changemaker }; + } + if (resolution.kind === 'ambiguous') { + return { action: 'skip', reason: `EIN ${ein} matches multiple changemakers; pass --changemaker-id to choose one` }; + } + return { + action: 'skip', + reason: `EIN ${ein} not found in the PDC; pass --create-changemaker to create it or --changemaker-id to choose one`, + }; +}; + +/** + * Decide which changemaker the package goes to, without writing anything. + * `--create-changemaker` refuses to create a duplicate of an EIN already in the + * PDC; `--changemaker-id` must name an existing changemaker; otherwise the + * package EIN must match exactly one changemaker. + */ +const planImport = (pkg: TechSoupPackage, changemakers: ChangemakerBundle, target: ChangemakerTarget): ImportPlan => { + if (target.kind === 'id') { + return planForExplicitId(pkg, changemakers, target.changemakerId); + } + if (pkg.ein === null) { + return { + action: 'skip', + reason: `${pkg.slug} has no valid IRS EIN (organization_tax_id); pass --changemaker-id to choose a changemaker`, + }; + } + const resolution = resolveChangemaker(pkg.ein, changemakers); + return target.kind === 'create' ? planForCreate(pkg, pkg.ein, resolution) : planForEin(pkg.ein, resolution); +}; + +/** One line naming the changemaker the plan targets (or why nothing will be imported). */ +const describeTarget = (plan: ImportPlan): string => { + if (plan.action === 'update') { + return `changemaker ${plan.changemaker.id} (${JSON.stringify(plan.changemaker.name)}, taxId ${plan.changemaker.taxId})`; + } + if (plan.action === 'create') { + return `NEW changemaker (${JSON.stringify(plan.name)}, taxId ${plan.taxId})`; + } + return `nothing — ${plan.reason}`; +}; + +/** + * The `--dry-run` report: the target changemaker, every organization field that + * would be imported into it, and the proposal fields that are present but not + * imported. Pure, so it can be unit-tested and logged line by line. + */ +const formatImportPreview = (pkg: TechSoupPackage, plan: ImportPlan): string[] => { + const lines = [`${pkg.slug} → ${describeTarget(plan)}`]; + if (plan.action === 'skip') { + return lines; + } + lines.push(`Fields that would be imported (${pkg.organizationFields.length}, goodAsOf=${pkg.goodAsOf ?? 'null'}):`); + for (const field of pkg.organizationFields) { + lines.push(` ${field.baseFieldShortCode} = ${JSON.stringify(field.value)} [${field.sourceNote}]`); + } + if (pkg.proposalFields.length > 0) { + lines.push(`Proposal (non-organization) fields NOT imported (${pkg.proposalFields.length}):`); + for (const field of pkg.proposalFields) { + lines.push(` ${field.baseFieldShortCode} = ${JSON.stringify(field.value)} [${field.sourceNote}]`); + } + } + return lines; +}; + +// --------------------------------------------------------------------------- +// Import write path +// --------------------------------------------------------------------------- + +/** Base URL + OIDC credentials needed to start a write run (before authentication). */ +interface ImportWriteContext { + pdcApiBaseUrl: string; + oidcBaseUrl: string; + oidcClientId: string; + oidcClientSecret: string; +} + +/** An authenticated write session: where to POST, with what token, into which field-value batch. */ +interface WriteSession { + baseUrl: string; + token: AccessTokenSet; + batchId: number; +} + +/** POST every organization field of the package to one changemaker, sequentially (proposals stay stubbed). */ +const postOrganizationFields = async ( + session: WriteSession, + changemaker: Changemaker, + pkg: TechSoupPackage, +): Promise => { + warnProposalsStubbed(pkg.slug, pkg.proposalFields); + const warnedChangemakers = new Set(); + /* eslint-disable no-await-in-loop -- sequential POSTs avoid PDC API connection saturation. */ + for (const field of pkg.organizationFields) { + await postChangemakerFieldValueWarnOnForbidden( + session.baseUrl, + session.token, + { + changemakerId: changemaker.id, + batchId: session.batchId, + baseFieldShortCode: field.baseFieldShortCode, + value: field.value, + goodAsOf: pkg.goodAsOf, + }, + warnedChangemakers, + ); + } + /* eslint-enable no-await-in-loop */ + if (warnedChangemakers.size > 0) { + logger.warn(`No permission for at least one field of changemaker ${changemaker.id} (so not fully updated)`); + } +}; + +/** + * Carry out an `update`/`create` plan: authenticate, resolve the TechSoup + * source, create the changemaker if planned, open a field-value batch, then + * POST the organization fields. The source is resolved first so a missing + * source fails the run before anything (such as a new changemaker) is written. + */ +const runImportWrite = async ( + pkg: TechSoupPackage, + plan: Exclude, + ctx: ImportWriteContext, +): Promise => { + // Up to this point we didn't need PDC authentication. Now we do. + const token = await getToken(ctx.oidcBaseUrl, ctx.oidcClientId, ctx.oidcClientSecret); + const source = await getOrCreateSource(ctx.pdcApiBaseUrl, token); + logger.info(source, 'The PDC Source for TechSoup was found'); + const changemaker = + plan.action === 'create' + ? await createChangemakerForPackage(ctx.pdcApiBaseUrl, token, plan.taxId, pkg) + : plan.changemaker; + if (changemaker === null) { + return; + } + const fieldBatch = await postChangemakerFieldValueBatch(ctx.pdcApiBaseUrl, token, { + sourceId: source.id, + notes: `data-scripts techsoup.ts import of ${pkg.slug} ${Date.now()}`, + }); + await postOrganizationFields({ baseUrl: ctx.pdcApiBaseUrl, token, batchId: fieldBatch.id }, changemaker, pkg); + logger.info(`Imported ${pkg.organizationFields.length} field(s) into changemaker ${changemaker.id}`); +}; + +export { + type ChangemakerTarget, + type ImportPlan, + applyBaseFieldCategories, + formatImportPreview, + planImport, + resolveChangemaker, + runImportWrite, + selectTechSoupSource, + toChangemakerTarget, +}; diff --git a/src/fields.ts b/src/fields.ts new file mode 100644 index 0000000..a66d365 --- /dev/null +++ b/src/fields.ts @@ -0,0 +1,54 @@ +// An array-valued field in a `_pdc.json` is flattened into a single PDC string +// with this separator, since a ChangemakerFieldValue.value is a scalar string. +const ARRAY_VALUE_SEPARATOR = '; '; + +/** A resolved value ready to be posted as a PDC field, with a provenance note. */ +interface MappedField { + baseFieldShortCode: string; + value: string; + sourceNote: string; +} + +/** + * One organization as read from a `_pdc.json` file, before its fields are + * sorted by PDC base-field category. `ein`/`name` may be null; `goodAsOf` is + * currently always null because the JSON carries no generation date. + */ +interface TechSoupDocument { + slug: string; + ein: string | null; + name: string | null; + goodAsOf: string | null; + fields: MappedField[]; +} + +/** + * A document whose fields have been sorted by the PDC's base-field categories + * (see `applyBaseFieldCategories`): what the importer loads into the PDC. + * Only `organizationFields` are written; `proposalFields` are reported. + */ +interface TechSoupPackage extends Omit { + organizationFields: MappedField[]; + proposalFields: MappedField[]; +} + +/** Coerce an arbitrary source value to a single trimmed PDC field string (arrays joined), or null. */ +const toFieldValueString = (value: unknown): string | null => { + if (value === null || value === undefined) { + return null; + } + if (typeof value === 'string') { + const trimmed = value.trim(); + return trimmed === '' ? null : trimmed; + } + if (typeof value === 'number' || typeof value === 'boolean') { + return String(value); + } + if (Array.isArray(value)) { + const parts = value.map((item) => toFieldValueString(item)).filter((item): item is string => item !== null); + return parts.length === 0 ? null : parts.join(ARRAY_VALUE_SEPARATOR); + } + return null; +}; + +export { type MappedField, type TechSoupDocument, type TechSoupPackage, toFieldValueString }; diff --git a/src/index.ts b/src/index.ts index 782e16d..0b86349 100644 --- a/src/index.ts +++ b/src/index.ts @@ -4,9 +4,10 @@ import { hideBin } from 'yargs/helpers'; import { candid } from './candid.js'; import { charityNavigator } from './charityNavigator.js'; import { getMetrics } from './getMetrics.js'; -import { givingTuesday } from './gtdc990.js'; -import { logger } from './logger.js'; +import { givingTuesday } from './givingTuesday.js'; +import { logger, toLoggableError } from './logger.js'; import { getTokenCommand } from './oidc.js'; +import { techsoup } from './techsoup.js'; config(); @@ -19,7 +20,7 @@ const main = async (argv: string[]): Promise => { .env('DS') .config('config') .fail((msg, err, y) => { - logger.fatal({ m: msg, err, yargs: y }, 'Error encountered'); + logger.fatal({ m: msg, err: toLoggableError(err), yargs: y }, 'Error encountered'); }) .command( ['show-args', 'args'], @@ -37,11 +38,12 @@ const main = async (argv: string[]): Promise => { .command(charityNavigator) .command(getMetrics) .command(givingTuesday) + .command(techsoup) .demandCommand() .parse(); }; main(process.argv).catch((err: unknown) => { - logger.fatal({ err }, 'Error encountered while running `main`'); + logger.fatal({ err: toLoggableError(err) }, 'Error encountered while running `main`'); process.exit(1); }); diff --git a/src/logger.ts b/src/logger.ts index ffc4e53..83edc24 100644 --- a/src/logger.ts +++ b/src/logger.ts @@ -1,3 +1,4 @@ +import { AxiosError } from 'axios'; import pino from 'pino'; const logger = pino({ @@ -11,4 +12,33 @@ const logger = pino({ }, }); -export { logger }; +/** + * Reduce an error to what is useful in a log. An AxiosError otherwise serializes + * its whole request — including the `Authorization: Bearer …` header — so only + * its status, method, URL, and response body are kept. Plain errors keep their + * name, message, and stack, and any `cause` is reduced the same way. + */ +const toLoggableError = (err: unknown): unknown => { + if (err instanceof AxiosError) { + return { + name: err.name, + message: err.message, + code: err.code, + status: err.status, + method: err.config?.method, + url: err.config?.url, + responseData: err.response?.data as unknown, + }; + } + if (err instanceof Error) { + return { + name: err.name, + message: err.message, + stack: err.stack, + ...(err.cause === undefined ? {} : { cause: toLoggableError(err.cause) }), + }; + } + return err; +}; + +export { logger, toLoggableError }; diff --git a/src/pdc-api.ts b/src/pdc-api.ts index 3c83750..20734c0 100644 --- a/src/pdc-api.ts +++ b/src/pdc-api.ts @@ -1,6 +1,14 @@ import { client } from './client.js'; import type { AccessTokenSet } from './oidc.js'; -import type { BaseField, ProposalBundle, ChangemakerBundle, SourceBundle, Source, BaseFieldBundle } from '@pdc/sdk'; +import type { + BaseField, + Changemaker, + ProposalBundle, + ChangemakerBundle, + SourceBundle, + Source, + BaseFieldBundle, +} from '@pdc/sdk'; const callPdcApi = async ( baseUrl: string, @@ -20,7 +28,8 @@ const callPdcApi = async ( return response.data; }; -const getBaseFields = async (baseUrl: string, token: AccessTokenSet): Promise => +/** Get all base fields. `GET /baseFields` is public, so the token is optional. */ +const getBaseFields = async (baseUrl: string, token?: AccessTokenSet): Promise => await callPdcApi(baseUrl, '/baseFields', 'get', { params: { _page: '1', @@ -71,6 +80,23 @@ export interface WritableSource { const postSource = async (baseUrl: string, token: AccessTokenSet, data: WritableSource): Promise => await callPdcApi(baseUrl, '/sources', 'post', { params: {}, token, data }); +/** + * The POST /changemakers body. The SDK's `Writable` also pulls in + * `fields`/`fiscalSponsors`, which the create endpoint does not accept; only + * `taxId` and `name` are required (`keycloakOrganizationId` is optional). + */ +export interface WritableChangemaker { + taxId: string; + name: string; + keycloakOrganizationId?: string; +} + +const postChangemaker = async ( + baseUrl: string, + token: AccessTokenSet, + data: WritableChangemaker, +): Promise => await callPdcApi(baseUrl, '/changemakers', 'post', { params: {}, token, data }); + // TODO: use the SDK, delete these temp types copied from the service repo interface ChangemakerFieldValueBatch { readonly id: number; @@ -138,6 +164,7 @@ export { getChangemakers, getProposals, getSources, + postChangemaker, postChangemakerFieldValueBatch, postChangemakerFieldValue, postPlatformProviderData, diff --git a/src/pdc-json.ts b/src/pdc-json.ts new file mode 100644 index 0000000..7edaa91 --- /dev/null +++ b/src/pdc-json.ts @@ -0,0 +1,93 @@ +import { readFile } from 'node:fs/promises'; +import { basename } from 'node:path'; +import { isValidEin } from './ein.js'; +import { type MappedField, type TechSoupDocument, toFieldValueString } from './fields.js'; + +const isRecord = (value: unknown): value is Record => + value !== null && typeof value === 'object' && !Array.isArray(value); + +/** The provenance note for a `_pdc.json` field: its `batch.notes`, or null. */ +const extractFieldNote = (field: Record): string | null => { + const { batch } = field; + if (isRecord(batch) && typeof batch.notes === 'string' && batch.notes.trim() !== '') { + return batch.notes; + } + return null; +}; + +/** The changemaker name carried by a `_pdc.json` document (`changemaker.name`), or null. */ +const extractChangemakerName = (raw: Record): string | null => { + const { changemaker } = raw; + if (isRecord(changemaker)) { + return toFieldValueString(changemaker.name); + } + return null; +}; + +/** Convert one `_pdc.json` field entry to a `MappedField`, or null when malformed/empty. */ +const toMappedField = (field: unknown): MappedField | null => { + if (!isRecord(field) || typeof field.baseFieldShortCode !== 'string') { + return null; + } + const value = toFieldValueString(field.value); + if (value === null) { + return null; + } + return { + baseFieldShortCode: field.baseFieldShortCode, + value, + sourceNote: extractFieldNote(field) ?? field.baseFieldShortCode, + }; +}; + +/** The usable entries of a `_pdc.json` `fields[]` array (malformed or empty ones are dropped). */ +const parsePdcJsonFields = (raw: unknown): MappedField[] => { + if (!isRecord(raw) || !Array.isArray(raw.fields)) { + return []; + } + return raw.fields.map(toMappedField).filter((field): field is MappedField => field !== null); +}; + +/** + * Convert one parsed `_pdc.json` document into a `TechSoupDocument`. The EIN is + * taken from the `organization_tax_id` field and only kept when it is a valid + * US EIN (so non-US ids like Poland's `0000000000` yield null). + * `goodAsOf` is always null — the exporter output carries no generation date. + */ +const pdcJsonToDocument = (raw: unknown, slug: string): TechSoupDocument => { + const fields = parsePdcJsonFields(raw); + const taxIdValue = fields.find((f) => f.baseFieldShortCode === 'organization_tax_id')?.value ?? null; + return { + slug, + ein: taxIdValue !== null && isValidEin(taxIdValue) ? taxIdValue : null, + name: isRecord(raw) ? extractChangemakerName(raw) : null, + goodAsOf: null, + fields, + }; +}; + +/** The slug for a `_pdc.json` file: its basename with the `_pdc.json`/`.json` suffix removed. */ +const slugForJsonFile = (filePath: string): string => + basename(filePath) + .replace(/_pdc\.json$/iv, '') + .replace(/\.json$/iv, ''); + +/** + * Validate one parsed JSON value as a TechSoup PDC JSON document and normalize + * it. Throws when it is not `{ changemaker?, fields: [...] }` shaped, so a wrong + * file fails loudly instead of importing nothing. + */ +const parsePdcJsonDocument = (raw: unknown, slug: string): TechSoupDocument => { + if (!isRecord(raw) || !Array.isArray(raw.fields)) { + throw new Error(`${slug} is not a TechSoup PDC JSON document (expected an object with a "fields" array)`); + } + return pdcJsonToDocument(raw, slug); +}; + +/** Read and normalize the single TechSoup PDC JSON file at `filePath` (the import's only input). */ +const readPdcJsonFile = async (filePath: string): Promise => { + const content = await readFile(filePath, 'utf8'); + return parsePdcJsonDocument(JSON.parse(content), slugForJsonFile(filePath)); +}; + +export { parsePdcJsonDocument, pdcJsonToDocument, readPdcJsonFile }; diff --git a/src/techsoup.ts b/src/techsoup.ts new file mode 100644 index 0000000..c1646a6 --- /dev/null +++ b/src/techsoup.ts @@ -0,0 +1,78 @@ +import { + applyBaseFieldCategories, + formatImportPreview, + planImport, + runImportWrite, + toChangemakerTarget, +} from './changemakers.js'; +import { logger } from './logger.js'; +import { oidcOptions } from './oidc.js'; +import { getBaseFields, getChangemakers } from './pdc-api.js'; +import { readPdcJsonFile } from './pdc-json.js'; +import type { CommandModule } from 'yargs'; + +// The importer reads exactly one TechSoup PDC JSON file per run. + +interface ImportCommandArgs { + file: string; + 'pdc-api-base-url': string; + 'create-changemaker'?: boolean; + 'changemaker-id'?: number; + 'dry-run': boolean; + 'oidc-base-url': string; + 'oidc-client-id': string; + 'oidc-client-secret': string; +} + +const techsoup: CommandModule = { + command: 'techsoup ', + describe: 'Import one TechSoup PDC JSON file into a PDC changemaker', + builder: { + ...oidcOptions, + // Filled from the `` positional in `command`. + file: { describe: 'Path to the TechSoup PDC JSON file to import', normalize: true, type: 'string' }, + 'pdc-api-base-url': { describe: 'Location of PDC API', demandOption: true, type: 'string' }, + 'create-changemaker': { + describe: 'Create a new changemaker from the file (EIN + name); fails if the EIN already exists in the PDC', + // No `default`: yargs counts a defaulted flag as "set" and would trip `conflicts`. + type: 'boolean', + conflicts: 'changemaker-id', + }, + 'changemaker-id': { + describe: 'Load into the existing changemaker with this ID (default: match the file EIN)', + demandOption: false, + type: 'number', + }, + 'dry-run': { + describe: 'Print the target changemaker and the fields that would be imported; write nothing', + default: false, + type: 'boolean', + }, + }, + handler: async (args): Promise => { + // Split fields by the PDC's base-field categories: only `organization` + // fields can be posted to a changemaker. + const pkg = applyBaseFieldCategories(await readPdcJsonFile(args.file), await getBaseFields(args.pdcApiBaseUrl)); + const changemakers = await getChangemakers(args.pdcApiBaseUrl); + const plan = planImport( + pkg, + changemakers, + toChangemakerTarget(args.createChangemaker === true, args.changemakerId), + ); + + if (args.dryRun) { + logger.info('[dry-run] No data will be written to the PDC.'); + for (const line of formatImportPreview(pkg, plan)) { + logger.info(line); + } + return; + } + + if (plan.action === 'skip') { + throw new Error(`Not importing ${args.file}: ${plan.reason}`); + } + await runImportWrite(pkg, plan, args); + }, +}; + +export { techsoup }; diff --git a/src/techsoup.unit.test.ts b/src/techsoup.unit.test.ts new file mode 100644 index 0000000..a5d913f --- /dev/null +++ b/src/techsoup.unit.test.ts @@ -0,0 +1,485 @@ +import { mkdtemp, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { afterAll, beforeAll, describe, expect, it } from '@jest/globals'; +import { AxiosError, AxiosHeaders } from 'axios'; +import { + applyBaseFieldCategories, + formatImportPreview, + planImport, + resolveChangemaker, + selectTechSoupSource, + toChangemakerTarget, +} from './changemakers.js'; +import { toFieldValueString } from './fields.js'; +import { toLoggableError } from './logger.js'; +import { parsePdcJsonDocument, pdcJsonToDocument, readPdcJsonFile } from './pdc-json.js'; +import type { TechSoupDocument, TechSoupPackage } from './fields.js'; +import type { Changemaker, ChangemakerBundle, Source } from '@pdc/sdk'; + +// Minimal Changemaker fixture; the helpers under test only read `id`/`taxId`, +// but the SDK type requires the full shape, so fill the rest with inert defaults. +const makeChangemaker = (id: number, taxId: string): Changemaker => ({ + id, + taxId, + name: `Org ${id}`, + createdAt: '2024-01-01T00:00:00Z', + createdBy: 'test', + fiscalSponsors: [], + fields: [], +}); + +// Minimal Source fixture; `selectTechSoupSource` only reads `id` and +// `dataProviderShortCode`, the rest are inert defaults the SDK type requires. +const makeSource = (id: number, dataProviderShortCode: string): Source => ({ + id, + label: `Source ${id}`, + createdAt: '2024-01-01T00:00:00Z', + createdBy: 'test', + dataProviderShortCode, + dataProvider: { + shortCode: dataProviderShortCode, + name: dataProviderShortCode, + keycloakOrganizationId: '', + createdAt: '2024-01-01T00:00:00Z', + createdBy: 'test', + }, + funderShortCode: '', + funder: { shortCode: '', name: '', createdAt: '2024-01-01T00:00:00Z', createdBy: 'test', isCollaborative: false }, + changemakerId: 0, + changemaker: makeChangemaker(0, ''), +}); + +describe('toFieldValueString', () => { + it('trims a string', () => { + expect(toFieldValueString(' hello ')).toBe('hello'); + }); + + it('returns null for an empty or whitespace-only string', () => { + expect(toFieldValueString(' ')).toBeNull(); + }); + + it('returns null for null and undefined', () => { + expect(toFieldValueString(null)).toBeNull(); + expect(toFieldValueString(undefined)).toBeNull(); + }); + + it('stringifies numbers and booleans', () => { + expect(toFieldValueString(42)).toBe('42'); + expect(toFieldValueString(true)).toBe('true'); + }); + + it('joins array values with a separator, dropping empties', () => { + expect(toFieldValueString(['a', '', 'b'])).toBe('a; b'); + }); + + it('returns null for an empty array', () => { + expect(toFieldValueString([])).toBeNull(); + }); + + it('returns null for a plain object', () => { + expect(toFieldValueString({ a: 1 })).toBeNull(); + }); +}); + +describe('resolveChangemaker', () => { + const bundle: ChangemakerBundle = { + entries: [makeChangemaker(1, '00-1000008'), makeChangemaker(2, '22-3456789')], + total: 2, + }; + + it('resolves a unique match (hyphen-insensitive)', () => { + const resolution = resolveChangemaker('001000008', bundle); + expect(resolution.kind).toBe('matched'); + expect(resolution.kind === 'matched' ? resolution.changemaker.id : null).toBe(1); + }); + + it('resolves to missing when there is no match', () => { + expect(resolveChangemaker('99-9999999', bundle).kind).toBe('missing'); + }); + + it('resolves to ambiguous when more than one changemaker matches', () => { + const ambiguous: ChangemakerBundle = { + entries: [makeChangemaker(1, '00-1000008'), makeChangemaker(2, '001000008')], + total: 2, + }; + expect(resolveChangemaker('00-1000008', ambiguous).kind).toBe('ambiguous'); + }); +}); + +describe('toChangemakerTarget', () => { + it('targets a new changemaker when --create-changemaker is set', () => { + expect(toChangemakerTarget(true, undefined)).toStrictEqual({ kind: 'create' }); + }); + + it('targets the explicit ID when --changemaker-id is given', () => { + expect(toChangemakerTarget(false, 7)).toStrictEqual({ kind: 'id', changemakerId: 7 }); + }); + + it('falls back to EIN matching when neither flag is given', () => { + expect(toChangemakerTarget(false, undefined)).toStrictEqual({ kind: 'ein' }); + }); +}); + +// A normalized package as read from a single PDC JSON file. +const makePackage = (overrides: Partial = {}): TechSoupPackage => ({ + slug: 'synthetic-org', + ein: '00-1000008', + name: 'Synthetic Org', + goodAsOf: null, + organizationFields: [ + { baseFieldShortCode: 'organization_name', value: 'Synthetic Org', sourceNote: 'README.md#title' }, + { baseFieldShortCode: 'organization_tax_id', value: '00-1000008', sourceNote: 'organization_tax_id' }, + ], + proposalFields: [], + ...overrides, +}); + +describe('planImport', () => { + const bundle: ChangemakerBundle = { + entries: [makeChangemaker(1, '001000008'), makeChangemaker(2, '22-3456789')], + total: 2, + }; + const ambiguous: ChangemakerBundle = { + entries: [makeChangemaker(1, '00-1000008'), makeChangemaker(2, '001000008')], + total: 2, + }; + const empty: ChangemakerBundle = { entries: [], total: 0 }; + + describe('matching by EIN (default)', () => { + it('updates the changemaker whose taxId matches the file EIN', () => { + const plan = planImport(makePackage(), bundle, { kind: 'ein' }); + expect(plan.action).toBe('update'); + expect(plan.action === 'update' ? plan.changemaker.id : null).toBe(1); + }); + + it('skips when the EIN is not in the PDC, pointing at --create-changemaker', () => { + const plan = planImport(makePackage({ ein: '99-9999999' }), bundle, { kind: 'ein' }); + expect(plan.action).toBe('skip'); + expect(plan.action === 'skip' ? plan.reason : '').toContain('--create-changemaker'); + }); + + it('skips when the EIN matches more than one changemaker', () => { + const plan = planImport(makePackage(), ambiguous, { kind: 'ein' }); + expect(plan.action).toBe('skip'); + expect(plan.action === 'skip' ? plan.reason : '').toContain('multiple'); + }); + + it('skips when the file has no valid EIN', () => { + const plan = planImport(makePackage({ ein: null }), bundle, { kind: 'ein' }); + expect(plan.action).toBe('skip'); + expect(plan.action === 'skip' ? plan.reason : '').toContain('--changemaker-id'); + }); + }); + + describe('explicit --changemaker-id', () => { + it('updates the changemaker with that ID, regardless of EIN', () => { + const plan = planImport(makePackage(), bundle, { kind: 'id', changemakerId: 2 }); + expect(plan.action === 'update' ? plan.changemaker.id : null).toBe(2); + }); + + it('works when the file has no EIN', () => { + const plan = planImport(makePackage({ ein: null }), bundle, { kind: 'id', changemakerId: 1 }); + expect(plan.action === 'update' ? plan.changemaker.id : null).toBe(1); + }); + + it('skips when no changemaker has that ID', () => { + const plan = planImport(makePackage(), bundle, { kind: 'id', changemakerId: 999 }); + expect(plan.action).toBe('skip'); + expect(plan.action === 'skip' ? plan.reason : '').toContain('999'); + }); + }); + + describe('--create-changemaker', () => { + it('plans a create with the file EIN (verbatim) and name when the EIN is new', () => { + expect(planImport(makePackage(), empty, { kind: 'create' })).toStrictEqual({ + action: 'create', + taxId: '00-1000008', + name: 'Synthetic Org', + }); + }); + + it('refuses to create a duplicate when the EIN already exists', () => { + const plan = planImport(makePackage(), bundle, { kind: 'create' }); + expect(plan.action).toBe('skip'); + expect(plan.action === 'skip' ? plan.reason : '').toContain('changemaker 1'); + }); + + it('refuses to create when the EIN already matches several changemakers', () => { + expect(planImport(makePackage(), ambiguous, { kind: 'create' }).action).toBe('skip'); + }); + + it('skips when the file has no name', () => { + const plan = planImport(makePackage({ name: null }), empty, { kind: 'create' }); + expect(plan.action).toBe('skip'); + expect(plan.action === 'skip' ? plan.reason : '').toContain('name'); + }); + + it('skips when the file has no valid EIN', () => { + expect(planImport(makePackage({ ein: null }), empty, { kind: 'create' }).action).toBe('skip'); + }); + }); +}); + +describe('applyBaseFieldCategories', () => { + // Only `shortCode` and `category` are read. + const makeBaseField = (shortCode: string, category: string): { shortCode: string; category: string } => ({ + shortCode, + category, + }); + const field = (baseFieldShortCode: string): { baseFieldShortCode: string; value: string; sourceNote: string } => ({ + baseFieldShortCode, + value: 'v', + sourceNote: baseFieldShortCode, + }); + const baseFields = { + entries: [ + makeBaseField('organization_name', 'organization'), + makeBaseField('community_definition_overview', 'project'), + makeBaseField('proposal_name', 'project'), + ], + total: 3, + }; + const doc: TechSoupDocument = { + slug: 'example', + ein: null, + name: null, + goodAsOf: null, + fields: [ + field('organization_name'), + field('proposal_name'), + field('community_definition_overview'), + field('not_a_base_field'), + ], + }; + + it('keeps only organization-category fields for the changemaker', () => { + expect(applyBaseFieldCategories(doc, baseFields).organizationFields.map((f) => f.baseFieldShortCode)).toStrictEqual( + ['organization_name'], + ); + }); + + it('sends fields of other categories to the proposal fields, whatever their prefix', () => { + expect(applyBaseFieldCategories(doc, baseFields).proposalFields.map((f) => f.baseFieldShortCode)).toStrictEqual([ + 'proposal_name', + 'community_definition_overview', + ]); + }); + + it('drops fields that are not base fields in the PDC', () => { + const result = applyBaseFieldCategories(doc, baseFields); + const codes = [...result.organizationFields, ...result.proposalFields].map((f) => f.baseFieldShortCode); + expect(codes).not.toContain('not_a_base_field'); + }); +}); + +describe('formatImportPreview', () => { + const pkg = makePackage({ + proposalFields: [{ baseFieldShortCode: 'proposal_name', value: 'A funding ask', sourceNote: 'proposal_name' }], + }); + + it('names the existing target changemaker and lists every field to import', () => { + const lines = formatImportPreview(pkg, { action: 'update', changemaker: makeChangemaker(1, '00-1000008') }); + expect(lines[0]).toBe('synthetic-org → changemaker 1 ("Org 1", taxId 00-1000008)'); + expect(lines).toContain('Fields that would be imported (2, goodAsOf=null):'); + expect(lines).toContain(' organization_name = "Synthetic Org" [README.md#title]'); + expect(lines).toContain(' organization_tax_id = "00-1000008" [organization_tax_id]'); + }); + + it('names a new changemaker when the plan creates one', () => { + const lines = formatImportPreview(pkg, { action: 'create', taxId: '00-1000008', name: 'Synthetic Org' }); + expect(lines[0]).toBe('synthetic-org → NEW changemaker ("Synthetic Org", taxId 00-1000008)'); + }); + + it('lists proposal fields separately as not imported', () => { + const lines = formatImportPreview(pkg, { action: 'update', changemaker: makeChangemaker(1, '00-1000008') }); + expect(lines).toContain('Proposal (non-organization) fields NOT imported (1):'); + expect(lines).toContain(' proposal_name = "A funding ask" [proposal_name]'); + }); + + it('prints only the skip reason when nothing would be imported', () => { + expect(formatImportPreview(pkg, { action: 'skip', reason: 'no match' })).toStrictEqual([ + 'synthetic-org → nothing — no match', + ]); + }); +}); + +describe('pdcJsonToDocument', () => { + const raw = { + changemaker: { name: 'synthetic-Black Mountain Workforce Partnership' }, + fields: [ + { + baseFieldShortCode: 'organization_name', + value: 'synthetic-Black Mountain Workforce Partnership', + batch: { notes: 'slug/README.md#title' }, + }, + { baseFieldShortCode: 'organization_tax_id', value: '00-1000008' }, + { baseFieldShortCode: 'organization_dba_name', value: ['Alias One', 'Alias Two'] }, + { baseFieldShortCode: 'proposal_name', value: 'A funding ask' }, + { baseFieldShortCode: 'organization_overview', value: ' ' }, + ], + }; + const doc = pdcJsonToDocument(raw, 'synthetic-black-mountain-workforce-partnership'); + + it('takes the EIN from organization_tax_id (hyphenated, verbatim)', () => { + expect(doc.ein).toBe('00-1000008'); + }); + + it('takes the name from changemaker.name and leaves goodAsOf null', () => { + expect(doc.name).toBe('synthetic-Black Mountain Workforce Partnership'); + expect(doc.goodAsOf).toBeNull(); + }); + + it('keeps every usable field in file order, without sorting them by prefix', () => { + expect(doc.fields.map((f) => f.baseFieldShortCode)).toStrictEqual([ + 'organization_name', + 'organization_tax_id', + 'organization_dba_name', + 'proposal_name', + ]); + }); + + it('flattens array values and uses batch.notes as the provenance note', () => { + const dba = doc.fields.find((f) => f.baseFieldShortCode === 'organization_dba_name'); + expect(dba?.value).toBe('Alias One; Alias Two'); + const name = doc.fields.find((f) => f.baseFieldShortCode === 'organization_name'); + expect(name?.sourceNote).toBe('slug/README.md#title'); + }); + + it('drops blank-valued fields', () => { + expect(doc.fields.some((f) => f.baseFieldShortCode === 'organization_overview')).toBe(false); + }); + + it('yields a null EIN for a non-US tax id', () => { + const polish = pdcJsonToDocument( + { + changemaker: { name: 'Fundacja' }, + fields: [{ baseFieldShortCode: 'organization_tax_id', value: '0000000000' }], + }, + 'synthetic-fundacja', + ); + expect(polish.ein).toBeNull(); + }); +}); + +describe('parsePdcJsonDocument', () => { + it('normalizes a PDC JSON document', () => { + const doc = parsePdcJsonDocument( + { changemaker: { name: 'Org' }, fields: [{ baseFieldShortCode: 'organization_tax_id', value: '12-3456789' }] }, + 'org', + ); + expect(doc.ein).toBe('12-3456789'); + expect(doc.name).toBe('Org'); + }); + + it('throws when the document has no fields array', () => { + expect(() => parsePdcJsonDocument({ changemaker: { name: 'Org' } }, 'org')).toThrow('"fields" array'); + }); + + it('throws when the document is not an object', () => { + expect(() => parsePdcJsonDocument([], 'org')).toThrow('not a TechSoup PDC JSON document'); + }); +}); + +describe('readPdcJsonFile', () => { + let dir = ''; + + beforeAll(async () => { + dir = await mkdtemp(join(tmpdir(), 'techsoup-test-')); + }); + + afterAll(async () => { + await rm(dir, { recursive: true, force: true }); + }); + + it('reads one file and derives the slug from its name', async () => { + const filePath = join(dir, 'synthetic-org_pdc.json'); + await writeFile( + filePath, + JSON.stringify({ + changemaker: { name: 'Synthetic Org' }, + fields: [ + { baseFieldShortCode: 'organization_name', value: 'Synthetic Org' }, + { baseFieldShortCode: 'organization_tax_id', value: '00-1000008' }, + { baseFieldShortCode: 'proposal_name', value: 'A funding ask' }, + ], + }), + ); + const doc = await readPdcJsonFile(filePath); + expect(doc.slug).toBe('synthetic-org'); + expect(doc.ein).toBe('00-1000008'); + expect(doc.fields).toHaveLength(3); + }); + + it('rejects a JSON file that is not PDC-export shaped', async () => { + const filePath = join(dir, 'package.json'); + await writeFile(filePath, JSON.stringify({ name: 'not-a-pdc-file' })); + await expect(readPdcJsonFile(filePath)).rejects.toThrow('"fields" array'); + }); + + it('rejects a file that is not valid JSON', async () => { + const filePath = join(dir, 'broken.json'); + await writeFile(filePath, '{ not json'); + await expect(readPdcJsonFile(filePath)).rejects.toThrow(SyntaxError); + }); +}); + +describe('selectTechSoupSource', () => { + it('returns the single TechSoup source', () => { + const sources = { entries: [makeSource(1, 'charitynavigator'), makeSource(2, 'techsoup')], total: 2 }; + expect(selectTechSoupSource(sources)?.id).toBe(2); + }); + + it('returns null when there is no TechSoup source', () => { + expect(selectTechSoupSource({ entries: [makeSource(1, 'charitynavigator')], total: 1 })).toBeNull(); + }); + + it('uses the oldest (lowest ID) when several TechSoup sources exist', () => { + const sources = { entries: [makeSource(9, 'techsoup'), makeSource(4, 'techsoup')], total: 2 }; + expect(selectTechSoupSource(sources)?.id).toBe(4); + }); +}); + +describe('toLoggableError', () => { + const NOT_FOUND = 404; + const config = { + method: 'post', + url: 'https://api.example.org/sources', + headers: new AxiosHeaders({ Authorization: 'Bearer secret-token' }), + }; + const axiosError = new AxiosError('Request failed with status code 404', 'ERR_BAD_REQUEST', config, null, { + data: { name: 'NotFoundError', message: 'Entity not found' }, + status: NOT_FOUND, + statusText: 'Not Found', + headers: {}, + config, + }); + + it('keeps the status, method, URL, and response body of an AxiosError', () => { + expect(toLoggableError(axiosError)).toStrictEqual({ + name: 'AxiosError', + message: 'Request failed with status code 404', + code: 'ERR_BAD_REQUEST', + status: NOT_FOUND, + method: 'post', + url: 'https://api.example.org/sources', + responseData: { name: 'NotFoundError', message: 'Entity not found' }, + }); + }); + + it('drops the request headers, so the bearer token is never logged', () => { + expect(JSON.stringify(toLoggableError(axiosError))).not.toContain('secret-token'); + }); + + it('reduces an AxiosError found in an Error cause', () => { + const wrapped = new Error('Cannot create the TechSoup source', { cause: axiosError }); + const loggable = toLoggableError(wrapped); + expect(JSON.stringify(loggable)).not.toContain('secret-token'); + expect(loggable).toMatchObject({ message: 'Cannot create the TechSoup source', cause: { status: NOT_FOUND } }); + }); + + it('passes non-Error values through unchanged', () => { + expect(toLoggableError('plain string')).toBe('plain string'); + expect(toLoggableError(undefined)).toBeUndefined(); + }); +});