diff --git a/cip/cip0001.md b/cip/cip0001.md index dccc43c..df01cbd 100644 --- a/cip/cip0001.md +++ b/cip/cip0001.md @@ -6,7 +6,8 @@ last_updated: "2026-11-13" status: In Progress compressed: false related_requirements: [] -related_cips: [] +related_cips: +- "0002" related_gpy_cips: - "0010" - "000F" @@ -203,8 +204,9 @@ pluggable reducer in R14, so the MPI-specific code shrinks to that one place. - Porting PyDeepGP to JAX / GPy2 (GPy CIP-0010 frames that; on JAX, sharding and `vmap` replace both chunking and MPI). - Changing models, objectives or default hyperparameters. -- MRD and prediction-only SVI helpers beyond removing their MPI code (separate - revival work; see README limitations). +- MRD and prediction-only SVI helpers beyond removing their MPI code (MRD + revival is CIP-0002; prediction-only SVI remains separate; see README + limitations). ### Alternatives considered diff --git a/cip/cip0002.md b/cip/cip0002.md new file mode 100644 index 0000000..f60baad --- /dev/null +++ b/cip/cip0002.md @@ -0,0 +1,249 @@ +--- +author: Neil Lawrence +created: "2026-11-13" +id: "0002" +last_updated: "2026-11-13" +status: Proposed +compressed: false +related_requirements: [] +related_cips: +- "0001" +tags: +- cip +- mrd +- multi-view +- deepgp +- testing +- damianou2013 +title: Revive Manifold Relevance Determination (MRD) for multi-view DeepGPs +--- + +# CIP-0002: Revive Manifold Relevance Determination (MRD) + +## Status + +- [x] Proposed - Initial idea documented (open for review) +- [ ] Accepted - Approved, ready to start work +- [ ] In Progress - Actively being implemented +- [ ] Implemented - Work complete, awaiting verification +- [ ] Closed - Verified and complete +- [ ] Rejected - Will not be implemented +- [ ] Deferred - Postponed + +## Summary + +Bring the multi-view / shared-latent path (`ObservedMRDLayer` and friends) back +onto the maintained DeepGP surface so that models with conditionally independent +observation views — including the Damianou & Lawrence (2013) CMU high-five +experiment — construct, optimise, and are covered by tests. + +CIP-0001 explicitly deferred MRD beyond stripping MPI (R20). This CIP is that +revival. It does **not** depend on finishing CIP-0001 Phase 4; MRD can be fixed +on the current serial stack first, then cleaned of MPI in line with R20 when +Phase 4 lands. + +## Motivation + +### Why it matters + +- **Paper fidelity.** Damianou & Lawrence AISTATS 2013 §4.2 uses a two-view + hierarchy (two subjects, shared parent latent layer). The + `examples/damianou2013/` suite (PR #41) can recreate the toy and USPS + experiments; high-five is blocked on MRD. +- **Public API already claims the feature.** `DeepGP` accepts `Y` as a list and + builds `ObservedMRDLayer`, but the path is omitted from `.coveragerc` and has + no unit tests. That violates the *test-what-you-ship* and + *honest-public-surface* tenets: either we maintain it or we stop advertising it. +- **Concrete breakage today.** A minimal two-view unsupervised `DeepGP` fails + during `initialize_parameter` with: + + ```text + ValueError: setting an array element with a sequence + ``` + + in `DeepGP.parameters_changed` when summing layer bounds. + +### Evidence (2026-11-13, `main` / CIP-0001 post Phase 3) + +Diagnosed on synthetic correlated views (`Y=[Y1,Y2]`, dims `[[D,D], Q1, Q2]`, +`back_constraint=False`): + +| Layer | `_log_marginal_likelihood` | +| --- | --- | +| `ObservedMRDLayer` | `numpy.float64` scalar (from `np.sum` over views) | +| `TopHiddenLayer` | `ndarray` shape `(1,)` (same as serial `ObservedLayer`) | + +`np.sum([scalar, array(1,)])` raises under current NumPy. Serial stacks work +because every layer keeps a length-1 array. MRD's aggregation is the mismatch. + +Additional known debt in `deepgp/layers/mrd.py`: + +1. **Init-string bug.** `_init_X` uses `init in "PCA_concat"` / `"PCA_single"` + (substring / character tests) instead of equality. Default `"PCA"` happens to + take the concat branch; `"PCA_single"` does not match as intended. +2. **Stale encoder import.** `MRDView` does `from .mlp import MLP` under + `back_constraint=True`; the NumPy encoder lives in `deepgp.encoder.mlp` + (CIP-0001 Phase 1). Back-constrained MRD is broken independently of the + bound bug. +3. **MPI still wired.** Views subclass `SparseGP_MPI`; `update_qX_gradients` + still calls `reduceArrays`. Covered by CIP-0001 R20, but MRD must remain + correct with `mpi_comm=None` after this CIP. +4. **API friction.** Observation kernels must be a **list of kernels** (one per + view); undocumented. Wrong shapes fail with an assert deep in `MRDLayer`. +5. **No tests, omitted from coverage.** `.coveragerc` omits + `*/deepgp/layers/mrd.py`. + +## Detailed Description + +### Target behaviour + +After this CIP, the following must work on the serial path (`mpi_comm=None`): + +```python +m = deepgp.DeepGP( + [[D1, D2], Q_shared, Q_top], + Y=[Y1, Y2], + kernels=[ + [kern_view1, kern_view2], # each input_dim == Q_shared + kern_top, + ], + num_inducing=M, + back_constraint=False, +) +m.optimize(...) +float(m.log_likelihood()) # finite scalar +``` + +Optional stretch (same CIP if cheap; else follow-up task): +`back_constraint=True` with the NumPy MLP encoder per view, aggregating `q(X)` +as already sketched in `_aggregate_qX`. + +### Design decisions + +**D1 — Bound dtype contract.** Every layer exposes +`_log_marginal_likelihood` as a NumPy array of shape `(1,)` (or any shape that +`np.sum` can reduce uniformly). `ObservedMRDLayer.parameters_changed` must +aggregate view bounds into that contract, e.g.: + +```python +self._log_marginal_likelihood = np.sum( + [np.asarray(v._log_marginal_likelihood) for v in self.views], axis=0 +) +``` + +Optionally harden `DeepGP.parameters_changed` to coerce with +`float(np.sum([...]))` so one divergent layer cannot crash the model — keep the +layer contract as the source of truth; the model-level coerce is defence in +depth. + +**D2 — Construction API.** Document and test the kernel list-of-lists rule. +Consider a thin helper (e.g. `deepgp.util.mrd_kernels(Q, n_views, **kw)`) only +if examples stay awkward; prefer clear docs + tests over new API surface. + +**D3 — Init strings.** Replace `init in "PCA_..."` with explicit equality +(`init == "PCA_concat"`, etc.) and map legacy `"PCA"` → `"PCA_concat"`. + +**D4 — Encoder path.** Point MRD back-constraint at `deepgp.encoder.mlp.MLP`, +matching `enclayer.py`. Default for revival work remains +`back_constraint=False` (matches high-five-style ARD exploration without +encoders). + +**D5 — MPI.** Do not re-enable MPI for MRD. Keep `mpi_comm=None` behaviour +correct; leave R20 (strip MPI args/branches) to CIP-0001 Phase 4 or a small +follow-up commit in this CIP if Phase 4 has not started — strip only dead +branches that confuse readers, without changing CIP-0001's deprecation +timeline. + +**D6 — Coverage.** Remove `mrd.py` from `.coveragerc` omit list once tests +exist. Update README / `examples/damianou2013/highfive_shared.py` so +`--force-mrd` becomes a real demo (or rename to the default high-five script). + +**D7 — Scope limit.** This CIP covers **observed** multi-view leaves +(`ObservedMRDLayer`) stacked under ordinary hidden/top layers — the paper §4.2 +architecture. Nested MRD mid-layers and product-of-experts variants mentioned +in comments stay out of scope unless tests prove they already work. + +### Non-goals + +- Faithful bit-reproduction of 2013 Matlab high-five figures (optimisation + schedules differ); a working qualitative recreation is enough. +- Multi-machine MPI for views (CIP-0001). +- Prediction-only SVI helpers (`inferenceX`, `svi_ratio`) — still separate. + +## Implementation Plan + +1. **Phase A — Make construction work (serial, no back-constraint)** + - Fix bound aggregation dtype in `MRDLayer.parameters_changed` (D1). + - Fix `_init_X` string comparisons (D3). + - Add `deepgp/testing/test_mrd.py`: two-view construct, finite bound, + short `optimize`, optional `checkgrad` smoke on a tiny model. + - Harden `DeepGP.parameters_changed` if still useful after D1. + +2. **Phase B — Public surface honesty** + - Document multi-view construction in README and + `examples/damianou2013/README.md`. + - Turn `highfive_shared.py` into a runnable demo (synthetic views always; + real CMU high-five when data helpers exist). + - Remove `mrd.py` from `.coveragerc` omit; ensure CI coverage does not + regress the new tests. + +3. **Phase C — Back-constraint + encoder (optional in first PR)** + - Fix MLP import (D4); add one back-constrained MRD test. + - Skip or xfail if encoder dims make the smoke test too heavy. + +4. **Phase D — Align with CIP-0001 R20** + - Remove or no-op MPI branches in `mrd.py` consistent with Phase 4 policy. + - Keep accepting `mpi_comm=None` only until CIP-0001 R23 says otherwise. + +## Backward Compatibility + +- **Intended break:** none for serial single-view users. +- **Multi-view users:** construction that already passed the kernel-list assert + but never successfully initialised will start working; no parameter-file + format change. +- **Init names:** `"PCA"` remains accepted as alias of `"PCA_concat"`; document + `"PCA_single"` and `"random"` as the supported spellings. + +## Testing Strategy + +| Test | Intent | +| --- | --- | +| `test_mrd_constructs_two_view` | `DeepGP([[D,D],Q1,Q2], [Y1,Y2], ...)` returns finite `log_likelihood()` | +| `test_mrd_optimize_smoke` | Few L-BFGS steps increase or leave bound finite | +| `test_mrd_kernel_list_required` | Clear error if obs kernels are not a list of length `n_views` | +| `test_mrd_init_pca_concat_and_single` | Init dispatch uses equality, not substring | +| `test_mrd_back_constraint` (Phase C) | Encoder import path works | +| Example smoke | `highfive_shared.py --quick` exit 0 | + +Manual: run USPS/toys unchanged; run high-five synthetic demo; if MOCAP +available, optional qualitative ARD plot for shared dimensions. + +## Related Requirements + +No formal `requirements/` entries yet in this repository. This CIP supports: + +- Honest public surface (tenet): multi-view API either works under CI or is + clearly marked unavailable. +- Damianou & Lawrence 2013 recreation completeness (examples suite). + +If a requirement ID is added later, set `related_requirements` in the +frontmatter. + +## Implementation Status + +- [ ] Phase A: bound dtype + init fix + unit tests +- [ ] Phase B: docs, coverage omit removal, high-five example +- [ ] Phase C: back-constraint encoder path (optional first PR) +- [ ] Phase D: MPI cleanup aligned with CIP-0001 R20 +- [ ] CIP review → Accepted → backlog tasks + +## References + +- Damianou & Lawrence, *Deep Gaussian Processes*, AISTATS 2013, §4.2 + (https://proceedings.mlr.press/v31/damianou13a.html) +- Damianou et al., shared / MRD GP-LVM structure (2012) +- Code: `deepgp/layers/mrd.py`, `deepgp/models/model.py` (`mrd_flag` branch) +- CIP-0001 non-goals and R20 (MPI strip in `mrd.py`) +- Examples: `examples/damianou2013/highfive_shared.py` (PR #41) +- Tenets: `tenets/maintained-core-first.md`, `tenets/test-what-you-ship.md`, + `tenets/honest-public-surface.md`