Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions cip/cip0001.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ last_updated: "2026-11-13"
status: In Progress
compressed: false
related_requirements: []
related_cips: []
related_cips:
- "0002"
related_gpy_cips:
- "0010"
- "000F"
Expand Down Expand Up @@ -203,8 +204,9 @@ pluggable reducer in R14, so the MPI-specific code shrinks to that one place.
- Porting PyDeepGP to JAX / GPy2 (GPy CIP-0010 frames that; on JAX, sharding
and `vmap` replace both chunking and MPI).
- Changing models, objectives or default hyperparameters.
- MRD and prediction-only SVI helpers beyond removing their MPI code (separate
revival work; see README limitations).
- MRD and prediction-only SVI helpers beyond removing their MPI code (MRD
revival is CIP-0002; prediction-only SVI remains separate; see README
limitations).

### Alternatives considered

Expand Down
249 changes: 249 additions & 0 deletions cip/cip0002.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,249 @@
---
author: Neil Lawrence
created: "2026-11-13"
id: "0002"
last_updated: "2026-11-13"
status: Proposed
compressed: false
related_requirements: []
related_cips:
- "0001"
tags:
- cip
- mrd
- multi-view
- deepgp
- testing
- damianou2013
title: Revive Manifold Relevance Determination (MRD) for multi-view DeepGPs
---

# CIP-0002: Revive Manifold Relevance Determination (MRD)

## Status

- [x] Proposed - Initial idea documented (open for review)
- [ ] Accepted - Approved, ready to start work
- [ ] In Progress - Actively being implemented
- [ ] Implemented - Work complete, awaiting verification
- [ ] Closed - Verified and complete
- [ ] Rejected - Will not be implemented
- [ ] Deferred - Postponed

## Summary

Bring the multi-view / shared-latent path (`ObservedMRDLayer` and friends) back
onto the maintained DeepGP surface so that models with conditionally independent
observation views — including the Damianou & Lawrence (2013) CMU high-five
experiment — construct, optimise, and are covered by tests.

CIP-0001 explicitly deferred MRD beyond stripping MPI (R20). This CIP is that
revival. It does **not** depend on finishing CIP-0001 Phase 4; MRD can be fixed
on the current serial stack first, then cleaned of MPI in line with R20 when
Phase 4 lands.

## Motivation

### Why it matters

- **Paper fidelity.** Damianou & Lawrence AISTATS 2013 §4.2 uses a two-view
hierarchy (two subjects, shared parent latent layer). The
`examples/damianou2013/` suite (PR #41) can recreate the toy and USPS
experiments; high-five is blocked on MRD.
- **Public API already claims the feature.** `DeepGP` accepts `Y` as a list and
builds `ObservedMRDLayer`, but the path is omitted from `.coveragerc` and has
no unit tests. That violates the *test-what-you-ship* and
*honest-public-surface* tenets: either we maintain it or we stop advertising it.
- **Concrete breakage today.** A minimal two-view unsupervised `DeepGP` fails
during `initialize_parameter` with:

```text
ValueError: setting an array element with a sequence
```

in `DeepGP.parameters_changed` when summing layer bounds.

### Evidence (2026-11-13, `main` / CIP-0001 post Phase 3)

Diagnosed on synthetic correlated views (`Y=[Y1,Y2]`, dims `[[D,D], Q1, Q2]`,
`back_constraint=False`):

| Layer | `_log_marginal_likelihood` |
| --- | --- |
| `ObservedMRDLayer` | `numpy.float64` scalar (from `np.sum` over views) |
| `TopHiddenLayer` | `ndarray` shape `(1,)` (same as serial `ObservedLayer`) |

`np.sum([scalar, array(1,)])` raises under current NumPy. Serial stacks work
because every layer keeps a length-1 array. MRD's aggregation is the mismatch.

Additional known debt in `deepgp/layers/mrd.py`:

1. **Init-string bug.** `_init_X` uses `init in "PCA_concat"` / `"PCA_single"`
(substring / character tests) instead of equality. Default `"PCA"` happens to
take the concat branch; `"PCA_single"` does not match as intended.
2. **Stale encoder import.** `MRDView` does `from .mlp import MLP` under
`back_constraint=True`; the NumPy encoder lives in `deepgp.encoder.mlp`
(CIP-0001 Phase 1). Back-constrained MRD is broken independently of the
bound bug.
3. **MPI still wired.** Views subclass `SparseGP_MPI`; `update_qX_gradients`
still calls `reduceArrays`. Covered by CIP-0001 R20, but MRD must remain
correct with `mpi_comm=None` after this CIP.
4. **API friction.** Observation kernels must be a **list of kernels** (one per
view); undocumented. Wrong shapes fail with an assert deep in `MRDLayer`.
5. **No tests, omitted from coverage.** `.coveragerc` omits
`*/deepgp/layers/mrd.py`.

## Detailed Description

### Target behaviour

After this CIP, the following must work on the serial path (`mpi_comm=None`):

```python
m = deepgp.DeepGP(
[[D1, D2], Q_shared, Q_top],
Y=[Y1, Y2],
kernels=[
[kern_view1, kern_view2], # each input_dim == Q_shared
kern_top,
],
num_inducing=M,
back_constraint=False,
)
m.optimize(...)
float(m.log_likelihood()) # finite scalar
```

Optional stretch (same CIP if cheap; else follow-up task):
`back_constraint=True` with the NumPy MLP encoder per view, aggregating `q(X)`
as already sketched in `_aggregate_qX`.

### Design decisions

**D1 — Bound dtype contract.** Every layer exposes
`_log_marginal_likelihood` as a NumPy array of shape `(1,)` (or any shape that
`np.sum` can reduce uniformly). `ObservedMRDLayer.parameters_changed` must
aggregate view bounds into that contract, e.g.:

```python
self._log_marginal_likelihood = np.sum(
[np.asarray(v._log_marginal_likelihood) for v in self.views], axis=0
)
```

Optionally harden `DeepGP.parameters_changed` to coerce with
`float(np.sum([...]))` so one divergent layer cannot crash the model — keep the
layer contract as the source of truth; the model-level coerce is defence in
depth.

**D2 — Construction API.** Document and test the kernel list-of-lists rule.
Consider a thin helper (e.g. `deepgp.util.mrd_kernels(Q, n_views, **kw)`) only
if examples stay awkward; prefer clear docs + tests over new API surface.

**D3 — Init strings.** Replace `init in "PCA_..."` with explicit equality
(`init == "PCA_concat"`, etc.) and map legacy `"PCA"` → `"PCA_concat"`.

**D4 — Encoder path.** Point MRD back-constraint at `deepgp.encoder.mlp.MLP`,
matching `enclayer.py`. Default for revival work remains
`back_constraint=False` (matches high-five-style ARD exploration without
encoders).

**D5 — MPI.** Do not re-enable MPI for MRD. Keep `mpi_comm=None` behaviour
correct; leave R20 (strip MPI args/branches) to CIP-0001 Phase 4 or a small
follow-up commit in this CIP if Phase 4 has not started — strip only dead
branches that confuse readers, without changing CIP-0001's deprecation
timeline.

**D6 — Coverage.** Remove `mrd.py` from `.coveragerc` omit list once tests
exist. Update README / `examples/damianou2013/highfive_shared.py` so
`--force-mrd` becomes a real demo (or rename to the default high-five script).

**D7 — Scope limit.** This CIP covers **observed** multi-view leaves
(`ObservedMRDLayer`) stacked under ordinary hidden/top layers — the paper §4.2
architecture. Nested MRD mid-layers and product-of-experts variants mentioned
in comments stay out of scope unless tests prove they already work.

### Non-goals

- Faithful bit-reproduction of 2013 Matlab high-five figures (optimisation
schedules differ); a working qualitative recreation is enough.
- Multi-machine MPI for views (CIP-0001).
- Prediction-only SVI helpers (`inferenceX`, `svi_ratio`) — still separate.

## Implementation Plan

1. **Phase A — Make construction work (serial, no back-constraint)**
- Fix bound aggregation dtype in `MRDLayer.parameters_changed` (D1).
- Fix `_init_X` string comparisons (D3).
- Add `deepgp/testing/test_mrd.py`: two-view construct, finite bound,
short `optimize`, optional `checkgrad` smoke on a tiny model.
- Harden `DeepGP.parameters_changed` if still useful after D1.

2. **Phase B — Public surface honesty**
- Document multi-view construction in README and
`examples/damianou2013/README.md`.
- Turn `highfive_shared.py` into a runnable demo (synthetic views always;
real CMU high-five when data helpers exist).
- Remove `mrd.py` from `.coveragerc` omit; ensure CI coverage does not
regress the new tests.

3. **Phase C — Back-constraint + encoder (optional in first PR)**
- Fix MLP import (D4); add one back-constrained MRD test.
- Skip or xfail if encoder dims make the smoke test too heavy.

4. **Phase D — Align with CIP-0001 R20**
- Remove or no-op MPI branches in `mrd.py` consistent with Phase 4 policy.
- Keep accepting `mpi_comm=None` only until CIP-0001 R23 says otherwise.

## Backward Compatibility

- **Intended break:** none for serial single-view users.
- **Multi-view users:** construction that already passed the kernel-list assert
but never successfully initialised will start working; no parameter-file
format change.
- **Init names:** `"PCA"` remains accepted as alias of `"PCA_concat"`; document
`"PCA_single"` and `"random"` as the supported spellings.

## Testing Strategy

| Test | Intent |
| --- | --- |
| `test_mrd_constructs_two_view` | `DeepGP([[D,D],Q1,Q2], [Y1,Y2], ...)` returns finite `log_likelihood()` |
| `test_mrd_optimize_smoke` | Few L-BFGS steps increase or leave bound finite |
| `test_mrd_kernel_list_required` | Clear error if obs kernels are not a list of length `n_views` |
| `test_mrd_init_pca_concat_and_single` | Init dispatch uses equality, not substring |
| `test_mrd_back_constraint` (Phase C) | Encoder import path works |
| Example smoke | `highfive_shared.py --quick` exit 0 |

Manual: run USPS/toys unchanged; run high-five synthetic demo; if MOCAP
available, optional qualitative ARD plot for shared dimensions.

## Related Requirements

No formal `requirements/` entries yet in this repository. This CIP supports:

- Honest public surface (tenet): multi-view API either works under CI or is
clearly marked unavailable.
- Damianou & Lawrence 2013 recreation completeness (examples suite).

If a requirement ID is added later, set `related_requirements` in the
frontmatter.

## Implementation Status

- [ ] Phase A: bound dtype + init fix + unit tests
- [ ] Phase B: docs, coverage omit removal, high-five example
- [ ] Phase C: back-constraint encoder path (optional first PR)
- [ ] Phase D: MPI cleanup aligned with CIP-0001 R20
- [ ] CIP review → Accepted → backlog tasks

## References

- Damianou & Lawrence, *Deep Gaussian Processes*, AISTATS 2013, §4.2
(https://proceedings.mlr.press/v31/damianou13a.html)
- Damianou et al., shared / MRD GP-LVM structure (2012)
- Code: `deepgp/layers/mrd.py`, `deepgp/models/model.py` (`mrd_flag` branch)
- CIP-0001 non-goals and R20 (MPI strip in `mrd.py`)
- Examples: `examples/damianou2013/highfive_shared.py` (PR #41)
- Tenets: `tenets/maintained-core-first.md`, `tenets/test-what-you-ship.md`,
`tenets/honest-public-surface.md`
Loading