Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,3 +86,10 @@ linear psi-statistics (CIP-0001 Phase 3)::
Defaults (``psi_chunk_size=None``, ``psi_threads=1``) preserve previous
behaviour. Threading runs inside GPy's pure psi helpers; PyDeepGP itself stays
serial at the kernel API.

Examples
--------

* ``examples/`` — short supervised / unsupervised demos (oil, CMU walk/jog).
* ``examples/damianou2013/`` — recreations of the AISTATS 2013 experiments
(toy hierarchy, warped regression, USPS depth/ARD, high-five / MRD).
3 changes: 2 additions & 1 deletion cip/cip0002.md
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,8 @@ frontmatter.
## Implementation Status

- [x] Phase A: bound dtype + init fix + unit tests
- [x] Phase B: docs, coverage omit removal, `examples/example_mrd.py`
- [x] Phase B: docs, coverage omit removal, `examples/example_mrd.py`,
`examples/damianou2013/highfive_shared.py` runnable
- [x] Phase C: back-constraint encoder path (included in first implementation PR)
- [ ] Phase D: MPI cleanup aligned with CIP-0001 R20
- [x] CIP review → Accepted (2026-11-13); implementation on `cip0002-mrd-revival`
Expand Down
73 changes: 73 additions & 0 deletions examples/damianou2013/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Damianou & Lawrence (AISTATS 2013) — PyDeepGP recreations

Scripts that recreate the **feasible** experiments from
[Damianou & Lawrence, *Deep Gaussian Processes*](https://proceedings.mlr.press/v31/damianou13a.html)
(AISTATS 2013) on the maintained `DeepGP` path.

These are **modern recreations**, not bit-identical paper scripts: optimiser
schedules, inducing-point counts, and initialisation differ from the 2013
Matlab stack. Use `--quick` for a short smoke run.

## Mapping

| Paper section | Script | Status |
| --- | --- | --- |
| §4.1 Toy hierarchy (fig. 2) | `toy_hierarchy.py` | Runnable |
| §4.1 Warped regression (fig. 3) | `toy_warped_regression.py` | Runnable |
| §4.3 USPS {0,1,6} depth / ARD / 1-NN (figs. 6–8) | `usps_digits.py` | Runnable (downloads ESL `zip.train`) |
| §4.2 CMU high-five shared views (figs. 4–5) | `highfive_shared.py` | Runnable via CIP-0002 MRD (synthetic or pods stand-in) |

## Setup

From the repository root (Python ≥ 3.11, GPy ≥ 1.15):

```bash
python -m pip install -e .
# optional: matplotlib for figures; already a project dependency
```

USPS data are cached under `examples/damianou2013/data/zip.train.gz` on first run
(ElemStatLearn `zip.train`).

## Quick smoke tests

```bash
cd examples/damianou2013
python toy_hierarchy.py --quick --no-plot
python toy_warped_regression.py --quick --no-plot
python usps_digits.py --quick --no-plot --depths 1 2
python highfive_shared.py --quick --no-plot --synthetic
```

Save figures without interactive display:

```bash
python toy_hierarchy.py --quick --no-plot --outdir /tmp/dgp2013
```

## Paper-scale runs

Omit `--quick` (slower; especially USPS depth 1–5):

```bash
python toy_warped_regression.py --repeats 10 --outdir /tmp/dgp2013
python usps_digits.py --depths 1 2 3 4 5 --outdir /tmp/dgp2013
```

## High-five / MRD

The high-five experiment uses **conditionally independent observation views**
with a shared parent latent layer (`ObservedMRDLayer`, CIP-0002).
`highfive_shared.py` fits that model; use `--synthetic` when MOCAP data are
unavailable. See also ``examples/example_mrd.py``.

## Citation

```
@inproceedings{damianou2013deep,
title={Deep {G}aussian {P}rocesses},
author={Damianou, Andreas and Lawrence, Neil},
booktitle={AISTATS},
year={2013}
}
```
183 changes: 183 additions & 0 deletions examples/damianou2013/_common.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,183 @@
"""Shared helpers for Damianou & Lawrence (AISTATS 2013) recreation scripts."""

from __future__ import annotations

import argparse
import os
from typing import List, Optional, Tuple

import numpy as np

import GPy
import deepgp


def add_common_args(parser: argparse.ArgumentParser) -> argparse.ArgumentParser:
parser.add_argument(
"--quick",
action="store_true",
help="Fewer iterations / smaller models for a smoke test (not paper-faithful).",
)
parser.add_argument(
"--seed",
type=int,
default=42,
help="RNG seed (default: 42).",
)
parser.add_argument(
"--messages",
action="store_true",
help="Show optimiser progress messages.",
)
parser.add_argument(
"--no-plot",
action="store_true",
help="Skip matplotlib figures.",
)
parser.add_argument(
"--outdir",
type=str,
default=None,
help="If set, save figures under this directory.",
)
return parser


def ensure_outdir(outdir: Optional[str]) -> Optional[str]:
if outdir is None:
return None
os.makedirs(outdir, exist_ok=True)
return outdir


def save_or_show(fig, outdir: Optional[str], name: str, no_plot: bool) -> None:
import matplotlib.pyplot as plt

if outdir is not None:
path = os.path.join(outdir, name)
fig.savefig(path, bbox_inches="tight", dpi=120)
print(f"Wrote {path}")
if no_plot:
plt.close(fig)
else:
plt.show()


def rbf_bias(input_dim: int, ard: bool = True) -> GPy.kern.Kern:
return GPy.kern.RBF(input_dim, ARD=ard) + GPy.kern.Bias(input_dim)


def _output_variance(y) -> float:
"""Variance of observed data or of a variational posterior mean."""
from GPy.core.parameterization.variational import NormalPosterior, VariationalPosterior

if isinstance(y, (NormalPosterior, VariationalPosterior)):
return float(np.asarray(y.mean).var())
return float(np.asarray(y).var())


def init_noise(model: deepgp.DeepGP, fraction: float = 0.01, fix: bool = False) -> None:
for layer in model.layers:
if hasattr(layer, "views"):
for view in layer.views:
view.Gaussian_noise.variance = _output_variance(view.Y) * fraction
if fix:
view.Gaussian_noise.variance.fix()
else:
layer.Gaussian_noise.variance = _output_variance(layer.Y) * fraction
if fix:
layer.Gaussian_noise.variance.fix()


def unfix_noise(model: deepgp.DeepGP) -> None:
for layer in model.layers:
if hasattr(layer, "views"):
for view in layer.views:
view.Gaussian_noise.variance.unfix()
else:
layer.Gaussian_noise.variance.unfix()


def optimize_staged(
model: deepgp.DeepGP,
iters_fixed: int,
iters_free: int,
messages: bool = False,
) -> None:
"""Fix noise briefly, then free it — matches the shipped supervised examples."""
init_noise(model, fix=True)
if iters_fixed > 0:
model.optimize(max_iters=iters_fixed, messages=messages)
unfix_noise(model)
if iters_free > 0:
model.optimize(max_iters=iters_free, messages=messages)


def lengthscales(kern: GPy.kern.Kern) -> np.ndarray:
"""Return ARD lengthscales from an RBF(+Bias) kernel."""
if hasattr(kern, "rbf"):
return np.asarray(kern.rbf.lengthscale).ravel()
if hasattr(kern, "lengthscale"):
return np.asarray(kern.lengthscale).ravel()
# Additive kernels: prefer the first RBF-like part
if hasattr(kern, "parts"):
for part in kern.parts:
if hasattr(part, "lengthscale"):
return np.asarray(part.lengthscale).ravel()
raise AttributeError("Could not find lengthscale on kernel")


def ard_weights(kern: GPy.kern.Kern) -> np.ndarray:
"""Inverse lengthscale (larger ⇒ more relevant), as used in the paper plots."""
ls = lengthscales(kern)
return 1.0 / np.maximum(ls, 1e-12)


def print_ard(model: deepgp.DeepGP) -> None:
for i, layer in enumerate(model.layers):
try:
w = ard_weights(layer.kern)
except AttributeError:
continue
print(f"Layer {i} ARD (1/lengthscale): {np.array2string(w, precision=3)}")


def nearest_neighbour_error(X: np.ndarray, labels: np.ndarray) -> Tuple[int, float]:
"""Leave-one-out 1-NN error count and rate in Euclidean feature space."""
X = np.asarray(X, dtype=float)
labels = np.asarray(labels).ravel()
n = X.shape[0]
# Squared distances via (x-y)^2 = x^2 + y^2 - 2xy
sq = np.sum(X * X, axis=1, keepdims=True)
d2 = sq + sq.T - 2.0 * X @ X.T
np.fill_diagonal(d2, np.inf)
nn = np.argmin(d2, axis=1)
mistakes = int(np.sum(labels[nn] != labels))
return mistakes, mistakes / float(n)


def dominant_dims(weights: np.ndarray, k: int = 2) -> List[int]:
order = np.argsort(-weights)
return [int(i) for i in order[:k]]


def mse(pred: np.ndarray, target: np.ndarray) -> float:
return float(np.mean((np.asarray(pred) - np.asarray(target)) ** 2))


def rmse(pred: np.ndarray, target: np.ndarray) -> float:
return float(np.sqrt(mse(pred, target)))


def sample_gp_prior(
X: np.ndarray,
kernel: GPy.kern.Kern,
output_dim: int,
noise: float = 1e-6,
) -> np.ndarray:
"""Draw ``output_dim`` independent prior samples at inputs ``X``."""
K = kernel.K(X)
K = K + noise * np.eye(X.shape[0])
L = np.linalg.cholesky(K)
Z = np.random.randn(X.shape[0], output_dim)
return L @ Z
2 changes: 2 additions & 0 deletions examples/damianou2013/data/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Cached ElemStatLearn USPS split (downloaded by usps_digits.py)
zip.train.gz
Empty file.
Loading
Loading