Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,4 +92,5 @@ Examples

* ``examples/`` — short supervised / unsupervised demos (oil, CMU walk/jog).
* ``examples/damianou2013/`` — recreations of the AISTATS 2013 experiments
(toy hierarchy, warped regression, USPS depth/ARD, high-five / MRD).
(toy hierarchy, warped regression, USPS depth/ARD, high-five / MRD),
including Colab notebooks under ``examples/damianou2013/notebooks/``.
74 changes: 31 additions & 43 deletions examples/damianou2013/README.md
Original file line number Diff line number Diff line change
@@ -1,66 +1,54 @@
# Damianou & Lawrence (AISTATS 2013) — PyDeepGP recreations
# Damianou & Lawrence (AISTATS 2013) — PyDeepGP ports

Scripts that recreate the **feasible** experiments from
[Damianou & Lawrence, *Deep Gaussian Processes*](https://proceedings.mlr.press/v31/damianou13a.html)
(AISTATS 2013) on the maintained `DeepGP` path.
Ports of the official Matlab demos in
[SheffieldML/deepGP](https://github.com/SheffieldML/deepGP)
(`deepGP/matlab/`). See [SOURCE.md](SOURCE.md) for the file-level mapping.

These are **modern recreations**, not bit-identical paper scripts: optimiser
schedules, inducing-point counts, and initialisation differ from the 2013
Matlab stack. Use `--quick` for a short smoke run.
These are **not** bit-identical to Matlab (optimiser / kernel stack differ), but
they use the same reference data files, model widths, and evaluation metrics
as the demos — not invented protocols.

## Mapping
Paper: [Damianou & Lawrence, AISTATS 2013](https://proceedings.mlr.press/v31/damianou13a.html).

| Paper section | Script | Status |
## Notebooks (Colab)

| Paper | Notebook | Matlab demo |
| --- | --- | --- |
| §4.1 Toy hierarchy (fig. 2) | `toy_hierarchy.py` | Runnable |
| §4.1 Warped regression (fig. 3) | `toy_warped_regression.py` | Runnable |
| §4.3 USPS {0,1,6} depth / ARD / 1-NN (figs. 6–8) | `usps_digits.py` | Runnable (downloads ESL `zip.train`) |
| §4.2 CMU high-five shared views (figs. 4–5) | `highfive_shared.py` | Runnable via CIP-0002 MRD (synthetic or pods stand-in) |
| §4.1 fig. 2 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb) [01](notebooks/01_toy_hierarchy.ipynb) | `demToyUnsupervised.m` |
| §4.1 fig. 3 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb) [02](notebooks/02_toy_warped_regression.ipynb) | `demToyRegressionSimple.m` |
| §4.3 figs. 6–8 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/03_usps_digits.ipynb) [03](notebooks/03_usps_digits.ipynb) | `demUsps.m` / `demDigitsDemonstration.m` |
| §4.2 figs. 4–5 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/04_highfive_mrd.ipynb) [04](notebooks/04_highfive_mrd.ipynb) | `demHighFiveDemonstration.m` |

## Setup
Notebooks are walkthroughs (data → fit → plots), not wrappers around the
`.py` scripts. Until this lands on `main`, open Colab from the branch, e.g.
`…/blob/examples-damianou2013/examples/damianou2013/notebooks/…`.

From the repository root (Python ≥ 3.11, GPy ≥ 1.15):
## Scripts

```bash
python -m pip install -e .
# optional: matplotlib for figures; already a project dependency
```

USPS data are cached under `examples/damianou2013/data/zip.train.gz` on first run
(ElemStatLearn `zip.train`).

## Quick smoke tests
| Script | Data | Notes |
| --- | --- | --- |
| `toy_hierarchy.py` | `data/hgplvmSampleDataTr1.mat` | Two-view DeepGP Q={6,4}; vs stacked PCA & Isomap |
| `toy_warped_regression.py` | `hierGpsNEW` generator | Defaults = Matlab simple (N=150,D=15,Ntr=25); `--paper` → N=120,D=10 |
| `highfive_shared.py` | `data/demHighFiveHgplvm1.mat` | YA/YB 78×62; plots vs stored hgplvm latents |
| `usps_digits.py` | ESL `zip.train` | `vercat2` 50×{0,6,1}; Q={15,12,10,8,6}; K=80 |

```bash
cd examples/damianou2013
python toy_hierarchy.py --quick --no-plot
python toy_warped_regression.py --quick --no-plot
python usps_digits.py --quick --no-plot --depths 1 2
python highfive_shared.py --quick --no-plot --synthetic
python highfive_shared.py --quick --no-plot
python usps_digits.py --quick --no-plot --depths 1
```

Save figures without interactive display:
Omit `--quick` for demo-scale optimisation (slow, especially USPS depth 5).

```bash
python toy_hierarchy.py --quick --no-plot --outdir /tmp/dgp2013
```

## Paper-scale runs

Omit `--quick` (slower; especially USPS depth 1–5):
## Setup

```bash
python toy_warped_regression.py --repeats 10 --outdir /tmp/dgp2013
python usps_digits.py --depths 1 2 3 4 5 --outdir /tmp/dgp2013
python -m pip install -e . # from repo root
# Isomap comparison in toy_hierarchy: pip install scikit-learn
```

## High-five / MRD

The high-five experiment uses **conditionally independent observation views**
with a shared parent latent layer (`ObservedMRDLayer`, CIP-0002).
`highfive_shared.py` fits that model; use `--synthetic` when MOCAP data are
unavailable. See also ``examples/example_mrd.py``.

## Citation

```
Expand Down
20 changes: 20 additions & 0 deletions examples/damianou2013/SOURCE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Provenance: Matlab deepGP demos

Python scripts and notebooks in this directory are ports of the official
AISTATS 2013 Matlab package:

**https://github.com/SheffieldML/deepGP** (`deepGP/matlab/`)

| Paper | Matlab demo | Python entry |
| --- | --- | --- |
| §4.1 fig. 2 hierarchy | `demToyUnsupervised.m` + `hgplvmSampleDataTr1.mat` | `toy_hierarchy.py` |
| §4.1 fig. 3 regression | `demToyRegressionSimple.m` + `demToyDynamicsCreateData.m` (`hierGpsNEW`) | `toy_warped_regression.py` |
| §4.2 figs. 4–5 high-five | `demHighFiveDemonstration.m` + `demHighFiveHgplvm1.mat` | `highfive_shared.py` |
| §4.3 figs. 6–8 USPS | `demUsps.m` / `demDigitsDemonstration.m` (`vercat2`, H=5, Q={15,12,10,8,6}) | `usps_digits.py` |

Walkthrough narrative: `tutorial.m`.

These are still **not** bit-identical to Matlab (different optimiser stack,
kernel implementations, and PyDeepGP API), but data construction, model
depth/width, and evaluation metrics follow the demos above rather than
ad-hoc settings.
161 changes: 161 additions & 0 deletions examples/damianou2013/_matlab_data.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
"""Loaders and generators matching SheffieldML/deepGP Matlab demos."""

from __future__ import annotations

import os
from typing import Dict, List, Optional, Tuple

import numpy as np
import scipy.io as sio

_DATA_DIR = os.path.join(os.path.dirname(__file__), "data")


def data_path(name: str) -> str:
return os.path.join(_DATA_DIR, name)


def load_toy_hierarchy(mat_name: str = "hgplvmSampleDataTr1.mat") -> Dict[str, np.ndarray]:
"""Ground-truth hierarchy from ``demToyUnsupervised.m`` / ``hgplvmSampleDataTr1.mat``.

Returns arrays ``X2`` (top), ``XA``, ``XB`` (intermediate), ``YA``, ``YB`` (leaves).
"""
path = data_path(mat_name)
d = sio.loadmat(path, squeeze_me=True)
out = {k: np.asarray(d[k], dtype=float) for k in ("X2", "XA", "XB", "YA", "YB")}
for k, arr in out.items():
if arr.ndim == 1:
out[k] = arr[:, None]
return out


def load_highfive(mat_name: str = "demHighFiveHgplvm1.mat") -> Dict[str, np.ndarray]:
"""CMU high-five views from ``demHighFiveDemonstration.m``.

Returns ``YA``, ``YB`` (78×62) and optional hgplvm latents ``Xparent``,
``Xchild1``, ``Xchild2``.
"""
path = data_path(mat_name)
d = sio.loadmat(path, struct_as_record=False, squeeze_me=True)
Y = d["Y"]
out = {
"YA": np.asarray(Y.YA, dtype=float),
"YB": np.asarray(Y.YB, dtype=float),
}
for name in ("Xparent", "Xchild1", "Xchild2"):
if hasattr(Y, name):
out[name] = np.asarray(getattr(Y, name), dtype=float)
return out


def scale_data(X: np.ndarray) -> np.ndarray:
"""Zero-mean unit-variance columns (Matlab ``scaleData(X,1)``)."""
X = np.asarray(X, dtype=float)
X = X - X.mean(axis=0, keepdims=True)
std = X.std(axis=0, keepdims=True)
std[std < 1e-12] = 1.0
return X / std


def generate_hier_gps_new(
n_toy: int = 150,
d_toy: int = 15,
seed: int = 2,
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
"""Match ``demToyDynamicsCreateData.m`` branch ``hierGpsNEW``.

Returns ``(X, Y1_warp, Y)`` where ``X`` is the observed input,
``Y1_warp`` is the hidden 1D warp, and ``Y`` is the D-dimensional output.
"""
import GPy

rng = np.random.RandomState(seed)
# Matlab: X = linspace(1,2*pi,Ntoy)'; scaleData
X = np.linspace(1.0, 2.0 * np.pi, n_toy)[:, None]
X = scale_data(X)

# First GP: rbf + lin + white, inverseWidth=4 → lengthscale = 0.5
kern1 = (
GPy.kern.RBF(1, variance=1.0, lengthscale=0.5)
+ GPy.kern.Linear(1, variances=1.0)
+ GPy.kern.White(1, variance=1e-6)
)
K1 = kern1.K(X)
# gsamp: one sample from N(0,K)
Y1 = rng.multivariate_normal(np.zeros(n_toy), K1)[:, None]

# Second GP: rbf on Y1, inverseWidth=16 → lengthscale=0.25
kern2 = GPy.kern.RBF(1, variance=1.0, lengthscale=0.25) + GPy.kern.White(
1, variance=1e-6
)
K2 = kern2.K(Y1)
# D independent samples
L2 = np.linalg.cholesky(K2 + 1e-10 * np.eye(n_toy))
Y = L2 @ rng.randn(n_toy, d_toy)
return X, Y1, Y


def train_test_split_indices(n_toy: int, n_tr: int, seed: int) -> Tuple[np.ndarray, np.ndarray]:
"""Match ``demToyDynamicsSplitDataset.m`` (non-block): random ``Ntr`` train indices."""
rng = np.random.RandomState(seed)
perm = rng.permutation(n_toy)
ind_tr = np.sort(perm[:n_tr])
ind_ts = np.sort(perm[n_tr:])
return ind_tr, ind_ts


def load_usps_vercat2(
per_class: int = 50,
cache_dir: Optional[str] = None,
) -> Tuple[np.ndarray, np.ndarray]:
"""USPS digits {0,1,6} as in ``demUsps.m`` / ``demDigitsDemonstration.m`` ``vercat2``.

Matlab indexing (1-based contiguous blocks in the ESL / GPmat layout) is
approximated by taking the first ``per_class`` examples of each digit from
ESL ``zip.train`` (same counts; order may differ slightly from GPmat).
"""
import gzip
import urllib.request

if cache_dir is None:
cache_dir = _DATA_DIR
os.makedirs(cache_dir, exist_ok=True)
path = os.path.join(cache_dir, "zip.train.gz")
if not (os.path.isfile(path) and os.path.getsize(path) > 0):
url = "https://web.stanford.edu/~hastie/ElemStatLearn/datasets/zip.train.gz"
req = urllib.request.Request(url, headers={"User-Agent": "PyDeepGP-damianou2013"})
with urllib.request.urlopen(req, timeout=120) as resp, open(path, "wb") as out:
out.write(resp.read())

rows = []
with gzip.open(path, "rt") as f:
for line in f:
parts = line.strip().split()
if parts:
rows.append([float(x) for x in parts])
data = np.asarray(rows, dtype=float)
labels = data[:, 0].astype(int)
Y = data[:, 1:]

pieces_Y: List[np.ndarray] = []
pieces_L: List[np.ndarray] = []
# Matlab vercat2 order: 0, 6, 1
for digit in (0, 6, 1):
idx = np.where(labels == digit)[0]
if idx.size < per_class:
raise RuntimeError(f"Need {per_class} examples of digit {digit}, found {idx.size}")
# Take the first per_class in file order (stable, closer to block loads)
choose = idx[:per_class]
pieces_Y.append(Y[choose])
pieces_L.append(np.full(per_class, digit, dtype=int))
return np.vstack(pieces_Y), np.concatenate(pieces_L)


# Matlab demDigitsDemonstration experiment 19 widths
USPS_DEPTH_Q = {
1: [15],
2: [15, 12],
3: [15, 12, 10],
4: [15, 12, 10, 8],
5: [15, 12, 10, 8, 6],
}
8 changes: 8 additions & 0 deletions examples/damianou2013/data/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Reference data from SheffieldML/deepGP (Matlab)

| File | Demo | Role |
| --- | --- | --- |
| `hgplvmSampleDataTr1.mat` | `demToyUnsupervised.m` (`toyType='hgplvmSampleTr1'`) | Ground-truth hierarchy: `X2`, `XA`, `XB`, `YA`, `YB` |
| `demHighFiveHgplvm1.mat` | `demHighFiveDemonstration.m` / `demHsvargplvmHighFive1.m` | CMU high-five views `Y.YA`, `Y.YB` (78×62 each) |

Source repo: https://github.com/SheffieldML/deepGP
Binary file added examples/damianou2013/data/demHighFiveHgplvm1.mat
Binary file not shown.
Binary file not shown.
Loading
Loading