diff --git a/README.md b/README.md index 18d49b1..52bbaea 100644 --- a/README.md +++ b/README.md @@ -92,4 +92,5 @@ Examples * ``examples/`` — short supervised / unsupervised demos (oil, CMU walk/jog). * ``examples/damianou2013/`` — recreations of the AISTATS 2013 experiments - (toy hierarchy, warped regression, USPS depth/ARD, high-five / MRD). + (toy hierarchy, warped regression, USPS depth/ARD, high-five / MRD), + including Colab notebooks under ``examples/damianou2013/notebooks/``. diff --git a/examples/damianou2013/README.md b/examples/damianou2013/README.md index a461aa3..c6c84d7 100644 --- a/examples/damianou2013/README.md +++ b/examples/damianou2013/README.md @@ -1,66 +1,54 @@ -# Damianou & Lawrence (AISTATS 2013) — PyDeepGP recreations +# Damianou & Lawrence (AISTATS 2013) — PyDeepGP ports -Scripts that recreate the **feasible** experiments from -[Damianou & Lawrence, *Deep Gaussian Processes*](https://proceedings.mlr.press/v31/damianou13a.html) -(AISTATS 2013) on the maintained `DeepGP` path. +Ports of the official Matlab demos in +[SheffieldML/deepGP](https://github.com/SheffieldML/deepGP) +(`deepGP/matlab/`). See [SOURCE.md](SOURCE.md) for the file-level mapping. -These are **modern recreations**, not bit-identical paper scripts: optimiser -schedules, inducing-point counts, and initialisation differ from the 2013 -Matlab stack. Use `--quick` for a short smoke run. +These are **not** bit-identical to Matlab (optimiser / kernel stack differ), but +they use the same reference data files, model widths, and evaluation metrics +as the demos — not invented protocols. -## Mapping +Paper: [Damianou & Lawrence, AISTATS 2013](https://proceedings.mlr.press/v31/damianou13a.html). -| Paper section | Script | Status | +## Notebooks (Colab) + +| Paper | Notebook | Matlab demo | | --- | --- | --- | -| §4.1 Toy hierarchy (fig. 2) | `toy_hierarchy.py` | Runnable | -| §4.1 Warped regression (fig. 3) | `toy_warped_regression.py` | Runnable | -| §4.3 USPS {0,1,6} depth / ARD / 1-NN (figs. 6–8) | `usps_digits.py` | Runnable (downloads ESL `zip.train`) | -| §4.2 CMU high-five shared views (figs. 4–5) | `highfive_shared.py` | Runnable via CIP-0002 MRD (synthetic or pods stand-in) | +| §4.1 fig. 2 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb) [01](notebooks/01_toy_hierarchy.ipynb) | `demToyUnsupervised.m` | +| §4.1 fig. 3 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb) [02](notebooks/02_toy_warped_regression.ipynb) | `demToyRegressionSimple.m` | +| §4.3 figs. 6–8 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/03_usps_digits.ipynb) [03](notebooks/03_usps_digits.ipynb) | `demUsps.m` / `demDigitsDemonstration.m` | +| §4.2 figs. 4–5 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/04_highfive_mrd.ipynb) [04](notebooks/04_highfive_mrd.ipynb) | `demHighFiveDemonstration.m` | -## Setup +Notebooks are walkthroughs (data → fit → plots), not wrappers around the +`.py` scripts. Until this lands on `main`, open Colab from the branch, e.g. +`…/blob/examples-damianou2013/examples/damianou2013/notebooks/…`. -From the repository root (Python ≥ 3.11, GPy ≥ 1.15): +## Scripts -```bash -python -m pip install -e . -# optional: matplotlib for figures; already a project dependency -``` - -USPS data are cached under `examples/damianou2013/data/zip.train.gz` on first run -(ElemStatLearn `zip.train`). - -## Quick smoke tests +| Script | Data | Notes | +| --- | --- | --- | +| `toy_hierarchy.py` | `data/hgplvmSampleDataTr1.mat` | Two-view DeepGP Q={6,4}; vs stacked PCA & Isomap | +| `toy_warped_regression.py` | `hierGpsNEW` generator | Defaults = Matlab simple (N=150,D=15,Ntr=25); `--paper` → N=120,D=10 | +| `highfive_shared.py` | `data/demHighFiveHgplvm1.mat` | YA/YB 78×62; plots vs stored hgplvm latents | +| `usps_digits.py` | ESL `zip.train` | `vercat2` 50×{0,6,1}; Q={15,12,10,8,6}; K=80 | ```bash cd examples/damianou2013 python toy_hierarchy.py --quick --no-plot python toy_warped_regression.py --quick --no-plot -python usps_digits.py --quick --no-plot --depths 1 2 -python highfive_shared.py --quick --no-plot --synthetic +python highfive_shared.py --quick --no-plot +python usps_digits.py --quick --no-plot --depths 1 ``` -Save figures without interactive display: +Omit `--quick` for demo-scale optimisation (slow, especially USPS depth 5). -```bash -python toy_hierarchy.py --quick --no-plot --outdir /tmp/dgp2013 -``` - -## Paper-scale runs - -Omit `--quick` (slower; especially USPS depth 1–5): +## Setup ```bash -python toy_warped_regression.py --repeats 10 --outdir /tmp/dgp2013 -python usps_digits.py --depths 1 2 3 4 5 --outdir /tmp/dgp2013 +python -m pip install -e . # from repo root +# Isomap comparison in toy_hierarchy: pip install scikit-learn ``` -## High-five / MRD - -The high-five experiment uses **conditionally independent observation views** -with a shared parent latent layer (`ObservedMRDLayer`, CIP-0002). -`highfive_shared.py` fits that model; use `--synthetic` when MOCAP data are -unavailable. See also ``examples/example_mrd.py``. - ## Citation ``` diff --git a/examples/damianou2013/SOURCE.md b/examples/damianou2013/SOURCE.md new file mode 100644 index 0000000..56b2bc2 --- /dev/null +++ b/examples/damianou2013/SOURCE.md @@ -0,0 +1,20 @@ +# Provenance: Matlab deepGP demos + +Python scripts and notebooks in this directory are ports of the official +AISTATS 2013 Matlab package: + +**https://github.com/SheffieldML/deepGP** (`deepGP/matlab/`) + +| Paper | Matlab demo | Python entry | +| --- | --- | --- | +| §4.1 fig. 2 hierarchy | `demToyUnsupervised.m` + `hgplvmSampleDataTr1.mat` | `toy_hierarchy.py` | +| §4.1 fig. 3 regression | `demToyRegressionSimple.m` + `demToyDynamicsCreateData.m` (`hierGpsNEW`) | `toy_warped_regression.py` | +| §4.2 figs. 4–5 high-five | `demHighFiveDemonstration.m` + `demHighFiveHgplvm1.mat` | `highfive_shared.py` | +| §4.3 figs. 6–8 USPS | `demUsps.m` / `demDigitsDemonstration.m` (`vercat2`, H=5, Q={15,12,10,8,6}) | `usps_digits.py` | + +Walkthrough narrative: `tutorial.m`. + +These are still **not** bit-identical to Matlab (different optimiser stack, +kernel implementations, and PyDeepGP API), but data construction, model +depth/width, and evaluation metrics follow the demos above rather than +ad-hoc settings. diff --git a/examples/damianou2013/_matlab_data.py b/examples/damianou2013/_matlab_data.py new file mode 100644 index 0000000..cedefc1 --- /dev/null +++ b/examples/damianou2013/_matlab_data.py @@ -0,0 +1,161 @@ +"""Loaders and generators matching SheffieldML/deepGP Matlab demos.""" + +from __future__ import annotations + +import os +from typing import Dict, List, Optional, Tuple + +import numpy as np +import scipy.io as sio + +_DATA_DIR = os.path.join(os.path.dirname(__file__), "data") + + +def data_path(name: str) -> str: + return os.path.join(_DATA_DIR, name) + + +def load_toy_hierarchy(mat_name: str = "hgplvmSampleDataTr1.mat") -> Dict[str, np.ndarray]: + """Ground-truth hierarchy from ``demToyUnsupervised.m`` / ``hgplvmSampleDataTr1.mat``. + + Returns arrays ``X2`` (top), ``XA``, ``XB`` (intermediate), ``YA``, ``YB`` (leaves). + """ + path = data_path(mat_name) + d = sio.loadmat(path, squeeze_me=True) + out = {k: np.asarray(d[k], dtype=float) for k in ("X2", "XA", "XB", "YA", "YB")} + for k, arr in out.items(): + if arr.ndim == 1: + out[k] = arr[:, None] + return out + + +def load_highfive(mat_name: str = "demHighFiveHgplvm1.mat") -> Dict[str, np.ndarray]: + """CMU high-five views from ``demHighFiveDemonstration.m``. + + Returns ``YA``, ``YB`` (78×62) and optional hgplvm latents ``Xparent``, + ``Xchild1``, ``Xchild2``. + """ + path = data_path(mat_name) + d = sio.loadmat(path, struct_as_record=False, squeeze_me=True) + Y = d["Y"] + out = { + "YA": np.asarray(Y.YA, dtype=float), + "YB": np.asarray(Y.YB, dtype=float), + } + for name in ("Xparent", "Xchild1", "Xchild2"): + if hasattr(Y, name): + out[name] = np.asarray(getattr(Y, name), dtype=float) + return out + + +def scale_data(X: np.ndarray) -> np.ndarray: + """Zero-mean unit-variance columns (Matlab ``scaleData(X,1)``).""" + X = np.asarray(X, dtype=float) + X = X - X.mean(axis=0, keepdims=True) + std = X.std(axis=0, keepdims=True) + std[std < 1e-12] = 1.0 + return X / std + + +def generate_hier_gps_new( + n_toy: int = 150, + d_toy: int = 15, + seed: int = 2, +) -> Tuple[np.ndarray, np.ndarray, np.ndarray]: + """Match ``demToyDynamicsCreateData.m`` branch ``hierGpsNEW``. + + Returns ``(X, Y1_warp, Y)`` where ``X`` is the observed input, + ``Y1_warp`` is the hidden 1D warp, and ``Y`` is the D-dimensional output. + """ + import GPy + + rng = np.random.RandomState(seed) + # Matlab: X = linspace(1,2*pi,Ntoy)'; scaleData + X = np.linspace(1.0, 2.0 * np.pi, n_toy)[:, None] + X = scale_data(X) + + # First GP: rbf + lin + white, inverseWidth=4 → lengthscale = 0.5 + kern1 = ( + GPy.kern.RBF(1, variance=1.0, lengthscale=0.5) + + GPy.kern.Linear(1, variances=1.0) + + GPy.kern.White(1, variance=1e-6) + ) + K1 = kern1.K(X) + # gsamp: one sample from N(0,K) + Y1 = rng.multivariate_normal(np.zeros(n_toy), K1)[:, None] + + # Second GP: rbf on Y1, inverseWidth=16 → lengthscale=0.25 + kern2 = GPy.kern.RBF(1, variance=1.0, lengthscale=0.25) + GPy.kern.White( + 1, variance=1e-6 + ) + K2 = kern2.K(Y1) + # D independent samples + L2 = np.linalg.cholesky(K2 + 1e-10 * np.eye(n_toy)) + Y = L2 @ rng.randn(n_toy, d_toy) + return X, Y1, Y + + +def train_test_split_indices(n_toy: int, n_tr: int, seed: int) -> Tuple[np.ndarray, np.ndarray]: + """Match ``demToyDynamicsSplitDataset.m`` (non-block): random ``Ntr`` train indices.""" + rng = np.random.RandomState(seed) + perm = rng.permutation(n_toy) + ind_tr = np.sort(perm[:n_tr]) + ind_ts = np.sort(perm[n_tr:]) + return ind_tr, ind_ts + + +def load_usps_vercat2( + per_class: int = 50, + cache_dir: Optional[str] = None, +) -> Tuple[np.ndarray, np.ndarray]: + """USPS digits {0,1,6} as in ``demUsps.m`` / ``demDigitsDemonstration.m`` ``vercat2``. + + Matlab indexing (1-based contiguous blocks in the ESL / GPmat layout) is + approximated by taking the first ``per_class`` examples of each digit from + ESL ``zip.train`` (same counts; order may differ slightly from GPmat). + """ + import gzip + import urllib.request + + if cache_dir is None: + cache_dir = _DATA_DIR + os.makedirs(cache_dir, exist_ok=True) + path = os.path.join(cache_dir, "zip.train.gz") + if not (os.path.isfile(path) and os.path.getsize(path) > 0): + url = "https://web.stanford.edu/~hastie/ElemStatLearn/datasets/zip.train.gz" + req = urllib.request.Request(url, headers={"User-Agent": "PyDeepGP-damianou2013"}) + with urllib.request.urlopen(req, timeout=120) as resp, open(path, "wb") as out: + out.write(resp.read()) + + rows = [] + with gzip.open(path, "rt") as f: + for line in f: + parts = line.strip().split() + if parts: + rows.append([float(x) for x in parts]) + data = np.asarray(rows, dtype=float) + labels = data[:, 0].astype(int) + Y = data[:, 1:] + + pieces_Y: List[np.ndarray] = [] + pieces_L: List[np.ndarray] = [] + # Matlab vercat2 order: 0, 6, 1 + for digit in (0, 6, 1): + idx = np.where(labels == digit)[0] + if idx.size < per_class: + raise RuntimeError(f"Need {per_class} examples of digit {digit}, found {idx.size}") + # Take the first per_class in file order (stable, closer to block loads) + choose = idx[:per_class] + pieces_Y.append(Y[choose]) + pieces_L.append(np.full(per_class, digit, dtype=int)) + return np.vstack(pieces_Y), np.concatenate(pieces_L) + + +# Matlab demDigitsDemonstration experiment 19 widths +USPS_DEPTH_Q = { + 1: [15], + 2: [15, 12], + 3: [15, 12, 10], + 4: [15, 12, 10, 8], + 5: [15, 12, 10, 8, 6], +} diff --git a/examples/damianou2013/data/README.md b/examples/damianou2013/data/README.md new file mode 100644 index 0000000..3a6ce46 --- /dev/null +++ b/examples/damianou2013/data/README.md @@ -0,0 +1,8 @@ +# Reference data from SheffieldML/deepGP (Matlab) + +| File | Demo | Role | +| --- | --- | --- | +| `hgplvmSampleDataTr1.mat` | `demToyUnsupervised.m` (`toyType='hgplvmSampleTr1'`) | Ground-truth hierarchy: `X2`, `XA`, `XB`, `YA`, `YB` | +| `demHighFiveHgplvm1.mat` | `demHighFiveDemonstration.m` / `demHsvargplvmHighFive1.m` | CMU high-five views `Y.YA`, `Y.YB` (78×62 each) | + +Source repo: https://github.com/SheffieldML/deepGP diff --git a/examples/damianou2013/data/demHighFiveHgplvm1.mat b/examples/damianou2013/data/demHighFiveHgplvm1.mat new file mode 100644 index 0000000..f09b2f4 Binary files /dev/null and b/examples/damianou2013/data/demHighFiveHgplvm1.mat differ diff --git a/examples/damianou2013/data/hgplvmSampleDataTr1.mat b/examples/damianou2013/data/hgplvmSampleDataTr1.mat new file mode 100644 index 0000000..91d935a Binary files /dev/null and b/examples/damianou2013/data/hgplvmSampleDataTr1.mat differ diff --git a/examples/damianou2013/highfive_shared.py b/examples/damianou2013/highfive_shared.py index afd26a9..c58f9fe 100644 --- a/examples/damianou2013/highfive_shared.py +++ b/examples/damianou2013/highfive_shared.py @@ -1,15 +1,9 @@ #!/usr/bin/env python -"""Section 4.2 (figs. 4–5): CMU high-five — shared / MRD hierarchy. +"""§4.2 / figs. 4–5 — CMU high-five shared MRD (Matlab ``demHighFiveDemonstration.m``). -Paper: Damianou & Lawrence, Deep Gaussian Processes, AISTATS 2013. - -Two observation views share a parent latent layer (conditionally independent -given the shared features). With CIP-0002, this uses the maintained -``ObservedMRDLayer`` path. - -Real CMU high-five MOCAP (two subjects, ~78 frames × 62 dims) is used when -``pods`` can supply a suitable sequence; otherwise the script falls back to -synthetic correlated views so the shared-hierarchy fit still runs. +Loads ``demHighFiveHgplvm1.mat`` (subjects A/B, 78×62 each) and fits a two-view +DeepGP. Compares ARD / shared latents to the stored hgplvm reference spaces in +the same mat file when present. """ from __future__ import annotations @@ -25,92 +19,56 @@ from _common import ( add_common_args, ard_weights, + dominant_dims, ensure_outdir, init_noise, - print_ard, rbf_bias, save_or_show, unfix_noise, ) +from _matlab_data import load_highfive def build_parser() -> argparse.ArgumentParser: p = argparse.ArgumentParser(description=__doc__) add_common_args(p) - p.add_argument( - "--synthetic", - action="store_true", - help="Force synthetic correlated views (skip pods MOCAP).", - ) return p -def load_two_views(seed: int, synthetic: bool): - """Return (Y1, Y2, source_label).""" - if not synthetic: - try: - import pods - - if hasattr(pods.datasets, "cmu_mocap_35_walk_jog"): - data = pods.datasets.cmu_mocap_35_walk_jog() - Y = np.asarray(data["Y"], dtype=float) - n = min(80, Y.shape[0]) - Y = Y[:n] - mid = Y.shape[1] // 2 - print( - "Note: using walk/jog feature split as a two-view stand-in; " - "paper §4.2 used the CMU high-five pair of subjects." - ) - return Y[:, :mid], Y[:, mid:], "pods:cmu_mocap_35_walk_jog" - except Exception as exc: - print(f"pods MOCAP unavailable ({exc}); using synthetic views.") - - rng = np.random.RandomState(seed) - n, q, d = 60, 3, 20 - X = rng.randn(n, q) - Y1 = X @ rng.randn(q, d) + 0.05 * rng.randn(n, d) - Y2 = X @ rng.randn(q, d) + 0.05 * rng.randn(n, d) - return Y1, Y2, "synthetic" - - -def fit_mrd(Y1, Y2, quick: bool, messages: bool) -> deepgp.DeepGP: - q1, q2 = (4, 3) if quick else (6, 4) - num_inducing = 15 if quick else 30 - iters_fixed = 40 if quick else 200 - iters_free = 80 if quick else 800 - kernels = [ - [rbf_bias(q1, ard=True), rbf_bias(q1, ard=True)], - rbf_bias(q2, ard=True), - ] +def main(argv: list[str] | None = None) -> int: + args = build_parser().parse_args(argv) + outdir = ensure_outdir(args.outdir) + + data = load_highfive() + YA, YB = data["YA"].copy(), data["YB"].copy() + YA = (YA - YA.mean(0)) / (YA.std() + 1e-12) + YB = (YB - YB.mean(0)) / (YB.std() + 1e-12) + print(f"Loaded demHighFiveHgplvm1: YA={YA.shape} YB={YB.shape}") + + # Two-level hierarchy as in the paper / demHighFiveDemonstration + q1, q2 = (4, 3) if args.quick else (6, 4) + num_inducing = 15 if args.quick else 30 + iters_fixed = 40 if args.quick else 200 + iters_free = 80 if args.quick else 1500 + m = deepgp.DeepGP( - [[Y1.shape[1], Y2.shape[1]], q1, q2], - [Y1, Y2], - kernels=kernels, + [[YA.shape[1], YB.shape[1]], q1, q2], + [YA, YB], + kernels=[[rbf_bias(q1), rbf_bias(q1)], rbf_bias(q2)], num_inducing=num_inducing, back_constraint=False, + inits="PCA", ) init_noise(m, fix=True) - m.optimize(max_iters=iters_fixed, messages=messages) + m.optimize(max_iters=iters_fixed, messages=args.messages) unfix_noise(m) - m.optimize(max_iters=iters_free, messages=messages) - return m - - -def main(argv: list[str] | None = None) -> int: - args = build_parser().parse_args(argv) - outdir = ensure_outdir(args.outdir) - - Y1, Y2, source = load_two_views(args.seed, synthetic=args.synthetic or args.quick) - Y1 = (Y1 - Y1.mean(0)) / (Y1.std() + 1e-12) - Y2 = (Y2 - Y2.mean(0)) / (Y2.std() + 1e-12) - print(f"Views: {Y1.shape}, {Y2.shape} (source={source})") - - m = fit_mrd(Y1, Y2, quick=args.quick, messages=args.messages) + m.optimize(max_iters=iters_free, messages=args.messages) print(f"MRD DeepGP variational bound: {float(m.log_likelihood()):.3f}") + for i, view in enumerate(m.layers[0].views): w = ard_weights(view.kern) print(f"View {i} ARD (1/lengthscale): {np.array2string(w, precision=3)}") - print_ard(m) + print(f"Top ARD: {np.array2string(ard_weights(m.layers[-1].kern), precision=3)}") if not args.no_plot or outdir: import matplotlib.pyplot as plt @@ -119,16 +77,41 @@ def main(argv: list[str] | None = None) -> int: for i, view in enumerate(m.layers[0].views): w = ard_weights(view.kern) axes[i].bar(np.arange(1, len(w) + 1), w) - axes[i].set_title(f"view {i} ARD") + axes[i].set_title(f"view {i} ARD (fY)") axes[i].set_xlabel("shared dim") w_top = ard_weights(m.layers[-1].kern) axes[2].bar(np.arange(1, len(w_top) + 1), w_top) - axes[2].set_title("top layer ARD") - axes[2].set_xlabel("dimension") - fig.suptitle("Damianou & Lawrence 2013 — high-five / shared MRD (fig. 4 style)") + axes[2].set_title("top ARD (fX)") + fig.suptitle("High-five MRD ARD (fig. 4 style)") fig.tight_layout() save_or_show(fig, outdir, "highfive_ard.png", args.no_plot) + Xshare = np.asarray(m.layers[0].X.mean) + Xtop = np.asarray(m.layers[-1].X.mean) + d_share = dominant_dims(ard_weights(m.layers[0].views[0].kern), k=2) + d_top = dominant_dims(ard_weights(m.layers[-1].kern), k=2) + + fig2, axes = plt.subplots(1, 2, figsize=(8, 3.5)) + axes[0].plot(Xshare[:, d_share[0]], Xshare[:, d_share[1]], ".-") + axes[0].set_title("DeepGP shared (dominant)") + axes[1].plot(Xtop[:, d_top[0]], Xtop[:, d_top[1]], ".-") + axes[1].set_title("DeepGP top (dominant)") + fig2.suptitle("Latent projections (fig. 5 style)") + fig2.tight_layout() + save_or_show(fig2, outdir, "highfive_latents.png", args.no_plot) + + if "Xparent" in data: + fig3, axes = plt.subplots(1, 3, figsize=(10, 3.2)) + axes[0].plot(data["Xparent"][:, 0], data["Xparent"][:, 1], ".-") + axes[0].set_title("hgplvm Xparent (ref)") + axes[1].plot(data["Xchild1"][:, 0], data["Xchild1"][:, 1], ".-") + axes[1].set_title("hgplvm Xchild1 (ref)") + axes[2].plot(data["Xchild2"][:, 0], data["Xchild2"][:, 1], ".-") + axes[2].set_title("hgplvm Xchild2 (ref)") + fig3.suptitle("Reference latents from demHighFiveHgplvm1.mat") + fig3.tight_layout() + save_or_show(fig3, outdir, "highfive_hgplvm_ref.png", args.no_plot) + return 0 diff --git a/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb b/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb new file mode 100644 index 0000000..a1f4104 --- /dev/null +++ b/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb @@ -0,0 +1,254 @@ +{ + "nbformat": 4, + "nbformat_minor": 5, + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + } + }, + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Toy hierarchy \u2014 \u00a74.1 / fig. 2\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb)\n", + "\n", + "Port of Matlab [`demToyUnsupervised.m`](https://github.com/SheffieldML/deepGP/blob/master/deepGP/matlab/demToyUnsupervised.m) using\n", + "`hgplvmSampleDataTr1.mat`: two observation views `YA`, `YB` share a parent latent\n", + "hierarchy. We fit a two-view DeepGP (Q={6,4}) and compare stacked PCA / Isomap.\n", + "\n", + "Paper: [Damianou & Lawrence, AISTATS 2013](https://proceedings.mlr.press/v31/damianou13a.html).\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import sys, subprocess, os, urllib.request\n", + "from pathlib import Path\n", + "\n", + "IN_COLAB = \"google.colab\" in sys.modules\n", + "ROOT = Path.cwd().resolve()\n", + "# Prefer sibling of notebooks/ (repo layout)\n", + "for c in [ROOT.parent, ROOT, ROOT / \"examples\" / \"damianou2013\"]:\n", + " if (c / \"_common.py\").is_file():\n", + " ROOT = c\n", + " break\n", + "\n", + "if IN_COLAB:\n", + " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", + " \"numpy\", \"scipy\", \"matplotlib\", \"GPy>=1.15\", \"scikit-learn\"])\n", + " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", + " f\"git+https://github.com/SheffieldML/PyDeepGP.git@examples-damianou2013\"])\n", + " ROOT = Path.cwd().resolve()\n", + " (ROOT / \"data\").mkdir(exist_ok=True)\n", + " for dest, url in [\n", + " (\"_common.py\", f\"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_common.py\"),\n", + " (\"_matlab_data.py\", f\"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_matlab_data.py\"),\n", + " (\"data/hgplvmSampleDataTr1.mat\",\n", + " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/hgplvmSampleDataTr1.mat\"),\n", + " (\"data/demHighFiveHgplvm1.mat\",\n", + " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/demHighFiveHgplvm1.mat\"),\n", + " ]:\n", + " p = ROOT / dest\n", + " if not p.is_file():\n", + " print(\"Fetching\", dest)\n", + " urllib.request.urlretrieve(url, str(p))\n", + "\n", + "os.chdir(ROOT)\n", + "sys.path.insert(0, str(ROOT))\n", + "print(\"Working directory:\", ROOT)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%matplotlib inline\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "import deepgp\n", + "from _common import (\n", + " ard_weights, dominant_dims, init_noise, unfix_noise, rbf_bias, optimize_staged,\n", + ")\n", + "from _matlab_data import load_toy_hierarchy\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "QUICK = True # set False for Matlab-demo scale (Q={6,4}, longer optimisation)\n", + "q_shared, q_top = (4, 3) if QUICK else (6, 4)\n", + "num_inducing = 15 if QUICK else 25\n", + "iters_fixed = 50 if QUICK else 200\n", + "iters_free = 150 if QUICK else 2000\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Load Matlab ground-truth hierarchy\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "data = load_toy_hierarchy()\n", + "YA, YB = data[\"YA\"], data[\"YB\"]\n", + "X2, XA, XB = data[\"X2\"], data[\"XA\"], data[\"XB\"]\n", + "print(f\"YA={YA.shape} YB={YB.shape} XA={XA.shape} XB={XB.shape} X2={X2.shape}\")\n", + "\n", + "fig, axes = plt.subplots(3, 2, figsize=(9, 8))\n", + "axes[0, 0].plot(X2); axes[0, 0].set_title(\"True top X2\")\n", + "axes[0, 1].axis(\"off\")\n", + "axes[1, 0].plot(XA); axes[1, 0].set_title(\"True XA\")\n", + "axes[1, 1].plot(XB); axes[1, 1].set_title(\"True XB\")\n", + "axes[2, 0].plot(YA); axes[2, 0].set_title(\"YA\")\n", + "axes[2, 1].plot(YB); axes[2, 1].set_title(\"YB\")\n", + "fig.suptitle(\"Matlab hgplvmSampleDataTr1 ground truth\")\n", + "fig.tight_layout()\n", + "plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Fit two-view DeepGP (Q shared \u2192 top)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "kernels = [[rbf_bias(q_shared), rbf_bias(q_shared)], rbf_bias(q_top)]\n", + "m = deepgp.DeepGP(\n", + " [[YA.shape[1], YB.shape[1]], q_shared, q_top],\n", + " [YA, YB],\n", + " kernels=kernels,\n", + " num_inducing=num_inducing,\n", + " back_constraint=False,\n", + " inits=\"PCA\",\n", + ")\n", + "optimize_staged(m, iters_fixed, iters_free, messages=False)\n", + "print(f\"variational bound: {float(m.log_likelihood()):.3f}\")\n", + "for i, view in enumerate(m.layers[0].views):\n", + " print(f\"View {i} ARD:\", np.array2string(ard_weights(view.kern), precision=3))\n", + "print(\"Top ARD:\", np.array2string(ard_weights(m.layers[-1].kern), precision=3))\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Recovered latents (dominant ARD dims)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "Xshare = np.asarray(m.layers[0].X.mean)\n", + "Xtop = np.asarray(m.layers[-1].X.mean)\n", + "d_top = dominant_dims(ard_weights(m.layers[-1].kern), k=min(2, Xtop.shape[1]))\n", + "d_a = dominant_dims(ard_weights(m.layers[0].views[0].kern), k=min(2, Xshare.shape[1]))\n", + "d_b = dominant_dims(ard_weights(m.layers[0].views[1].kern), k=min(2, Xshare.shape[1]))\n", + "\n", + "fig, axes = plt.subplots(2, 2, figsize=(8, 7))\n", + "axes[0, 0].plot(Xtop[:, d_top]); axes[0, 0].set_title(\"DeepGP top (dominant dims)\")\n", + "axes[0, 1].axis(\"off\")\n", + "axes[1, 0].plot(Xshare[:, d_a]); axes[1, 0].set_title(\"shared (view0 ARD dims)\")\n", + "axes[1, 1].plot(Xshare[:, d_b]); axes[1, 1].set_title(\"shared (view1 ARD dims)\")\n", + "fig.suptitle(\"demToyUnsupervised-style DeepGP recovery\")\n", + "fig.tight_layout()\n", + "plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Baselines: stacked PCA and Isomap\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def stacked_pca(Ya, Yb, q1, q2):\n", + " def pca(Y, q):\n", + " Yc = Y - Y.mean(0)\n", + " U, S, _ = np.linalg.svd(Yc, full_matrices=False)\n", + " q = min(q, U.shape[1])\n", + " return U[:, :q] * S[:q]\n", + " xa, xb = pca(Ya, q1), pca(Yb, q1)\n", + " return xa, xb, pca(np.hstack([xa, xb]), q2)\n", + "\n", + "pca_xa, pca_xb, pca_x2 = stacked_pca(YA, YB, q_shared, q_top)\n", + "fig, axes = plt.subplots(2, 2, figsize=(8, 7))\n", + "axes[0, 0].plot(pca_x2[:, :min(2, pca_x2.shape[1])]); axes[0, 0].set_title(\"Stacked PCA top\")\n", + "axes[0, 1].axis(\"off\")\n", + "axes[1, 0].plot(pca_xa[:, :min(2, pca_xa.shape[1])]); axes[1, 0].set_title(\"Stacked PCA XA\")\n", + "axes[1, 1].plot(pca_xb[:, :min(2, pca_xb.shape[1])]); axes[1, 1].set_title(\"Stacked PCA XB\")\n", + "fig.suptitle(\"Stacked PCA\")\n", + "fig.tight_layout()\n", + "plt.show()\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from sklearn.manifold import Isomap\n", + "\n", + "def stacked_isomap(Ya, Yb, q1, q2, n_neighbors=5):\n", + " xa = Isomap(n_components=min(q1, Ya.shape[0] - 1), n_neighbors=n_neighbors).fit_transform(Ya)\n", + " xb = Isomap(n_components=min(q1, Yb.shape[0] - 1), n_neighbors=n_neighbors).fit_transform(Yb)\n", + " x2 = Isomap(n_components=min(q2, xa.shape[0] - 1), n_neighbors=n_neighbors).fit_transform(\n", + " np.hstack([xa, xb])\n", + " )\n", + " return xa, xb, x2\n", + "\n", + "iso_xa, iso_xb, iso_x2 = stacked_isomap(YA, YB, min(2, q_shared), min(2, q_top))\n", + "fig, axes = plt.subplots(2, 2, figsize=(8, 7))\n", + "axes[0, 0].plot(iso_x2); axes[0, 0].set_title(\"Stacked Isomap top\")\n", + "axes[0, 1].axis(\"off\")\n", + "axes[1, 0].plot(iso_xa); axes[1, 0].set_title(\"Stacked Isomap XA\")\n", + "axes[1, 1].plot(iso_xb); axes[1, 1].set_title(\"Stacked Isomap XB\")\n", + "fig.suptitle(\"Stacked Isomap\")\n", + "fig.tight_layout()\n", + "plt.show()\n" + ] + } + ] +} diff --git a/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb b/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb new file mode 100644 index 0000000..789c1bc --- /dev/null +++ b/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb @@ -0,0 +1,222 @@ +{ + "nbformat": 4, + "nbformat_minor": 5, + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + } + }, + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Warped regression \u2014 \u00a74.1 / fig. 3\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb)\n", + "\n", + "Port of Matlab [`demToyRegressionSimple.m`](https://github.com/SheffieldML/deepGP/blob/master/deepGP/matlab/demToyRegressionSimple.m) with\n", + "data from `hierGpsNEW` in `demToyDynamicsCreateData.m`.\n", + "\n", + "Defaults: Ntoy=150, D=15, Ntr=25, H=2, Q=6, K=10 (Matlab simple). Set `PAPER = True`\n", + "for the AISTATS text sizes (N=120, D=10).\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import sys, subprocess, os, urllib.request\n", + "from pathlib import Path\n", + "\n", + "IN_COLAB = \"google.colab\" in sys.modules\n", + "ROOT = Path.cwd().resolve()\n", + "# Prefer sibling of notebooks/ (repo layout)\n", + "for c in [ROOT.parent, ROOT, ROOT / \"examples\" / \"damianou2013\"]:\n", + " if (c / \"_common.py\").is_file():\n", + " ROOT = c\n", + " break\n", + "\n", + "if IN_COLAB:\n", + " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", + " \"numpy\", \"scipy\", \"matplotlib\", \"GPy>=1.15\", \"scikit-learn\"])\n", + " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", + " f\"git+https://github.com/SheffieldML/PyDeepGP.git@examples-damianou2013\"])\n", + " ROOT = Path.cwd().resolve()\n", + " (ROOT / \"data\").mkdir(exist_ok=True)\n", + " for dest, url in [\n", + " (\"_common.py\", f\"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_common.py\"),\n", + " (\"_matlab_data.py\", f\"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_matlab_data.py\"),\n", + " (\"data/hgplvmSampleDataTr1.mat\",\n", + " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/hgplvmSampleDataTr1.mat\"),\n", + " (\"data/demHighFiveHgplvm1.mat\",\n", + " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/demHighFiveHgplvm1.mat\"),\n", + " ]:\n", + " p = ROOT / dest\n", + " if not p.is_file():\n", + " print(\"Fetching\", dest)\n", + " urllib.request.urlretrieve(url, str(p))\n", + "\n", + "os.chdir(ROOT)\n", + "sys.path.insert(0, str(ROOT))\n", + "print(\"Working directory:\", ROOT)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%matplotlib inline\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "import GPy\n", + "import deepgp\n", + "from _common import mse, optimize_staged, rbf_bias\n", + "from _matlab_data import generate_hier_gps_new, train_test_split_indices\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "QUICK = True\n", + "PAPER = False # True \u2192 N=120, D=10\n", + "n_toy, d_toy = (120, 10) if PAPER else (150, 15)\n", + "n_tr = 25\n", + "q = 3 if QUICK else 6\n", + "num_inducing = 8 if QUICK else 10\n", + "repeats = 2 if QUICK else 10\n", + "iters_fixed = 40 if QUICK else 500\n", + "iters_free = 120 if QUICK else 2000\n", + "SEED = 42\n", + "print(f\"Ntoy={n_toy} D={d_toy} Ntr={n_tr} Q={q} K={num_inducing} repeats={repeats}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Generative process (`hierGpsNEW`)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "X, H, Y = generate_hier_gps_new(n_toy=n_toy, d_toy=d_toy, seed=SEED)\n", + "fig, axes = plt.subplots(2, 1, figsize=(7, 4), sharex=True)\n", + "axes[0].plot(X[:, 0], H[:, 0], \"k-\")\n", + "axes[0].set_ylabel(\"hidden warp\")\n", + "axes[0].set_title(\"hierGpsNEW generative process\")\n", + "axes[1].plot(X[:, 0], Y[:, 0], label=\"Y[:,0]\")\n", + "axes[1].plot(X[:, 0], Y[:, 1], label=\"Y[:,1]\")\n", + "axes[1].set_xlabel(\"X (scaled)\")\n", + "axes[1].legend(fontsize=8)\n", + "fig.tight_layout()\n", + "plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Fit sparse GP and DeepGP across repeats\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def fit_gp(Xtr, Ytr, num_inducing):\n", + " kern = GPy.kern.RBF(Xtr.shape[1]) + GPy.kern.Bias(Xtr.shape[1])\n", + " m = GPy.models.SparseGPRegression(Xtr, Ytr, kernel=kern, num_inducing=num_inducing)\n", + " m.Gaussian_noise.variance = Ytr.var() * 0.01\n", + " m.Gaussian_noise.variance.fix()\n", + " m.optimize(max_iters=100, messages=False)\n", + " m.Gaussian_noise.variance.unfix()\n", + " m.optimize(max_iters=400, messages=False)\n", + " return m\n", + "\n", + "def fit_dgp(Xtr, Ytr, q, num_inducing, iters_fixed, iters_free):\n", + " # H=2 hidden layers between observed X and Y\n", + " dims = [Ytr.shape[1], q, q, Xtr.shape[1]]\n", + " kernels = [rbf_bias(q, ard=True), rbf_bias(q, ard=True), rbf_bias(Xtr.shape[1], ard=False)]\n", + " m = deepgp.DeepGP(\n", + " dims, Ytr, X=Xtr, kernels=kernels,\n", + " num_inducing=num_inducing, back_constraint=False,\n", + " )\n", + " optimize_staged(m, iters_fixed, iters_free, messages=False)\n", + " return m\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "gp_mses, dgp_mses = [], []\n", + "for r in range(repeats):\n", + " seed = SEED + r\n", + " X, H, Y = generate_hier_gps_new(n_toy=n_toy, d_toy=d_toy, seed=seed)\n", + " ind_tr, ind_ts = train_test_split_indices(n_toy, n_tr, seed=seed + 1000)\n", + " Xtr, Ytr = X[ind_tr], Y[ind_tr]\n", + " Xte, Yte = X[ind_ts], Y[ind_ts]\n", + " m_gp = fit_gp(Xtr, Ytr, num_inducing)\n", + " m_dgp = fit_dgp(Xtr, Ytr, q, num_inducing, iters_fixed, iters_free)\n", + " gp_mses.append(mse(m_gp.predict(Xte)[0], Yte))\n", + " dgp_mses.append(mse(m_dgp.predict(Xte)[0], Yte))\n", + " print(f\"Repeat {r + 1}/{repeats}: MSE GP={gp_mses[-1]:.4f} DeepGP={dgp_mses[-1]:.4f}\")\n", + "\n", + "print(\n", + " f\"Mean MSE (\u00b1std): GP {np.mean(gp_mses):.4f}\u00b1{np.std(gp_mses):.4f} \"\n", + " f\"DeepGP {np.mean(dgp_mses):.4f}\u00b1{np.std(dgp_mses):.4f}\"\n", + ")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Fig. 3(b)-style MSE plot\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "fig, ax = plt.subplots(figsize=(5, 3.5))\n", + "xs = np.arange(1, repeats + 1)\n", + "ax.plot(xs, gp_mses, \"o-\", label=\"sparse GP\")\n", + "ax.plot(xs, dgp_mses, \"s-\", label=\"DeepGP\")\n", + "ax.set_xlabel(\"experiment #\")\n", + "ax.set_ylabel(\"test MSE\")\n", + "ax.set_title(\"Fig. 3(b) style / demToyRegressionSimple\")\n", + "ax.legend()\n", + "fig.tight_layout()\n", + "plt.show()\n" + ] + } + ] +} diff --git a/examples/damianou2013/notebooks/03_usps_digits.ipynb b/examples/damianou2013/notebooks/03_usps_digits.ipynb new file mode 100644 index 0000000..492af9a --- /dev/null +++ b/examples/damianou2013/notebooks/03_usps_digits.ipynb @@ -0,0 +1,239 @@ +{ + "nbformat": 4, + "nbformat_minor": 5, + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + } + }, + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# USPS digits \u2014 \u00a74.3 / figs. 6\u20138\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/03_usps_digits.ipynb)\n", + "\n", + "Port of Matlab [`demUsps.m`](https://github.com/SheffieldML/deepGP/blob/master/deepGP/matlab/demUsps.m) /\n", + "[`demDigitsDemonstration.m`](https://github.com/SheffieldML/deepGP/blob/master/deepGP/matlab/demDigitsDemonstration.m):\n", + "`vercat2` (50\u00d7{0,6,1}), depth-5 widths Q={15,12,10,8,6}, K=80.\n", + "\n", + "Metrics: variational bound and leave-one-out 1-NN error in the top latent space.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import sys, subprocess, os, urllib.request\n", + "from pathlib import Path\n", + "\n", + "IN_COLAB = \"google.colab\" in sys.modules\n", + "ROOT = Path.cwd().resolve()\n", + "# Prefer sibling of notebooks/ (repo layout)\n", + "for c in [ROOT.parent, ROOT, ROOT / \"examples\" / \"damianou2013\"]:\n", + " if (c / \"_common.py\").is_file():\n", + " ROOT = c\n", + " break\n", + "\n", + "if IN_COLAB:\n", + " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", + " \"numpy\", \"scipy\", \"matplotlib\", \"GPy>=1.15\", \"scikit-learn\"])\n", + " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", + " f\"git+https://github.com/SheffieldML/PyDeepGP.git@examples-damianou2013\"])\n", + " ROOT = Path.cwd().resolve()\n", + " (ROOT / \"data\").mkdir(exist_ok=True)\n", + " for dest, url in [\n", + " (\"_common.py\", f\"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_common.py\"),\n", + " (\"_matlab_data.py\", f\"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_matlab_data.py\"),\n", + " (\"data/hgplvmSampleDataTr1.mat\",\n", + " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/hgplvmSampleDataTr1.mat\"),\n", + " (\"data/demHighFiveHgplvm1.mat\",\n", + " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/demHighFiveHgplvm1.mat\"),\n", + " ]:\n", + " p = ROOT / dest\n", + " if not p.is_file():\n", + " print(\"Fetching\", dest)\n", + " urllib.request.urlretrieve(url, str(p))\n", + "\n", + "os.chdir(ROOT)\n", + "sys.path.insert(0, str(ROOT))\n", + "print(\"Working directory:\", ROOT)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%matplotlib inline\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "import deepgp\n", + "from _common import (\n", + " ard_weights, dominant_dims, nearest_neighbour_error, optimize_staged, print_ard, rbf_bias,\n", + ")\n", + "from _matlab_data import USPS_DEPTH_Q, load_usps_vercat2\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "QUICK = True # True \u2192 fewer digits / smaller Q / depths 1\u20132\n", + "DEPTHS = [1, 2] if QUICK else [1, 2, 3, 4, 5]\n", + "per_class = 20 if QUICK else 50\n", + "num_inducing = 25 if QUICK else 80\n", + "iters_fixed = 40 if QUICK else 500\n", + "iters_free = 100 if QUICK else 3000\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Load USPS `vercat2` subset\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "Y, labels = load_usps_vercat2(per_class=per_class)\n", + "Y = Y - Y.mean(0)\n", + "Y = Y / (Y.std() + 1e-12)\n", + "print(f\"Y={Y.shape}, classes={sorted(set(labels.tolist()))}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Fit DeepGPs at increasing depth\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def depth_dimensions(depth, data_dim):\n", + " return [data_dim] + list(USPS_DEPTH_Q[depth])\n", + "\n", + "results = []\n", + "models = {}\n", + "for depth in DEPTHS:\n", + " dims = depth_dimensions(depth, Y.shape[1])\n", + " if QUICK:\n", + " dims = [Y.shape[1]] + [max(2, q // 2) for q in dims[1:]]\n", + " print(f\"\\n=== depth {depth} dims={dims} ===\")\n", + " m = deepgp.DeepGP(\n", + " dims, Y,\n", + " kernels=[rbf_bias(q) for q in dims[1:]],\n", + " num_inducing=num_inducing,\n", + " back_constraint=False,\n", + " inits=\"PCA\",\n", + " )\n", + " optimize_staged(m, iters_fixed, iters_free, messages=False)\n", + " bound = float(m.log_likelihood())\n", + " Xtop = np.asarray(m.layers[-1].X.mean)\n", + " mistakes, rate = nearest_neighbour_error(Xtop, labels)\n", + " print(f\"bound={bound:.3f} 1-NN mistakes={mistakes}/{Y.shape[0]} ({rate:.3%})\")\n", + " print_ard(m)\n", + " results.append((depth, bound, mistakes, rate))\n", + " models[depth] = m\n", + "\n", + "print(\"\\ndepth bound 1-NN errors\")\n", + "for depth, bound, mistakes, rate in results:\n", + " print(f\"{depth:5d} {bound:9.3f} {mistakes:3d} ({rate:.3%})\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Top latent scatter and ARD (figs. 6\u20138 style)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "deepest = max(DEPTHS)\n", + "m = models[deepest]\n", + "Xtop = np.asarray(m.layers[-1].X.mean)\n", + "dims_plot = dominant_dims(ard_weights(m.layers[-1].kern), k=2)\n", + "colors = {0: \"C0\", 1: \"C1\", 6: \"C2\"}\n", + "markers = {0: \"o\", 1: \"s\", 6: \"^\"}\n", + "\n", + "fig, ax = plt.subplots(figsize=(4.5, 4.5))\n", + "for d in (0, 1, 6):\n", + " mask = labels == d\n", + " ax.scatter(\n", + " Xtop[mask, dims_plot[0]], Xtop[mask, dims_plot[1]],\n", + " c=colors[d], marker=markers[d], s=28, label=str(d),\n", + " )\n", + "ax.set_title(f\"USPS top latent (depth {deepest})\")\n", + "ax.legend()\n", + "fig.tight_layout()\n", + "plt.show()\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "fig, axes = plt.subplots(1, len(m.layers), figsize=(3.0 * len(m.layers), 3))\n", + "if len(m.layers) == 1:\n", + " axes = [axes]\n", + "for i, layer in enumerate(m.layers):\n", + " w = ard_weights(layer.kern)\n", + " axes[i].bar(np.arange(1, len(w) + 1), w)\n", + " axes[i].set_title(f\"layer {i}\")\n", + "fig.suptitle(\"ARD (fig. 6 style)\")\n", + "fig.tight_layout()\n", + "plt.show()\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "fig, ax = plt.subplots(figsize=(5, 3.5))\n", + "ax.plot([r[0] for r in results], [r[1] for r in results], \"o-\", label=\"bound\")\n", + "ax.set_xlabel(\"depth\")\n", + "ax.set_ylabel(\"variational bound\")\n", + "ax2 = ax.twinx()\n", + "ax2.plot([r[0] for r in results], [r[2] for r in results], \"s--\", color=\"C1\")\n", + "ax2.set_ylabel(\"1-NN mistakes\")\n", + "ax.set_title(\"Depth vs bound / 1-NN\")\n", + "fig.tight_layout()\n", + "plt.show()\n" + ] + } + ] +} diff --git a/examples/damianou2013/notebooks/04_highfive_mrd.ipynb b/examples/damianou2013/notebooks/04_highfive_mrd.ipynb new file mode 100644 index 0000000..c8d3002 --- /dev/null +++ b/examples/damianou2013/notebooks/04_highfive_mrd.ipynb @@ -0,0 +1,239 @@ +{ + "nbformat": 4, + "nbformat_minor": 5, + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + } + }, + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# High-five MRD \u2014 \u00a74.2 / figs. 4\u20135\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/04_highfive_mrd.ipynb)\n", + "\n", + "Port of Matlab [`demHighFiveDemonstration.m`](https://github.com/SheffieldML/deepGP/blob/master/deepGP/matlab/demHighFiveDemonstration.m)\n", + "using `demHighFiveHgplvm1.mat` (subjects A/B, 78\u00d762). Fits a two-view DeepGP and\n", + "compares ARD / latents to the stored hgplvm reference spaces when present.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import sys, subprocess, os, urllib.request\n", + "from pathlib import Path\n", + "\n", + "IN_COLAB = \"google.colab\" in sys.modules\n", + "ROOT = Path.cwd().resolve()\n", + "# Prefer sibling of notebooks/ (repo layout)\n", + "for c in [ROOT.parent, ROOT, ROOT / \"examples\" / \"damianou2013\"]:\n", + " if (c / \"_common.py\").is_file():\n", + " ROOT = c\n", + " break\n", + "\n", + "if IN_COLAB:\n", + " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", + " \"numpy\", \"scipy\", \"matplotlib\", \"GPy>=1.15\", \"scikit-learn\"])\n", + " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", + " f\"git+https://github.com/SheffieldML/PyDeepGP.git@examples-damianou2013\"])\n", + " ROOT = Path.cwd().resolve()\n", + " (ROOT / \"data\").mkdir(exist_ok=True)\n", + " for dest, url in [\n", + " (\"_common.py\", f\"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_common.py\"),\n", + " (\"_matlab_data.py\", f\"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_matlab_data.py\"),\n", + " (\"data/hgplvmSampleDataTr1.mat\",\n", + " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/hgplvmSampleDataTr1.mat\"),\n", + " (\"data/demHighFiveHgplvm1.mat\",\n", + " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/demHighFiveHgplvm1.mat\"),\n", + " ]:\n", + " p = ROOT / dest\n", + " if not p.is_file():\n", + " print(\"Fetching\", dest)\n", + " urllib.request.urlretrieve(url, str(p))\n", + "\n", + "os.chdir(ROOT)\n", + "sys.path.insert(0, str(ROOT))\n", + "print(\"Working directory:\", ROOT)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%matplotlib inline\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "import deepgp\n", + "from _common import (\n", + " ard_weights, dominant_dims, init_noise, unfix_noise, rbf_bias,\n", + ")\n", + "from _matlab_data import load_highfive\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "QUICK = True\n", + "q1, q2 = (4, 3) if QUICK else (6, 4)\n", + "num_inducing = 15 if QUICK else 30\n", + "iters_fixed = 40 if QUICK else 200\n", + "iters_free = 80 if QUICK else 1500\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Load high-five views\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "data = load_highfive()\n", + "YA = data[\"YA\"].copy()\n", + "YB = data[\"YB\"].copy()\n", + "YA = (YA - YA.mean(0)) / (YA.std() + 1e-12)\n", + "YB = (YB - YB.mean(0)) / (YB.std() + 1e-12)\n", + "print(f\"YA={YA.shape} YB={YB.shape}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Fit two-view MRD DeepGP\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "m = deepgp.DeepGP(\n", + " [[YA.shape[1], YB.shape[1]], q1, q2],\n", + " [YA, YB],\n", + " kernels=[[rbf_bias(q1), rbf_bias(q1)], rbf_bias(q2)],\n", + " num_inducing=num_inducing,\n", + " back_constraint=False,\n", + " inits=\"PCA\",\n", + ")\n", + "init_noise(m, fix=True)\n", + "m.optimize(max_iters=iters_fixed, messages=False)\n", + "unfix_noise(m)\n", + "m.optimize(max_iters=iters_free, messages=False)\n", + "print(f\"MRD DeepGP variational bound: {float(m.log_likelihood()):.3f}\")\n", + "for i, view in enumerate(m.layers[0].views):\n", + " print(f\"View {i} ARD:\", np.array2string(ard_weights(view.kern), precision=3))\n", + "print(\"Top ARD:\", np.array2string(ard_weights(m.layers[-1].kern), precision=3))\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## ARD bars (fig. 4 style)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "fig, axes = plt.subplots(1, 3, figsize=(10, 3.2))\n", + "for i, view in enumerate(m.layers[0].views):\n", + " w = ard_weights(view.kern)\n", + " axes[i].bar(np.arange(1, len(w) + 1), w)\n", + " axes[i].set_title(f\"view {i} ARD (fY)\")\n", + " axes[i].set_xlabel(\"shared dim\")\n", + "w_top = ard_weights(m.layers[-1].kern)\n", + "axes[2].bar(np.arange(1, len(w_top) + 1), w_top)\n", + "axes[2].set_title(\"top ARD (fX)\")\n", + "fig.suptitle(\"High-five MRD ARD (fig. 4 style)\")\n", + "fig.tight_layout()\n", + "plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Latent projections (fig. 5 style)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "Xshare = np.asarray(m.layers[0].X.mean)\n", + "Xtop = np.asarray(m.layers[-1].X.mean)\n", + "d_share = dominant_dims(ard_weights(m.layers[0].views[0].kern), k=2)\n", + "d_top = dominant_dims(ard_weights(m.layers[-1].kern), k=2)\n", + "\n", + "fig, axes = plt.subplots(1, 2, figsize=(8, 3.5))\n", + "axes[0].plot(Xshare[:, d_share[0]], Xshare[:, d_share[1]], \".-\")\n", + "axes[0].set_title(\"DeepGP shared (dominant)\")\n", + "axes[1].plot(Xtop[:, d_top[0]], Xtop[:, d_top[1]], \".-\")\n", + "axes[1].set_title(\"DeepGP top (dominant)\")\n", + "fig.suptitle(\"Latent projections (fig. 5 style)\")\n", + "fig.tight_layout()\n", + "plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Reference hgplvm latents from the Matlab mat (if present)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "if \"Xparent\" in data:\n", + " fig, axes = plt.subplots(1, 3, figsize=(10, 3.2))\n", + " axes[0].plot(data[\"Xparent\"][:, 0], data[\"Xparent\"][:, 1], \".-\")\n", + " axes[0].set_title(\"hgplvm Xparent (ref)\")\n", + " axes[1].plot(data[\"Xchild1\"][:, 0], data[\"Xchild1\"][:, 1], \".-\")\n", + " axes[1].set_title(\"hgplvm Xchild1 (ref)\")\n", + " axes[2].plot(data[\"Xchild2\"][:, 0], data[\"Xchild2\"][:, 1], \".-\")\n", + " axes[2].set_title(\"hgplvm Xchild2 (ref)\")\n", + " fig.suptitle(\"Reference latents from demHighFiveHgplvm1.mat\")\n", + " fig.tight_layout()\n", + " plt.show()\n", + "else:\n", + " print(\"No Xparent/Xchild* fields in mat.\")\n" + ] + } + ] +} diff --git a/examples/damianou2013/notebooks/README.md b/examples/damianou2013/notebooks/README.md new file mode 100644 index 0000000..771f63b --- /dev/null +++ b/examples/damianou2013/notebooks/README.md @@ -0,0 +1,9 @@ +# Colab notebooks (Matlab-aligned) + +Walkthrough notebooks for the Damianou & Lawrence (AISTATS 2013) ports. +Each notebook expands the experiment step by step (data → fit → plots). +Shared helpers live in `../_common.py` and `../_matlab_data.py`. + +See [../SOURCE.md](../SOURCE.md) and [../README.md](../README.md). + +Matlab demos: [SheffieldML/deepGP](https://github.com/SheffieldML/deepGP). diff --git a/examples/damianou2013/toy_hierarchy.py b/examples/damianou2013/toy_hierarchy.py index ab5548f..be64949 100644 --- a/examples/damianou2013/toy_hierarchy.py +++ b/examples/damianou2013/toy_hierarchy.py @@ -1,10 +1,9 @@ #!/usr/bin/env python -"""Section 4.1 (fig. 2): unsupervised recovery of a stacked-GP hierarchy. +"""§4.1 / fig. 2 — unsupervised hierarchy (Matlab ``demToyUnsupervised.m``). -Paper: Damianou & Lawrence, Deep Gaussian Processes, AISTATS 2013. - -Generate a three-level stack (top latent → 2D intermediate → 10D observations), -fit a DeepGP with overcomplete ARD widths, and contrast against stacked PCA. +Loads ``hgplvmSampleDataTr1.mat`` (top ``X2`` → intermediates ``XA``,``XB`` → +leaves ``YA``,``YB``), fits a two-view DeepGP with latent widths Q={6,4} +matching the Matlab demo, and compares against stacked PCA and Isomap. """ from __future__ import annotations @@ -15,7 +14,6 @@ import numpy as np -import GPy import deepgp from _common import ( @@ -24,54 +22,41 @@ dominant_dims, ensure_outdir, optimize_staged, - print_ard, rbf_bias, - sample_gp_prior, save_or_show, ) +from _matlab_data import load_toy_hierarchy -def _pca_project(Y: np.ndarray, q: int) -> np.ndarray: - try: - from sklearn.decomposition import PCA - - return PCA(n_components=q).fit_transform(Y) - except Exception: +def stacked_pca(Ya: np.ndarray, Yb: np.ndarray, q1: int, q2: int): + def pca(Y, q): Yc = Y - Y.mean(0) U, S, _ = np.linalg.svd(Yc, full_matrices=False) q = min(q, U.shape[1]) return U[:, :q] * S[:q] + xa, xb = pca(Ya, q1), pca(Yb, q1) + x2 = pca(np.hstack([xa, xb]), q2) + return xa, xb, x2 -def stacked_pca(Y: np.ndarray, dims: list[int]) -> list[np.ndarray]: - """Greedy PCA stack returning latent matrices from bottom to top.""" - spaces = [] - cur = Y - for q in dims: - cur = _pca_project(cur, q) - spaces.append(cur) - return spaces +def stacked_isomap(Ya: np.ndarray, Yb: np.ndarray, q1: int, q2: int, n_neighbors: int = 5): + try: + from sklearn.manifold import Isomap + except ImportError as exc: + raise SystemExit("stacked Isomap requires scikit-learn") from exc -def generate_hierarchy(n: int, seed: int) -> tuple[np.ndarray, np.ndarray, np.ndarray]: - """Return (Z_top, X_mid, Y_obs) for a 1→2→10 stacked GP.""" - rng = np.random.RandomState(seed) - np.random.seed(seed) - - Z = np.linspace(-1.0, 1.0, n)[:, None] - kern_mid = GPy.kern.RBF(1, lengthscale=0.4, variance=1.0) - X = sample_gp_prior(Z, kern_mid, output_dim=2) - X += 0.02 * rng.randn(*X.shape) - kern_obs = GPy.kern.RBF(2, lengthscale=0.8, variance=1.0, ARD=False) - Y = sample_gp_prior(X, kern_obs, output_dim=10) - Y += 0.05 * rng.randn(*Y.shape) - return Z, X, Y + xa = Isomap(n_components=min(q1, Ya.shape[0] - 1), n_neighbors=n_neighbors).fit_transform(Ya) + xb = Isomap(n_components=min(q1, Yb.shape[0] - 1), n_neighbors=n_neighbors).fit_transform(Yb) + x2 = Isomap( + n_components=min(q2, xa.shape[0] - 1), n_neighbors=n_neighbors + ).fit_transform(np.hstack([xa, xb])) + return xa, xb, x2 def build_parser() -> argparse.ArgumentParser: p = argparse.ArgumentParser(description=__doc__) add_common_args(p) - p.add_argument("--n", type=int, default=80, help="Number of observations.") p.add_argument("--num-inducing", type=int, default=25) return p @@ -79,64 +64,111 @@ def build_parser() -> argparse.ArgumentParser: def main(argv: list[str] | None = None) -> int: args = build_parser().parse_args(argv) outdir = ensure_outdir(args.outdir) - n = 40 if args.quick else args.n - q_mid, q_top = (4, 3) if args.quick else (5, 4) - iters_fixed = 50 if args.quick else 200 - iters_free = 150 if args.quick else 1500 - num_inducing = 15 if args.quick else args.num_inducing - Z, X_true, Y = generate_hierarchy(n, args.seed) - print(f"Generated hierarchy: Y={Y.shape}, true mid={X_true.shape}, true top={Z.shape}") + data = load_toy_hierarchy() + YA, YB = data["YA"], data["YB"] + X2, XA, XB = data["X2"], data["XA"], data["XB"] + print( + f"Loaded hgplvmSampleDataTr1: YA={YA.shape} YB={YB.shape} " + f"XA={XA.shape} XB={XB.shape} X2={X2.shape}" + ) + + # Matlab demToyUnsupervised: Q = {6,4} + q_shared, q_top = (4, 3) if args.quick else (6, 4) + num_inducing = 15 if args.quick else args.num_inducing + iters_fixed = 50 if args.quick else 200 + iters_free = 150 if args.quick else 2000 - kernels = [rbf_bias(q_mid, ard=True), rbf_bias(q_top, ard=True)] + kernels = [ + [rbf_bias(q_shared), rbf_bias(q_shared)], + rbf_bias(q_top), + ] m = deepgp.DeepGP( - [Y.shape[1], q_mid, q_top], - Y, + [[YA.shape[1], YB.shape[1]], q_shared, q_top], + [YA, YB], kernels=kernels, num_inducing=num_inducing, back_constraint=False, + inits="PCA", ) optimize_staged(m, iters_fixed, iters_free, messages=args.messages) print(f"DeepGP variational bound: {float(m.log_likelihood()):.3f}") - print_ard(m) + for i, view in enumerate(m.layers[0].views): + print(f"View {i} ARD: {np.array2string(ard_weights(view.kern), precision=3)}") + print(f"Top ARD: {np.array2string(ard_weights(m.layers[-1].kern), precision=3)}") - pca_spaces = stacked_pca(Y, [q_mid, q_top]) + pca_xa, pca_xb, pca_x2 = stacked_pca(YA, YB, q_shared, q_top) + try: + iso_xa, iso_xb, iso_x2 = stacked_isomap(YA, YB, min(2, q_shared), min(2, q_top)) + have_iso = True + except SystemExit: + have_iso = False + print("sklearn not available — skipping Isomap comparison") if not args.no_plot or outdir: import matplotlib.pyplot as plt - fig, axes = plt.subplots(1, 3, figsize=(10, 3.2)) - t = np.arange(n) - axes[0].plot(t, Z[:, 0], "k-", label="top") - axes[0].plot(t, X_true[:, 0], "C0-", label="mid0") - axes[0].plot(t, X_true[:, 1], "C1-", label="mid1") - axes[0].set_title("Ground truth") - axes[0].legend(fontsize=8) - - for ax, layer_idx, title in ( - (axes[1], 0, "DeepGP layer 0 (near data)"), - (axes[2], 1, "DeepGP layer 1 (top)"), - ): - Xmean = m.layers[layer_idx].X.mean - dims = dominant_dims(ard_weights(m.layers[layer_idx].kern), k=min(2, Xmean.shape[1])) - for d in dims: - ax.plot(t, Xmean[:, d], label=f"dim {d}") - ax.set_title(title) - ax.legend(fontsize=8) - - fig.suptitle("Damianou & Lawrence 2013 — toy hierarchy (fig. 2 style)") + t = np.arange(YA.shape[0]) + fig, axes = plt.subplots(3, 2, figsize=(9, 8)) + axes[0, 0].plot(X2) + axes[0, 0].set_title("True top X2") + axes[0, 1].axis("off") + axes[1, 0].plot(XA) + axes[1, 0].set_title("True XA") + axes[1, 1].plot(XB) + axes[1, 1].set_title("True XB") + axes[2, 0].plot(YA) + axes[2, 0].set_title("YA") + axes[2, 1].plot(YB) + axes[2, 1].set_title("YB") + fig.suptitle("Matlab hgplvmSampleDataTr1 ground truth") fig.tight_layout() - save_or_show(fig, outdir, "toy_hierarchy.png", args.no_plot) - - fig2, ax = plt.subplots(figsize=(5, 3)) - ax.plot(t, pca_spaces[0][:, 0], label="PCA mid 0") - if pca_spaces[0].shape[1] > 1: - ax.plot(t, pca_spaces[0][:, 1], label="PCA mid 1") - ax.plot(t, pca_spaces[1][:, 0], label="PCA top 0") - ax.set_title("Stacked PCA (greedy)") - ax.legend(fontsize=8) + save_or_show(fig, outdir, "toy_hierarchy_truth.png", args.no_plot) + + Xshare = np.asarray(m.layers[0].X.mean) + Xtop = np.asarray(m.layers[-1].X.mean) + d_top = dominant_dims(ard_weights(m.layers[-1].kern), k=min(2, Xtop.shape[1])) + w0 = ard_weights(m.layers[0].views[0].kern) + w1 = ard_weights(m.layers[0].views[1].kern) + d_a = dominant_dims(w0, k=min(2, Xshare.shape[1])) + d_b = dominant_dims(w1, k=min(2, Xshare.shape[1])) + + fig2, axes = plt.subplots(2, 2, figsize=(8, 7)) + axes[0, 0].plot(Xtop[:, d_top]) + axes[0, 0].set_title("DeepGP top (dominant dims)") + axes[0, 1].axis("off") + axes[1, 0].plot(Xshare[:, d_a]) + axes[1, 0].set_title("DeepGP shared (view0 ARD dims)") + axes[1, 1].plot(Xshare[:, d_b]) + axes[1, 1].set_title("DeepGP shared (view1 ARD dims)") + fig2.suptitle("demToyUnsupervised-style DeepGP recovery") fig2.tight_layout() - save_or_show(fig2, outdir, "toy_hierarchy_pca.png", args.no_plot) + save_or_show(fig2, outdir, "toy_hierarchy_deepgp.png", args.no_plot) + + fig3, axes = plt.subplots(2, 2, figsize=(8, 7)) + axes[0, 0].plot(pca_x2[:, : min(2, pca_x2.shape[1])]) + axes[0, 0].set_title("Stacked PCA top") + axes[1, 0].plot(pca_xa[:, : min(2, pca_xa.shape[1])]) + axes[1, 0].set_title("Stacked PCA XA") + axes[1, 1].plot(pca_xb[:, : min(2, pca_xb.shape[1])]) + axes[1, 1].set_title("Stacked PCA XB") + axes[0, 1].axis("off") + fig3.suptitle("Stacked PCA") + fig3.tight_layout() + save_or_show(fig3, outdir, "toy_hierarchy_pca.png", args.no_plot) + + if have_iso: + fig4, axes = plt.subplots(2, 2, figsize=(8, 7)) + axes[0, 0].plot(iso_x2) + axes[0, 0].set_title("Stacked Isomap top") + axes[1, 0].plot(iso_xa) + axes[1, 0].set_title("Stacked Isomap XA") + axes[1, 1].plot(iso_xb) + axes[1, 1].set_title("Stacked Isomap XB") + axes[0, 1].axis("off") + fig4.suptitle("Stacked Isomap") + fig4.tight_layout() + save_or_show(fig4, outdir, "toy_hierarchy_isomap.png", args.no_plot) return 0 diff --git a/examples/damianou2013/toy_warped_regression.py b/examples/damianou2013/toy_warped_regression.py index 0ef8a97..e8a8c92 100644 --- a/examples/damianou2013/toy_warped_regression.py +++ b/examples/damianou2013/toy_warped_regression.py @@ -1,11 +1,9 @@ #!/usr/bin/env python -"""Section 4.1 (fig. 3): warped toy regression — DeepGP vs sparse GP. +"""§4.1 / fig. 3 — warped regression (Matlab ``demToyRegressionSimple.m``). -Paper: Damianou & Lawrence, Deep Gaussian Processes, AISTATS 2013. - -Stack Linear+RBF → RBF to make a non-stationary 10D regression problem from a -1D input. Train on 25 points; compare MSE of a standard sparse GP against a -DeepGP that is one hidden layer deeper than the generative process. +Data: ``hierGpsNEW`` in ``demToyDynamicsCreateData.m`` (RBF+Linear warp → RBF). +Defaults follow the Matlab simple demo (Ntoy=150, D=15, Ntr=25, H=2, Q=6, K=10). +Use ``--paper`` for the AISTATS text settings (N=120, D=10). """ from __future__ import annotations @@ -25,55 +23,30 @@ mse, optimize_staged, rbf_bias, - sample_gp_prior, save_or_show, ) +from _matlab_data import generate_hier_gps_new, train_test_split_indices -def generate_warped( - n: int = 120, - seed: int = 0, -) -> tuple[np.ndarray, np.ndarray, np.ndarray]: - """Return (X, H_hidden, Y) for the paper's warped process.""" - rng = np.random.RandomState(seed) - np.random.seed(seed) - X = np.linspace(-1.0, 1.0, n)[:, None] - kern_warp = GPy.kern.Linear(1) + GPy.kern.RBF(1, lengthscale=0.3, variance=1.0) - H = sample_gp_prior(X, kern_warp, output_dim=1) - H += 0.01 * rng.randn(*H.shape) - kern_obs = GPy.kern.RBF(1, lengthscale=0.25, variance=1.0) - Y = sample_gp_prior(H, kern_obs, output_dim=10) - Y += 0.05 * rng.randn(*Y.shape) - return X, H, Y - - -def train_test_split( - X: np.ndarray, - Y: np.ndarray, - n_train: int, - seed: int, -) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]: - rng = np.random.RandomState(seed) - idx = rng.permutation(X.shape[0]) - tr, te = np.sort(idx[:n_train]), np.sort(idx[n_train:]) - return X[tr], Y[tr], X[te], Y[te] - - -def fit_deepgp( - Xtr: np.ndarray, - Ytr: np.ndarray, - q1: int, - q2: int, - num_inducing: int, - iters_fixed: int, - iters_free: int, - messages: bool, -) -> deepgp.DeepGP: - # Two hidden layers between X and Y (paper: one more than the generative depth) - dims = [Ytr.shape[1], q1, q2, Xtr.shape[1]] +def fit_gp(Xtr, Ytr, num_inducing, messages): + kern = GPy.kern.RBF(Xtr.shape[1]) + GPy.kern.Bias(Xtr.shape[1]) + m = GPy.models.SparseGPRegression(Xtr, Ytr, kernel=kern, num_inducing=num_inducing) + m.Gaussian_noise.variance = Ytr.var() * 0.01 + m.Gaussian_noise.variance.fix() + m.optimize(max_iters=100, messages=messages) + m.Gaussian_noise.variance.unfix() + m.optimize(max_iters=400, messages=messages) + return m + + +def fit_dgp(Xtr, Ytr, q, num_inducing, iters_fixed, iters_free, messages): + # H=2 hidden layers between observed X and Y (Matlab simple / paper) + dims = [Ytr.shape[1], q, q, Xtr.shape[1]] kernels = [ - rbf_bias(q1, ard=True), - rbf_bias(q2, ard=True), + rbf_bias(q, ard=True), + rbf_bias(q, ard=True), + # Top mapping sees inputs; Matlab uses lin+white+bias ("dynamics") — + # approximate with RBF+Bias on the observed X layer. rbf_bias(Xtr.shape[1], ard=False), ] m = deepgp.DeepGP( @@ -88,28 +61,15 @@ def fit_deepgp( return m -def fit_gp( - Xtr: np.ndarray, - Ytr: np.ndarray, - num_inducing: int, - messages: bool, -) -> GPy.core.Model: - kern = GPy.kern.RBF(Xtr.shape[1]) + GPy.kern.Bias(Xtr.shape[1]) - m = GPy.models.SparseGPRegression(Xtr, Ytr, kernel=kern, num_inducing=num_inducing) - m.Gaussian_noise.variance = Ytr.var() * 0.01 - m.Gaussian_noise.variance.fix() - m.optimize(max_iters=100, messages=messages) - m.Gaussian_noise.variance.unfix() - m.optimize(max_iters=400, messages=messages) - return m - - def build_parser() -> argparse.ArgumentParser: p = argparse.ArgumentParser(description=__doc__) add_common_args(p) - p.add_argument("--repeats", type=int, default=10, help="Paper uses 10 repeats.") - p.add_argument("--n-train", type=int, default=25) - p.add_argument("--num-inducing", type=int, default=20) + p.add_argument( + "--paper", + action="store_true", + help="Use AISTATS text sizes (N=120, D=10) instead of Matlab simple (150, 15).", + ) + p.add_argument("--repeats", type=int, default=10) return p @@ -117,38 +77,37 @@ def main(argv: list[str] | None = None) -> int: args = build_parser().parse_args(argv) outdir = ensure_outdir(args.outdir) + if args.paper: + n_toy, d_toy = 120, 10 + else: + n_toy, d_toy = 150, 15 + n_tr = 25 + q = 3 if args.quick else 6 + num_inducing = 8 if args.quick else 10 repeats = 2 if args.quick else args.repeats - n_train = 20 if args.quick else args.n_train - q1, q2 = (3, 2) if args.quick else (4, 3) - iters_fixed = 40 if args.quick else 200 - iters_free = 120 if args.quick else 1200 - num_inducing = 12 if args.quick else args.num_inducing + iters_fixed = 40 if args.quick else 500 + iters_free = 120 if args.quick else 2000 - gp_mses = [] - dgp_mses = [] + print( + f"Config: Ntoy={n_toy} D={d_toy} Ntr={n_tr} Q={q} K={num_inducing} " + f"repeats={repeats} ({'paper' if args.paper else 'matlab-simple'})" + ) + gp_mses, dgp_mses = [], [] for r in range(repeats): seed = args.seed + r - X, H, Y = generate_warped(n=120, seed=seed) - Xtr, Ytr, Xte, Yte = train_test_split(X, Y, n_train=n_train, seed=seed + 1000) - - m_gp = fit_gp(Xtr, Ytr, num_inducing=num_inducing, messages=False) - m_dgp = fit_deepgp( - Xtr, - Ytr, - q1=q1, - q2=q2, - num_inducing=num_inducing, - iters_fixed=iters_fixed, - iters_free=iters_free, - messages=args.messages, + X, H, Y = generate_hier_gps_new(n_toy=n_toy, d_toy=d_toy, seed=seed) + ind_tr, ind_ts = train_test_split_indices(n_toy, n_tr, seed=seed + 1000) + Xtr, Ytr = X[ind_tr], Y[ind_tr] + Xte, Yte = X[ind_ts], Y[ind_ts] + + m_gp = fit_gp(Xtr, Ytr, num_inducing, messages=False) + m_dgp = fit_dgp( + Xtr, Ytr, q, num_inducing, iters_fixed, iters_free, messages=args.messages ) - - mse_gp = mse(m_gp.predict(Xte)[0], Yte) - mse_dgp = mse(m_dgp.predict(Xte)[0], Yte) - gp_mses.append(mse_gp) - dgp_mses.append(mse_dgp) - print(f"Repeat {r + 1}/{repeats}: MSE GP={mse_gp:.4f} DeepGP={mse_dgp:.4f}") + gp_mses.append(mse(m_gp.predict(Xte)[0], Yte)) + dgp_mses.append(mse(m_dgp.predict(Xte)[0], Yte)) + print(f"Repeat {r + 1}/{repeats}: MSE GP={gp_mses[-1]:.4f} DeepGP={dgp_mses[-1]:.4f}") print( f"Mean MSE (±std): GP {np.mean(gp_mses):.4f}±{np.std(gp_mses):.4f} " @@ -158,27 +117,25 @@ def main(argv: list[str] | None = None) -> int: if not args.no_plot or outdir: import matplotlib.pyplot as plt - # Illustrate one generative draw - X, H, Y = generate_warped(n=120, seed=args.seed) + X, H, Y = generate_hier_gps_new(n_toy=n_toy, d_toy=d_toy, seed=args.seed) fig, axes = plt.subplots(2, 1, figsize=(7, 4), sharex=True) axes[0].plot(X[:, 0], H[:, 0], "k-") - axes[0].set_ylabel("hidden warp") - axes[0].set_title("Warped process (one draw)") + axes[0].set_ylabel("hidden warp Y1") + axes[0].set_title("hierGpsNEW generative process") axes[1].plot(X[:, 0], Y[:, 0], label="Y[:,0]") axes[1].plot(X[:, 0], Y[:, 1], label="Y[:,1]") - axes[1].set_xlabel("X") - axes[1].set_ylabel("outputs") + axes[1].set_xlabel("X (scaled)") axes[1].legend(fontsize=8) fig.tight_layout() save_or_show(fig, outdir, "toy_warped_process.png", args.no_plot) fig2, ax = plt.subplots(figsize=(5, 3.5)) xs = np.arange(1, repeats + 1) - ax.plot(xs, gp_mses, "o-", label="GP") + ax.plot(xs, gp_mses, "o-", label="sparse GP") ax.plot(xs, dgp_mses, "s-", label="DeepGP") ax.set_xlabel("experiment #") - ax.set_ylabel("MSE") - ax.set_title("Damianou & Lawrence 2013 — fig. 3(b) style") + ax.set_ylabel("test MSE") + ax.set_title("Fig. 3(b) style / demToyRegressionSimple") ax.legend() fig2.tight_layout() save_or_show(fig2, outdir, "toy_warped_mse.png", args.no_plot) diff --git a/examples/damianou2013/usps_digits.py b/examples/damianou2013/usps_digits.py index aab1dd3..67bf247 100644 --- a/examples/damianou2013/usps_digits.py +++ b/examples/damianou2013/usps_digits.py @@ -1,21 +1,17 @@ #!/usr/bin/env python -"""Section 4.3 (figs. 6–8): USPS digits {0,1,6} — depth, ARD, 1-NN in latent space. +"""§4.3 / figs. 6–8 — USPS {0,1,6} (Matlab ``demUsps.m`` / ``demDigitsDemonstration.m``). -Paper: Damianou & Lawrence, Deep Gaussian Processes, AISTATS 2013. - -Downloads the ESL ``zip.train`` USPS split (16×16), keeps 50 examples of each of -digits {0,1,6}, and trains unsupervised DeepGPs of increasing depth. Reports the -variational bound and leave-one-out 1-NN error in the top latent features. +``vercat2``: 50 examples of digits 0, 6, 1. Depth-5 demo uses +Q = {15, 12, 10, 8, 6} and K = 80 inducing points (experimentNo 19 comment). +Reports variational bound and leave-one-out 1-NN error in the top latent space. """ from __future__ import annotations import argparse -import gzip import os import sys -import urllib.request -from typing import List, Sequence, Tuple +from typing import List import numpy as np @@ -32,104 +28,13 @@ rbf_bias, save_or_show, ) +from _matlab_data import USPS_DEPTH_Q, load_usps_vercat2 -ESL_ZIP_TRAIN = "https://web.stanford.edu/~hastie/ElemStatLearn/datasets/zip.train.gz" -DIGITS = (0, 1, 6) - - -def _cache_path(cache_dir: str) -> str: - return os.path.join(cache_dir, "zip.train.gz") - - -def download_usps(cache_dir: str) -> str: - os.makedirs(cache_dir, exist_ok=True) - path = _cache_path(cache_dir) - if os.path.isfile(path) and os.path.getsize(path) > 0: - return path - print(f"Downloading USPS zip.train → {path}") - req = urllib.request.Request(ESL_ZIP_TRAIN, headers={"User-Agent": "PyDeepGP-examples"}) - with urllib.request.urlopen(req, timeout=120) as resp, open(path, "wb") as out: - out.write(resp.read()) - return path - - -def load_usps_matrix(path: str) -> Tuple[np.ndarray, np.ndarray]: - """Return (Y [N,256], labels [N]) with pixels in roughly [-1, 1].""" - rows = [] - with gzip.open(path, "rt") as f: - for line in f: - parts = line.strip().split() - if not parts: - continue - rows.append([float(x) for x in parts]) - data = np.asarray(rows, dtype=float) - labels = data[:, 0].astype(int) - Y = data[:, 1:] - return Y, labels - - -def subsample_digits( - Y: np.ndarray, - labels: np.ndarray, - digits: Sequence[int] = DIGITS, - per_class: int = 50, - seed: int = 42, -) -> Tuple[np.ndarray, np.ndarray]: - rng = np.random.RandomState(seed) - pieces_Y = [] - pieces_L = [] - for d in digits: - idx = np.where(labels == d)[0] - if idx.size < per_class: - raise RuntimeError(f"Digit {d}: need {per_class} examples, found {idx.size}") - choose = rng.choice(idx, size=per_class, replace=False) - pieces_Y.append(Y[choose]) - pieces_L.append(np.full(per_class, d, dtype=int)) - return np.vstack(pieces_Y), np.concatenate(pieces_L) - -def depth_dimensions(depth: int, data_dim: int, quick: bool) -> List[int]: - """Latent widths from data up to the top layer (length = depth + 1 entries for DeepGP). - - ``DeepGP(nDims, Y)`` expects ``nDims = [Y_dim, H1, H2, ..., H_depth]``. - """ - if quick: - # Narrower / shallower budgets for smoke tests - widths = {1: [8], 2: [8, 5], 3: [8, 6, 4]} - else: - # Overcomplete relative to the true manifold; ARD prunes - widths = { - 1: [12], - 2: [12, 8], - 3: [12, 10, 6], - 4: [12, 10, 8, 5], - 5: [12, 10, 8, 6, 4], - } - if depth not in widths: - raise ValueError(f"Unsupported depth {depth}; choose from {sorted(widths)}") - return [data_dim] + widths[depth] - - -def fit_depth( - Y: np.ndarray, - depth: int, - num_inducing: int, - iters_fixed: int, - iters_free: int, - messages: bool, - quick: bool, -) -> deepgp.DeepGP: - dims = depth_dimensions(depth, Y.shape[1], quick=quick) - kernels = [rbf_bias(q, ard=True) for q in dims[1:]] - m = deepgp.DeepGP( - dims, - Y, - kernels=kernels, - num_inducing=num_inducing, - back_constraint=False, - ) - optimize_staged(m, iters_fixed, iters_free, messages=messages) - return m +def depth_dimensions(depth: int, data_dim: int) -> List[int]: + if depth not in USPS_DEPTH_Q: + raise ValueError(f"depth must be in {sorted(USPS_DEPTH_Q)}") + return [data_dim] + list(USPS_DEPTH_Q[depth]) def build_parser() -> argparse.ArgumentParser: @@ -140,16 +45,10 @@ def build_parser() -> argparse.ArgumentParser: type=int, nargs="+", default=None, - help="Hierarchy depths to fit (default: 1..5, or 1..2 with --quick).", + help="Default: 1..5 (or 1..2 with --quick).", ) p.add_argument("--per-class", type=int, default=50) - p.add_argument("--num-inducing", type=int, default=40) - p.add_argument( - "--cache-dir", - type=str, - default=os.path.join(os.path.dirname(__file__), "data"), - help="Directory for zip.train.gz cache.", - ) + p.add_argument("--num-inducing", type=int, default=80) return p @@ -162,33 +61,34 @@ def main(argv: list[str] | None = None) -> int: depths = [1, 2] if args.quick else [1, 2, 3, 4, 5] per_class = 20 if args.quick else args.per_class - num_inducing = 20 if args.quick else args.num_inducing - iters_fixed = 40 if args.quick else 200 - iters_free = 100 if args.quick else 2000 + # Matlab uses K=80; keep a reduced inducing set only for --quick + num_inducing = 25 if args.quick else args.num_inducing + iters_fixed = 40 if args.quick else 500 + iters_free = 100 if args.quick else 3000 - path = download_usps(args.cache_dir) - Y_all, labels_all = load_usps_matrix(path) - Y, labels = subsample_digits(Y_all, labels_all, per_class=per_class, seed=args.seed) - # Zero-mean unit-ish scaling helps optimisation + Y, labels = load_usps_vercat2(per_class=per_class) Y = Y - Y.mean(0) Y = Y / (Y.std() + 1e-12) - print(f"USPS subset: Y={Y.shape}, classes={sorted(set(labels.tolist()))}") + print(f"USPS vercat2-style: Y={Y.shape}, classes={sorted(set(labels.tolist()))}") results = [] models = {} for depth in depths: - print(f"\n=== depth {depth} ===") - m = fit_depth( + print(f"\n=== depth {depth} dims={depth_dimensions(depth, Y.shape[1])} ===") + dims = depth_dimensions(depth, Y.shape[1]) + # For quick runs, shrink Q to keep optimisation tractable + if args.quick: + dims = [Y.shape[1]] + [max(2, q // 2) for q in dims[1:]] + m = deepgp.DeepGP( + dims, Y, - depth=depth, + kernels=[rbf_bias(q) for q in dims[1:]], num_inducing=num_inducing, - iters_fixed=iters_fixed, - iters_free=iters_free, - messages=args.messages, - quick=args.quick, + back_constraint=False, + inits="PCA", ) + optimize_staged(m, iters_fixed, iters_free, messages=args.messages) bound = float(m.log_likelihood()) - # Top latent features = last layer's X Xtop = np.asarray(m.layers[-1].X.mean) mistakes, rate = nearest_neighbour_error(Xtop, labels) print(f"bound={bound:.3f} 1-NN mistakes={mistakes}/{Y.shape[0]} ({rate:.3%})") @@ -196,7 +96,7 @@ def main(argv: list[str] | None = None) -> int: results.append((depth, bound, mistakes, rate)) models[depth] = m - print("\nSummary") + print("\nSummary (Matlab demDigitsDemonstration metrics)") print("depth bound 1-NN errors") for depth, bound, mistakes, rate in results: print(f"{depth:5d} {bound:9.3f} {mistakes:3d} ({rate:.3%})") @@ -207,49 +107,50 @@ def main(argv: list[str] | None = None) -> int: import matplotlib.pyplot as plt Xtop = np.asarray(m.layers[-1].X.mean) - w = ard_weights(m.layers[-1].kern) - dims = dominant_dims(w, k=2) - colors = ["C0", "C1", "C2"] - markers = ["o", "s", "^"] - + dims_plot = dominant_dims(ard_weights(m.layers[-1].kern), k=2) + colors = {0: "C0", 1: "C1", 6: "C2"} + markers = {0: "o", 1: "s", 6: "^"} fig, ax = plt.subplots(figsize=(4.5, 4.5)) - for i, digit in enumerate(DIGITS): - mask = labels == digit + for d in (0, 1, 6): + mask = labels == d ax.scatter( - Xtop[mask, dims[0]], - Xtop[mask, dims[1]], - c=colors[i], - marker=markers[i], + Xtop[mask, dims_plot[0]], + Xtop[mask, dims_plot[1]], + c=colors[d], + marker=markers[d], s=28, - label=str(digit), + label=str(d), ) - ax.set_xlabel(f"top dim {dims[0]}") - ax.set_ylabel(f"top dim {dims[1]}") ax.set_title(f"USPS top latent (depth {deepest})") ax.legend() fig.tight_layout() save_or_show(fig, outdir, "usps_top_latent.png", args.no_plot) - fig2, axes = plt.subplots(1, len(m.layers), figsize=(3.2 * len(m.layers), 3)) + fig2, axes = plt.subplots(1, len(m.layers), figsize=(3.0 * len(m.layers), 3)) if len(m.layers) == 1: axes = [axes] for i, layer in enumerate(m.layers): w = ard_weights(layer.kern) axes[i].bar(np.arange(1, len(w) + 1), w) - axes[i].set_title(f"layer {i} ARD") - axes[i].set_xlabel("dimension") - fig2.suptitle("ARD weights (1/lengthscale)") + axes[i].set_title(f"layer {i}") + fig2.suptitle("ARD (fig. 6 style)") fig2.tight_layout() save_or_show(fig2, outdir, "usps_ard.png", args.no_plot) fig3, ax = plt.subplots(figsize=(5, 3.5)) - ax.plot([r[0] for r in results], [r[1] for r in results], "o-", label="variational bound") + ax.plot([r[0] for r in results], [r[1] for r in results], "o-", label="bound") ax.set_xlabel("depth") - ax.set_ylabel("bound") + ax.set_ylabel("variational bound") ax2 = ax.twinx() - ax2.plot([r[0] for r in results], [r[2] for r in results], "s--", color="C1", label="1-NN mistakes") + ax2.plot( + [r[0] for r in results], + [r[2] for r in results], + "s--", + color="C1", + label="1-NN mistakes", + ) ax2.set_ylabel("1-NN mistakes") - ax.set_title("Depth selection via bound / 1-NN") + ax.set_title("Depth vs bound / 1-NN") fig3.tight_layout() save_or_show(fig3, outdir, "usps_depth.png", args.no_plot)