From 9158465f3ff07f8a481668a593a55c2e1ebb56b0 Mon Sep 17 00:00:00 2001 From: Neil Lawrence Date: Fri, 13 Nov 2026 21:33:13 +0000 Subject: [PATCH 1/3] Add Colab notebooks for Damianou & Lawrence 2013 examples. Four self-contained notebooks with Open-in-Colab badges, auto-install on Colab, paper citations, QUICK-mode plots, covering hierarchy, warped regression, USPS, and shared-view MRD. --- README.md | 3 +- examples/damianou2013/README.md | 20 +- .../notebooks/01_toy_hierarchy.ipynb | 234 +++++++++++++++ .../notebooks/02_toy_warped_regression.ipynb | 222 ++++++++++++++ .../notebooks/03_usps_digits.ipynb | 275 ++++++++++++++++++ .../notebooks/04_highfive_mrd.ipynb | 224 ++++++++++++++ examples/damianou2013/notebooks/README.md | 19 ++ 7 files changed, 993 insertions(+), 4 deletions(-) create mode 100644 examples/damianou2013/notebooks/01_toy_hierarchy.ipynb create mode 100644 examples/damianou2013/notebooks/02_toy_warped_regression.ipynb create mode 100644 examples/damianou2013/notebooks/03_usps_digits.ipynb create mode 100644 examples/damianou2013/notebooks/04_highfive_mrd.ipynb create mode 100644 examples/damianou2013/notebooks/README.md diff --git a/README.md b/README.md index 18d49b1..52bbaea 100644 --- a/README.md +++ b/README.md @@ -92,4 +92,5 @@ Examples * ``examples/`` — short supervised / unsupervised demos (oil, CMU walk/jog). * ``examples/damianou2013/`` — recreations of the AISTATS 2013 experiments - (toy hierarchy, warped regression, USPS depth/ARD, high-five / MRD). + (toy hierarchy, warped regression, USPS depth/ARD, high-five / MRD), + including Colab notebooks under ``examples/damianou2013/notebooks/``. diff --git a/examples/damianou2013/README.md b/examples/damianou2013/README.md index a461aa3..05fffdb 100644 --- a/examples/damianou2013/README.md +++ b/examples/damianou2013/README.md @@ -1,14 +1,28 @@ # Damianou & Lawrence (AISTATS 2013) — PyDeepGP recreations -Scripts that recreate the **feasible** experiments from +Scripts and Colab notebooks that recreate experiments from [Damianou & Lawrence, *Deep Gaussian Processes*](https://proceedings.mlr.press/v31/damianou13a.html) (AISTATS 2013) on the maintained `DeepGP` path. These are **modern recreations**, not bit-identical paper scripts: optimiser schedules, inducing-point counts, and initialisation differ from the 2013 -Matlab stack. Use `--quick` for a short smoke run. +Matlab stack. Use `--quick` / `QUICK = True` for a short smoke run. -## Mapping +## Notebooks (Colab) + +Each notebook starts with an **Open in Colab** badge and a cell that installs +GPy + PyDeepGP when run on Colab. + +| Paper section | Notebook | +| --- | --- | +| §4.1 fig. 2 — toy hierarchy | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb) [`01_toy_hierarchy.ipynb`](notebooks/01_toy_hierarchy.ipynb) | +| §4.1 fig. 3 — warped regression | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb) [`02_toy_warped_regression.ipynb`](notebooks/02_toy_warped_regression.ipynb) | +| §4.3 figs. 6–8 — USPS depth / ARD | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/03_usps_digits.ipynb) [`03_usps_digits.ipynb`](notebooks/03_usps_digits.ipynb) | +| §4.2 figs. 4–5 — high-five / MRD | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/04_highfive_mrd.ipynb) [`04_highfive_mrd.ipynb`](notebooks/04_highfive_mrd.ipynb) | + +Locally: `jupyter notebook notebooks/01_toy_hierarchy.ipynb` (with PyDeepGP installed). + +## Mapping (scripts) | Paper section | Script | Status | | --- | --- | --- | diff --git a/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb b/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb new file mode 100644 index 0000000..428f0b6 --- /dev/null +++ b/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb @@ -0,0 +1,234 @@ +{ + "nbformat": 4, + "nbformat_minor": 5, + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "pygments_lexer": "ipython3" + } + }, + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Toy stacked-GP hierarchy (Damianou & Lawrence 2013, \u00a74.1 / fig. 2)\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb)\n", + "\n", + "**Paper:** [Damianou, A. and Lawrence, N. D. (2013). *Deep Gaussian Processes*. In *AISTATS* (pp. 207\u2013215).](https://proceedings.mlr.press/v31/damianou13a.html)\n", + "\n", + "Generate observations from a three-level stack of GPs (top latent \u2192 2D intermediate \u2192 10D data), fit an unsupervised DeepGP with overcomplete ARD widths, and compare recovered signals to stacked PCA. In the paper, only the deep GP recovers latent signals close to the ground truth while learning dimensionalities via ARD.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Install dependencies when running on Google Colab\n", + "import sys\n", + "import subprocess\n", + "\n", + "IN_COLAB = \"google.colab\" in sys.modules\n", + "if IN_COLAB:\n", + " pkgs = [\"numpy>=1.7\", \"scipy>=0.12\", \"matplotlib\", \"GPy>=1.15\"]\n", + " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", *pkgs])\n", + " subprocess.check_call([\n", + " sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", + " \"git+https://github.com/SheffieldML/PyDeepGP.git@main\",\n", + " ])\n", + " print(\"Installed GPy and PyDeepGP for Colab.\")\n", + "else:\n", + " print(\"Local environment \u2014 ensure PyDeepGP and GPy are importable.\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Set QUICK=True for a short smoke run (fewer iterations / smaller models).\n", + "# Set QUICK=False for a paper-scale demonstration (slower).\n", + "QUICK = True\n", + "SEED = 42\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "import GPy\n", + "import deepgp\n", + "\n", + "np.random.seed(SEED)\n", + "\n", + "def sample_gp_prior(X, kernel, output_dim, noise=1e-6):\n", + " K = kernel.K(X) + noise * np.eye(X.shape[0])\n", + " L = np.linalg.cholesky(K)\n", + " return L @ np.random.randn(X.shape[0], output_dim)\n", + "\n", + "def rbf_bias(q, ard=True):\n", + " return GPy.kern.RBF(q, ARD=ard) + GPy.kern.Bias(q)\n", + "\n", + "def ard_weights(kern):\n", + " if hasattr(kern, \"rbf\"):\n", + " ls = np.asarray(kern.rbf.lengthscale).ravel()\n", + " else:\n", + " ls = np.asarray(kern.lengthscale).ravel()\n", + " return 1.0 / np.maximum(ls, 1e-12)\n", + "\n", + "def dominant_dims(w, k=2):\n", + " return list(np.argsort(-w)[:k])\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Generative hierarchy\n", + "\n", + "Top latent $Z \\in \\mathbb{R}$ drives a 2D intermediate layer, which drives 10D observations (paper fig. 2a).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "n = 40 if QUICK else 80\n", + "Z = np.linspace(-1.0, 1.0, n)[:, None]\n", + "X_true = sample_gp_prior(Z, GPy.kern.RBF(1, lengthscale=0.4), output_dim=2)\n", + "X_true += 0.02 * np.random.randn(*X_true.shape)\n", + "Y = sample_gp_prior(X_true, GPy.kern.RBF(2, lengthscale=0.8), output_dim=10)\n", + "Y += 0.05 * np.random.randn(*Y.shape)\n", + "\n", + "fig, ax = plt.subplots(figsize=(7, 3))\n", + "t = np.arange(n)\n", + "ax.plot(t, Z[:, 0], \"k-\", label=\"top $Z$\")\n", + "ax.plot(t, X_true[:, 0], label=\"mid 0\")\n", + "ax.plot(t, X_true[:, 1], label=\"mid 1\")\n", + "ax.set_title(\"Ground-truth hierarchy (fig. 2a style)\")\n", + "ax.legend()\n", + "plt.show()\n", + "print(\"Y\", Y.shape)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Fit DeepGP\n", + "\n", + "Overcomplete latent widths; ARD should down-weight unused dimensions (paper: dimensionalities learnt automatically).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "q_mid, q_top = (4, 3) if QUICK else (5, 4)\n", + "num_inducing = 15 if QUICK else 25\n", + "iters_fixed, iters_free = (50, 150) if QUICK else (200, 1500)\n", + "\n", + "from GPy.core.parameterization.variational import NormalPosterior, VariationalPosterior\n", + "\n", + "def out_var(y):\n", + " if isinstance(y, (NormalPosterior, VariationalPosterior)):\n", + " return float(np.asarray(y.mean).var())\n", + " return float(np.asarray(y).var())\n", + "\n", + "m = deepgp.DeepGP(\n", + " [Y.shape[1], q_mid, q_top],\n", + " Y,\n", + " kernels=[rbf_bias(q_mid), rbf_bias(q_top)],\n", + " num_inducing=num_inducing,\n", + " back_constraint=False,\n", + ")\n", + "for layer in m.layers:\n", + " layer.Gaussian_noise.variance = out_var(layer.Y) * 0.01\n", + " layer.Gaussian_noise.variance.fix()\n", + "m.optimize(max_iters=iters_fixed, messages=False)\n", + "for layer in m.layers:\n", + " layer.Gaussian_noise.variance.unfix()\n", + "m.optimize(max_iters=iters_free, messages=True)\n", + "print(f\"Variational bound: {float(m.log_likelihood()):.3f}\")\n", + "for i, layer in enumerate(m.layers):\n", + " print(f\"Layer {i} ARD:\", np.array2string(ard_weights(layer.kern), precision=3))\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Recovered latents vs stacked PCA\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def pca_project(Y, q):\n", + " Yc = Y - Y.mean(0)\n", + " U, S, _ = np.linalg.svd(Yc, full_matrices=False)\n", + " q = min(q, U.shape[1])\n", + " return U[:, :q] * S[:q]\n", + "\n", + "pca_mid = pca_project(Y, q_mid)\n", + "pca_top = pca_project(pca_mid, q_top)\n", + "\n", + "fig, axes = plt.subplots(1, 3, figsize=(11, 3.2))\n", + "axes[0].plot(t, Z[:, 0], \"k-\", label=\"top\")\n", + "axes[0].plot(t, X_true[:, 0], label=\"mid0\")\n", + "axes[0].plot(t, X_true[:, 1], label=\"mid1\")\n", + "axes[0].set_title(\"Ground truth\")\n", + "axes[0].legend(fontsize=8)\n", + "\n", + "for ax, li, title in ((axes[1], 0, \"DeepGP near data\"), (axes[2], 1, \"DeepGP top\")):\n", + " Xm = m.layers[li].X.mean\n", + " for d in dominant_dims(ard_weights(m.layers[li].kern), k=min(2, Xm.shape[1])):\n", + " ax.plot(t, Xm[:, d], label=f\"dim {d}\")\n", + " ax.set_title(title)\n", + " ax.legend(fontsize=8)\n", + "fig.suptitle(\"Fig. 2 style \u2014 DeepGP recovery\")\n", + "fig.tight_layout()\n", + "plt.show()\n", + "\n", + "fig, ax = plt.subplots(figsize=(6, 3))\n", + "ax.plot(t, pca_mid[:, 0], label=\"PCA mid 0\")\n", + "if pca_mid.shape[1] > 1:\n", + " ax.plot(t, pca_mid[:, 1], label=\"PCA mid 1\")\n", + "ax.plot(t, pca_top[:, 0], label=\"PCA top 0\")\n", + "ax.set_title(\"Stacked PCA (greedy)\")\n", + "ax.legend(fontsize=8)\n", + "plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## References\n", + "\n", + "- Damianou, A. and Lawrence, N. D. (2013). *Deep Gaussian Processes*. In *AISTATS* (pp. 207\u2013215). \u2014 [PDF](https://proceedings.mlr.press/v31/damianou13a.html)\n", + "- Script counterpart: `examples/damianou2013/toy_hierarchy.py`\n" + ] + } + ] +} \ No newline at end of file diff --git a/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb b/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb new file mode 100644 index 0000000..302fdc5 --- /dev/null +++ b/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb @@ -0,0 +1,222 @@ +{ + "nbformat": 4, + "nbformat_minor": 5, + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "pygments_lexer": "ipython3" + } + }, + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Warped toy regression (Damianou & Lawrence 2013, \u00a74.1 / fig. 3)\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb)\n", + "\n", + "**Paper:** [Damianou, A. and Lawrence, N. D. (2013). *Deep Gaussian Processes*. In *AISTATS* (pp. 207\u2013215).](https://proceedings.mlr.press/v31/damianou13a.html)\n", + "\n", + "A non-stationary regression problem is created by stacking Linear+RBF \u2192 RBF GPs. Only 25 of 120 points are observed. A standard sparse GP is compared to a DeepGP with **two** hidden layers (one more than the generative depth). The paper reports lower test MSE for the deep model across repeats (fig. 3b).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Install dependencies when running on Google Colab\n", + "import sys\n", + "import subprocess\n", + "\n", + "IN_COLAB = \"google.colab\" in sys.modules\n", + "if IN_COLAB:\n", + " pkgs = [\"numpy>=1.7\", \"scipy>=0.12\", \"matplotlib\", \"GPy>=1.15\"]\n", + " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", *pkgs])\n", + " subprocess.check_call([\n", + " sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", + " \"git+https://github.com/SheffieldML/PyDeepGP.git@main\",\n", + " ])\n", + " print(\"Installed GPy and PyDeepGP for Colab.\")\n", + "else:\n", + " print(\"Local environment \u2014 ensure PyDeepGP and GPy are importable.\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Set QUICK=True for a short smoke run (fewer iterations / smaller models).\n", + "# Set QUICK=False for a paper-scale demonstration (slower).\n", + "QUICK = True\n", + "SEED = 42\n", + "\n", + "REPEATS = 2 if QUICK else 10\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "import GPy\n", + "import deepgp\n", + "\n", + "np.random.seed(SEED)\n", + "\n", + "def sample_gp_prior(X, kernel, output_dim, noise=1e-6):\n", + " K = kernel.K(X) + noise * np.eye(X.shape[0])\n", + " return np.linalg.cholesky(K) @ np.random.randn(X.shape[0], output_dim)\n", + "\n", + "def rbf_bias(q, ard=True):\n", + " return GPy.kern.RBF(q, ARD=ard) + GPy.kern.Bias(q)\n", + "\n", + "def generate_warped(n=120, seed=0):\n", + " rng = np.random.RandomState(seed)\n", + " np.random.seed(seed)\n", + " X = np.linspace(-1.0, 1.0, n)[:, None]\n", + " H = sample_gp_prior(X, GPy.kern.Linear(1) + GPy.kern.RBF(1, lengthscale=0.3), 1)\n", + " H += 0.01 * rng.randn(*H.shape)\n", + " Y = sample_gp_prior(H, GPy.kern.RBF(1, lengthscale=0.25), 10)\n", + " Y += 0.05 * rng.randn(*Y.shape)\n", + " return X, H, Y\n", + "\n", + "def mse(a, b):\n", + " return float(np.mean((np.asarray(a) - np.asarray(b)) ** 2))\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## One draw of the warped process (fig. 3a style)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "X, H, Y = generate_warped(seed=SEED)\n", + "fig, axes = plt.subplots(2, 1, figsize=(8, 4), sharex=True)\n", + "axes[0].plot(X[:, 0], H[:, 0], \"k-\")\n", + "axes[0].set_ylabel(\"hidden warp\")\n", + "axes[0].set_title(\"Warped generative process\")\n", + "axes[1].plot(X[:, 0], Y[:, 0], label=\"Y[:,0]\")\n", + "axes[1].plot(X[:, 0], Y[:, 1], label=\"Y[:,1]\")\n", + "axes[1].set_xlabel(\"X\")\n", + "axes[1].set_ylabel(\"outputs\")\n", + "axes[1].legend(fontsize=8)\n", + "fig.tight_layout()\n", + "plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## GP vs DeepGP over repeats\n", + "\n", + "Training size 25 (paper); DeepGP has two hidden layers between $X$ and $Y$.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from GPy.core.parameterization.variational import NormalPosterior, VariationalPosterior\n", + "\n", + "def out_var(y):\n", + " if isinstance(y, (NormalPosterior, VariationalPosterior)):\n", + " return float(np.asarray(y.mean).var())\n", + " return float(np.asarray(y).var())\n", + "\n", + "def fit_gp(Xtr, Ytr, M):\n", + " m = GPy.models.SparseGPRegression(\n", + " Xtr, Ytr, kernel=GPy.kern.RBF(1) + GPy.kern.Bias(1), num_inducing=M\n", + " )\n", + " m.Gaussian_noise.variance = Ytr.var() * 0.01\n", + " m.Gaussian_noise.variance.fix()\n", + " m.optimize(max_iters=100, messages=False)\n", + " m.Gaussian_noise.variance.unfix()\n", + " m.optimize(max_iters=400, messages=False)\n", + " return m\n", + "\n", + "def fit_dgp(Xtr, Ytr, M, q1, q2, it_f, it_u):\n", + " m = deepgp.DeepGP(\n", + " [Ytr.shape[1], q1, q2, Xtr.shape[1]],\n", + " Ytr,\n", + " X=Xtr,\n", + " kernels=[rbf_bias(q1), rbf_bias(q2), rbf_bias(Xtr.shape[1], ard=False)],\n", + " num_inducing=M,\n", + " back_constraint=False,\n", + " )\n", + " for layer in m.layers:\n", + " layer.Gaussian_noise.variance = out_var(layer.Y) * 0.01\n", + " layer.Gaussian_noise.variance.fix()\n", + " m.optimize(max_iters=it_f, messages=False)\n", + " for layer in m.layers:\n", + " layer.Gaussian_noise.variance.unfix()\n", + " m.optimize(max_iters=it_u, messages=False)\n", + " return m\n", + "\n", + "n_train = 20 if QUICK else 25\n", + "q1, q2 = (3, 2) if QUICK else (4, 3)\n", + "M = 12 if QUICK else 20\n", + "it_f, it_u = (40, 120) if QUICK else (200, 1200)\n", + "\n", + "gp_mses, dgp_mses = [], []\n", + "for r in range(REPEATS):\n", + " seed = SEED + r\n", + " X, _, Y = generate_warped(seed=seed)\n", + " rng = np.random.RandomState(seed + 1000)\n", + " idx = rng.permutation(X.shape[0])\n", + " tr, te = np.sort(idx[:n_train]), np.sort(idx[n_train:])\n", + " Xtr, Ytr, Xte, Yte = X[tr], Y[tr], X[te], Y[te]\n", + " m_gp = fit_gp(Xtr, Ytr, M)\n", + " m_dgp = fit_dgp(Xtr, Ytr, M, q1, q2, it_f, it_u)\n", + " gp_mses.append(mse(m_gp.predict(Xte)[0], Yte))\n", + " dgp_mses.append(mse(m_dgp.predict(Xte)[0], Yte))\n", + " print(f\"Repeat {r+1}/{REPEATS}: GP={gp_mses[-1]:.4f} DeepGP={dgp_mses[-1]:.4f}\")\n", + "\n", + "print(f\"Mean MSE (\u00b1std): GP {np.mean(gp_mses):.4f}\u00b1{np.std(gp_mses):.4f} \"\n", + " f\"DeepGP {np.mean(dgp_mses):.4f}\u00b1{np.std(dgp_mses):.4f}\")\n", + "\n", + "fig, ax = plt.subplots(figsize=(5.5, 3.5))\n", + "xs = np.arange(1, REPEATS + 1)\n", + "ax.plot(xs, gp_mses, \"o-\", label=\"GP\")\n", + "ax.plot(xs, dgp_mses, \"s-\", label=\"DeepGP\")\n", + "ax.set_xlabel(\"experiment #\")\n", + "ax.set_ylabel(\"test MSE\")\n", + "ax.set_title(\"Fig. 3(b) style \u2014 MSE by repeat\")\n", + "ax.legend()\n", + "plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## References\n", + "\n", + "- Damianou, A. and Lawrence, N. D. (2013). *Deep Gaussian Processes*. In *AISTATS* (pp. 207\u2013215). \u2014 [PDF](https://proceedings.mlr.press/v31/damianou13a.html)\n", + "- Script counterpart: `examples/damianou2013/toy_warped_regression.py`\n" + ] + } + ] +} \ No newline at end of file diff --git a/examples/damianou2013/notebooks/03_usps_digits.ipynb b/examples/damianou2013/notebooks/03_usps_digits.ipynb new file mode 100644 index 0000000..8ff4971 --- /dev/null +++ b/examples/damianou2013/notebooks/03_usps_digits.ipynb @@ -0,0 +1,275 @@ +{ + "nbformat": 4, + "nbformat_minor": 5, + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "pygments_lexer": "ipython3" + } + }, + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# USPS digits depth / ARD / 1-NN (Damianou & Lawrence 2013, \u00a74.3 / figs. 6\u20138)\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/03_usps_digits.ipynb)\n", + "\n", + "**Paper:** [Damianou, A. and Lawrence, N. D. (2013). *Deep Gaussian Processes*. In *AISTATS* (pp. 207\u2013215).](https://proceedings.mlr.press/v31/damianou13a.html)\n", + "\n", + "Subsample 50 examples of each digit $\\{0,1,6\\}$ from USPS ($16\\times16$). Train unsupervised DeepGPs of increasing depth and evaluate (i) the variational bound and (ii) leave-one-out 1-NN error in the top latent features. The paper finds both the bound and nearest-neighbour quality improve with depth, with a 5-layer model nearly separating the classes in a low-dimensional top space.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Install dependencies when running on Google Colab\n", + "import sys\n", + "import subprocess\n", + "\n", + "IN_COLAB = \"google.colab\" in sys.modules\n", + "if IN_COLAB:\n", + " pkgs = [\"numpy>=1.7\", \"scipy>=0.12\", \"matplotlib\", \"GPy>=1.15\"]\n", + " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", *pkgs])\n", + " subprocess.check_call([\n", + " sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", + " \"git+https://github.com/SheffieldML/PyDeepGP.git@main\",\n", + " ])\n", + " print(\"Installed GPy and PyDeepGP for Colab.\")\n", + "else:\n", + " print(\"Local environment \u2014 ensure PyDeepGP and GPy are importable.\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Set QUICK=True for a short smoke run (fewer iterations / smaller models).\n", + "# Set QUICK=False for a paper-scale demonstration (slower).\n", + "QUICK = True\n", + "SEED = 42\n", + "\n", + "DEPTHS = [1, 2] if QUICK else [1, 2, 3, 4, 5]\n", + "PER_CLASS = 20 if QUICK else 50\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import gzip\n", + "import os\n", + "import urllib.request\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "import GPy\n", + "import deepgp\n", + "\n", + "np.random.seed(SEED)\n", + "DIGITS = (0, 1, 6)\n", + "ESL_URL = \"https://web.stanford.edu/~hastie/ElemStatLearn/datasets/zip.train.gz\"\n", + "CACHE = os.path.join(\"/tmp\", \"pydeepgp_usps_zip.train.gz\")\n", + "\n", + "def download_usps(path=CACHE):\n", + " if os.path.isfile(path) and os.path.getsize(path) > 0:\n", + " return path\n", + " print(\"Downloading USPS zip.train \u2026\")\n", + " req = urllib.request.Request(ESL_URL, headers={\"User-Agent\": \"PyDeepGP-notebooks\"})\n", + " with urllib.request.urlopen(req, timeout=120) as resp, open(path, \"wb\") as out:\n", + " out.write(resp.read())\n", + " return path\n", + "\n", + "def load_usps(path):\n", + " rows = []\n", + " with gzip.open(path, \"rt\") as f:\n", + " for line in f:\n", + " parts = line.strip().split()\n", + " if parts:\n", + " rows.append([float(x) for x in parts])\n", + " data = np.asarray(rows)\n", + " return data[:, 1:], data[:, 0].astype(int)\n", + "\n", + "def subsample(Y, labels, per_class=50, seed=42):\n", + " rng = np.random.RandomState(seed)\n", + " Ys, Ls = [], []\n", + " for d in DIGITS:\n", + " idx = np.where(labels == d)[0]\n", + " choose = rng.choice(idx, size=per_class, replace=False)\n", + " Ys.append(Y[choose]); Ls.append(np.full(per_class, d))\n", + " return np.vstack(Ys), np.concatenate(Ls)\n", + "\n", + "def rbf_bias(q):\n", + " return GPy.kern.RBF(q, ARD=True) + GPy.kern.Bias(q)\n", + "\n", + "def ard_weights(kern):\n", + " ls = np.asarray(kern.rbf.lengthscale).ravel() if hasattr(kern, \"rbf\") else np.asarray(kern.lengthscale).ravel()\n", + " return 1.0 / np.maximum(ls, 1e-12)\n", + "\n", + "def nn_error(X, labels):\n", + " sq = np.sum(X * X, axis=1, keepdims=True)\n", + " d2 = sq + sq.T - 2 * X @ X.T\n", + " np.fill_diagonal(d2, np.inf)\n", + " nn = np.argmin(d2, axis=1)\n", + " mistakes = int(np.sum(labels[nn] != labels))\n", + " return mistakes, mistakes / len(labels)\n", + "\n", + "def depth_dims(depth, data_dim, quick):\n", + " widths = {1: [8], 2: [8, 5], 3: [8, 6, 4]} if quick else {\n", + " 1: [12], 2: [12, 8], 3: [12, 10, 6], 4: [12, 10, 8, 5], 5: [12, 10, 8, 6, 4]\n", + " }\n", + " return [data_dim] + widths[depth]\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "path = download_usps()\n", + "Y_all, lab_all = load_usps(path)\n", + "Y, labels = subsample(Y_all, lab_all, per_class=PER_CLASS, seed=SEED)\n", + "Y = (Y - Y.mean(0)) / (Y.std() + 1e-12)\n", + "print(\"Subset\", Y.shape, \"classes\", sorted(set(labels.tolist())))\n", + "\n", + "# Show a few digits\n", + "fig, axes = plt.subplots(1, 6, figsize=(8, 1.6))\n", + "for ax, i in zip(axes, range(6)):\n", + " ax.imshow(Y[i].reshape(16, 16), cmap=\"gray\")\n", + " ax.set_title(str(labels[i]))\n", + " ax.axis(\"off\")\n", + "plt.suptitle(\"USPS subset samples\")\n", + "plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Fit depths and score bound / 1-NN\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from GPy.core.parameterization.variational import NormalPosterior, VariationalPosterior\n", + "\n", + "def out_var(y):\n", + " if isinstance(y, (NormalPosterior, VariationalPosterior)):\n", + " return float(np.asarray(y.mean).var())\n", + " return float(np.asarray(y).var())\n", + "\n", + "num_inducing = 20 if QUICK else 40\n", + "iters_fixed, iters_free = (40, 100) if QUICK else (200, 2000)\n", + "\n", + "results, models = [], {}\n", + "for depth in DEPTHS:\n", + " print(f\"=== depth {depth} ===\")\n", + " dims = depth_dims(depth, Y.shape[1], QUICK)\n", + " m = deepgp.DeepGP(\n", + " dims, Y,\n", + " kernels=[rbf_bias(q) for q in dims[1:]],\n", + " num_inducing=num_inducing,\n", + " back_constraint=False,\n", + " )\n", + " for layer in m.layers:\n", + " layer.Gaussian_noise.variance = out_var(layer.Y) * 0.01\n", + " layer.Gaussian_noise.variance.fix()\n", + " m.optimize(max_iters=iters_fixed, messages=False)\n", + " for layer in m.layers:\n", + " layer.Gaussian_noise.variance.unfix()\n", + " m.optimize(max_iters=iters_free, messages=False)\n", + " bound = float(m.log_likelihood())\n", + " Xtop = np.asarray(m.layers[-1].X.mean)\n", + " mistakes, rate = nn_error(Xtop, labels)\n", + " print(f\"bound={bound:.3f} 1-NN mistakes={mistakes}/{Y.shape[0]} ({rate:.3%})\")\n", + " results.append((depth, bound, mistakes, rate))\n", + " models[depth] = m\n", + "\n", + "print(\"\\ndepth bound 1-NN errors\")\n", + "for depth, bound, mistakes, rate in results:\n", + " print(f\"{depth:5d} {bound:9.3f} {mistakes:3d} ({rate:.3%})\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Visualisations (figs. 6\u20137 style)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "deepest = max(DEPTHS)\n", + "m = models[deepest]\n", + "Xtop = np.asarray(m.layers[-1].X.mean)\n", + "w = ard_weights(m.layers[-1].kern)\n", + "dims = list(np.argsort(-w)[:2])\n", + "colors = {0: \"C0\", 1: \"C1\", 6: \"C2\"}\n", + "markers = {0: \"o\", 1: \"s\", 6: \"^\"}\n", + "\n", + "fig, ax = plt.subplots(figsize=(4.5, 4.5))\n", + "for d in DIGITS:\n", + " mask = labels == d\n", + " ax.scatter(Xtop[mask, dims[0]], Xtop[mask, dims[1]],\n", + " c=colors[d], marker=markers[d], s=28, label=str(d))\n", + "ax.set_xlabel(f\"top dim {dims[0]}\")\n", + "ax.set_ylabel(f\"top dim {dims[1]}\")\n", + "ax.set_title(f\"Top latent (depth {deepest})\")\n", + "ax.legend()\n", + "plt.show()\n", + "\n", + "fig, axes = plt.subplots(1, len(m.layers), figsize=(3.0 * len(m.layers), 2.8))\n", + "if len(m.layers) == 1:\n", + " axes = [axes]\n", + "for i, layer in enumerate(m.layers):\n", + " ww = ard_weights(layer.kern)\n", + " axes[i].bar(np.arange(1, len(ww) + 1), ww)\n", + " axes[i].set_title(f\"layer {i} ARD\")\n", + "fig.suptitle(\"ARD weights (1/lengthscale) \u2014 fig. 6 style\")\n", + "fig.tight_layout()\n", + "plt.show()\n", + "\n", + "fig, ax = plt.subplots(figsize=(5.5, 3.5))\n", + "ax.plot([r[0] for r in results], [r[1] for r in results], \"o-\", label=\"variational bound\")\n", + "ax.set_xlabel(\"depth\"); ax.set_ylabel(\"bound\")\n", + "ax2 = ax.twinx()\n", + "ax2.plot([r[0] for r in results], [r[2] for r in results], \"s--\", color=\"C1\", label=\"1-NN mistakes\")\n", + "ax2.set_ylabel(\"1-NN mistakes\")\n", + "ax.set_title(\"Depth selection via bound / 1-NN\")\n", + "plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## References\n", + "\n", + "- Damianou, A. and Lawrence, N. D. (2013). *Deep Gaussian Processes*. In *AISTATS* (pp. 207\u2013215). \u2014 [PDF](https://proceedings.mlr.press/v31/damianou13a.html)\n", + "- USPS via ElemStatLearn `zip.train`\n", + "- Script counterpart: `examples/damianou2013/usps_digits.py`\n" + ] + } + ] +} \ No newline at end of file diff --git a/examples/damianou2013/notebooks/04_highfive_mrd.ipynb b/examples/damianou2013/notebooks/04_highfive_mrd.ipynb new file mode 100644 index 0000000..2bddc26 --- /dev/null +++ b/examples/damianou2013/notebooks/04_highfive_mrd.ipynb @@ -0,0 +1,224 @@ +{ + "nbformat": 4, + "nbformat_minor": 5, + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "pygments_lexer": "ipython3" + } + }, + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Shared-view / high-five MRD (Damianou & Lawrence 2013, \u00a74.2 / figs. 4\u20135)\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/04_highfive_mrd.ipynb)\n", + "\n", + "**Paper:** [Damianou, A. and Lawrence, N. D. (2013). *Deep Gaussian Processes*. In *AISTATS* (pp. 207\u2013215).](https://proceedings.mlr.press/v31/damianou13a.html)\n", + "\n", + "Two conditionally independent observation views share a parent latent layer (manifold relevance determination / shared GP-LVM style). In the paper this models two CMU MOCAP subjects performing a high-five; ARD on each view reveals shared vs private dimensions (fig. 4).\n", + "\n", + "This notebook uses **synthetic correlated views** by default (reliable in Colab). Optionally try a pods MOCAP stand-in. Multi-view DeepGP requires CIP-0002 MRD support.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Install dependencies when running on Google Colab\n", + "import sys\n", + "import subprocess\n", + "\n", + "IN_COLAB = \"google.colab\" in sys.modules\n", + "if IN_COLAB:\n", + " pkgs = [\"numpy>=1.7\", \"scipy>=0.12\", \"matplotlib\", \"GPy>=1.15\"]\n", + " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", *pkgs])\n", + " subprocess.check_call([\n", + " sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", + " \"git+https://github.com/SheffieldML/PyDeepGP.git@main\",\n", + " ])\n", + " print(\"Installed GPy and PyDeepGP for Colab.\")\n", + "else:\n", + " print(\"Local environment \u2014 ensure PyDeepGP and GPy are importable.\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Set QUICK=True for a short smoke run (fewer iterations / smaller models).\n", + "# Set QUICK=False for a paper-scale demonstration (slower).\n", + "QUICK = True\n", + "SEED = 42\n", + "\n", + "USE_PODS = False # set True to try pods CMU walk/jog feature split\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "import GPy\n", + "import deepgp\n", + "\n", + "np.random.seed(SEED)\n", + "\n", + "def rbf_bias(q, ard=True):\n", + " return GPy.kern.RBF(q, ARD=ard) + GPy.kern.Bias(q)\n", + "\n", + "def ard_weights(kern):\n", + " ls = np.asarray(kern.rbf.lengthscale).ravel() if hasattr(kern, \"rbf\") else np.asarray(kern.lengthscale).ravel()\n", + " return 1.0 / np.maximum(ls, 1e-12)\n", + "\n", + "def load_views(seed=42, use_pods=False):\n", + " if use_pods:\n", + " try:\n", + " import pods\n", + " data = pods.datasets.cmu_mocap_35_walk_jog()\n", + " Y = np.asarray(data[\"Y\"], dtype=float)[:80]\n", + " mid = Y.shape[1] // 2\n", + " print(\"Using pods walk/jog feature split as a two-view stand-in.\")\n", + " return Y[:, :mid], Y[:, mid:], \"pods\"\n", + " except Exception as exc:\n", + " print(\"pods unavailable:\", exc)\n", + " rng = np.random.RandomState(seed)\n", + " n, q, d = 60, 3, 20\n", + " X = rng.randn(n, q)\n", + " Y1 = X @ rng.randn(q, d) + 0.05 * rng.randn(n, d)\n", + " Y2 = X @ rng.randn(q, d) + 0.05 * rng.randn(n, d)\n", + " return Y1, Y2, \"synthetic\"\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "Y1, Y2, source = load_views(SEED, use_pods=USE_PODS)\n", + "Y1 = (Y1 - Y1.mean(0)) / (Y1.std() + 1e-12)\n", + "Y2 = (Y2 - Y2.mean(0)) / (Y2.std() + 1e-12)\n", + "print(f\"Views {Y1.shape}, {Y2.shape} (source={source})\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Fit shared-hierarchy DeepGP\n", + "\n", + "Observation kernels are a **list of two kernels** (one per view), each with input dimension equal to the shared latent width.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from GPy.core.parameterization.variational import NormalPosterior, VariationalPosterior\n", + "\n", + "def out_var(y):\n", + " if isinstance(y, (NormalPosterior, VariationalPosterior)):\n", + " return float(np.asarray(y.mean).var())\n", + " return float(np.asarray(y).var())\n", + "\n", + "q1, q2 = (4, 3) if QUICK else (6, 4)\n", + "M = 15 if QUICK else 30\n", + "it_f, it_u = (40, 80) if QUICK else (200, 800)\n", + "\n", + "m = deepgp.DeepGP(\n", + " [[Y1.shape[1], Y2.shape[1]], q1, q2],\n", + " [Y1, Y2],\n", + " kernels=[[rbf_bias(q1), rbf_bias(q1)], rbf_bias(q2)],\n", + " num_inducing=M,\n", + " back_constraint=False,\n", + ")\n", + "for layer in m.layers:\n", + " if hasattr(layer, \"views\"):\n", + " for view in layer.views:\n", + " view.Gaussian_noise.variance = out_var(view.Y) * 0.01\n", + " view.Gaussian_noise.variance.fix()\n", + " else:\n", + " layer.Gaussian_noise.variance = out_var(layer.Y) * 0.01\n", + " layer.Gaussian_noise.variance.fix()\n", + "m.optimize(max_iters=it_f, messages=False)\n", + "for layer in m.layers:\n", + " if hasattr(layer, \"views\"):\n", + " for view in layer.views:\n", + " view.Gaussian_noise.variance.unfix()\n", + " else:\n", + " layer.Gaussian_noise.variance.unfix()\n", + "m.optimize(max_iters=it_u, messages=True)\n", + "print(f\"Variational bound: {float(m.log_likelihood()):.3f}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## ARD per view and top layer (fig. 4 style)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "fig, axes = plt.subplots(1, 3, figsize=(10, 3.2))\n", + "for i, view in enumerate(m.layers[0].views):\n", + " w = ard_weights(view.kern)\n", + " axes[i].bar(np.arange(1, len(w) + 1), w)\n", + " axes[i].set_title(f\"view {i} ARD\")\n", + " axes[i].set_xlabel(\"shared dim\")\n", + " print(f\"View {i} ARD:\", np.array2string(w, precision=3))\n", + "w_top = ard_weights(m.layers[-1].kern)\n", + "axes[2].bar(np.arange(1, len(w_top) + 1), w_top)\n", + "axes[2].set_title(\"top layer ARD\")\n", + "fig.suptitle(\"Shared MRD ARD weights (fig. 4 style)\")\n", + "fig.tight_layout()\n", + "plt.show()\n", + "\n", + "# 2D projection of dominant shared dimensions from view 0's posterior mean\n", + "Xshare = np.asarray(m.layers[0].X.mean)\n", + "w0 = ard_weights(m.layers[0].views[0].kern)\n", + "dims = list(np.argsort(-w0)[:2])\n", + "fig, ax = plt.subplots(figsize=(4.5, 4))\n", + "ax.plot(Xshare[:, dims[0]], Xshare[:, dims[1]], \"o-\", markersize=4)\n", + "ax.set_xlabel(f\"shared dim {dims[0]}\")\n", + "ax.set_ylabel(f\"shared dim {dims[1]}\")\n", + "ax.set_title(\"Shared latent trajectory (fig. 5 style)\")\n", + "plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## References\n", + "\n", + "- Damianou, A. and Lawrence, N. D. (2013). *Deep Gaussian Processes*. In *AISTATS* (pp. 207\u2013215). \u2014 [PDF](https://proceedings.mlr.press/v31/damianou13a.html), \u00a74.2\n", + "- Damianou et al. (2012), shared / MRD GP-LVM\n", + "- Lawrence & Moore (2007), hierarchical GP-LVM (high-five MOCAP)\n", + "- Scripts: `examples/damianou2013/highfive_shared.py`, `examples/example_mrd.py`\n", + "- CIP-0002: MRD revival on the maintained DeepGP path\n" + ] + } + ] +} \ No newline at end of file diff --git a/examples/damianou2013/notebooks/README.md b/examples/damianou2013/notebooks/README.md new file mode 100644 index 0000000..494d1b3 --- /dev/null +++ b/examples/damianou2013/notebooks/README.md @@ -0,0 +1,19 @@ +# Damianou & Lawrence 2013 — Colab notebooks + +Interactive recreations of the AISTATS 2013 Deep GP experiments. + +| Notebook | Paper | Colab | +| --- | --- | --- | +| [01_toy_hierarchy.ipynb](01_toy_hierarchy.ipynb) | §4.1 fig. 2 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb) | +| [02_toy_warped_regression.ipynb](02_toy_warped_regression.ipynb) | §4.1 fig. 3 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb) | +| [03_usps_digits.ipynb](03_usps_digits.ipynb) | §4.3 figs. 6–8 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/03_usps_digits.ipynb) | +| [04_highfive_mrd.ipynb](04_highfive_mrd.ipynb) | §4.2 figs. 4–5 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/04_highfive_mrd.ipynb) | + +Each notebook: + +1. Shows an **Open in Colab** badge at the top. +2. Installs GPy + PyDeepGP automatically when `google.colab` is detected. +3. Uses `QUICK = True` by default for a short run; set `False` for paper-scale settings. +4. Cites Damianou & Lawrence (AISTATS 2013) and points at the matching `.py` script. + +Paper: https://proceedings.mlr.press/v31/damianou13a.html From 23ab9ed7c4179a5e1b1d2de1f1d08bb5a9e6f0f4 Mon Sep 17 00:00:00 2001 From: Neil Lawrence Date: Fri, 13 Nov 2026 21:48:39 +0000 Subject: [PATCH 2/3] Align Damianou 2013 examples with SheffieldML/deepGP Matlab demos. Rewrite scripts against demToyUnsupervised, demToyRegressionSimple, demHighFiveDemonstration, and demUsps; ship reference .mat files and thin Colab notebooks that call the ports. --- examples/damianou2013/README.md | 85 ++---- examples/damianou2013/SOURCE.md | 20 ++ examples/damianou2013/_matlab_data.py | 161 +++++++++++ examples/damianou2013/data/README.md | 8 + .../damianou2013/data/demHighFiveHgplvm1.mat | Bin 0 -> 66925 bytes .../damianou2013/data/hgplvmSampleDataTr1.mat | Bin 0 -> 7016 bytes examples/damianou2013/highfive_shared.py | 135 ++++----- .../notebooks/01_toy_hierarchy.ipynb | 222 +++------------ .../notebooks/02_toy_warped_regression.ipynb | 210 +++----------- .../notebooks/03_usps_digits.ipynb | 263 +++--------------- .../notebooks/04_highfive_mrd.ipynb | 212 +++----------- examples/damianou2013/notebooks/README.md | 21 +- examples/damianou2013/toy_hierarchy.py | 188 +++++++------ .../damianou2013/toy_warped_regression.py | 165 ++++------- examples/damianou2013/usps_digits.py | 205 ++++---------- 15 files changed, 683 insertions(+), 1212 deletions(-) create mode 100644 examples/damianou2013/SOURCE.md create mode 100644 examples/damianou2013/_matlab_data.py create mode 100644 examples/damianou2013/data/README.md create mode 100644 examples/damianou2013/data/demHighFiveHgplvm1.mat create mode 100644 examples/damianou2013/data/hgplvmSampleDataTr1.mat diff --git a/examples/damianou2013/README.md b/examples/damianou2013/README.md index 05fffdb..e29aa74 100644 --- a/examples/damianou2013/README.md +++ b/examples/damianou2013/README.md @@ -1,80 +1,53 @@ -# Damianou & Lawrence (AISTATS 2013) — PyDeepGP recreations +# Damianou & Lawrence (AISTATS 2013) — PyDeepGP ports -Scripts and Colab notebooks that recreate experiments from -[Damianou & Lawrence, *Deep Gaussian Processes*](https://proceedings.mlr.press/v31/damianou13a.html) -(AISTATS 2013) on the maintained `DeepGP` path. +Ports of the official Matlab demos in +[SheffieldML/deepGP](https://github.com/SheffieldML/deepGP) +(`deepGP/matlab/`). See [SOURCE.md](SOURCE.md) for the file-level mapping. -These are **modern recreations**, not bit-identical paper scripts: optimiser -schedules, inducing-point counts, and initialisation differ from the 2013 -Matlab stack. Use `--quick` / `QUICK = True` for a short smoke run. +These are **not** bit-identical to Matlab (optimiser / kernel stack differ), but +they use the same reference data files, model widths, and evaluation metrics +as the demos — not invented protocols. -## Notebooks (Colab) - -Each notebook starts with an **Open in Colab** badge and a cell that installs -GPy + PyDeepGP when run on Colab. - -| Paper section | Notebook | -| --- | --- | -| §4.1 fig. 2 — toy hierarchy | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb) [`01_toy_hierarchy.ipynb`](notebooks/01_toy_hierarchy.ipynb) | -| §4.1 fig. 3 — warped regression | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb) [`02_toy_warped_regression.ipynb`](notebooks/02_toy_warped_regression.ipynb) | -| §4.3 figs. 6–8 — USPS depth / ARD | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/03_usps_digits.ipynb) [`03_usps_digits.ipynb`](notebooks/03_usps_digits.ipynb) | -| §4.2 figs. 4–5 — high-five / MRD | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/04_highfive_mrd.ipynb) [`04_highfive_mrd.ipynb`](notebooks/04_highfive_mrd.ipynb) | +Paper: [Damianou & Lawrence, AISTATS 2013](https://proceedings.mlr.press/v31/damianou13a.html). -Locally: `jupyter notebook notebooks/01_toy_hierarchy.ipynb` (with PyDeepGP installed). - -## Mapping (scripts) +## Notebooks (Colab) -| Paper section | Script | Status | +| Paper | Notebook | Matlab demo | | --- | --- | --- | -| §4.1 Toy hierarchy (fig. 2) | `toy_hierarchy.py` | Runnable | -| §4.1 Warped regression (fig. 3) | `toy_warped_regression.py` | Runnable | -| §4.3 USPS {0,1,6} depth / ARD / 1-NN (figs. 6–8) | `usps_digits.py` | Runnable (downloads ESL `zip.train`) | -| §4.2 CMU high-five shared views (figs. 4–5) | `highfive_shared.py` | Runnable via CIP-0002 MRD (synthetic or pods stand-in) | +| §4.1 fig. 2 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb) [01](notebooks/01_toy_hierarchy.ipynb) | `demToyUnsupervised.m` | +| §4.1 fig. 3 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb) [02](notebooks/02_toy_warped_regression.ipynb) | `demToyRegressionSimple.m` | +| §4.3 figs. 6–8 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/03_usps_digits.ipynb) [03](notebooks/03_usps_digits.ipynb) | `demUsps.m` / `demDigitsDemonstration.m` | +| §4.2 figs. 4–5 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/04_highfive_mrd.ipynb) [04](notebooks/04_highfive_mrd.ipynb) | `demHighFiveDemonstration.m` | -## Setup - -From the repository root (Python ≥ 3.11, GPy ≥ 1.15): - -```bash -python -m pip install -e . -# optional: matplotlib for figures; already a project dependency -``` +Until this PR is on `main`, open Colab from the branch, e.g. +`…/blob/examples-damianou2013/examples/damianou2013/notebooks/…`. -USPS data are cached under `examples/damianou2013/data/zip.train.gz` on first run -(ElemStatLearn `zip.train`). +## Scripts -## Quick smoke tests +| Script | Data | Notes | +| --- | --- | --- | +| `toy_hierarchy.py` | `data/hgplvmSampleDataTr1.mat` | Two-view DeepGP Q={6,4}; vs stacked PCA & Isomap | +| `toy_warped_regression.py` | `hierGpsNEW` generator | Defaults = Matlab simple (N=150,D=15,Ntr=25); `--paper` → N=120,D=10 | +| `highfive_shared.py` | `data/demHighFiveHgplvm1.mat` | YA/YB 78×62; plots vs stored hgplvm latents | +| `usps_digits.py` | ESL `zip.train` | `vercat2` 50×{0,6,1}; Q={15,12,10,8,6}; K=80 | ```bash cd examples/damianou2013 python toy_hierarchy.py --quick --no-plot python toy_warped_regression.py --quick --no-plot -python usps_digits.py --quick --no-plot --depths 1 2 -python highfive_shared.py --quick --no-plot --synthetic +python highfive_shared.py --quick --no-plot +python usps_digits.py --quick --no-plot --depths 1 ``` -Save figures without interactive display: - -```bash -python toy_hierarchy.py --quick --no-plot --outdir /tmp/dgp2013 -``` +Omit `--quick` for demo-scale optimisation (slow, especially USPS depth 5). -## Paper-scale runs - -Omit `--quick` (slower; especially USPS depth 1–5): +## Setup ```bash -python toy_warped_regression.py --repeats 10 --outdir /tmp/dgp2013 -python usps_digits.py --depths 1 2 3 4 5 --outdir /tmp/dgp2013 +python -m pip install -e . # from repo root +# Isomap comparison in toy_hierarchy: pip install scikit-learn ``` -## High-five / MRD - -The high-five experiment uses **conditionally independent observation views** -with a shared parent latent layer (`ObservedMRDLayer`, CIP-0002). -`highfive_shared.py` fits that model; use `--synthetic` when MOCAP data are -unavailable. See also ``examples/example_mrd.py``. - ## Citation ``` diff --git a/examples/damianou2013/SOURCE.md b/examples/damianou2013/SOURCE.md new file mode 100644 index 0000000..56b2bc2 --- /dev/null +++ b/examples/damianou2013/SOURCE.md @@ -0,0 +1,20 @@ +# Provenance: Matlab deepGP demos + +Python scripts and notebooks in this directory are ports of the official +AISTATS 2013 Matlab package: + +**https://github.com/SheffieldML/deepGP** (`deepGP/matlab/`) + +| Paper | Matlab demo | Python entry | +| --- | --- | --- | +| §4.1 fig. 2 hierarchy | `demToyUnsupervised.m` + `hgplvmSampleDataTr1.mat` | `toy_hierarchy.py` | +| §4.1 fig. 3 regression | `demToyRegressionSimple.m` + `demToyDynamicsCreateData.m` (`hierGpsNEW`) | `toy_warped_regression.py` | +| §4.2 figs. 4–5 high-five | `demHighFiveDemonstration.m` + `demHighFiveHgplvm1.mat` | `highfive_shared.py` | +| §4.3 figs. 6–8 USPS | `demUsps.m` / `demDigitsDemonstration.m` (`vercat2`, H=5, Q={15,12,10,8,6}) | `usps_digits.py` | + +Walkthrough narrative: `tutorial.m`. + +These are still **not** bit-identical to Matlab (different optimiser stack, +kernel implementations, and PyDeepGP API), but data construction, model +depth/width, and evaluation metrics follow the demos above rather than +ad-hoc settings. diff --git a/examples/damianou2013/_matlab_data.py b/examples/damianou2013/_matlab_data.py new file mode 100644 index 0000000..cedefc1 --- /dev/null +++ b/examples/damianou2013/_matlab_data.py @@ -0,0 +1,161 @@ +"""Loaders and generators matching SheffieldML/deepGP Matlab demos.""" + +from __future__ import annotations + +import os +from typing import Dict, List, Optional, Tuple + +import numpy as np +import scipy.io as sio + +_DATA_DIR = os.path.join(os.path.dirname(__file__), "data") + + +def data_path(name: str) -> str: + return os.path.join(_DATA_DIR, name) + + +def load_toy_hierarchy(mat_name: str = "hgplvmSampleDataTr1.mat") -> Dict[str, np.ndarray]: + """Ground-truth hierarchy from ``demToyUnsupervised.m`` / ``hgplvmSampleDataTr1.mat``. + + Returns arrays ``X2`` (top), ``XA``, ``XB`` (intermediate), ``YA``, ``YB`` (leaves). + """ + path = data_path(mat_name) + d = sio.loadmat(path, squeeze_me=True) + out = {k: np.asarray(d[k], dtype=float) for k in ("X2", "XA", "XB", "YA", "YB")} + for k, arr in out.items(): + if arr.ndim == 1: + out[k] = arr[:, None] + return out + + +def load_highfive(mat_name: str = "demHighFiveHgplvm1.mat") -> Dict[str, np.ndarray]: + """CMU high-five views from ``demHighFiveDemonstration.m``. + + Returns ``YA``, ``YB`` (78×62) and optional hgplvm latents ``Xparent``, + ``Xchild1``, ``Xchild2``. + """ + path = data_path(mat_name) + d = sio.loadmat(path, struct_as_record=False, squeeze_me=True) + Y = d["Y"] + out = { + "YA": np.asarray(Y.YA, dtype=float), + "YB": np.asarray(Y.YB, dtype=float), + } + for name in ("Xparent", "Xchild1", "Xchild2"): + if hasattr(Y, name): + out[name] = np.asarray(getattr(Y, name), dtype=float) + return out + + +def scale_data(X: np.ndarray) -> np.ndarray: + """Zero-mean unit-variance columns (Matlab ``scaleData(X,1)``).""" + X = np.asarray(X, dtype=float) + X = X - X.mean(axis=0, keepdims=True) + std = X.std(axis=0, keepdims=True) + std[std < 1e-12] = 1.0 + return X / std + + +def generate_hier_gps_new( + n_toy: int = 150, + d_toy: int = 15, + seed: int = 2, +) -> Tuple[np.ndarray, np.ndarray, np.ndarray]: + """Match ``demToyDynamicsCreateData.m`` branch ``hierGpsNEW``. + + Returns ``(X, Y1_warp, Y)`` where ``X`` is the observed input, + ``Y1_warp`` is the hidden 1D warp, and ``Y`` is the D-dimensional output. + """ + import GPy + + rng = np.random.RandomState(seed) + # Matlab: X = linspace(1,2*pi,Ntoy)'; scaleData + X = np.linspace(1.0, 2.0 * np.pi, n_toy)[:, None] + X = scale_data(X) + + # First GP: rbf + lin + white, inverseWidth=4 → lengthscale = 0.5 + kern1 = ( + GPy.kern.RBF(1, variance=1.0, lengthscale=0.5) + + GPy.kern.Linear(1, variances=1.0) + + GPy.kern.White(1, variance=1e-6) + ) + K1 = kern1.K(X) + # gsamp: one sample from N(0,K) + Y1 = rng.multivariate_normal(np.zeros(n_toy), K1)[:, None] + + # Second GP: rbf on Y1, inverseWidth=16 → lengthscale=0.25 + kern2 = GPy.kern.RBF(1, variance=1.0, lengthscale=0.25) + GPy.kern.White( + 1, variance=1e-6 + ) + K2 = kern2.K(Y1) + # D independent samples + L2 = np.linalg.cholesky(K2 + 1e-10 * np.eye(n_toy)) + Y = L2 @ rng.randn(n_toy, d_toy) + return X, Y1, Y + + +def train_test_split_indices(n_toy: int, n_tr: int, seed: int) -> Tuple[np.ndarray, np.ndarray]: + """Match ``demToyDynamicsSplitDataset.m`` (non-block): random ``Ntr`` train indices.""" + rng = np.random.RandomState(seed) + perm = rng.permutation(n_toy) + ind_tr = np.sort(perm[:n_tr]) + ind_ts = np.sort(perm[n_tr:]) + return ind_tr, ind_ts + + +def load_usps_vercat2( + per_class: int = 50, + cache_dir: Optional[str] = None, +) -> Tuple[np.ndarray, np.ndarray]: + """USPS digits {0,1,6} as in ``demUsps.m`` / ``demDigitsDemonstration.m`` ``vercat2``. + + Matlab indexing (1-based contiguous blocks in the ESL / GPmat layout) is + approximated by taking the first ``per_class`` examples of each digit from + ESL ``zip.train`` (same counts; order may differ slightly from GPmat). + """ + import gzip + import urllib.request + + if cache_dir is None: + cache_dir = _DATA_DIR + os.makedirs(cache_dir, exist_ok=True) + path = os.path.join(cache_dir, "zip.train.gz") + if not (os.path.isfile(path) and os.path.getsize(path) > 0): + url = "https://web.stanford.edu/~hastie/ElemStatLearn/datasets/zip.train.gz" + req = urllib.request.Request(url, headers={"User-Agent": "PyDeepGP-damianou2013"}) + with urllib.request.urlopen(req, timeout=120) as resp, open(path, "wb") as out: + out.write(resp.read()) + + rows = [] + with gzip.open(path, "rt") as f: + for line in f: + parts = line.strip().split() + if parts: + rows.append([float(x) for x in parts]) + data = np.asarray(rows, dtype=float) + labels = data[:, 0].astype(int) + Y = data[:, 1:] + + pieces_Y: List[np.ndarray] = [] + pieces_L: List[np.ndarray] = [] + # Matlab vercat2 order: 0, 6, 1 + for digit in (0, 6, 1): + idx = np.where(labels == digit)[0] + if idx.size < per_class: + raise RuntimeError(f"Need {per_class} examples of digit {digit}, found {idx.size}") + # Take the first per_class in file order (stable, closer to block loads) + choose = idx[:per_class] + pieces_Y.append(Y[choose]) + pieces_L.append(np.full(per_class, digit, dtype=int)) + return np.vstack(pieces_Y), np.concatenate(pieces_L) + + +# Matlab demDigitsDemonstration experiment 19 widths +USPS_DEPTH_Q = { + 1: [15], + 2: [15, 12], + 3: [15, 12, 10], + 4: [15, 12, 10, 8], + 5: [15, 12, 10, 8, 6], +} diff --git a/examples/damianou2013/data/README.md b/examples/damianou2013/data/README.md new file mode 100644 index 0000000..3a6ce46 --- /dev/null +++ b/examples/damianou2013/data/README.md @@ -0,0 +1,8 @@ +# Reference data from SheffieldML/deepGP (Matlab) + +| File | Demo | Role | +| --- | --- | --- | +| `hgplvmSampleDataTr1.mat` | `demToyUnsupervised.m` (`toyType='hgplvmSampleTr1'`) | Ground-truth hierarchy: `X2`, `XA`, `XB`, `YA`, `YB` | +| `demHighFiveHgplvm1.mat` | `demHighFiveDemonstration.m` / `demHsvargplvmHighFive1.m` | CMU high-five views `Y.YA`, `Y.YB` (78×62 each) | + +Source repo: https://github.com/SheffieldML/deepGP diff --git a/examples/damianou2013/data/demHighFiveHgplvm1.mat b/examples/damianou2013/data/demHighFiveHgplvm1.mat new file mode 100644 index 0000000000000000000000000000000000000000..f09b2f4f6ce810557bcd2473a93f361f0a762140 GIT binary patch literal 66925 zcmb4}Q*$K@vxSq1Cbn(cP9_uEwr$(i#J0^H+qP}nJNVu@|KO~fuCCR0UDX%;ELmX{ z8R0)f>gMxTuXRCX1rR)O4PZEp?SlJ}P z@REsJ0C2mI6hk1%?WBo4qoF@f=*kRbGN7q$+SO1v@%pbv%XtDmp%M+xNDl1K0})$ z<0B{SOsqk>azx}+isO6R%H>@CJViP7I4$%#;MD%5iKtJpb>o@B2p5EO@9gG@%LVQJ z>y>7SNGWPXf`ZXrMv2%DbA=PS)c$?VGywsM7N7rv}Pf4!Q&J2$5yQ1PEa(D>fjzO8;D zw(@d!DrhC74TT>4Szg2zo1dr8?lCBR@C|J?t8X?6Sr1r=mA!#&Zvj47N28vSQ z>f~I@nK3e@)&a70nSOcB^js_#ud0A@goqFCdk+S75X#EhW9V*^^$?yeg~>lDhq<=hw`WhJFhnD&`xLkD!d&CfX}o<2n%t89ZHvmxr< z@Gm+$FHTStWkweoEx91^;aDYM<0K-$t>U=-Ds?>vrE}KVA$39((qqkl1q@r~Fd=qA zI=EWtXc|UOT=2hCd_yU~Xdmo6DJF%ErI7mWKH&b~w*?2AMp}*_oLyUUm#q&6I3Im| z*-nPA5N%b7E8rDuojN+!(YCwqgFtU%cNyNQ_|up!-X$={Tue}c@B*9{0=|38YqRe$ zoDV9#f<1_t3l-B$5sbmbswd}D8^P{QcU)>q5p;t?-y4ZpA9SEP>e49AA1tvD8izwz zk?1yE>u0;o2hEJtYu&0D<(hZi>;3Z@aw*M9yTTiQe}5(i*OPH8qTU()Txn}@Hu;{G zc&JS|*YSD6l*|^oCXX8$bYl6mH(j~=m}j1#`W((5&M;M&|3E1JrxuiCX z@$sUqHz>AT?L6w{Cd%k(tFk8~tSCSUPx23I4_x)m z1ELEI9G?9vKBPHor3kA2T!8q#PRvj5VLl^k^lQ>OSvs!*E&7)U%wKvH+Buw`fQYW1 z98P^1u!jayO23*s$l?~i(`nY9fcCV=k;$*`fY9@;Z}vw2fIrmIMmZ#CAOXyAjF`9Frut09F|015$b94GZeg1VTIVW4@#P9?of@b^ zYKbf}ex&w7Eo?)@H{q5RwKHD73%kQ6iYK^@PS=%;J#^M|#|Tlw+?&hmSExm~$k#>I zbbo_k->P%$dVW%NT)ajEGnT;dbGlJXi>VgkRU`=j0jq_*5dY-aci4+wPVy&nU~Mhx zB$6&_b@9gp)~wp?{Gq(QJNIi0^(lF()*0T(YH)p3eVZy-DiEpVIg4mHAr*O6Jo^Uu zqi!H>v>n5E!tfGQ7aQg!*>D+ZHP*RcWV^b5Nm(A`YTcwQDyw8R zjJ=S*o~gNC$y`A_dGV<`NLb@%Hg_N4C9;G+@iK6(di)cL#|ylhCnSs8TGKPT+2a0Y zAykHJ|jz9}#0@{{pP{rz{?p>0Z7F+B#*X?Sy0<6H(%`rZ-VKdRAvz&}T>6+sY% zwNits%cuQe2Tb!*0h$hiqZvg6^1;sXOK};DvVn6UVkqXr;1P1UQl(*wedHH2|5mjj z15y`(-m%2pLXYo~GY_A3L}#y_Dh-vW39@Y?EJ2(7-oCx|%GotSPaC0azWmQZcH7Zx zH!(C!lGf@zse0f9l0Hs#s}c~f;TROWZhk^HVV*nAUZzxZfK%D+SDvQ>^65Uq0(9Lq zq}~@vPpAdI$)a`t5-8%_nTaqUn1$7T{XsB3X}Lpy@vamsP9qBoDlJwSab9WKw}ATS z9Q|{NDq1>xQTmZ9+yOYi& zcuYYi$m8iw)u=+d-At^qDqAxb>EkOtFMJ}~Ap|K+Sfb|%PTr$D)I2ir?X;%z*V#-_ zpWw^G*O{R^p}w~`q0<8O=evct;K!-MY&+fyX!Bs)znm$-fR09=;t63v9u_Jze!qM~ zejq19S6e=&4jjiqEy02xO|U5JMvlPp;7LiSMfWw=-GS^Zb(6#rR&nr$G+gum8j|<9VA^x z|E$YHm3%&Av$uyC-R33IT6Dv#@DI2v<)&!=LVI~V2)#GWNUt@0g!XbZBI#PfAVC{H zBHBQKy4y?XIleshEH63A4kzmVQ5qQ1so3SkZt|)|CnI3b5HF)(Ne`&AuMrV*tiRXTbP!xHQCqFVWsM zWV>I3CBwCYu1_IdL#B%CAVs zsmUS%#1f5eDaRfj$oDIR_0SndJ_ZCn7f{tp8ygUtGx+fyIw({j%ksEuJZR9tC77_p z4c_XO(HwIQ7g(k#$YGs@5OjsIXw=`8 zON#HXiL=rHCz4ze;x&bVxU%DKqAne8Zo;R|kJU4a%~Pv;L76}HN<2bg$ zdn^5*cnXAn&AI8QS$h?Fi|*Rp1xQYK%SH4VD1sgI8b>&Rj8<~9rp!;T=e%Igv3S`a z+(C-g6K2L|@XEV+i>J6ufu=5TNpv`c;!{_Ne$)fFk$r&Nb&IjZ5AdCJLyRqi7rzAV z1->YlSg$j6`&=Bz0Ofd>>`p2fzG%kD&{BjBFb8(W;vxgg1FC%8$h{r~{DU_yeN=qR z-d7*G)sb)a&>s;8-t*ObSRT$Rt?K=Xu9}W1vxLqeHqcD=Pj_{LY^@+gA5&<4j+QyQwHpmxv)GgSJW=}KqX$YQgym1`ub;bG)quW ziMp)wQ(E2e?rcgy-0PT$KKXTdnqweu>}vJ&$mGFJ{9p#Ps>A}WD}+PMPKE1T4xDR& z8iaTAExq3VmSDsDwGyfrU4k3j%hqS?&VDI6GKecX^8_5;cGR{c+V(X+9ubu}u_94< z-e_*BLy-`OPv}?Ffex^q+JWcR{=!hDIwsbK(0Car^^}!@UNdWW86<#!TR|+fNWilX zdWLvm2g7o^W?n+ z7H7tj?%sLHeTPt!Rr4DKWmgD80gu~5T*vhgUA zf044PPr1X4)Z`fUT002NHB4^YKw%6Mp&R|({65XrX2_jlh9>~4hkpscMa_bWyHON~ z>%;_kSI>CJ=z-WXN1{=wssC+a`syEFrw8Jus(;0C#{e=t$YUo*>TFO+$60cIPj|-K zvutPk!o~#Ttg*IRMboH#X2QFZ4#&a{RSfSwg9~BWn$KO6>Pt9zV0!3kpzz~eQrJ& zTo-~vY7hdGb8Fc643C8t0dsDoT-Wjin#eE6?<}$yZ+Cd|d+yq*|&whPLlr;XK zzc*xT)ZN|pHfV)H$3*J_X6{BKDEkraxn79>Mw%bt3ba1d-wcf@wz}<|md=bHFkMkTtrH(G@`@ z1eXEm^(Tq69@NMvdvL{OE6Y%Omg4e9Kn808K#Q@D*l_QuFlLjTOnc5QU%YAgJRtNs zg~Gv$bfrHUg~CpXOiBmgVL3BEp@_>CZW}M}=ZHc?M18fEwUh{!bjU@1_{_jtMvlbQ zq>SO#&;54e4&i`@<&{O4TPG7o8`*?>A+#|yd6Aee0!o~XLKh*0_yDF7OAZ5e!X0eW zROB!i+2@Jv7c-$eN^VYSDq4rs0H;5ODWAXafYFOR%Hx9MTf$!K;{`Wa*LG3Sr4cf; z^Z7px?3RRjsl)#3rZ2MW%?>U%m#C2E7JJ5{hR8klicSO%m?(qZ0B~1SZsbqu>@11~ zIqXyA^1kH4zi7zYkp) z&72gy-|yE}XN~=Pzcq}i-}MrGuy7hODE_x0{#Ys!m7s8K(e&8n&Q!6Q!i)KQ9ZYeH zz1y;3AHTsq45Uy7jS$WjV=mZ;OHH0nl2#ecy#ld4^c#IJ591GImd#KDGP{~LwK$$N z(29y$Cd$}(XETfBiQEA;-e@Eih~l9{vFUOV4_@2!F>Gl{8VShwJWiHL9jbf*JeGNs z+%fhXPTcq@wUM8k1MPE^)~L+&oBx%v)q1!*2B@^M#brG1`lsnb-fEghrMQ?nOe_Lx zyu?lGnfT`xO|Z_&eHEhhaVuB(t`gIh!m`%g_0d&`j?Nn|RcTxH!ygZCUni_Yu&cz@ zh5m7LQPtgA@kobVgBnr)G$-wMN=&$(=9!$4*;ps&$%KK^bW%Ev*NerHY$u9_g%WfK z6YeG%VkD{=Mw&?2kRVYEN|C@}68_siGK$srXT#>vw^}Fg4$Jzf)F$huN>`L-(!j*LUu(6Mey#TuxlF3m1X_*XK~ob`*2TdQUUo^WR#A#98}{qeK(eSrcHj5pBrszf81mwd zI}p*x?#ZxX_Xv7_-Y^mW=rW?Kiulx($=)=RKFv+LQx+#c?jsl6zJ&)mWA~0F+`OdF zusjv(0{sHocC!wN>*x&QrToTdaR&rQa3~2e5)H=+E*Nt{zH&sF*XKj@x6+Da?0ktu z{!~4OqOqvJrdUvU4|m*>2S%%|Ojv{4@v{BU(4+LiX0g^nj{Mt?C+&>#{N2^-9{je8 zY$9Q@%mb7*F%xJ*;aY%yGC4;i6OpF6cwqeo`*+B%Zh5FX z=9A>e&8v`)Xz+a@@m7EK-##V-Ng=;h!hT#52H#x9-5Krow^96`ZuMkUzLG;bBpAtn&}(I(PruyGP#8lrAEky0<=_wK!@+bUuY*c4@Ny~ zwUE^bIfTf+)PD+LZ0$is(`oKw1`9Mny(&tp^Hx;?0^?J|zT4YT9!ze0>UQZ6#Su|b z>Sz$pkLyn9p#U3swrbOfp>ngSx@sPTAI3D!S>K9ij-<@Qj%gs7nNxJ0u+th&As>z{(>vp2>h?D~d$+4@46;`7J z|1`yF??|el?tE=7Tl&P4=i7_*HFLEw1|%3sxCgaLz^M%ACZRw#AV`5>ic(hEnE@UgtXP z<#q~9$R}Y6#%4Hay-NL#kH4^((8cV3S_dftiPzTYD z9Ocg+UICp{%_kMn?{wtx<2%Q3o z$wj!@jXSlUsj|=+sls=boDDru!zv25~l5pY+g47;e!1>Bl{Xg64EMZTa!^G~co6nr6ea|##}Nty>%|LC zD7XDqse%75$Q_B)pAeI|Pi2f;>T0gyh8I$p9qPUU3@wR%m-HnmO~*IM{!drOGJuMr z!?Rss5u3u@r1&n{ENkMG z-IU+22yB?RsU{r4grvZ_Pyxw2rR3n6?TPln!i5B8fNSI*(g%ZnJz|D6zy17^!04QO z=n~c8XxXABf=H$AB0ddUflzlzc{D-vh&J=Vq5yW8BS>B{HRHlHy@&8>;~8=QJ5!=* zxX#kp+f7t}>^L&do7vyhwm-{aU2mhE*Y~>oGG+K*`d%6xgK7_?nuM?HT^UXORq@La zFN2&#@$9!lGYReROzKdp!aS6lX6ptiCexry|)O9=7Ut zi+Op^4x|MmMcVQ0_9+6W0C0XYKKt{1`-;r61EpuCdEL?2$^Qy@<#Op$y7M%suZ+p5 zhH9LE*<;s~CR1hn_L+Y*b6^7ezHe_2I)9zIf=nDD9?uve*8I;El)xG`Sp^P9^v9^^c%epIy!+$ZE zU=S1sZY)o)kjP~43Pk7U4Q_K7lTuo=D|NLjDxD7u& zsI$46t~3x|X1CJni4dZ3v>iZ@Y;~@7uz02%MPYx1t)egbYdQYcTE|3nM>1)=DrCGb z$qi4n|C?Cal`&mWhe**37pzFXc`+Mc{CFw1p(%}*uTv_Q*^wwR;*oVJQ+6Z(l{RBZ z@zjS)IUn_;DFp<^s?a3q$0=lVb;nN}X`|AVc9R5bNDRk#s^v@^!Tw?KFb7M?;M`C# z6R@s#P&3}H`WC!|)8SCe_>?{~o(siD8UHa^>7Yr`G*?og@I#^F3(@duE0@tDFuPwy ziiU#LGdUa2H|W^>bVQ@ji`V4Cls6n<4;UEYPaUq?rrRO8Qp^Xt_&ZGS6kT1Bes|Dc zQAWr6P%*{jXS%*BHe82~aHU)TYNk7dvrnGXzAr2aR@##t+m>ZOJiEIgll#krf-!F` z06KNU%2@-9b%H?@9#u0h;jj&X^+e&4eU0&J3Lx?#m45F(S`I!U1@#XnsQv!LzOTEN z1wTQ}Ypi6&NmyzCVOang+QTp0*=r~O?UTsiWV%M&=?;O}_Rf>$PmoVqi7aiu_e+Q= zP8AVgudv*AjpmH|*@NY^mZdme|AURZyw$txeiSOo;jJRk3%-{mSwB@R?aPMTWJx3m z@4u05byGtbo%!-nbIG=jMi<>DL+144GTQtE8o2W(SKbuNn~E;!t)Y>wRWmq`&n)}7 zJ8}Z%52GtMrJr;L(uYLcM{vDlB`qX_m17j>P5=1c;qk-wJ^1oLD<{&1y`zk3zKv;v zkG|}R$CfZ2&Y0}czADH+nE&y=pC7&qPa>6T;D~BtT22FFFg6gYY2&1o7KZ$q7ukMe zk%rv1*+Th(zQk9)Gqv4PH43BVpNdMkgjVt^kni;E>m*(*o7;KjaqHF9w-(r@+k@M2 zSAnl~zkw%E>fmVl5+R=E&3%xuzsuM&=QQxD`oicRyT6aJjCt@y6-)Ju%#poO2}4Cb)onT z?7;j19;+ek?08vDdRZdLsbzHhebys#i_3H>x@5}tyn>AmZ2+~{f1ePhpT(5^IDc~S zr5SG1M0o#!{`=OJ-+uwJi#qLCY@3gX2(?*w@ zv!~&BQg?~0x);;*4k&J;)L`FQ7N`JQ3B#g)^qv&W{MJ4%FE%%ngsmN6+e!-)4=B2Wk$Gbp=vuFnOp&2H^>D=LPqPjYZu!lN z1?dww&tXpG1f_lcxm8%9&BE9x|6JqZSOl*?3HTmI8;tg`qF40K97#x~O!ZK$ zpAKzWN3s5@ye{shyPM2{w}+DB7!63RlSlnXKMb*7EWa&h^ zn{fl>?a4;-GW>l1!b366^5SJOQ_JFBgg&`FfhR)=8=tn4rk0t;n}WhD)T z^koJ|&_ATv?<~GDVQLF8!y=>_b2GjaCL?z$g<{r!)v zbW|#2nHPAy8c!0!9aP)h;aHZMwRI_bXSI8~^rXiUmB7Af3H>d{?U$n%*q*mAmuV$l zK|D8ckFO;;L7!F=v$fK%Zx6fmfy%|>Z;#Kdkm1?e_oBygSJU0_cjM#ue*4$j4=CN`t|Ug(dp$a#c8WK=yyN~p8ZO; zZmUBk+_ONg;CwXYU}b{2{-(b)bGeg}>)~(e7Z-9Vq1xVHd4|1e@bgjAa}>7Ni1$HH z+K~cMdFrg1VT~F|@PpvrmYXeRkq+F6X@IOQ$5X7+POsBYn;dxLgNr140Xw_L0c7Z& zx}xoFh3}&gw8UJHpJ^8w?cAd{*haphmsj`Ugk5@Bi<6+Qb4I%d#Wjbmd;GL;(^9!$ zZ*%TKSvTdY&>RE{DHAJ)dN70zgPiT++kiX{%u5;jlY^{zg6E#Yb}m-luLCJv299U~ zgiQ1P;kAJ?_b(0yW~#lx$D$JikL9~3tejh&=FENFr1t3O_TA#nXF11thlS|cX(NW0 z>t0tU%2|R}^QDn%^o>J@`QD$4n1<1(s?I5X@y$czLdb6L)-{x#kC$;Y+^HQmJ4d5( z@|7z3sl6^{;UZ65vu#*rv-oJ@F9m#q7P&Yq?C8VY+iOc=EgZn)zwI23ga5lxbU3~#v7U$mV?s!ZAs`gXfyTBesCSZ#) z!v0f;5;aqYB~e&k$uppJ=NYo_B>vRzDePR_S&srUDqzCZI~hC%tr%rSV*e8?Wh2)K zyBg>6V$^q*N{EaG1fO#c1BQayWKd`qWitCfgH2^*8uF1RouJEnb@akFv z8>8B5kHNc(Zq`S(nmZtR_nUm?^_ny;O zJ$jeluul&tlV1T>7kiq}epkM6&%q({?1tT8 z@BMWH#fy1f?*mFCBKY`D<<}MM8z-4)Vo5ACI}Ab6m@DPJwn&2?v@QPH*8bG0<%V*L zeZWO82r4Jw0WWFQ!)x91{a|eebit_1{lVq-XC)ae+Y$-F^LE^F3AEf;HBj0-`HN1P ztNI|2IB9$W?_uaoS3Hg%_cF2!8e?e7b+pq9AjRHnHp@A>}krQYVE^&XRm9S-{3BMl9T`8$v`x%RNSYN`?K4JiexhmHe7pL zOy#}+AfbQ2xq)gYDy0sA0YhBT!92{aH&wfeK|RM2>p>6pY9QS>EPy1&$+C_qhrCX! zcLuxCTu}QVgwky&&b^etfxt@|xM!SXUABwd@cf|pJLU!m+gDFt@3er#i6x1<9gFBq zprPCA9|lW^8hg`$SP%K)Cri}zIVO{10*2MDK-<*v2enb&bhDE8TY(IGP0g0~QRjX7 zHpsrRSvrGHsuWNFp!nxsctQ5zLBD_3)@Ec3cNXise50A)$ViSTUVP0@IhgWXqr%x$ zbJOcPW(2VNavQm5C(P**Xxj_fmjA2Y+CkjX9+Q?UVd^`r%&C@_y_KNpXU=PKv7mJOL1AflQmC1Ca|g#%eQq$J1U~@9PQHowra`K>S#VE!RS#9x=m03=_gY-#B;fd{ z=0;Vonjpxl4z%-4Vi~&dRrpGFfBBLy-YWFEQ!pw zvSMCnLRU7Ovfl^gS=7-ZnK+>p&#q+hk;*tMDYz56YfCb$vKsfPzDuU@ukyXhArzci zZdN`y--=H$iWbF3Fqh7x1U%Xci87B|II5e|EwUlHyajbXqUo<03d~=w)Un+fG~~=9(YOSo3HJ6tC)w6xW%SmS7EWFJdN>(7QOd zUC~=oj%P}vQGeXHsdEq2k{iR^^zOZOL*rjeCAmvsHrwJwjQno&cwLqzkc@lc{`R`C zzH0eM4z}$GY>yL(@IucAm=fhUflLZ1J6@UQfnnO0zFHG&87uBGUwND0kV?d(VeNxU z>o~x-uSHl4VrAwbny25_J_mw}pGQ}F<20NAMzXa2ipC&*(5|#wW}xQb5>Y1$tijhR zd|}0dH|h_s?5~tm5Dyg(b#LKMl8%4Bh!{@lT>D_vkhn2sQ2O)zLOep0@=Y^86sJB8 zK(Lvfnfjox;vQ1_w2e?nAxI3~q(s2s82RoYOA*aIgspi&yMliieB8BSGV#!0{u&Nf z^zYs)Cnu(tCtmxxPS;gGgoqLF_G<7Giz`|?PoxNvq@|3j^&~O$bt%A$14k0CNDltm zSYMQqg+MqSIuW1LIR`4O79uC|O z5C<)9KkJLnOw|H+FBqr9Q^aO-W&#M$X?eg_FK5W#GJ`_Qdz=;PX zdyZc7ey?eQH}rf`An}hidgZ{HxH8Zp#abv}p+`JMz*`JKkQDfm($)!a>`sC;xRF9* zZ3K?p9XT5~3nS;FgMDvv3m~n1=0j?@r(n%0L3GMj!l1Dd;q~1B75#nw)ITgJ3%>Mx zeb!T=7qaR5SP8+gOhExJksfl$M#-~+oxjI#XIP<%*Em0hN{2R4rmlY^RhLA9-~eHf zigGjG){cOk-6t`evL9OX1HFY(d2_G**J$uD4%md$$U#p;w{C*Y{lAUZk0%`@hmXd( z3HuHDKA)0C>o47WWXWZIKZU4D>sOs4$ct(GT2>q3Z@c#)+J+f;)nlCa=Ef!A4-!y- zt7~)gufdzN>#agWekzM@Hxr-@X#8<6l{!_P2p;_gSo*tgCg!BO>K}uL-dfDbmPEO^ z40A?4Ei(|ZlxZKUJL&UGk~3%eEgUdyO0^g_*Z%mUm#eMr#C>^2>W^~rY>C|x#CAa_ zwJAYU@OkvdZjNcTkY_Q^!&v#Vf%ZHx8vDGe;2Y5n@t z_;Yu7Lhk(JP`>T0=UPLsnKu6+VauXk8?Pq*lhv-e3etUs*~2=3tt%(?Vh{yj$?ElA zVIR&vkeSZ>g$h>)t2iLMF>L)zmoOKb-Mfjx?JNk4XQ>}~G@pX^<7@hp2B=FC8?U}Y zN<JPE%lz%M)8(!#>v#?Qvxs03 zG1cLHVGLxC4r%sUbL4}-5lj2n}Byrf`F%d&G(&e^RqpSL&ugg);twjEDkn<+Ui51kYgd#{>3GG!wEt+}nEI}!Qa1Ts_4orslILaY7C%188S z+BmVK1j@8Uj{GZ@GYVI4Pvnk*R_k|A`InaAZs;7IcRvsBGH$v~cOx&Nx+hpldl(#Y zR&Z^kkbTdQgiz7%u`)dKTV$}JL17dFmna7y1IlV8FInH&tqV0(dA{aEK*l9qBIw{a zAhM40w@|hYQ5*y`7)ow{ifbZn>4H`wUdNZ0J06z`g`0h}i2mFDvIAltaE42Aq(E&GNVq zNajWG_BSIYn0L2V-%T(tH{vzp!|R`*WdtzL)HGD$4REH=YQIZJ17|}7kfI{XzMUUM zkda$Ruu0G(uX295ZJ55Z17@dhUv(*2RuHV8@0tLOAbegkAx?P+kilEVPUlZ3>MUu?$jx2Jr`BA96DE;L$vdp6Lrn5!?3ZwYljn-qOrC>5 z+YTelL4=Nen&==qjl8ju!4}@1WC-R2;aENnNCXs_&RT5v=yRx1QBeH4BmOuL9*!A$ z=s?xL;(S;T48^@o3iDn-9_R0L;Q-9$ACWQpUL-6(Q2Tu5hQkU@?@yBfe;2)HF>pA2 zsEGCF84R!_TAHD4A9BM}?mY6ow-+($xrVeB4M>0JI6hyYiu3@ep+u?J?j}_F`++bw)920Zv_IJ>%VUrT zi@{0aqlpmWA#mIP#W!OtAfvZxa_ze&=~ohOoK)ccm+JQIU`bzdRSsJEDWq?)jqj%P z`Ukr7{zK|>$Kzn}Toz<3=(f(5JWTUfn3z<5@#D?uL%&kKTaQyAT6(*_n){C}GH8z1 zOcIRHSC!;nnOIaCJ(j2OK(S6*n(aF60&AmQE()Eux9Nn$6i^Se}`);g(yzzwIeFOb%sMWcWW?U*Qt5xdoqKVet2|B7OJBW%4}aEXUUwtHUENt-3s=B(meCxcYz z({LJ{A8)(wTvw?SgLYkyJ-^R|%4}<3+yr(?9RzL`Jbg^FNWQeUqHuEZU`#L9i|HqO zXa@-R8S8l8)aJ9%1X=X1>se%I?`4&hyZ`MUwCJalq<&*76Mm_%=(IFSdjbjYKg$cV zv7anRwo9#rJIZ_L_j9tcK$WI6Y%D{Xw{6O>N%8Y^ELp{7ceN0m?bG)JNjdoe{L_Y> zH0-KOvuyW+Uu4Yld%9PP58EV57L1 z|BXZ9ez`8f*h&Q{&%E#9Ym~9F|NYX%N(eJv+jnc4YZzD&Xh++r*yrNp?^@1=q~u*Prwn(%HcK`lCp?> z>Cli(f-u{7Wf-i__IY0mU?e?D{$(ukR{Tm52W$*@8`&#I3Qm^=L+;+RYcnFN&{0S^ z3?uvO%67~hwtyZn@B&C1TO`Mi@y4jTfu8l2jIFlibxUHse!{>C&+&#FMN&=0xXP z^%jb;sRC))hMt4J!4r0<)eFZAu(xaj^$o?1Z(&q;$O1zSbpm!oSh|9JAe=igtvCmE z0humHdxg=00y8xMe`PrcWtN1ZhGS`6683Febw61pn3sD2f>L(WY9-_@ zN9mQ8ci->y(^^Iu;>a}Br`ipe{HGDCNBkG2FoZcN8$)J@lg(BI%OmH51%>rXC&;))g&*Xx4FlS>LSZ{BSI#9>_d ziO%LqfleNdk4S5J@H$STDZ_HB#r;%C^LH`mWty-)dZ<6?)3nUe-BUL^A$vJizbZO z5Wz;N^j@0!prbe`^khb>Qfe=Oghn31b?#430!UJ43%StkK^_zHWs}wR?LA2YVxu?0|KmYynx1?rySN;zFWXZFSCd$q zYboT%x&Iaz&{8;Tn^A0gQlyj=B@_Q{v?N;W&P1y;OD%r|2@fXH@gFi1J7(8r5OT&YnV$g!@ffds-;ORr$fAGdMPK!xRB%1Nzow-q zlNZsv^q=cU7edHCl5BGnPXNc8r|oh=m_XZ07S!x{F7)uoSw?Q+N8Id#3VXcybG`9n z$=j>B%81(}H*!sq@K*+B!F8N`eiY$4hU0q2A_gEDp{;kB$fPT%63Zs#ZU_G@MK= zh7$juFdi26_lCYhI>ZdV-Nw%2PRI{*s#a5fdc?@lFAvT136Tdcnw>CVJ~mgp_6HJI z6_m^Gckcw82Lb9%sknlu?|-A*XoJGmjBBwsnv-HM?~D4m0#m}3JSK)QDvA31j5qDH znB9EWj^{E^Q0!C0yUn4&wK~^ZehGu3BAsI?8ahjcB|C{1L!t~V)`ZM7?PoXAYV$4T zm4!T^4_Es?tSOZ**LIC@wIEOGwnj@+(+VH)}){?AIypT@EddVE12W?Mpg4#b-Gk>B}uR-Z|g?r?Q~hayib_l zyX5YUkMD?WUhYY|%k{l&eCIwZ(rSblyzTB5e}F^;+zPoZlA)lkJ!CUzUk-@&7{0=#~E{lbZ%^_N}A)%j2LR z*t8-IEpU2KYy@W#L+*Fyd36OQugV8VY0J*_rg~4F}k{Z3zo2MT;NzFz38?y z^T|H#YZOgY8@jT!Js5cJ(J-*IJCIdj}ohXA$ zu7vIG7&TmOwCq`^xf&0s!Kbr5H%{)KUet@M4|n$K!&BjSKZk=&vF~JD*U%WWIhwyM zZ_+sQar?9#pyMuaoyytlHT6qvt-Bl(n;H;)Z;+nm!x6CcQ;WnP#`EEb{`NI8ysFP0 z0xq{my_Sv*w*MK;>+KmAnY2P7-f!INo+M|qXhfUiD;Wrq&U4Z8GnhjNtEJI;?Kpy3W*Aw$tLPw89^-P> zuZBT}^Q^4#5Lg3FX7%CT$mWf)KSDsCva685N+P&TSM{6v4p%G10IVnpxKadbrqD>W z0L8KRRPc&k;|*8l1EI#RzdxZV!`k~wBwH_AtTn;a3JkuNI?9m%C zaF<|2T5W`vQis0F)-Bn=LfkvAp6$rA@cf#_SU~V%OX;o~Vc*Q_j*Y(}H3XU!emaf8 zANU1m8?L_t?;p~t=~%X>%6pGs;&^HbS=tZKw_I=U0Ux=y9^Bkv(knlGw2NOO(&2uC zj>jmhru3Ef$DGmIc6@cC{xUsn7ugoW%VhNSoM(OUH}x0BvYkwtDO*^pe%T^%X%X5* z9*R?ZVbh+FRCw>RM)F{&P87J-T$9WNZO#Xzm<42A-DltV*700tSLu=jIb=47^Gf9t z3BN|IDKPMfpbulUZr#+Ih)nmQTLt1Fs?kwN5o5m(mq{q#{#_$y4FtUF1C(@SG8x&~=Di2X(`0&a@Lt=1!47qXZp@nzz zfhplg2x(3YC->DIkaTrO3LTHksqTCJtJm+^3#ln$OIThrB4C6csZ*AAq8uB5G`!7l ziu*M;`XUBuA>gPOAjbK;5Po|^{CvsTKP|wGbPkJsmb{9`woVOyRM0}zCI!a-Lr&hm zS|jGJR>EZ?v6JXzi4IG(hLOP}`n59_kQ?nJD?Roaiznlvmq~?i#K6_~I4ta$wbQfE za)bGnaZh>=vl>WYMoGSE7rY%f$f>ZqN^qDLuIP9+CUKTR* z82@cU%F5{F&^O-hJ@<1dK((cV;(q`@K)=7##I@oT@kQLwxIb7u`R}-rk{)d1F?jFj zXdS+-p;2o!r3$ae_%Yrf<12Prv-yQec`o64I&eSTWJeLNPim;&}AzJHT-~?MxSJ0*S6z~@3$mDc3+3Q<`Z$a+D8(!Jy|yo zyhrB|i9-?=r=&)3>xUnk)HMgO68(<>EPY#h@oJYooOb-Ryw9{Ayl8XKc%#id_!a*6 zDELA*=4MW>4(GPx+UM)nky{oUy}6*F3``~8mKag(+yQ$sT~-_ z%rw1e(n^C^`)`edCiLQg=W`-@FLz?ai!qu#Z!?kq?d!SYW;6cz1O_^{m{g)i z#t0DKT>hRThK_|Fp0-1Il%U|#J#U1#+}KP`re5OteBtcM4VA>3|GpF*tn4CghwVsJ z*i}uKP6Y!Wvua{WtKsbVR^`Nw-UO*evl?PU_xgP$#q|UQljx{sLOqSSVj`=FaEjh= zKYCIvao*osKgYg~aDH5Q(X_aZn5sG|zhz+^5i+50;hE3%#2d+yf@=x2gmvx>C8I6B ziTdD--YoTM(f@y5MO>U7dwxk^9WikEx_w_pI>KwJq$Jwp ze5U56ofJAZsaRHikrcX672vLLO$t#qByG8r$d`PRM6@oQEs5?s_UEm>DS?)^jqbdY zERM2M+tOdfuuu^>j;_pQA&S0b@3BPlk0p){#96Fh?hr?m%t&n%M_FWBqhl=88{YT& zzb~Uiiv6W~j{Uua1TC=oh*CvrK(`UP$J|D&HntF=FLn^LU&(4G7>A#Rt7L;40a!(v3yh(es{~{Buez$bio7XJVJg&rRmkU$mUkL`}f145)87~1+(sd0T zE|Sn$GABSeNfM~3Qm|7B=sx$6hFV+7S;<2A{e!!UL*+qCr|zwsjS_UsBAu+X3V2r9 zk<`P6JG&F>b0=}(<{FX_;!-Xj4qQ zDr}_;T9qp7|ER=2_*xZcxwTFW%*9^W2fk5<@SAR%51rEhiWa2YHDMf8&M8{p)!C%Y zir0bNCTpMbX6gbZcT6tpK%Tm>w*+4YD3|tTmo|h@g@W7I!m7MK7K%ABN zpZ}_XLX~CQg(Ov|<&uUvjSKgWoEp+8;(*-(8}l2X9N6--*gt(f7ij!`=fdN>r~WU` zaN#65J#X#hf_~cqA?F1LPLMSHDoq7WI2`wjnWqA+r0trPtO67)&AMeki++$4$jL7% zZ*m^Mcce!j#_NpX?^)NfO`DrBsnXYeS=Wub2T%3?l4--e0aS7@hN&3g$f>S>KBfoH zTS_tG2p%on*=T&X8%s|*w;)Y8f}{Se^(pXYZjeL*~J z0l|fAF zSd)HiD~NAc!|lT~o{hWlP}oFL5_RIGFLwl##kAuT2wVJcaw~rPYF&1OZUer)*=qEG z$sb&iVex6v+FCp)v+iMHVGTAMT{-K-wJLmd_WD3pRux{XwA}cpauxoPegB0=;8(oh z=bZLApFd*1*e7Kb>F=7iIX<-+48mCO>h; z+z+0=Iw~>U2eP#|@GiH|msyXu50pjsylTJ%GopW>tr17$ea~2v-Gq;n-irykR)-6> zc^7RE{Kg>$C#t3um1B*#=M^Iz-*DZIT4;Z#E+B!mqS*x(UL^z{?$vZ z0!E2c>)vI%Ka~@!CskCzlsT%$|9?)zLmw(hLp!XAuAO_SpT*Wj%6!tHtkXtG z*4E&ftc^NSlX?P7br7ZY=8Ngj?EJJ>3(<46n}g<30o_zhq&D=nwUDna%Fn7*ky5}q z*;HRH67$Vo#n0lRZHs1abp60ZSa@J8qn3lBPN(?{g|U%_I8|X~L`>NBC+C?WqGO^^ zd9-A?(Y-$@a_Eappy9@Tc|`lhO>(G0k8FHz8Fb~Q%%xVAG*XS7*(5$r8l{jfV;)Z$ z#gg{G*FXx<@VqD`+V5p)RMVJ{E#Dy}%0t&kq2gj9x1dN01@9=^{4`D)DV@%<|lj?5K!BNcs+IS3;t`P)6a$mR<^%Q$Z2Ao4-A{u7X^K$1l2Ep@Jsb z{w?s#R6*n0oZP2xRY4S8=83By+6QrzQ6}3$ah<=b9z!#N1O>e9r;}x z@K^lTe%+q>e>~EGNp(Ovsa{OyevC1!O%F&ha3g7P@l7$f_;Z%#r(U+1Xx&hAQ4TZB9F~r{4Yu4egY9%sYzA~L=xGLNw=%vh zS2Kf9I&d4VChCG^Ijup zCdWY31|xW3bJm=9UjQS$+p1FT8pEHJ7t6+-5I}7I+;`zBd`L_%zIq`>AD)lTT5#Zx z0l4n+yFOmW5dO%vZB@}TfXK_WU!0BgVNAwqVd)AzNK#*+mm01QlhfXux*}%?4%8QO z4S~+7ZX3dbu94E68ivr>W^bjPWB?Qm@GcmFf@4)zK93Kz7R^D?Ck%j6#V#Cu2=sSS zj*rrX;&VrRc;|E=&US~TL4*$2)ny85^R;2#wm0vC^Y!3Wx)|@H7#qfR-dTLslMRpj z=0^tk@kDh!uK{^%CDU^yS}?6t78 zp#AJ~130+9z9K)t5Hv{J)47rlcdN<9j5PwXOINif8^IPT(A#eeyo6Gjz#3B^r*r-K z4MK?ayt-B?ObR3hr=B;`kpjxKG=F13DiyQTi@`DX&)*9#Fd={vMuFm>x@EQQfSWi> zvM5ciOO*sUI^dTE>amlR6rtChoTyRC;I-GQ<+CjtB+Y+Uu1n#VuQL*|2ki0>_YFtH__X%Qb1){st8+X$%RT71a}w5-dGBpCwuP115=umB|4 zT;f%VK=l0y0_dwH0eH^{h*!VvAM_W19~C_53V_a^(u|;vAPw2vf4EgOf*z`E%J@L% zUD%+O77GRa=$WkFi7tADmk!}O@Qo9d1j0WP;%^6iz!g@X|KE)RFJH+*xL;D>Nu^N z8e|4^zG!9&{F(nnT3Z;4^!*h+(DR@`AEJ+jj2d@q1CJs&Lmp7H0Z_-iU)5N|GhQY@^SqI%=ssZ7SfpR=O@Y3Uh&2P>X6vUWTZ~0KmS+N%)I!@$ zL7pO*vt~eFGrTni%4MNbX3$5{hKjodP;~dx*HXkEcPxQgMoG>RigOnAJWjTN=j3(@ z3oJnW`-eX7NK2q~X^16I@>&>WA*$04Axxk){80!&Vq5l$OIX0HMTeZq{s^H|ND}xX zA*`hUvRw$fse$*y9O&zTNFj`_A^%WqA>z9r3)mTc)xvF9D7vq3viO&uSO7(T4&?h# z*M}#yErDcbcBH-qI8yd~rG+SO{b~sm%quQg{##Ed!daaeWCT@&9NlaVF(c#ytA!#y zejpU>RgbJPs> z8|NKLe`p4j{gawxE~;0F8C-pyt8V<+3_@&iX2Z^ac;Bv6Vug?@^x$ z|MjgUq`klS@yi1%SV95lxs`~YR@%UPDhLd)fgE{qrCGM3{t?)T`fAJ;oJkchfNkMs z2W2d6fl{&QYpsEL-Rt))MZBpd{Fi^4iu8A`xyWy~3enWT7Sa(4Mfa8m%|-XK^MxW^ z(kDd!<`|;al@ubnuPe;a0t!fACb~DiWQ>L;1b;Z3ZHf+_CpVH{f;_6jxtS&=NWWyu zaw#hlMDxTGBSg>b^#VlS3-OJS3pHpnjS-E9%f^V#&78>f{5KVhZZ#G?Z?CcFKD1VV z=zZD^VjCD_I(TYn1GJf`sArLhM+vr<^0{- z`cThbHL>`oE@Z@hY76q!2LGK+Z+fEj;ML(tsz;;rVb;_JKXGRx*g6#d>%(;;cq-B; zCM)+#Xun|O{@A_U!qK-g>mOYFC|sSKt7mxhzK|xZyGC73bIK@4lyVXw7#av1O~g<0;eh-C|Z=JgD#B4_J~ zioKqiD3@R`KY6ka!r*R$(L6o;T(-VU^|wBTZAjE~7c;=Pp(8ia5)JVAMSxb#uXtt8gt8xQDpa;L04syKkJ*IgNh?h zu*U@BZalM$XPaU}g2{aZF5HobhKkw8x!wj$AKjwSU-VCo^-Y(U3(F{#K zY&iUAt~s(>CbBM_HAm5)jS3+)7C3p9TT@7x1wK%|bLH}O3p}Y->+{Oi0%v}k?B(8W zfftgpMrXp1S1CGOF0 zY%%N1yN(=HJ3Kb0;?q(|JB(UgesPA-7THUQmGo`!#!s`Doz<4uGgOz{A!UZOao4Iu z-Wa05_0GC4g?gxY?#lC}PdONPWp?+Yi=yy;^MyVpN|Zo`?a9I!KSlHfsqNvtuG4`|inYl@du%mO7lOwC|^fd3*1k+r^^r z(&m-In<#1UtXnM-eoz5QN0#old8`V`r$4j4d{l>NG4I$=uvxw&*_#a~YFBNa znyy9HW2O$6nZBM7P|1Z^I{h2gXL6uontZE|8yg-@B{nKx!M@@~&a3-u_}t>iK`k!i z{Kv1Ao6i9?C9{J|3N?Wh?|9c}0|z!7|M1*pwiet?eSUxA2n(pXxUCMM1U;y}XF+CB z^!?FBT2Ps!XT+GO1^G|3Z`Ralg7fH|zpsmEf2nC>w1NhAyY zTbY+{SkD66^;1^1AAn(-tLkq}b^l#xpw0DcnI1g>; z%4%L~S!n=Ui1dn77=rq?v7aO&jX>-_8Lh+a#;|C;#I2o+O~GKt^2J(T&EVGXxzz4j zb9m`MAW_;1NYtu!TY&$Pex;6=)_`9Q^Q`J@;7t}mShGffQl5SO`IlDUOqN@-Io$BS znH^AV2<=t3M}?K?L(!@`=C0>-V6?6-UG@(yT<7xFEQ;U) z5#`uVnQTzs9jCr%mKN+EPn>Wp2cG0EH7~ay^4LiPU@Hg8j-QgvuVI1BfZWq<#%%bW zKp;Mu4dqu?o${_>LzDXTk*IDqbP035yprL7^G1s?sna>2vw(Y6BbEc<&n3cs5%^hM z@o0NTA{$Ow9%)?_$_CX9c>h?dCM+id+Q@?S=ggy%SF#{*)QJAj3l>;X1i=ETUfyZa z>o&9L^Bb^X!LC5RnrclDe=NuxH;MyNSL5E~mukVs_V1|&4{O0?f=;#(`*9`0w3EP7 zM9hTVsVW?Zmet6#7ivMs>FoH;Gqr%|HY=`;upyQVKo%RuZ!t>^zy{8<6LkqguJj`^fNSqF$J)Glq(fdjA7VzfSK z15w2q@x9tWu*YCKQwOZRuRCY{OPj{sE*+rxRRzMdO&tLi-%kq-)w!} zZ=4QX+?ZBdQK0)Gj>GHEyWtv6+4HMPfxB?MKqaL=k1_| z8QDi72eZ}i;Xc-M-61t3<%DJqlkQJ^b^K~k@VnrOI;zR3B+002pvC*9IWZG8Fr}&Q zwf$BNWRJMNuNGyYh%2$eQ5Mc7UnZJ35Vx@_!B7*I$={vU*~r53u2HP-t2I$P=Xwoh?Y_ATRjI?wDyD6olvM?@;F;tM1anj&p3Z9e+a3FN2N6+twage2k5~=WoRB zy}-t|7YHEiWn&I;>}6?e6@3g2t!vV>=(q~Ckd!Z_23+*4Dvv$mrj6so>gvut=i*gUqQRSJ z(|O9)K^t^;U2d$4X=^m~c`3R`%8OH-x)|=WX+o8!9tNtgqrRKy;gfQ&`yuyrvF-iq zP`e%4IEpCWxY-Ghye0?!ks?xM&Ql~h0nXp|LgOU(L61k5IPFbAshFS?D0>X1G)a zh^!eePnU*CqySta4HRy)WxywKQCG$m8L%gXpS>*XsW?-lJuCxMd+(EmW3Qv?6fTSX%K)qr5fvHQ0Wy5~xuycmNCYp*YS zb@8qS%w|5YE<2$K6#g%>;0V!O|MO#k2n7^%2*|p7CX1yC{?69PPgp82@&4$WXUi4A zKC3l1>V`bj>*hWhr(Mh2zAtsSwq>|#^9tRix7|bqQN_N=Pg`^Z^QNVW@5tcr!$e)i zu74vZ=&x1W)a{}yVCj54Q&wZh_npfQ*X8Q+ZEjvGW5uWm&OZxWc{9k6uMj_Oc=(GR zU$E`7`BEDbzR+Z4;9YBTLD1Ka%aoG_s?2)}dN1rX;kWRnT-Z=)DTo@?%VB#<^6xfj z{$8PE#Sa*fj$E#x$&dcxblA4qNHAyDtAZU_`uxTVS%EdmqJp%hk(60!l7bD2m&KbK z$M6?!{J7)lFFk?n>MvIw$^PV7CjQqNbC@Zp8FTc+=zGe1qYp|;CRUmXE~!cO=DZxu z4^5YQkeF7;)0mUiBX>?saIjD=Q{*v&zv!vlokbqvf*Q4=y&LM)`KbmIA7++zRqY!d znt1vSOVDxTi+X*zwZPK<*weCdwxI1*Zeog`t)Ta5L zf$!Tx4Ef`Ks-5g)DGL}?Gt0P99Kmpdz3IlG0p8=cyF0c9mT3uFr5i7bd4XWu+l&KdJ+=bowZ&8BYnbxe zg@p{pCYd{)}7dYN=wPd@VVp^<(x) z{QVv8&*y?ap9{R~zU8R}YyHI|e|GXNV@OiudyYwPcTF1RHJtBSwl_|S|L2^%o%_|F zG4}NHM7xaG0xl9aUfWg}3vhb;j$$DjtnD$$$0n=XX^jCv;YCn=jAbR#LmEr0pFqR?79NS)G*N z?>YaTE4tvpixM2H62?#Hzv-qTn3I&((rwwt`+Hvcxw?F_81~EQIfnn@m$N?lMNyl+ zAg8~hC8$^SU!2i^;vd{-f7xq|e+bN!lY z7}Ppfi9heyv&gd-{^p~L_h%nGe%8B%)qc|a{w=({#ckhtr;^u^suDUKz>-uV0E_oqRx zJu&FVj)-LiK<7op9W_V;MU9U!dY<2BrnwjeP zYI`6lU+xj{rF~vPs# z0x=vB3PcIAb`Tp-RfV8qKR8!?)L}6J)8n=*Xw%#!Xw+tbL9_CyI(bcSP5Iq;B9;s0 zq%qc~3tASp`wu4RLs;K@!&^AZ}_Nx)Zl3#Q+0TJ?b)C5Eu#bmSAngWsK z9Uq>W!MI7AoL{t>1EuqaE$DOZwFJt}+G7dD#Q3*G)C$Nk*!Q9}kZ_q}WJ`Zv(GDnF z-z46BFa8+XI2uS>Fv;B>R0{>ySsD&t^VUdeeX9ddxXvE~r;Q0kJ~RfhGczVX<2l0A z1!QGA0#T(Y%)^dA`(92^TQTWpz7htd075Ju-UBn&=!6f|A)3#0Sz9!BFxCyd5pXDFN^XkrG20_8LB z2nA|>jtd3KS9J^piYG-v>H57C0;D~^Vp<4Ly!s=U#_9QBI8VUpb$T$IN)HP>6dDZV zIPyp>7^KKRH3R|0E6G7HhuoNA5NxX?7Hl35_1CJdK0s z@j&?lF2Qu3B}0KgLrEEgK~DVUnNt(OfpzJLNaNTDp!{M+BvAakGLp{gP$bwBbh(f< zf%Z4#C%`tM?JYVR3AC?Iypsi45kdPW--vvXKJm0ro zy1rs~V5v!TC}(#%KZ*dP+vj-+fTU}SN>7+e5`mRBkYNpf;R9g=?d6U21JX@TPW1<> zza$0#<*%y+!X)w_n*w14S%4D)fk07ft!f~Uu!XAuu;poEGvk>*^hLZcJb2R|Jkwk5 z>@V>LGHgnV{b^oh`qMt%eLuQxGyQ<_#qIoPUds2SpQq&u;<8z~GcWqkKG+l=p!&$L zHxx$`9cGI+?KAaz0d-H1;{`v*5`-4)1+4^~FS_IjMJfQlH6%Q7WMB3s;-vjlz0d?Oo;6~s3+;XFN?zS5|Pwp_W zD|Q18XFNR*^k1p$7SS$`erFq=&b`aPll9_dSSyVx zAC7K3utEV-l2*Q0I4F;@!VYSr+um27B4 z2@@BQMsA!2PMF&ZmbI!#+KmNKDmXu9`eJ8$CU#68I&!0miIJ`oN7`)Fa1l9>t1xlZ zLx}&K%0%94=NZ=1)G;T0A<@}cSYopIiqdoqB(F#2omE5aCH4*Ll$c1`t0NWan7=Q~ z>QXifCAeNW|M5BKG=EBV{d6uGt@4>m$YM-wbHomu`gC`sB@&2~zbu&6M`pVl7aQykI+QP3!7>TO{c>qRtkJ+5<+7 z#97k6&o{#q_N^u3#u?!IJNvKZ*6L#*d0?5N9m+|3&nlD9_%idejS6sUnjUcFQ!7x+j>yLbngIFN+E_uw~Et;Etmj znEZ_&vxLjSPss!jxtbU@d&BtAPg!`#kEoiHp$TUf~r#3Fv{U&;w&j#(DGqTX1+A0MC_KE`9KPj(!L%r7L}pzgZIng zkXlS>v8XJrJXRTONYv+XIC6LI2N~RBFuJrwfq`EyG9E4G$YTgqKn%=l=nQ4HGjP*A z(y(XH&oN?PLX5NW9$z^uBH*BrE{o)K1ZS!ot|!K&qJ6ShhP$4ZHOitId4XIgi=?fl zxl;in$9UGQMh3QplNTd$Na-|jDRg0^zJAv!gG+?oVMljL&~@Dwu80JBu2`KI z5nh;IW}`Z}Tj;dV&cn%}UPxUd>2wLHepoJr69$*@X6wnI-vNcu_wI_|{{R30|NrcH z=U-D<)U~46u!9IH(wj&R2_+YH5?W}vp*q$<2SMz;S7cN~#ezEah6Ou)9Z2{1xU8!%f47CAn0y&O_-ws`X5)f)^k$8+2N8|M?w{_ z`HtkBca8!E)6X5UZnxfl`PnO>Q)bx9rk4~Ttc}~ad$aG= zpv#D@EtPO_bI{?BW=beJ+1W0plM)t>AKkT0FC|PeuyL4?uY|X+(8oGY3EH7+Ca#;K zgatRsDh<aEg3gpX|5K!Q$5&BuemPA0?<=ep>!0Qh@6tn`X1)6mWI$?aI{U3V1s$ zz^ng#1+<_Wys3bdmSm-G3Lsfcnor`78c257n(W)`{PO_068eXqYc7N-~*LJ%1R|Xyojsp8^`(^MV1Z zBRXXsmq1h6NReL9W;8)s2QNr19Jky=B7uCe?fYLnVaH?=pp^vPc?X4zag;#b`gc3d z8c4um(A(b1E1uBp)e~u#3Qst<`ozjCR|z~eAPBHo0%If zLj2z2|~%tdHgnd$H*T?5N~$x4C<(dzZH?EFQ@i2}n! zl?d*1phSp#UQ86>oA0PA^Ib%kN&g`cA?47^u?6J!tR9YQfR_Emc^XKu8a8RlA`NW3 zoO|{5VhyaAm)`s49u4TC$z*CQLcw z)6+u)vijOMvL805BSqd%Y7r)pvOhLV1Ukm1k~mr2uM)xfLEVhhX(BkZGbq@u6@jL0 z9EtzMJ@R1kyEwNVy9`4`Xt&ODOTK{!0S8ot*9mb8=<#d#nh-w&V5IOcH^PF_T1*&(k6iO2TCe6wgH1cX;5; zd$&cPs39XfJ{sKcGgi~+l(NOUnH0j zI#X^R1)V8jQfZ*dj-13W4a83;E0}A5!p2e(|EK}YubBQJg5v$4OZ~cvaL;_@<#4lW z@4EPeIx-CtbWtZ5LCOzHqMGySZ>*bE{&UJ-)AV&|ZI3poDDx6V%>8fiiO&`~ zx9P8sUvBwH?lff0^S&F5bXUAKZTC3eUa)!AZcMLZ_PQg6`nB1m9)dWh)x{yxa^%Lz z(`s7f8S2*Nw3^rGs(~=_MyryM0ZnvQ40kL~yVpsWYSrt~ygk*0tLlIB+F#OHSGDnV zr+fRHbxt9hZ|vP?s;j)aa>$fmH(l+G*cn@Y{E#=>Xq)je<-UB5iFNaHxt795gQiJs zzc>gk#ksP>8Me9|Gv{1OFVfRRzT0}nx^YioeQmQrU#hwb-M3D($vb2r^swDoTwl~( z*z;7n_fWQha5oX(*}gE-HN+<@DAfg4+_n)$tEVIg!Av`ag zwaYNMs&JNX&Pk)oCc5x3Ka|}pjdkCemmStqf-d|~`acdeZaSl)(S3#)+Y2vR&)syr ze}M1^TUKZS{DkBt34S+Ay6Wc4+djS8!$^0yNu^P!Lo;D+(2GVb@|pZNb_vR_kOarw6R84tb|=A4&gok%m)^&a57sPs}tVc?lghEw$#3td~ao9(r#>960( z8}92G?(3B9Z`T<{b^TS3!r)0+wlJ!#pzuoZ_Z?L)f0$oAZh2xep&?HBlGUAV8Z4`m zH?FoZjWDwi`1&v9IcJW)U6EK@_^a-{%X@DOGpi{4wO?LepX3eq<+gpd%&0GEr+b_7 zA^u*@Yx$C0&gu7a3gx`s^ndq9!|w`zZtp%lb69I3bN98scUL}^4`}9l!r@jk-LE)* z#g(w_;+@_2t*|&LuI^vI4!Yz~>z;gY`Xc`oFJISOaOi7V`e(NF@BC%YPPnt9(o&aQ zSM?;aw&m|Uzfi|?d$PphcV6^Tj@W*1a`=574f`+T`Dm{jy8e$t8Ou91?CZUJ+Jp3a zLhp{cdv%M8lH-2LFU>UTm^AH7;iNqVN0%kq>9XPz;;lPW6*k=eou6O(ot%>Oa;3&d z_}V11I&t2&!e7t7`Ebvzv};Dc9i7%Ln@lI{nUD)VVg<&WM+?S!G5Z^j-h_;nuD@`ioLg^b4|P6am?>K4eC@2q?;&mE;GYQ4}*XW;KY{;iXV zj_IN1!MJN18H5LekqDrMb5L-I5uVJn`r~o$n?@sHBwmB6olfS znPgJ$48_(|0Fmd2M5cD!s=^F&LX`3Z+}?^HeY6KYFiw)5n`?@cW)#e|g)+UC8Rj>A zfR3-pN93H}O%4DmH}_@{9lUA5fyd`nK*y3WCkVdIAPmhQ1YlcvQQOQ=m_f!sV4WJA zDKYCF4vgmbMZi&*^Q6QglF#qZNFc0JzvH?{*nRbO{OgfXKGb8% z(63(g^-X0A=XvF^&}iw5Z@yw2u>6dVgDphn{&*P2dDyo&I6>%3_1id@K;&+PE)EJ; zX0@BLAr2-@$?2)ejDulB&L{VY<8jo*az2<9%lTJ}SV%H`(yMcy82CbD;o5W2e2%H3 z;iLJDkpoU>Au0DyLTR)>AMZR@1aC^1H)(*7-`erBqTmCeTkTgz0y&icufq{QY4nwM z;XFUz)gYAJkEOXSm6g$opaKEIZyLSIVoN(0e{(neC3{}5W;u2l$5r$S;# zC|)46cgO}cGM;7-j^heCeY@B{0^3&|+23SGBu*^a*v0jE6f(ZEUxUnVxS>J2XK5QQ z-;ctXl;EF>M5eFxBa!(iSrIsy5&}sCGGFFaIMyc-;{8WBQu}64Bn+J? zF%AyHJ~s=W8WpJVX$c|zCTjd|pY5}>3882}3C)KPyjfOZuzYd|uBdG`VDXd`KRNo)gJ0r{lm_Pv z^T7W|x{Ca*@YsUhYx-DT3;5fjx$UJuOJI1?#RIC1L&fi3e1IO?+eCm3q+G&(0`E_o z0HE!1DAxrxhh6TTqW`s+`Cl40WPSTnidpVtqf5Mz@$8Z~4&4#tx^NrM{%B6Z_>Tvu=$ zeB+NypW68G{ANp$@zm!MWc>7+1es16E<>hgg8exDB?2;d3k*aSZ=+CNH`_yz>5w2b zGC!$D7&8AaB^;SA7!-jN=33cDa-21f!YE2;KSm)zlWwmvG?;k$rc+V}5s7+`%uUN0iOBrD&mu1TK?-?vG!CQ%Fe(;VpDW|B>xTZ07kedOS8A~B zO27yb&YzPLcwcNzK&D6fCm_?YCGos24dRjMr`d5x%lL?C9IvK}hRO$UB_tQyUEznj@zTT?s*;qs=Y(X zM!$`Pzb^F)Kes;)Xt~z;#sjO5vGG9VigrXi_X}RdaX;d9Ec_s_Ub-t5k^>2%CB*_! zXaAWQ#&TVo7sGwbuoz(YT^Y^w;(};kcx4gIb@E;<&~|+3r{((ZN`vn1B|>vr*y}W) zq6=<`0-E0Ju2EcPlKIMfi5Q}%uFSttR;cCk^KvxziHF1h;|uvQaFWQk!?IXD|2D<) zd7K{0eSt=?Kw+il*ck2yw2uLXlZT_hlp3(Zqq$$Ajpp+)G#dIS{a@UZMDzLEBAV;L zvs$jVv$R0RU_q*u`%2+jVB#1Ua0w5KrjH-9cJl z__MzuA9-3}b0bB|{q|gu`_X-(fX%--Vcf^R5CqMs;Z&mlBWj$i@CA0;%8UC0z8*lT z;`Gg-?!e|wq8s-u0$hRhSqFFSzdZB=cT#TIHZsm{Qhb5oZIKr;U%ZPC$MrP=Qkb)B zrodyQU5u+0$atQ)3RzrFRmh%mD-appwhBVVJ5C02yxH0?k8_pCd6W{V{n5!yiYrWYWnGZD34Vf?Z!jjK7h@&y?QwP;yRaw3-q`@aM~6K zYnk&z=E409PYH;$G4Jt!YF-nt@7UI#&wHN=u)H{D zK|n@janUyS-XXU?7JC{;CflcdF7}=By=?o;mgrqndubV(pi`<_o6{i{$b8R7Hu!gM zLO>+8C_QImoxIcn9beQgI~!_-BR{RHDLQD1hab2|O3$02c22!=iOIua#*>G1;XHel zJLqp+Mmk&q)CM~>LBikL1bM?BosV2^G(y6CtpZDEqKZEMqeKe7 z-%Auo$AW=GffPoY-v&Nk0~)UH{P{ew7Lf7sYB?$?F<2`{*5}4@JT#Ro(AAssNu@L1 zy=$9$#I7T@zC6Cmkg68&l9cV}Omo<{yi8U!!5ns1Zu`7wWD~H=9vJded==mjh?cP} z4bhE^kKpvqST3UnTN}XUevTve&9-_19b?yOB*6Av=>g#{n@Tzyc824dijtysSi@!O znQqnVI>P{F^csOow%xv`wj1PzrXNhNu!5&j3Rrghe4Xui{kz#gz*2j4=|gJ>>%V>w z*jYku+he99EX<%xe%n%JZ32&HI^8?i;d=3#e{3749=E_yR`9M!<`no<+y&D}8O$HlItG^I zO2LEZsy|$&8U$*Ob)Shp#%QFczF+iB{B4Xfj_UM5*XjzqyWZ) zHUlsoxYeKIiAK(S(sCc}$4>U)b35D{7|wQ)!Ae34DtAjEk`j(vUi`g;vjm1yV^PnO zzX!GO1g3{oZv1_3wKMlcZJqdgJ$n~EH+H*m|Gt+8u=c2N11f{sIl2PlaZ5db@&6AJ zV0nG$#r<|s0+i(8DLN)6)@cVIf_mK$Uz5u#e?1oSn*{;jrP zcyzo>6J+^6ejoP#00030|Lj=@R})>=rAzNU2_zKhZRjB|M-fyIM12$#6(0*&z=9Pi zR_p~uioHJy7DQ38V2&aRqKLhqs5GT2ih`i{-I?(Rd_UmLT3NGZCYd|;o^$rzXJ6l` zP63K5`QZD%Ud@19$0tzi67vm=itiQsjjbJ^HN0TfBJI;vjl!LWq7hP0uI zuq02XclYv2Y1MrB09sugzr0B02Yih^yP`z=;HPvQA zEY;}wHq@>8`zwUy4vdu zhOM^k8aE;aL(L}gApODcb+qkL zcv=M3liYvC_L%`2rzQ(%&=*uRuCNCtd>UqUpnRG?VLHza?3uvk+JM84oVS^G2SUxy zwAt4^Y@xTk6@5$X;qvnEgrSY?$|n>BT)eeS z@aD;5;c!6*-<4Y5-Tzn-SFYY{;F;SkO_@=f-&);9k7FM6M*&u6I8KpUssqy}y$O~# zHii26O_2-V8$wxA{p*NA4UjoI{ciA~PWt^mo#Q1pyQK9d9$h0{HL*JF*ar1e+8DYd zUZHNW0UmlkHAo}Y6bG__+Si179ao76v2~Se2ao-WkE5C+1 z;Hiht#UjH*u4uYmfmGP3a;d>lCdPuMg+%9w`neau*=4XT| zLYPx@Yp4#!G9M*vOp^WaD9=F-kH;BKsMow6`d{z(=v}Xrp)g@o?3xjp zQ2#5;b_lA&fyXy{>AR{BSZ78mNChs&pK;NhpbXlgv-?Uee4?*u!EC)aSDOLp!ktb@($jC(G$jB&>qp}QlWIl?2&%?<#fP7{DURNPsA^E-EtywoW z_lv-2HGh}u6A@gwb#3D96C#*Zzu{ufA`#T2H;K1;h+t-t*_-{1LQrKUkS~NI9r78k zr;~G?`O0JMg;0_=EWNPN9I`*&d^~m+3Spdhp79>jR1(5YrWv0b_0Vvtmzl*yrfW_u3pAI}LfMDhA zUfXg75WeDvWzbpygpz6NED?ayKrizbu>uHr?3p=loB&dHRA(<7E`WyD5pGqq00=t! zt9lFI-S5uH7byXpQXy~dDuCSHq%k|}1;Av*L2Ch=&iI-++gt!_TJQz1u&bvejGW)K zBMC^3ombOt&NUW*Y}i_jky-+nHuc+r1Z4pj&G0I>k{7@k%y@RDpAR36GfHZI^MS)% zJ}4bX*2?+C2aRnzBR}@=;kX;gly7`kQhF(9cRL>jUC!|cZso(JZ@-@|Yv#imqg2IN zP5i&>!3U$uk7R1=_@GOw+K5~)BrtrwjSr{Kyo_1$nGYGUj(e&)_#mIv`PrQuKje^! zb@I9QCF6Db=_~d&@S&|&@!G#NGLle?br!Trt3hgzk4*s3;Q@Ok9JPp#(51>`)R zgfX3*A60tdnsS>Dv`k*uzI}XnEVx~zznl-1+IO!OC-8wpefAF^^BeL0`CC383S!Rj zF17POwpLDMlavR7L^8rW9@HGUue5tQ53cP!A#PCOL4P1YN*M~+yw;##jmwDtR1GQk zPc@i8GX>oZ{~mtcUM(|^qG0BZ_7vKa;`qp$g1uu8o;u@0ftrwyDjynV15iiB=S231)75> zj^9ElxILJ>;V24LUM2z~k^;++>nV`s zzgvkQc{&B;q^pLU4^1SQLXIq7Z=`_5fAT*Mri@qrjce-PbtU8dTl&OdMiK=KY&DVh zQyTrEB>QHEWe&vcx4zp>+-{9*zO1&wo~;G`7uH&0tXtvF;&m37wa33&b%X`7b*XHD zC4KcJv&%%7ZfzFfl`g`t!{1D=4H4m9-V?oSWf3MbKe#8v=oHnJom+(HbK}SToe@G@ zRk5&1E(6{g~g%OylJX|3BSL`7+JhmHOBQv;9&VeBQ$sR?by{}h;hZT2hC#*aq^MnbEZ@paQrPX zz=m$G->(Yv@l{p#n0jk{e5>yL{@YPKth}NhnI+J}|EiO=PCBBCu8gp<(#0-nQ)70v z4tg>IK~4v4hrbwcBS{;J%*_7qv*ozishtPVJE@H+||%=JiXxP6b3pyD1{aCyJOn zXUZuBH6_%fD$2)4$)O#c_oKPzw{+>Zf7ZX*@=Z#jpMU=IJE?AWRByH7O*&ORlNa>- zK3!N7m$m!x9lEM_*R=D+QaU#7k)Of*hji@qGhMIaU(*h~f$k&EzM(I)7kOG3*U&a4 z^HSP(*V2n}U0m=+Bi*TeXJmbC3thWy{)8_UEp)u7e2|KJ2kpiRvhQ@7*vvh`lX>1@=wKX(9-u8gFbD%d)9t~@_W9kNm*75+qP!M=zMD+6tH zps%u{qI9(m*j3wnwW`noi&N45Hv4s9W3prFCks7b>)cikItsfEtL5u~U3T;Yjb?q= z&{gwd)e{2|M?T5eKy-VlEE zZBZMr+5qCSrY!KepbzrV)=~T_eZXiTfgA%aA812hb$6=~G(XLH_;sHV5Vm+3W5|OC zmMdJpyb*x>!;wToeUXOdI5^+5q^^+6`5m!5&5ZNvrJ2rgp&?9%?eq*iL%Dw&VNL|4F}@ospP_|fq2N2bjs^& zkg&C}f}t%YFpba=TV(llq8&b4MgXwf4(C^GcwCEi9PY~5bN#f)9$Eaav_sORI{#T^ zhb;eu+o8sS)0=CQ?U2(cw#f3zaa+`5h&{^|r~ADdxi!)j6&L~V+Xl^;2fVXEmj5o> zAk%)1rEQSq;kJP&GjEkb+4g~WIF113cWeBTUh&V(Vb-XBcIb!YyR5KL^>d<5fh86< zKTJC|(*o~}Bnwb0M4~cN{{3o(jW*8$1TK7J+BadEF*djnCFf#*te%LL!@9zQWuvVw z(hSU{7yhCtAz_B2O7Lb{bD+^vbtr$l-uYyKCY1SS2P~4+0rq_g^kBNBc0sGR5vK=r zc+h=+r;O)!b5QwtDDJ{l5im6TYMUk0BpB^!{l^m28A19}1f$bNw?F9=fI`OW+ZB@x zfOKgaCpYOrN6h$Hqwi^O^;csIWBV3KVpp0#1_N_{On{xQhbd&t2`uw@X97F3@73s3 znZg@qj~BCsXmfq|;(prER{fsGo+|p$JcaP+@*-*Q;OMI3#huc|y$N$ijO>zT=si749Ay^D#&z^vGE%Ca@E4%T|YFC)4i&A`rZwP)hkqR(82P| zW34J2pYKt_`n$Q;va8jRxjZLU15-b(?#)ipzylX+oIP^XF`j9_)~F-fKi*Ws!z4}4 zT5BM)%=K@z|Herd3Cn3dm+Il)^}tojCP}AU*2g+Ox7M-@BecF>^xNT-As%SS*!!VL zAK!l~RsK+@hb$hp=wY_P$-Ehx4Y+>sfgzXwwMO{)!_NBfW+QZG0_ARmT_Z_@9Bahk zzP}OPZY9F~rUCA-Aw(FUkD7-=9+)ZX;ZXTyM0)CCMJ7Xh+PI=mOiDo$zZ@wLK6t5t z&+Ep>Z272xtZ&e*!i{@I5lcMJja^-)#PyL)1MtsB*O#Sq6;klEE2-b3+GMqz1$iIL@TSuv&%89s+xXWTUf1u;4>4Z$a0oWwLleZH8&20ofWQ(>f?sUO9dzS7y za?}aQXob$s_IQp=!#2SNS>I>31u}Ngxcw$L-`dYh&P^V(RisZJ@;*ul3JyxG)B-kN zw+VrrUyLm*VgmHX8Q6GzwShnPLNZ53@L(N*>yuVrXk|vAgleNjUi!`oyAJxvtBUuw zNj>!V!m}7W>>|zbv+E~%A$MleP0a!1o}6Ue*ip>Dpw7#O>?&CCQ=1)dyzA29_mb-oWMzNp;cKGSG8XIN_!02|XPtDgk+j_&^R&<~ zEY4<4y(->y9z~#H09qb)`Z#*u0F-A0b($Jx&bxHQtwalVKbhfHn52WOAG=r&TU0_N zha3&KcvTtThO3N0W{7UP^xEy$8X>zLDkk{-M(4#iLlZ1?Pp_JD-vl!w3DTvSqPUqU6F?1YT5xdqDHai95#7UAa7-7ybtn&GH!*>#{pa%RsXy7bD9SrTOZbe<0vUx^3GU02lA z|Llfr`8l{DQ$7p!x*%f{FKcx{78ey=k@ee)TsWTl;Eb$6KhXh6mAdW@vBe>q<3IKd zw?Rh&dr?L<$oeKp1OMU!(KUvQ!(xSk(|;D`7FeLz&xCIsLv-DOgpkIM1=S}ZdoHD~pho|n7s+c3IvyDY)G)zTuySoo%SV3;bP7g0)r)XTTy-@)$P&}GT$!UX-U_n= znFX*yMva~%I?jdx(AhS~?vD(xMb@VZw#AmL^Xd!k+o2yLJSRIKYugTYKo(zU2V6Cf z{PBeYQbZP+KXpXbC-HT~zkUkttXwfn8s^U7SBX0wA#nS9x(Bj4YNRL1>`3bqyLut( zXWaKj-{)ls9_Y>OFXeh8+n45hBhy}|RC^=SW_^9{&FSu5Z;nrDJh5jC0r;ox$fSXq zzB}fx9yNUbdlxijjQ?9+aGjv^@>qASPc3u#riBGxSz^;S74zCcud&Rj~&`BGJ4DA+Y) z!?Z9zSbN|>{=+gqAk`-uHblbtiKP-)H{r?kdJlhKmwAgnu=OkT=l0L;`hx{2J6?%D zFxwB+=l!|oAMu9=enjYz-!t1IqBMVQA1T=%Se{$p4{U!j>hE=)Kd`>c8ws#{yj}t$ znHLL`0AY7RF%sbTSHk(AQ4-ECOqT#F2m2+!EZ6A~{=kG|j)?>qy8qPd$L+W6lE8W* z3#hdcVC(*#g!9LSN;sZ$@#W%I?G7wYSGd9>3nE19ZGp|xr{#ZevgRH`H<+w z`IXoFfvBl3=^uu0>tz+f#iz=d^V`ZDIQ`UOhk}Ym`WS7E3@zr}5FulSYb5F++m8pS z17lN;${h$f;{xljhjz1Q>4GhWZV_wzaDyvI2o zy`s}+wzdlK(J2F`&#~61vMPP|r_uJ}Q!?`n@%k$lVWe0Wlb7_&nBJ|41{f$DvPuPKdfj@q-fsXNPY4^9w^0T= zekJ_5tgVRW)=l*oQml&0FX`2WFVMi{YZh*L+^UH|VylPY5414+cS`%C$vSxPb;RD; zX1Yk9A~!2q4}WTw1WMmGK=JWur>J39+gT%9{B zO0n4#&(-~0w6fg{boMs(W%#>l2S$J+4*?UWeIZ}e^GIy#bI-PpKs0i>+{U9 zKO?gBMxr_XL@&|UKy&=nKtC)s$2s@LC`%QZV{=@=)EsLI>^SoIVEhFO)V=dYqGG5e zmZf(GEswLtxTx2e|4p+)t=GdpHm{}e_g?vGho%$%UgeB;uS?hN>2$%7x7+%PFS+8( zOCB$M{M_*Qlc86a%yC7VLiPN6o=%uNUVG2j%eGj({p8Eb2^MHoHf?lQ+Ca?N{V2wL zyAB?C{z+rP?pksn^G2%iUv-$~Y?~%p zznH+=?{yWHx01v^mk%p2-DGIss|$DQRd9>9iQA-XZS=W(Ojz%yg*)PAy?gLY4=yYYu9r&j2-ac`qHXM}j$Xe!a z1Z9R%^IhXip>|5pw2qf%K;{iz7MLr792(xpbLKE%f%;6ZwH6>*A*Ha3v4rg2`1VgT ztl;ADT;Yw?)-d0d?ySHXhR&;bZPsH2%+vC$*m6tY>hXUTz{S}sc%21&XU>0lq-p^Zwswd{#ERg}57h-BiXvz^k^FnXT@ly^ z>sQU3VGdj#gUw-@VEAXbN)hM=4|--U62Z;0vhSRu%pjpR2iNqQ!Vbqv;YR~Zq3L^? zYW7Yc>=?O0-A5pVo69FJ-ag(0>|3^F*bFuV`j8VJ^z^~6vH8Oye?6GtSJ+mmstbA< z_e$1G(SoeX5p#!hYrv+J8{V(zQG+RWP2w6RE5cNXJt40r4S+pugU_f-N`lGPTa_ce zbTQExHZwNrwlk3uL&7U_Gl*F6!?~5>4q_EnJj~s@hrCu3#f~!UA+=B2V=YX2$@RH6 z3~xJhlC)EUwbO326Ve)Er*?&%b_GgmC}7_N>F^`7rsudRuB>AJ(bG}Q2(7RwM4_yU4^d&Wy( zNJ;HdyXg+d-sMn@6aI9%Z>6@oi)dXN^>tIVGs@KNv2NV#iEg5(YF6d3mZF|2nvgH$%=!R-3Zf9@R zx+7)7Ej{(_IN{CypuhAwZoP#LJb%jD;hb^LK3AONw^kF% zPh@>^JfsDMkL+UluWCS^^z(~B(Hfw*twq^jlNPKwF>;n|kv2^0S?jwPbs$dOGuAd( z2$__YCfQlQfuO&`y02Qot{Y$L4!c@I$nYiM%>%4Js<=2KYo8e`c=SM`C)pIV*|eC8 zAY-9O{rYh;ST}H6vE@D?^gkbaaLP&@_`aXf+xSZj%%-N5N;hhN>3YhzD$>Bx&)vO$ zNt%C3c~jy54Dhe;t_WSpb-JKYOfjePxQq1%G#ZF`EtKV7weNdCZAz?s%oBWcttWL1@<3v99ug7hgB) zy>MK;jJw8kZ&an_6a35zUu+#sHqZ9J`(WmbTZmy8YvA4?K31zHI#Pm^>A< z0x?oG=nZ(9w;}LGVOR*OgldT1UU}> zuCqclrL2n)R1d-5A+SM7h~*j+l;+&h!;|%i$G=)?;3I!ZBzY}lx54-)&HG!JMeGLS zdl@Oac%9N$1E9V0q5sxFs_?j*iSNqL1W$_McxN5BRTH%9O1(a8daS8tecTwPkM0_i zHCG6Y_su%LY%&G+xL096&zeE<3eD}PXbvR`p=QfGEx^HaVEeN77Ep08K5kZkCC_g= zt>A>lg@-x1Ho(>QL|a(AW8v)b#dg5LO>2TZT$=IRHFvTD*cTuBZk+E3SNvuK-c)l2 zPB%9@!|9#0kS;g_MT^V#InKc8;yNeb;(o#ro-TLac1P$4PZnLCH}!x61k<-E@eUxL zZZvDreFqqJbxFml=MKQipwvzWpw&>jw9p6igS`i7I0_NgiT+biJ;=Y`;9|IW-#Kuq7Ab|0Buv_%a!j` zG93RKYVkM}P~HAPH}KVg&#$^WCInDtli-O_tY9ql!JFjgN#8ww$fkYpB*3k#0jdN( zucrVPvOgf8JZmU?0G}n_PfC;noCwqj3}g8EEfgcWy3OM>gb-Qw^?Jc9Lr`W5Dcb-BxN8jZ{cHrYzs`27 z3N!?n{=kBLjk@r9J_WjuJ{-!K+*%)`2WNjtCzcJ@2KxtvBckF3us0=6+_+KzQ(Ehd zq+12BB<+Fm*LDFMDit2wQ!aqDHrH%>9JN4xRqR{3zv(3Mo&LYpTFh&4&`xryq=!dgVRrOLD@rA^5S=@3zHSaNq|^yjpM-6HBgc z(Sqy?C5d-Kw4uI;U7!veFBtr(Tu&Rwow}ke)td01`-v0Esv59NrfOZ^V^z?kux_8G z44FeL^}Pxupq-Ulxeg{hfAf`fRmEgFSC~CSUYND>V59;v5|kN(l<`dL(c4=`sN?ma z&nI5qtd3!$%NN!y6rkhu;&}!mrBSKmW!{&LVn!-B;9Bzv17IlZ$O{Xe_f-9!(MS*5+r7pbs5<67pfgv;%P$M`-3j!N<$KDRo2TnJqn)B-mG=)U1{qwc^h2T@Y z_^#v^Ghpd6MZ*#%haKCHVrIqTd!Q4K|G%z~G>j&6w+C>#cViH2qO^J7xF>M@>FUMT z?JqCjboi<_&l9>naLSi{O7aEkEqk)V{u2Y|Ta(4W+O=Dw#juVa>(U>@V9q|cM+_sV%#TeI150mCMt(56e5CZoPkvDJhu$#K4=z2WhWD!&_8y^x zxlIf#{f*o%2CCvpS>MG#&D32J4?j?(G`ew<9}Mz+I3;PmADnDTeL7Cg54KU7Gwu}g z_4t-v&-&f~F>wB)*cVFYuAgYK*cYT(TVdi0=MH#F`WO2E=iftp;2`}PJKG1$_N`lG zxY-9@zM!vOZ}{(jp6~PgyO5W;pR%J3Jh;j_7H^kI=Bp9d2cu=PGrxb>quETPMj zIRUWvPxbsW?>&K&vsyQxzvd-GyF=w8)fBgkL7+?F8L-eBG_|NBr}^aa66OcLv&(k% z<@tdV`+#YFK!bk4URn&STHP4p3#X1sVnBtoJ);tsR+_43_jYLiok?5Xntm}V;gO7hnezxnkwXd zlD`mMt(5j-_LuuO}0V@WQ>L$J|@s(|HnB$eiaq$SMxE+7Th$lRU0^cXXi2$2B%5Gwci_r>od;~>4Dujo zUq)~E-49$hP$!W)jK}k~(J-BM(;u70!DDVi(&TO zP)P0Q>%B%8GkxB(`uBRcy4=P%{kjlClVmQX1)E~Rmrv~>19dR!ZuQ*&?@lstW|d^{ zI~BYhl~kMa*$`8RLY({#Q@la9W4+fLc|Ep5uMxCUNU%e{2-;y^b;P+W(HwK;`vDIZ zTuc8Z8}5Rf&XzhM8STo+FSl6Y0JZ>^TOxEj6 zP=sHPbVme8njtMNb(2OR)|7b3E}BSvCk;n`rZEPkU&?V;*2gX_3EE<7W0NP{V74X} z96TRpIadt_{4agQ?{WpC-N^z6KY3K=48IIIa64-hwTc@` z+OY502Ag#S4NUG(>acF=BTGAZYL-~rlh*Rq!IAH~j@uy%=S*8`tg7A;e|DY$a`@US z!e&a}gVQXLmA$h8W;n;O%Eu+FoeZ&aUbOB?GsE>Ev6B2eYO(+q&CD}Any-%>59L@Q z$7e0p_&1mm-B+uB^9>8sJ=WmY}W1fN3_s|GrroG zp4fVD>0E83DxMTLO9xFjLn^@FYWtr{taNZx!Id@2JM{2roV4kGsEIY1zG}Z-36PzV z^hXG=jvYT3YvHTjrK;-o0({+QvG+}x0DHeZTpGMU3%8GVU9GcTfN07Rj{tvk9C#W! zo96q!kMC3&4cx*tqG~ur@!rGpEebe_wr5T=WssZqypiDNdxc$u-S((0ZRBG#3yfyc z_tBiGno&N(+QiP!cE|Xpt-h2li*)NS!SODXHGo zi#JL@SpqdWhx-_w-})G~J?>7G1Q{AlnH2+|Nys*=@=#5~^*CG^*zuJ<(SRU&yP3XP zpf5v_xIhR|*%S9khIl zc^<2P13&z4*D5~^jE+*!Hk$K^1TPQpb+oM_Gg z2y}{o`=6>E&&%r8u!DtHrVFp*Ppl#I+Pl%?EcD=B7)6YnE3nyD&$kEN(UW@h+T4H} zn}!bNuRG@f*?Z}Y^jzTpI|eu*0@|(pwhpp`vVuzMh!PPT9rbr!WrPX5zk2du#b6zn z!#<#}kEyXL@iSJFgw<~4+ltg=U;t-`iV_>uO*w6Y6$7c=EH!e)MXYfNb>?|{pbN5gW2?Rka`fZjgq(jk+(2aeysiCY>x8mFJO8EX#eCXPE zT?oJbIw-Y8fHT+uO@a=1WG%bubfBH2?oWFDX|fz#OmXU%6P3g8>uqC757|8^jFUjc z=b{xd$rq9@o@zua?X4WE> zk=vN^2dkze|M|`ctp50D#rL`aa1Lvmo~sBspINp*eQt3y>5n|+cz^a zipDe= zHMN+jYl%Gm_vZ&@rCj}?QG;JFCnbta;Pn+^^WTJ_bxMWgP?+MP^*RN_X!o5j4Oz#S zf`k3$N1d+{!_vb^`!`jQ+uJpIWF|F{%&bSFLknBTP}&YD_52|Xd6OpGvinQ8a||m< zWMS(aErU;NE1vmgNMk^fDRt%&c<#FJ<;+_W$ooDC4F7oBIy0xAh&sPeM*2%k5AI2c zd)Px-Da}}Uc9XD}OFa_iJ)}g2Ht0JfaISLA@cX-Dkem0sk;RlQb^s`kZ@n)c6WS`_ z=X$TY&F7SH)=9YyAr302VY1xlrj81l8GIh{bB{8{(efW!qKrikpg^%adhWJI*zeyf63vc$vu z*Uk1C5_>;${inEF$u*`m`yxz?Pydp;YVY`VhQqm6FQele9k`*bli}v~%~FstjyCK+rNRDGO~lGS(y-lq;hVJ`62Qu> zs7n$e*nw}P1iYs<<@$yW!uiJAy@axfIB0brVYfa0t0Yc#VLMxCe&0Q^_&)J&{@YRV z$ja;#FIi;i+DKJ__h0rhc=Y6rTxmr~TpCX-!-{(dhquNq;ul1n;fWT)%6voKPvSU( z8B~z)gJ>k=7QV}FB1=P(%cjIPkq>AoBAp(E zCLF0IrnP-rQ=Gq(p8aMt5?_hbg8{UaDJ6kX>?HOr5iJ?qovikTq*1ua-7FwAvOgAn zTYQ!ItT^8z=|(YgM)QNqwY$a4A88Xk=qzTA|7e>$vb=`rynI-JP@L>DA$RH~wlRKY z7l&(#e=|3C4R%e4`okEoK+5T4__;+llW8+!;}NAk=0{9-lGo>cX2p2waHAz*r`#d` zY$<8j_3Xy3!K-E9_$>C|Mi!DM+uylWCJU{}(a$^Y$ieV)K08j>DDZP36(#r)ukrYt zzcN1`u~dQV{I~IrmsEjm6Q@S2L9IlNk@%4sWGb&p-|eFgR@={2mt?C0=VQ0312;#t zQHOgIL=J~Hs6!2P-81_&;GJ{S@bPsT;JvB0;^HYykh=5hd)hAn$aq=Lzqni#JRfzu zI51BSxUuAG{eN>X17J~b>aijC(p9dVV*vYnE^PmON)ICHEM4Ts3m|((Q*QbZHK_hG zsIOE_7M7mB(7HZW0%TSC1VQQjOskx9(L=dzMl0gL+Lw!Z8QQ&l>hiNd)~;?DZ-INQ z8&w|NF#k6Ia$}^rRO>JDDZFv`n9<*f>y3Ti;>XhWRabbROam{vxPM;G*xTHWW456@Lx840ohRU}7Y<2iS`19Hp!p$c#TmI#_nxqHRM2>r}b8P#5*#WsAm`iUb4ux_&9Og2SiH{pG7HyKoKYVbLyhbZ4jm@vDmoBSUD0RR6y zS$8~D?;pR$#Wk;Sxw!VY_gaONG|rP!lGW58rL>784W%uWQjyZ$6b%*8cs`{eB#Dev zM#$cK{mwb}`}*}qdgYAge4fwey&kdNZ%O8QJyLZm0aIf5stTd;7JgKrPOQvcV z2Zj9zeCauc_2sAb={_9B56y-i%Iq7&f2}3rI<^kuh{&7hn)4`@+f!ckXzmyenfJgT z`1B~=+H<-4P4EbQF=cbQPa*@ZO|>XWFck-lSO3hNyTo93w9n1mo5f*i$o^HQev3ni z1}k84i8xGusKBWS7l(P`MMbGS;_$haxc%piB)n5m5?tEG1hLx_gSSOU!jRXJgxGhI zFi>tgPkfOSNLGiA9MqA5b=04`OxXF&D`s7@6mXKpr?LM^Lv*YDj=P@H0Fr^fzNkvW zvt>OQ4aXl>HN>S#ftq)6;@j;~5URJo$W)OD ziig*VslE`0h)u7ypS#eDC$E)g@LO1orCv`u%2-v6*I&~9uBu)`Jg;=}osqyoXSzlS zcWxEY?mm55T5U5CB^~RQ;n+!ZUO14`AJ$7OF<0G}bghT*3HUIYIy6lDT3N_VStEw_ zo+&sVA{-%B2itc&$QU5%xiPPQkF*n<+QR-LC5=S0M|B8)X$x@~vyRBT=ps&ul{NkP zBL)L%j2g{nOt@RHi>xpcY_->(Jrlx$MU0c{3kqamta;Ay>>wrBa9%Po=?)tdJs!pS zA5(-l^H;e}!)#cvdW%8!0XaCngR@|tiZqn`np$j-E)IQS&H>eH#o&chn#k54Kj#ePwtnuydvh;aonUq0_V>veGc$Uy zU=|oGN*cnM{*nEWDNT52%FkPqDq66T-+I|AUjMLK?9I462dc1S^u<}PD@*Z*lkL6w zk|j9RrQzaAzi(J)AusuM=2L>+_oAOfU0hRuEvJHz-QOXvHAw!vamcF5vxQjuC@kWY zS1KHM9-62<-Z45NW-jZw;F%)${Wp1~P1lq{h zdO(uNL^)ZGDI0aAQJAHHUbh7cSzk2c#;M7n#}f4?1(phEmEM)i&Si?ITkGYlp7Tm* zbmm}76R^=3%W6pxLj^slDBR>+p@QD@=AC7Psv_5{c4cdSHDo%xfY{ooh87ZA%@!DN z(Df}joZojiNR9Yd@qCs#+I;QQC;4s;`lq_e4IR>RZo22~86U7iekMdi>-!mY{aIh3h3}j!{@y)=xNpHGNsC>g74}~#h z%L;HuH}yd!czwk6inO*8oIGjMk%m3xxGOjyk-NSU8{NF{N^D&o4n5te$M^2VZ$nH6_ROTI$H56O<=TCGmAe$C^AHF_7(!nmRXA91_wZ z{O_K z9!_ugI;sS}o*xb`P*n!!fw|8;J}HBe-1ED;KB+)Z>??;)3^mXUDN*U1p$3t*I6-2q z8uTq4EMRgt@I?KZ_$r<{q;4QLE?fiTm;Gv-EaZaG&w+iLcp6Z=v%%DQ0|zRuJpXAm zK^b)K{d2bZqY7g?A1%KaqXMa!#UX`5s*vDVZ?|KkGElI#X8{|IDZjfFU84vq(tc>L zTI3+;;&*@RCrYrGyFz9gTLFBIKgfCWfC*FAUAGzCJA$wAn8yxYY{cs053C65DkGeF ze!!BbZk)H={__}sN>UX6a+$ueF{QjVN z9362jxC;8Q7fU_TmgvL%n^R{`n>C7O(7-f`&)GE^X!&&Fnbo@U^c;GzopI}v?kWa2 zy_6La8zsPBvwEpUx+M71g&V~bJbX|Z#g#5Qwn+MoV&(fq08>D$zs(1JjA7={-+I%3c;&oo z>1mfbaVQP^9r$&Pn$G9iPVCUIw`TL#ZcG-*>~?q$cD*Y*+oY!lC!Qn=vAPf6$Z$8Z z(skPv>Osru2yA_uz&Tdvk|FCb{=*~yw)i^Qv zZc~(19j^alH57WU0S|@ktxNX!hx3!xG7|mj@XW=ocdiAs;<8wZNIG!s>@&}gE@;N7 zN~snX`s?wjy6l{Bhw8DorO(@m{lZo{Ay=P_tHM*l)VT`VV+j4-3ry{v zl@VFFI&)rkJR*KfsZdmqYa;Uh9CnnfU?NYQ2eDTheiQqqM9GX977_-_&C7Z9KM5Zi zfb)ru>IZqT<6jbrx0mCQih|fH`^xbxFNxTrT5&`r`8wXab}ZuGZagvD*l2xKH~wZw4nku;o_l3K zsTc-vk@LLT7MU@8Z~NoXud5j_`+M2v%ES>Y>SqQ7e-0*pHHbZQ)i=3?^kSElSJt2% zop>&F8@~hBg}yh?cCN=4q6}{buCKuqtWM0V#Fgv4?b&B*vC^8tKl4`CVya4I{WbWy z*0^8BQZ?AgKPw^UW-XqvzVL42)h2wbD@gqNMiPH?@9qtW_=PQ3iKn;Lm0+4jz7*hz z4QaEi>_6d@$Z@Jp&L6PF<=X>mmSddjC+#q%ok%Qy+2R;-C67?;_%V>4n@1GAn1ol~ z{6Z{EaqYje?-$WHV}@|~$8W;?FPzw=^{I6OytPV$Fg z;_YN3mpu}rM4aM}vB|DuL`1KV)7lFRbZGGWWS<{msPnPs=7e-{#Bq4^WZohPM4_3` zK@zFX`X#OZoc#GBbdpDpBs$qO=?41%6Qy6C7raY zVWOIeTChT223^kz-z7Ft23b{4v2J3rkZ3Ms(FIq|NtaEMs5tfc8)s)WdRll}ZM(KI z^0S=Igr&+Tbcc<-%XAeKPr>s%W%S@{@segGHrnv-TBRTPoaqI!^2>ZA(UdPAE`9`&cLObzO1&COHmD8K+XdEuVurgp7~2|5VZK zXrmw1Pt_15*ZiAQ&>8Y|4^}FpsOSBp(qkc!{*png<9cga%Vp5~LmRo%{iF~D*RP(5 zBfA+zdl`lOMAx}Z*G|ppATDow*L%*ohM3$oR&$tDi0cd%PM0jM!DcDggc0!%lVWrd zM_7x))*Y^1z&DO&Q2xL!;MX* z$-gFAu}k$T}u<6x8qY}nHDnouwM6;C3~$#u$#>`_15L>dDOj85Ln5jSFi&J%^i^O%kgs&U=6n{= zd=nrGH2=(10Lc`HDNa#>DepGEJu#gPw5}XugQKbDWI1gW5l>dAz-4vP=p0jpZKV9D zf5d?=OD9~plB@>$7jAuOi&h4MWuz1FQ3Ba3C8Z|!S)%Km$FSJYzoIxL35Y+@@18d} zg2^r;5n~V^=NSgNn~?hC@sgH=n0#XX5hCl7-A7`+>u-jwZv##KJO9A{2K7Q(N zDjyWzjK64B$_I)K7f0~HJW{oy=g!At?VC`FvXD|JH{mlpQ>2`rrKmNZClP zAW#UN(eZ&Q66AW9Cfa?OA_Q_sf~J|7fj1@ST+M)z3#XNZP)*)0Z!Lg|*|b82*vKQx zzYp_)*6UaKkU3Xg-24+CD&HtxwiOpZ{ZLXzVE`YfsS6r06KVt%}lcHvNJmR}FOe5$SsIufZRFR?DPd?znx)P@C}jXa4|9iZi#rVgB5 zxZOCO&>Ir{=PUT4--l@ z-o9fa2jujs<)*Mi^URV3>83GWT`mjm@+s#d3)yV}gHL@~K-q*qvAAg*}B<%^tX9z5ir z{IuGc)f{yyMHF8D!y<8LAZggqAG}Z$8 zvma`~dDA=Bf68eCe0>q5|5qDcbN)E*$^z z>G_RnU0D1!@wPid4^~pfagRQ%h%M7U=WPg4g{0w@FcRHQ{$x{w7;g+~C_u3{1!_EG z<4xggZiF));e&nuJJN{>|I2O6q5a6`MQ@!gfET{nW-;3WzP}oUzLdx(&XtDOAFbxU4wZ%-i%7P6-;>QMW1s#0sK8pI_`E09fAh7ZXaGbeYjA^SPm z=-1egspb{8w~q}8;ZMIrHz-4%Q#gN}jtW>UDO#m<$A^sI-i^ z&gH|R1^nw#p2qP0iB;gO)utkT@Dc!tT4OP9%tiS6%N#EJIv06F%l!X*msx&%#6Mji z!tpHv5alyMNc=c&Qq&EBh}ZlDu#6%w1pyQ#)al&IHbPszTChA-jYPW9-Uyjdf!YN_ zG+IO&=U@{=(X7pLQ)KXX+lDhO0@Qwwd|-naY6_tm)e_PC=4Od#yUrSi(*4N6i5ibS zk*}SB#v?knlpT-!Zv2?gc3~V^L~pPqT1yE{4Rb`%(IJL8BH63(x4s$bUr1hHEJTje z|CJ>>2oZVr@sU~~`b^TlmWKdUY#<%&Z9d9t{ha0KCP1P&6o}Rr%}3(`Z;8Ef;v@2J zrb>|jQDcu(g@`KOzL#cb{j7~!+m8rQz>>WW)5Ha6z`E|C$t)qF_0u;C6s%3|M8P;@ znR=z|-fTDM2A?f$01(+ft;H^DPnM%)TLq0!t*~Yb#_mW-!tlYK2IZ zGH2T^E3}ueTbbEliD;cMYJq%(6cShc1Cel$aLPBUzVN)Lq0 zj#W29bRQfLqPt}EA8#{9WcMG4Q?)@f&T5$5+HI8LN9=Uw? z5oMtBI~5g?9<5dZylT>x6%#n1{&s9hs)V{orw3>N-@P?VkfQ-qI^EW(4bo3+kK~*4 zKq9WA%{EX64t}W0dvsS9=>EN_2UPm$>7ft6B9SOoUvxh4Mjtj56cBHF^r3bZ6$}}G z+-#Fj@dN|VHv2emN#79q^3F2sLJc8@LlwjjR!u#0babN;xXeh8pHOE6^EZgxsP4cslj%z{~jZ58Bxl5kSN)_+tX9Ffp~geyl_;yl<>Ljss858 zRbqif`ZFz+PlUAWt8*TwONsXdjeq4o)Dr7pnag?AHsVL%%L4Cvb;M5+hWpmG5++SS z+v8e)6Z<1v;skr@iQRYOxw+e$iPb}&xq9nr38`W!{!QOM1nuvq|0ZZWU*1JT`(@lI zj_D$HQ9;?}ZsHqtEV8?oD9$plC?nTxb;g)XwI;-aloIf6A&Ypmjru?#K5B5BfYk2a#(d2_NhqE3+}#qrR;x4)LX zbkjl76u`>siufi}4=HF3@?R{~M`zFH{8SFqM-+>*a`i>$5eY`3>ywR<4Q1@knIPIn zl$!jPOFJK)1;p{WRe=bE7_%B1p2w?y>$nY}fleMpB5nnwZUd0S*o4dy>vMC-ji zTa-)MxULji1v?P86?nt;-U=AK&a>v=8In0lc!+*=bhk#47C zg(mVcsO1-5<)Udyt@-6C8i>Y+zZxiGp~KQK@_jo=dz-=2K;+bHHx5%r^!Z<=Itr&6 zO;!WFdg%LDdx<()?8Xk&W2lSzjG=+HUTuuIWT1h@-C7#BF^-Ekj2g`eu^OWH^VAWo zGry}N`dGDA6VZCFfroV8=c_q9(n2Lij|Y#gQxm<VsORkMAVl^afqbtqXf|N;wldPni`6S|4M*54VYp8rkjgqP7o9M2LS_~ zdwg0WWyXMIu3M%*RA2xV3!5Kgz=y^De>h?cusArDvUbiG{*y%-!Ucnvjz!G|a6`i9 zNk6>@a43D_ApT3m`@{gI>s{521++jJ!Xz8a=%^XM&vam5cZ^stN>d;|K{g%rf!*m{<6CmF@pGLiXP4nCEbgam-^J z_Z!EtM@T9S?Lv~0q+PU2r9~Q|C6yAM+_|EXloaJ@q7+J!R3!S|gXd59{`B!fw_De_ z#(TWRbv%p<1uA?|^z+Gi_K6Q7C@ZI3GJTyW`pQTtY^)GRbGGy?dha&@J${*>d5kTA zR!ho{sK!X3{aM$poP9G9O-LRX#2-yWNgt;qZ)lT7J5toc*Qv;&Ye)V~kI|JwIYpBc zpRnYS>kJLmoX7I$N5Iq!UX%htotetj?-kJ3$!yhiixpAau~oUJ#g)*jJ9Q%pTb0n; z1DRW=H7OzYxUw0G6qM0*Lkr`Y3(9DYm4^h17;VwR_gqAIgicO>c~_mC%9G3yMG#u@FMg-r~U(w3`S5NwRDws&Wj>@r5F zK7Cc$F~-QbMt<F!QDAcVJ34*37J^ zF}Lf+4lnDnGML$goqeoOUVNklTeIm-$}f>;m`E-wLgVob%&m4q&yU_JtY4ti**@VG zw&$wq%uNxESkBZy(ZzpXU^V@nuQnCj!+bq1r+&(MiGA~DssuUS#}>n^uk!5kke4_TYN!@6Z|D2JxBW6SN5E;2IO zF_#;8MtaxYVY`nC@V2H-%q-jDBV*AgOd?*&E@TWjdj{I#G%Q8I7MS1tIl5DOO+YJ*GfL51grI^cQY{(&<) zwSjw3^^4^hEx5ktg88;IE$|<(+wtLsCd94HYCII74t+b?*0Z%$p&>|p_Vq>!{MO3S zXu*gSpUY1wXWcZ$?FF~OqfaZ~T%%~|lcqL!g+cKBa6=8eP%J;DC(R09S-3-%C1#4> z2$<2}tzv`U64|UVvQ3iDwqBzDR*H#>>KypO?~vf<+!_$K<0<3HW`k?e#GX`w*ypAP z#;Q1OG`DF%j4uC|+K!0cKx2L})1pavWqW1Q>!Rk(jcWL8*%0oI0yR9U(}eZ7PlGSo zXQ~~f&)|0jDkc12^t*F{O8pc!Nb&Qt^(QH3s^Zrg97X4*tMlveL!Opj)c75Pt(}=? zE$~k^wWl7nOvLAHo^ojaMkU<5dUu)ZVFurBLK>(0gB`y6WmD*YiVpwbdVQ7o>37{f zFUgL+(0SWEEBwv%KfPxB0M~Vs_e>%aUXC8y!x=OfIuB555#)_SVC4fpvKdzXKN-n+dFylelvSy-|X&fn3#v|6Z%fBCKS_CUNI zF4GwE=Ul%OZdjq%-}J7x@{hUUzk%jP_}aFkp(of%_^hOAWyiBzeyVBS9o{knzHYwA z!_bc$oO#N6_2HN%>5B{CMo%iV3&{p5&i4 zg&*(0fAQ_u8XaY6+>kFWkSejqO{`9omgX|?irmohI|4i0#+fgOIcmk9Hp^;v%R^(l zo_FilYB4VVVb4*SWGyTFW^qDC)ufS1&4}4Es+Ac03)hY++TI*=KRh}TeYsW>Z}iRC zk@Q5J&n!5nb1PL9|DL;{_d}vN?ye?x&8AL-e_@&DExDcY{Ovsh)>A8fRgT4xpWga6 zZC<(r-@x~J%fd`qzCr#f(~@gSe19X+g{G3?e4DLpBRV0nI7@fk^RS;omA#Alqh1vB zxQ{=>sw)s{HZK(e)@Y zwx1vWz0*%6KK(g4d~Ce?__(q8D!2G_iA*dJ$H((BK5zC=_q*Vq7UTQikB=X(Z~9%2 zKYos}JpH%s_&c396Fht4l;>wVjsDC3_~*yQ<74ssZ`{~A`1t(e@8jp+r*W|Od?y#D z>(Ux$lxBxrLTo-=_saPAzUg{r;$oboh9^uN`1LEKMjuK$@;~g4lX_FY89$eMvc%u* zp3{u+@%>B^bqUz`N{2uGq~oE@Ki>VPXlAhyJ{})>J~Lb^^wISyR?qd-Ro{|4Y&i=<>n&+vWy4 zqT|X1`0Y$5MA2_G54EWDdp=HaK_ZD8C)~x{(9gqf4rEHWqqW61=B!`sj;>bq!Wlw45U z8%3SCc2+#-k%WV? zlY6Nzy67=w`qfrnRMW(suPx(;)@-&3Dh&5S!>-F3w(0mFrS7uJ;(NW&o6;@mjd6h1 zVvaNa@%KQhNnsS^j>OoR8ZSazQHPp--G{$CR2%5L-P4wbC_W8#M*Hj6`&G?wLJ7q7 zZj?Bpk)JEua-TRL74je}?a^-57x`OZcF6nE;}ke*leF}|%aeI5lS(j&@Z@;-BPKx;yjf;^Kxwp=in_zQF(&tk>Qxh)zXy!!g-lKX0O9KWhU?<$tefWHbX zYHY-A`-njpu}ziDKd=(ck->acFP3iMz9?lzd(}Oy=-O}I|5mXC$SAT-9lYNe1RDOM zfU;!f%6cluqWf}@_p0_vpn>qK+{-3iRkYqy0^VTtybH)2UK3@w(ZPv6e~1?lXtS4l z*%yuksqYGSDTIt^O_lE-`T&y*q=gS0{3$u>NRa?Uvepf|rbogWwPN4)j7XsBcx5Co zt253{*)jI_b&=3~?4$nKMUjw9CNMD)sD9WL3I4}7m{;$Q1S=!Mf%vpYpm4)v?DK~r z;P;o$$mMT15Y@9WdviFD%PlLM1ffLx)O-yC0xh1togE6tjuJn&Lm-G~)7ktG`u%7y zJRo5A%$^{i@MU=*tPm1A*9?Ss>7kPg^Zmh)1h55uu#IqamGweUA%U&Io358^3|5LA zo2c8sr*-j1ci6KsNmm}+;jI`^I0sze`qUp4FCXxL!lO}V$oi5_AeA%3Musy_x;Oy| z+k4v`==xb;4-{S(xq^Ae=DbHT01pY6TD#Q?D4b*pU_A+di+tdL-JPq}g+ic6aGx)% zBpX-gN3ZkT4?ZWCst>#QgC_|L7yY5~5D63kK>4=Q0kDlIvyI=0I&V)?E5-tWv@@fd z1EFjV0YEPTL6>Y8+aQ=uq*ZNq5K#Et5(J$2+V7$@gMs*`?8%W3SVR&>a2UAw_%0t} zPXdxYka#$>korbm%LME^r$#$-6DLo~ct6#NrM1bR&rEEFtG z?2C;8iocFeru*bi1VkJ%+OtwR42W!8`;;R9EfOI50m@0|q{RmUwQMbqx&np!@h(95 zu?L<&@lTBx?Ct0`e0tU!{KEcdC2I=69S=Wbv{XQ^vs*ydH%kDPt^^}&^9BMf2VPzF z0*c>edcsuVm?Gvt$GOo1?jN~%Xy0RZxOIX!z&tkynWGf#a^01#7jGBvB93Rg(+O4( z;jeFSgiXmrL49<9iTs?&CqCQ5Vv?xU?Llf5(K#`8Ku-U*-8rVUuo_<4p67l>hVY^`W~U0?$gpS-sP!Or;3;chN8K8xzH`D_Q@i0x;v9DurS z&~t=pP4fteWJkL19G!sDFRPsC{+Y}JBI{f026*&5+g-qnYA9E5A_c&0S0G$vob+Nh zct+4d{5B6L%pnAbt0(kn5EZ=OGDJVUBBCbi}`UEZ5pz!^J zM6^1n5FZy=pay&n5&1VNK+^MXJq1|o>ZYw%AO_CuGmn;i`iG46mh><&w6o^utVbI) z(8AbIw?AeK#3LKziVm7MZKKh}HR`C?a=3WnTO~x&c;f?IbmoHe>D~hj^zTNZ;53Yo zgSFnoIy)0u7x*!0ohYhLcn0#@gPKAkC}J| zqhEx)QZ5aq^;t{+y7W6H;qC%oM5>DK1_ETaia_{zp0u7y^+06V|CDe=ABgr%G;~4~ z9?01vBAaXaH`pQxf__$A;?TO!f=TNxt}zlF?rP2sHAJz6UuI1`uZ@PW%%T2GYN($m zUnQO@^6qp@*&d*bLMY;tMI$fv5=<(IPOsmzQ{8d2s%n$N+$RGaRaAVUWPnWX$;s-F zTCwfxogQrn$$kAb><$CS>xx6q^`P7>u589O9Xj9FiT-P{O&7##L0dpd{Df*HSU_E9 ze5jgE3NkqjwD05~_hW-OG9cunHD-nCQV1kDF{azi1a#bm0#r=4W0U|Q3%MJpv#nhD=P&}hb4Cq&ASWtSX~ zEt%+N4rnPkK+FL-k-)go4lO4e;jukhPZDssBV7-~I8tVXe&HgDS6*=ug%4_WD1SdG zsO^v7Oz-qV%7o6{7mO#1>Kf zIok>;r4a~F#z7f8Vq~*5O6VLu|7|}TQE?VnA(G}Aez6fLm!rGb=vPPVsppX#WK9NG z(h>#Um*VG?GEp;OAKsNSX3?7~DJRYL>C6$Lx z5j2)uFqWr4XGEA5Ik9?YMB!5#4-smLkc>P!&eJ{U_)K%5)AqAMB&}26Iww2d~`;XzVUEDWV>2z za783N?wRa{nuvCZ-tI)#!z^1cBjlBLy$$G-!hF32t((8IX?+{R0m^4ITL4MpTctTb z!ufeF8!np>LMGS-DF6C~L*u*!8_rE36wEgZpyE?(4j+j0IqWs1b+(Erjl(BxVQYHh zwW;bhK>1rmF0I=Z+d?lP`%E5iXue5h)AyBeRzN;v&v0M?Sr9iYY2CJ&4V1rj-~zGA zSDv~DjlV_qK$i7xDF-03TGi8Q581>vXXM%gRX1G@u#Zas(<~<--A#CkJ*`JGxS%tM zbkep!=g$UKk%C~Y6_7GW@jM4KhUkR}GxlBu_C(Ccd0x19b$ecdcdox;R9cIDV)wLF``&qQl(PBa@ zX`ENH==BrWK;el22dMfh{$DU35s6947 z^{Im+P}4S+-lJGPTM4LZrXUsAw2w1e%vpiH%3 z?`dq(i`0e-ep|0@Wf+1gM`7VQbsgAD$chy=R3L-@rLTn;=U%Y+$NFLh(0t2)Ld6+j zt#cUg!t6qsq=q3pDfW60x#Yiq+NlkJQ%NS&g~ps6ky^TXaKOQEC3ADbl9P>LI2qz7GEC-B9LQ3G!`g#oq-V<;(U4#P!NHwP}wgh{^o-WO2 z(*1tW3@Cjt#S~7Y491m68bgKlQq;Lnf9#$>6D$*bc~;RX@TemvVX2%nMEMQ*mnuqt zop|=QulAB)m0f0%`&mzLTZb`@xH=2qB(m=wAUYrD+ zenNh+`5V(0FKM}niNWqU6$Q`24oA zsp_ckz)tR!msmn#nWi3q+Se!vX}^@<1Jph4kPl4zQSoxeT_2!yRIL!G`&qUSNITPX(}(7-R}vDWHAMpf{Z#o-dHJ7jsDnlulXZ3*rtgsNS8?>eeoGmd`nPzI^S&*Vm*WDsM!Wzk__0L;t3el+o=4DJ%@%PCxICy;c%=-tVFlB0KzX91&=sI!# z;)enmG>@no97?Y4vrfS5Dk=QxKjGR!6B)EVyjzfUM+VZx8BMg0pzjR}01+Fr#=<5P#($o7X<$kK=)5^`NcnFNgwO37c_=>+%6t5}@^PgMjJw2l zb99x#pIcMj*d$6p`t!p(Z1#u2<7)nt4e|%0ps3$H`$%ElyXcd>QwM-K_m2xQI$lEn z^y30h91Kn?wtN?6hCpcq=%;)Q0_z;5_3H0};96LZVaKl9vuAV4<JW++0|Q@)wTXUZK~3Z3Q&4if7w!>Wi3Y3k4iRW zz8Mt8ehjr_4%*t+>*hSIsDHHo)|~4G>@ckeVPUBO8{gE&?%&UbjN)f2<1jM;^Ey;# zT&v)Vs9#ulQ1@AC%=RL)Xx#<+?9po;iX-xSFjv?$lD;1dm>1t0nilM`XWu&4ZoPU} zogFzRVa=)n9oD95=?Sf30pr?bYQHxxUD>4%>LXk9OqjZvW_qdlHKKQ8isXBn&WTpW zJU;gOqcxLNF+SkMWIZ-x+c^98BLz%DZa}7XS$##>%|qK3ZT?=7q3=B2O!t08twi7V z*T+`Tl6}|b?`wK0dL10w>)CPv`+C*0q2eqZmMMIiSruT!e)y^LWOYU_R`YuD@7-V2 zSo2feztlhbSn=D&{9dSy1v|ZQU1W)a4m)&Wm9EDwAv48lbG7$`-i+yXg}TUB{aD+r zPI<-gjTP6X!^Tb761J**xAU=j7Y1BctZ&=MFqaNxd4v{Ou$4Bs!51Bk+28pmZ}`ic zm|2DMLmrj>7FDfR?Yg%~k+munRt?(c%MmJ9EEz z{WKF@4c44dma3Jxv3(p%%F8#}vZppio~)L-vd&(NGD@X@_KNq}Z+S{W zrtbYZ&8(h|?3sh(UQ|Z6R~W?(8(gI$VD=te*WKmJcTw@5cKM-NBlddWhQf(AdN8)7 zyG_nc(PzJIob~bLcpFw^pncT&k^-}Lg5<3B;;zi1_bpC|r@mEm#F5Ew_&arEz6#Sj z@Yvn71zJq+?YUNEM|GHB3&k`mRb|F`USqpSgcfUOI^#}M+mDKm>0jdSm%bNuUKKh% z-4y@HOv`^+p}*h6uc7pj==$~o^C?QYOlN#LfA4(HFfG4DvrM1qe2?N$p<_L*|J~E8 z4nyD5W;>o2bw1ZIUd3G5YX$W(WwtXfo#Ph$c=Ig0&AxLTOy~2R_@?*0n9hB433qliS^Ss(&i6Z>XFKBg-*X-Fu$|*~-goYQh2cL3_cRGvic2R}P+k;u z038^LdtJ73-4yS(tWv*B^Aam}=1R8OpE?Z>=H;rfYL83%b?!@)rSf-?B*~KPTu-#3 zPe{Q-6Q=V@<6+Efdh>h#kb@R%XME^*)@;wn1D7=xw~IRK>aLb;o6{u$dm_i*`Pdu{ zX2Gt6%dQ*!@nmm8jJqY6zVbu9*i3?RH(%ZRZ;c;z*DBwcSL};RLf&?9FYv~fU(9cg z|JNPUq@%}w&T&P;))&>L+hOccwWNY`mRR%Z`Gcq#1GHs4?SY|Br;r_8d z7RXbv*)6d!E`IvKw`80h2)BuR3VTLC{+Gj1V^_&w*Xtmszbn1LamtN^6|ee$V8~tPdrMoz&tA+} zu&q%Cqhhbz`B`OwL}jOaSz(I;2~qCpwrEMh(wJ<8TpE`vT4K^>wcNA=#+a>ramc3` z+E_whb$hZm?$N(4y@x({TO-{vW34ap@jos?Uy`Pq*Zh#|zG{9N;Q3a&~|6u?IRZcha_6o$aF6;XCTpEa5(~iE) zDhTk{xm5M@St)Y%Aj?RKBul!U(g8S%`#AYy)Y9Tiy{~|L zo<;%wB(^7efFDjXI2ic!r!Rg?^7ngv$p=TwBoR&bLgHfVl0UiQ*>grVvE{CG{8ui> z=kvJ}a$#Lx?1+j_lr##(j#!bsx@VWk&e)Q{0mTK&y}nnCOcG)j0@n{qg;==k#TWD0 zLadrk3UoNRHy`masZ2=Ys_TlJeQC;YMdE(SHZ{0nlNrg_5I5Yot3vuDRr1}bFA@12h4A2o5 zoP(FQ$_!m-TpgUzT9qJbtP>vinLMthwy)zif0jM`%B(aa@SNr2F64qzx0jNgG zyN5Is{|q3;G(8MCTWJ~*j(nXeM4&@k@#EFqBPssmQ8?zw#KL8wC>-hA5V`c+ea zXc~|O}Xe4Ng@3t-)Ia|G_AR77lu8+nO#GW(8G055JW#uur z?HB=CR4kqUMJ%pLCKKomL=Kkb+hZwS6l0N_zW7=UCcJk(W4j>+T?s9k4~xMg1je~N zV{jCq+eaob2s;|ic3TmHap!jLh)9mXr&_D`%}YNp{Pc}A5|BMrymry zr#gothr&U5;YenU4wgmGIvE~LaUc)J1EWbmyM@uXO^n8|T)?*vM84lBkkj>bA4KCO zOh%puFAn<(x1=ySbJ2zAd4o{X-9jTB57nYV5IXo{Ve^5_^ z8o}X7>57TK^W7ggXe&7^FnlA2YJalBt8(CI;8U#}{Qn(wyQxeLT$p|zmO~N;q;fg% zaVGbCf3;fycS7@y)=l1DID?4jih=!DIDyt=+vGL3WMSm-KTnnh5$kR<{`mB zw9hN;==R1nBG{wE!1MoX zF{pEf!c+`-ByH;2Vo1B)G{zxb44iGLz2EU%jF{^8YCqud|AGP1JzhFLn*qY+Qw?S? z5X}XG+#3*C`aG8bs%s4O1@jrY|0@jew0A`el&>Yg*UfW(kj#EoyH-mITzmQtBc=Yp z&L1A76GtNzQ(sf!P5WUzSKxWS%?bjJJi0kX-4Og<-{1dJqz6BUyjxotK)y6=k=0ZK zIQpruappcFxKK@u6e5+YLvdJUjqGy+vrDPg z_?6hUp|w`1FMX^gm}iOSY6?xlo?D=y$J+T%`pA6Gm~wZDUG(%pWuOM24&O5dapwzo(vU2^@VtD(VBis%{G!b~E^tM6IB3!!pRv$GlSy#2TzcwY65xFPfb-ke|e1L_AD zI0~LLGsFznKU1hwu7Re;EZZUFU%A4*e{aXXuXG2FKKK6hfF$DH3a`4-bC6~i zAWuy``_}>bat3OOJ!~fSFZ6C-dLFVz2t0q6x&hCV`@Mmq8HKw5mEAVFZPN~?=O?|w zf#=DX2;lkuaTvvsG>pEV6AnDT?~H&4`w7rKguviO0m@VKL%^6b61rZ<;~Tv2Y90A( z^hUm(w)MwD)@0=$d{K!@lbFH1!v@CK>WGn(;d%Ce98GE_0M5TO@Z9c!66EXFK!QEE z2A~dTzcF5d%`G8`^FK?Fv#m-uB*^oKvOgXqG9LLvf;?}Xks#Mj99SWt{%)lNYfls7 zHA{j-O+~Dwg!-^HKy6MSE&+--BklyqrFVss7@^wW&%<(ksjg-C&~;UN zAfM+t59Hu^=7JDw?`}@iyDmVUmsZ&0(P_ltEq0;x*W`wLe)Bw$KSvtmN&V|VSE{Sa zT=6%Nqwq=Y)X)C*p#HGd9m%R@{F>^Dz1}_ZH2Lg;BpXGyd_0iv2lTwCuW9n2`fDRZ z{#~-Kd{GC`5CP@FMymkMB=TG>=1E@pLe$th?^W*Hg3?a%F$TS%kbz80Lq( zer=M{y51wB{Qe{Wc>nc25cqub4hDXo=Y!!->bDmu!J*Kf6Vjn!Kv+-h+PZM6uZ0m1 zz$N5$1n{`n8A*AiHWIipYmbWro+oW1X}?73gV*n0VZhmpBPC(L!Q;faP^z;(iLR|9 zpS%#@afrdd`;tRJ;Bv;cyH-&kaCHA{W&m()hxJk!^#y07!1p)B1Ar^%5`TaCytM>2 zyyA>J0NZUd{4?RIcFCeCRye zy@Bszvb=$_`x=Qpz}MADAK>w&BLY6(%>XJ~qCfZnXT#X7{pmW?eZjOV=`2@>C~q9| zrMf;vMEl!SA|T!NhB&bZTH1+0nlGZ`bY*D2U&GKoI@6Ex`wl+{;fPdK3{|8}QgH)% zu51J}uK{8j*GT}D%W@`~wu|Y0Ya}4Q>{@+fhCe-@jRoM!CjYP>?e~fPE3BOHtA9@b z?iiurnm6?qJ-vaWm;JwesGoY|OV6*Re!%&N9Mx}dkt zDwVKdy12aIVN06WkoNN>M)*LmddWtaITn&~DT*<}W_yJTZg!@45{eaXGbsH8XIv=;d0Xqc5`@fe%R>!zlpyoVv@41BnQV2xwmu zB0&820L04TgTOyLRJU-x+*tkN-G7wBS0|~dQv`=8A93V+gUfKX~5+3$7SonRRJ})fVYU( zzZqiNntHD?ZowmMuVH`1PiH#ZKeAF2l&5G}+4&iR+tcEod9l49IQ_~g16vE=?3mUI zd&+AW*05pHyxdc}&EZYA;c1#?hLmp&G(d4dROzVhKPvBA?>(V9xhvJvr8aP7D|vwJ zOZ8%bGw|n5kA?7~`*yVD=|bG~qf>Z@2xe{T~5e=d9i zfRl%lkAeE-i2&TOrlSO)ZYFW!kNu#G*bl{iBH&~wLELh6g1o#?#reSz<9RBWjZjkg4=V=t2n9ZaCt5yKKC(51hdtH6ij zT32(D^k^MC(!gCR&ObF*nc<)57v?FanBZHDP|=npJ#f<|?PaS@7-IF%)gEup8RM3L zzp_O(dPtfvcWc)8IoFK1Z~jGAy{o2r{0^^Fg7@YU<)X&Yv=SB^X2toOiG2e|b6 zjQ%3NHOyrG@*qvTHNgG!9Jw0)X&zGU`n0uj>ELZs$8J%;7T3a;YwomF*4eEYl+?AY z@_gSX7e$kPReo7paQXYUw#wksqrP1A{7`AMWt_*mCAF2S6f?dmJ> z#%k_hpjo`ergY20E6OmG3qXhx@Z&eOio1F>em%cX4Q35n*1OV6fyOn^5V$by)9eMb zo>gE{O-=NvMNQ(_i+hlc@T*v{?ZzGl62U898$a%E*GKW5wl`DW_dt)EF?qgKU8zr- z`M0v=*PpA4_3w&bM6_E*<}GqSA4%NWS?U zSMHD?dCZ!6p(^CZ_Q?gUCQnRxJIa#SA1*0v zC>TFVke)UnWAeCug51&B>4HRN5E&xTf3TB4E*PC9NXf_=&kvK4lAW5FEy$XXGJ@Qs XrcBBtqYcmF|3*h};@0th9m*Br1;v__ literal 0 HcmV?d00001 diff --git a/examples/damianou2013/data/hgplvmSampleDataTr1.mat b/examples/damianou2013/data/hgplvmSampleDataTr1.mat new file mode 100644 index 0000000000000000000000000000000000000000..91d935aae35ecbba80b61cdbb408c0bffd142479 GIT binary patch literal 7016 zcmbt&WlWt-)GZv`OVOeSD^}d~!3q?2X=#Dt6pB+Q4h0Iu-HStu6)6r6?(TkY=in4S zm-pTu_xG1?Cu{bsWU@0^nVmH=s?s_t(lSs1PHw2Gv<`=*jh)3SsD_=XtEGdpy(m;c zMa@83kpC4_*4e_;)xsR=@L3e9<7NTXws3^<@IZO^MR|loc|@SR+&sL{|6w2@zfmPX zLP8RIf=9-NnTg4Fq>qBoZT|^q8}=3-goiI-8*P24xoRqRed=T=Aui%0^e54GCjU8d z+;7wxdxKX^k(QSxF8b1D*HqT!)g;W z>?@kVpoW1}x(Nb!CR5;nIiD`?PCH0QZBZT1RS5K`7tacf4gq$HWQ;}r0on*+7f&o8sA*T4w$zXU8wYk(%k`%Y8VIUo?mcGk4i z51{*hfDZu)z*0sJd*XT**m(YeH*aVftf!=ur4?KSeb$Fv)o(Y!E5^Ug(^z}pAv`#y zLHZDU)>Z2%8v+Lt5#@acoyVXG-FOe=<=z|q@0L(shgfG)M5w&YI% zd`rgGnno~SadXdTH*gb>0R-$IMZ18z&l+_2>Hu&(u4eDlg#&-)KB#+E90Q_;uSM0a zj{#=ul!_6KV_>|GFK0mS2(Zx!_(l=551iT+ERqgx152U&;XP{Wz|6D~dXmioFkO)I zUM+tFAa{1P;HfPG6eZJ1II-HmGfr9Dov)K%9zQvy%bOL@D%z$n0c#WVuc*g_N$-Np z4?D!tzYaj@U~~;zdN_FX^UFM5$}zY=c)FPSa10W7pA6@zAA{72Cd+6EM;iYT){>@c#?vLPRNO_KNg9HHE66*n>Qv7icd?xeLC zP{PMDtzW0-J8MecxR<1q{V+APV>3aA@II(l;|aLG!cAs;6iI8vEE&Bs2xHd~3$_Us zH96i=pJh@D`o&&s{y;=_?HC$z_TYhO?|8WP^^{bF{x}k?1Vb)89*q#4U<$*IF`Pgm zW;s2ETDS^nBUZFM8bY&=Rz$CPOHpntlFU@%)^=aemvX3y;_VwNaVKjlbhSMr&SA^H zj;%oOn2Ux&@3X^*B^DB!oRZCD{aS@Z8Emr4z4v-R#!{ z>b^kIy;yuFcXAAs*T1nZs}aX6m@E?};YYiqiCy2%a*Xq84J~6X)=rCY#Rh~>vn*Sy zps;p5jNxbouxQQnehBkyA@(9%jNebIS+E?=Nl7Cj_YE8u(ExNjkZ*X90Sig+|{cZ_+;v(R6aou*AhQuPuv|b`4%T;X}QEb z!DUrfZRHcSo0Kkmgvislv5itwh(v!iaEgy7>eXMA?;H){RY@pkGWVC>d1y=POb0dl z4vt**k~dXHFK~^26ni8XRF_|o_!%gkh9o5Qj=z3;`lN%LE*zmSe75I)ON=u9R_QK2O>a-Rj{~mKko#P@_aS> zlUSK08V-pb!n#}+A zG)RPRf9K*X2#F{g7{nO^As!QSNLG^|WK+4H$bS|@A#dJcn;QThn30A;>x5=3r9V+o8~1CgJ1yV6f?fyjeo z;-2=~AhNmcp)>a;*#6GCV)gkd2%#X66M@cvC|JeVo6G$m%GoZ@7Fj!pLO`i1GTsZK z)aP`d;thisfl0i&x3%Dny;5+dL@jXB-KF%;ss%vFHw=va)d2<^wVq^NZh~#Enhk?i z&p@f>dO}OS8&InBq|Igi7W9uG)XML^0^2a!TH2{j!2moYmW``j5Lp|CN-1drMD7e= zGM#(c4=*`VE6D-~G2pDL=y}p(Y3a)g$_WrsdOAa9`lOQwF3wk;aqs~coGJa@1|S$* z9)_pZ0mL2YE}`lc@Ms03Z)q<9H_0(0DmQfi%J)!Tldw?`dBa4};mLaf6kOzUq@VP- zMd|j5-39{^IdIEnH$iEsIBXW|H4rtWeYVJN40tBRY+tI{1fb?S=Z6!`0ucU4a)*H( z02v!)`*HgOKxWqJLR6jt5OiaN{J(twVkL}sA+`>nvi#Cy6PpDvHN2U-%K8AQ?1GDz zzYxDL%$%U)G0QVvIXcv6*v-lW<(DH+5-Z@B%f&E)fyGax0Emn) zmd5KX09iIS_m1dEk2&iZAG|XFq8Hu9@Nft~sPcbq!ZrZZ!mO0@s(ApDe}jG{=}gxl&EU05PP8e?FnAyBJ<)kR0p5!h_)CdD=}}Ub*f}&08uXWX{?}@& z82?=#ITAggQ#Z6d%tPb~_OE})L~$&?vClty)MC?r6J=BuGBs1-T=RG1ZEuoRRK6^R zOcbrMh%JkxjABYq2v5oj%amBBAvwXJ`+Hx|=g}i;_WbyKv>LaLg&tqw+V7qV!F~@c zAJTpZ)Nb>9yWz<8Lq+>T%?|+roewJtbZlMQA$ZiqNJTCt(~2(!lq?$J1^;$T9}zIL zm^HOumCY2GqEp&2%slH90Ay=SJ>HfJlgl?n80#;aWd|=jhoZl-;adPiM?BZb}D&Ihxhnu(|B}+D>%Ro(7@8vc7DiYNI;Rd z!=;WWzDDS`nZ?9 zrk07I;YPwxb5o@YCr^T(R!~7t*8+3uyO`sV^?UEK2+^^Kl)WYxbD>LpfsiB)uQR+eYN_}>A1d#DBA~9WyYaN7+ zt`b&<*~WIG3lLqw?mjh2EAY!GEo)=VvMAlYK*zlp=;1T}{O?(#j-U+{zS*wyJNw&h z7ouX#v4CSm;=HQu$qBw+dAqB7%-eKp41J5q;(qKg(OM3ndvZ)#bXY~(7Fz>BUj}8T ztOLZyGPH5ZnpW_QayM02_t3*f5Qcb3dU>Ucb!Bv~(YS9CGxS*UI`n+q$|Mu=4&$pL z%(|p?RJ-r4+^{v@9WF_1c+&AQ>?H9TGAQop4P9D}a5u_`Om!BMK$mGHe&G6TGL%da z;C#IipQ-~zUF{8kj_Ok9xNpnWDK|q0cw;f}aJ418LEK&M!opyDw+>CR3A-O|Lf0Xd zi+atbLGA~Zmu(g;CNqn1#Y6d>s<$d>@JlL>N$qWS>y&Wk;-ID)4nu2q3z6z8PoV{8 zWw?~HNp@<>`8v#HSF!j?a-}RhOsS^x1C`>g3_s#!^hbFKZiSjiRe^fPDl;xYS&DCa znRj29&|CxKGz+y8?g=l{IRE8!X=~`69O)?pTkDLQ&T28kk-|;FEL%Na9v+5GmvOz2 zN)rX9ooX>-XAhm4!prC{48FpI)Xbd7xPo=vDH!?;&8=RP3P-$6_gn6c-pq?mntV9d zBl-T~)c`)fXH-;-x3eHm^{PCE=;>O&mt*aBWoze0qyiGM5_`Uq?D4+&oe|h2(oci# zcFL4zOBNYY>CfA&+r}^4Zy7JxN8 zCkVV1h4@aTgwy%Zb=A})@kbB@<+-@5<2calnu*T%{(O<$;s|qpuZ%5jcpfS7kpKK4 zK|n_7O`q(oY9#wDvG%H27HRK|DqhVy+`U?Ifp)Q@^PvG?rQL4YXEMhNb4Fovs^|}S zb~_%Eu`6|;*loD;@gY{<@Hy-9Svo5H3T+Kg%8Rwga5HP9^_u&J!=5>`H`Ka0FuWg& z=KhGHxF`*VG37q%P?zKzto;u)R_ueKl;4h|nZUV&34$ziBZQ;vYkBBi&W5u`()Q?< zQBwQA2B*Mw*%#M|EF%B6=mjhDA7n*N?j8@5A8Z>R{18637qlFUR{Sxk}^IY;n&PAA-Hp~z8S zNCloNT=Y9CI6QCfu|l^pXg~EKDo3s~sfMG`mI;|iyk4`mS{19>u&1LYX@h5j@rv(( z93o%;MzFe}DB|XhZxlXD|B>_GoGFpB#VbwTm26oC6PfYs=^rNIg>#FC;{`?ENYZMV z&!SjO#=Wi-qWAk(>tI1)3p10B(_?+Srx~c_m!1j&&PArHcAX{MDWMB_LaM+2^{?p| zuBGNfUNpB7lk9IX2OYxKRrTDQHRiP}XbHnQry%R=O%ll_HX*S&ZUWeIo)&cynepLs zf!r?Y9tF>9-;c~hh(-3&AM`26L}1Nr9to5Ou_y4H(f*#FCEkl^ClTC#n84AeHMpcm zJIm&sANL)K!oXkGHZF#F<@9sRzsV`WDP5W-f)if`vR;B;-g>{Cy;QUqd z6Txpk)2xN&nGnmbOcV`@#K#n;vtD4uNL4g*{I;kjzNqVguf(ti8>{BVnPKER-YK1& zD$}znzSJ5fHdbD%jE^0D&xN(?`dg`tTAof{c)!YAfcp1@x_M!tP@o{4##s1Zs85w( zvXu(<^V{y-yO)Vjg@Eo+ALYTE*;D)A*(m)#6n^b=j0irJtr-A2$0^6LCqH3#_?r^P zniO&lvb9r55YI1(-U(OEpA$R6GK9q8U7Cf1=_K+sS9`~LJz!!9lWaln3H~Ep+P+Ih>4&wS0jYlfWJpudl2AybCHEoO(BsQW0SKn25bb#F^%t{o zZCZVh2vRNJW)|7Yc(K*9#b%e(_OhG3rHoC>n&-=rqDm}ed~_%;lr6X6c(4?;l`o4sZ*#GDHP6`OwYll{AMlp*)h{|`Mx;wumiS1 zY&a&Af+f9dc45 zu|b}|WehE*mIL;F+wEvLhTohn%3T**e}(Z1L<}l8duYu1%p$?1Lj}My*J&QLmM9JU zYK*erD$iH{3)9jjxehYOGo^gSgkp5)ry*+n_hhr9u{LVT_Rrux<6y~5$L2hB?Ym8K zXm1g}+W>!%ugW*PJr3&osCgrzv6m)-t?jFFDAXVCY-!7!j!{Rhy8bqHcS67DkFrg9 zQms&cP0|BLp%10{=fCJU{xEaNE)9Oe?|(-bKK+^K>rY(#tOujMODIPzvT;IfnkWa^^#?lQVC!ty!g8Y}uhOdZ($D zP#a{q7-f4awn%!Dc`bYN*UBHAh>w5r;+yNOrlfRT|D`z)71iYxi+VS{q{7_8{~5_a zUlQ&z&l`S6qFiKt_L94;&)29uHPRVe{OL~b;&^s)T2{hPq>;Jf@afSGX}SWwFkXH^ z!-^ZdU&M}B?>DV+!CPF@g!QwWPq?#K2DkJHT1aWP*}zO@qgPj)bK-mx@YTa}4+aMp z+YTDP^}ZU=Uz9YKB>mlVRo^Tg*>cnF@zyTr zBICcwskJbi_XsYV=g~YN;XsXVUXLdy_Z(#Q>L_`kElc{mK(oyZd5nChJ;avl~{c7wX*W_zkSS1tI3qRGXF0h&xuL# ziYt@^)7#kie9TG^5gS2#l@{_!@g6J7h#>EIMs#H%Qtv8*J=4C>yc!2&W8E$GnU`mp zO-VXgZWdU{w2st}A2$Z^C)@vh~JXBy-6_XSbQ1CK6Ws(-k6C<^~VB*?9q zE@}8*8pn$tvJ&5(=35g|-Y)yb*4O%z9Q&r~J@QAV9#B?HpLmjQy%1k%ba;`>*YY`M zk+(|eBZLz$>acGa>8%St;pX}k{gLe-9nxFn5i@LX))T>3%5CQHf}A8CH*$icy3>* zs3K`~p(3-m;;fz?;qMYfBx4L zq;r>+)Wir+3eH~|9C(MqF|gDRAr2INzD>QI*qV1=ba~*=~9X=DW@wEWE{)?%&PCTcfzlZiB`87MuwT9Njyg zOgN#359?71p}A*!@7?krq-S0i>o9nZH_S1I9Ggx$#ABu{lpz@1VehgLpZOMN)w`%k9^r$cm8U{{e?yip>B3 literal 0 HcmV?d00001 diff --git a/examples/damianou2013/highfive_shared.py b/examples/damianou2013/highfive_shared.py index afd26a9..c58f9fe 100644 --- a/examples/damianou2013/highfive_shared.py +++ b/examples/damianou2013/highfive_shared.py @@ -1,15 +1,9 @@ #!/usr/bin/env python -"""Section 4.2 (figs. 4–5): CMU high-five — shared / MRD hierarchy. +"""§4.2 / figs. 4–5 — CMU high-five shared MRD (Matlab ``demHighFiveDemonstration.m``). -Paper: Damianou & Lawrence, Deep Gaussian Processes, AISTATS 2013. - -Two observation views share a parent latent layer (conditionally independent -given the shared features). With CIP-0002, this uses the maintained -``ObservedMRDLayer`` path. - -Real CMU high-five MOCAP (two subjects, ~78 frames × 62 dims) is used when -``pods`` can supply a suitable sequence; otherwise the script falls back to -synthetic correlated views so the shared-hierarchy fit still runs. +Loads ``demHighFiveHgplvm1.mat`` (subjects A/B, 78×62 each) and fits a two-view +DeepGP. Compares ARD / shared latents to the stored hgplvm reference spaces in +the same mat file when present. """ from __future__ import annotations @@ -25,92 +19,56 @@ from _common import ( add_common_args, ard_weights, + dominant_dims, ensure_outdir, init_noise, - print_ard, rbf_bias, save_or_show, unfix_noise, ) +from _matlab_data import load_highfive def build_parser() -> argparse.ArgumentParser: p = argparse.ArgumentParser(description=__doc__) add_common_args(p) - p.add_argument( - "--synthetic", - action="store_true", - help="Force synthetic correlated views (skip pods MOCAP).", - ) return p -def load_two_views(seed: int, synthetic: bool): - """Return (Y1, Y2, source_label).""" - if not synthetic: - try: - import pods - - if hasattr(pods.datasets, "cmu_mocap_35_walk_jog"): - data = pods.datasets.cmu_mocap_35_walk_jog() - Y = np.asarray(data["Y"], dtype=float) - n = min(80, Y.shape[0]) - Y = Y[:n] - mid = Y.shape[1] // 2 - print( - "Note: using walk/jog feature split as a two-view stand-in; " - "paper §4.2 used the CMU high-five pair of subjects." - ) - return Y[:, :mid], Y[:, mid:], "pods:cmu_mocap_35_walk_jog" - except Exception as exc: - print(f"pods MOCAP unavailable ({exc}); using synthetic views.") - - rng = np.random.RandomState(seed) - n, q, d = 60, 3, 20 - X = rng.randn(n, q) - Y1 = X @ rng.randn(q, d) + 0.05 * rng.randn(n, d) - Y2 = X @ rng.randn(q, d) + 0.05 * rng.randn(n, d) - return Y1, Y2, "synthetic" - - -def fit_mrd(Y1, Y2, quick: bool, messages: bool) -> deepgp.DeepGP: - q1, q2 = (4, 3) if quick else (6, 4) - num_inducing = 15 if quick else 30 - iters_fixed = 40 if quick else 200 - iters_free = 80 if quick else 800 - kernels = [ - [rbf_bias(q1, ard=True), rbf_bias(q1, ard=True)], - rbf_bias(q2, ard=True), - ] +def main(argv: list[str] | None = None) -> int: + args = build_parser().parse_args(argv) + outdir = ensure_outdir(args.outdir) + + data = load_highfive() + YA, YB = data["YA"].copy(), data["YB"].copy() + YA = (YA - YA.mean(0)) / (YA.std() + 1e-12) + YB = (YB - YB.mean(0)) / (YB.std() + 1e-12) + print(f"Loaded demHighFiveHgplvm1: YA={YA.shape} YB={YB.shape}") + + # Two-level hierarchy as in the paper / demHighFiveDemonstration + q1, q2 = (4, 3) if args.quick else (6, 4) + num_inducing = 15 if args.quick else 30 + iters_fixed = 40 if args.quick else 200 + iters_free = 80 if args.quick else 1500 + m = deepgp.DeepGP( - [[Y1.shape[1], Y2.shape[1]], q1, q2], - [Y1, Y2], - kernels=kernels, + [[YA.shape[1], YB.shape[1]], q1, q2], + [YA, YB], + kernels=[[rbf_bias(q1), rbf_bias(q1)], rbf_bias(q2)], num_inducing=num_inducing, back_constraint=False, + inits="PCA", ) init_noise(m, fix=True) - m.optimize(max_iters=iters_fixed, messages=messages) + m.optimize(max_iters=iters_fixed, messages=args.messages) unfix_noise(m) - m.optimize(max_iters=iters_free, messages=messages) - return m - - -def main(argv: list[str] | None = None) -> int: - args = build_parser().parse_args(argv) - outdir = ensure_outdir(args.outdir) - - Y1, Y2, source = load_two_views(args.seed, synthetic=args.synthetic or args.quick) - Y1 = (Y1 - Y1.mean(0)) / (Y1.std() + 1e-12) - Y2 = (Y2 - Y2.mean(0)) / (Y2.std() + 1e-12) - print(f"Views: {Y1.shape}, {Y2.shape} (source={source})") - - m = fit_mrd(Y1, Y2, quick=args.quick, messages=args.messages) + m.optimize(max_iters=iters_free, messages=args.messages) print(f"MRD DeepGP variational bound: {float(m.log_likelihood()):.3f}") + for i, view in enumerate(m.layers[0].views): w = ard_weights(view.kern) print(f"View {i} ARD (1/lengthscale): {np.array2string(w, precision=3)}") - print_ard(m) + print(f"Top ARD: {np.array2string(ard_weights(m.layers[-1].kern), precision=3)}") if not args.no_plot or outdir: import matplotlib.pyplot as plt @@ -119,16 +77,41 @@ def main(argv: list[str] | None = None) -> int: for i, view in enumerate(m.layers[0].views): w = ard_weights(view.kern) axes[i].bar(np.arange(1, len(w) + 1), w) - axes[i].set_title(f"view {i} ARD") + axes[i].set_title(f"view {i} ARD (fY)") axes[i].set_xlabel("shared dim") w_top = ard_weights(m.layers[-1].kern) axes[2].bar(np.arange(1, len(w_top) + 1), w_top) - axes[2].set_title("top layer ARD") - axes[2].set_xlabel("dimension") - fig.suptitle("Damianou & Lawrence 2013 — high-five / shared MRD (fig. 4 style)") + axes[2].set_title("top ARD (fX)") + fig.suptitle("High-five MRD ARD (fig. 4 style)") fig.tight_layout() save_or_show(fig, outdir, "highfive_ard.png", args.no_plot) + Xshare = np.asarray(m.layers[0].X.mean) + Xtop = np.asarray(m.layers[-1].X.mean) + d_share = dominant_dims(ard_weights(m.layers[0].views[0].kern), k=2) + d_top = dominant_dims(ard_weights(m.layers[-1].kern), k=2) + + fig2, axes = plt.subplots(1, 2, figsize=(8, 3.5)) + axes[0].plot(Xshare[:, d_share[0]], Xshare[:, d_share[1]], ".-") + axes[0].set_title("DeepGP shared (dominant)") + axes[1].plot(Xtop[:, d_top[0]], Xtop[:, d_top[1]], ".-") + axes[1].set_title("DeepGP top (dominant)") + fig2.suptitle("Latent projections (fig. 5 style)") + fig2.tight_layout() + save_or_show(fig2, outdir, "highfive_latents.png", args.no_plot) + + if "Xparent" in data: + fig3, axes = plt.subplots(1, 3, figsize=(10, 3.2)) + axes[0].plot(data["Xparent"][:, 0], data["Xparent"][:, 1], ".-") + axes[0].set_title("hgplvm Xparent (ref)") + axes[1].plot(data["Xchild1"][:, 0], data["Xchild1"][:, 1], ".-") + axes[1].set_title("hgplvm Xchild1 (ref)") + axes[2].plot(data["Xchild2"][:, 0], data["Xchild2"][:, 1], ".-") + axes[2].set_title("hgplvm Xchild2 (ref)") + fig3.suptitle("Reference latents from demHighFiveHgplvm1.mat") + fig3.tight_layout() + save_or_show(fig3, outdir, "highfive_hgplvm_ref.png", args.no_plot) + return 0 diff --git a/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb b/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb index 428f0b6..2b8741d 100644 --- a/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb +++ b/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb @@ -6,10 +6,6 @@ "display_name": "Python 3", "language": "python", "name": "python3" - }, - "language_info": { - "name": "python", - "pygments_lexer": "ipython3" } }, "cells": [ @@ -17,13 +13,13 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Toy stacked-GP hierarchy (Damianou & Lawrence 2013, \u00a74.1 / fig. 2)\n", + "# Toy hierarchy \u2014 \u00a74.1 / fig. 2\n", "\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb)\n", "\n", - "**Paper:** [Damianou, A. and Lawrence, N. D. (2013). *Deep Gaussian Processes*. In *AISTATS* (pp. 207\u2013215).](https://proceedings.mlr.press/v31/damianou13a.html)\n", + "**Matlab:** [`demToyUnsupervised.m`](https://github.com/SheffieldML/deepGP/blob/master/deepGP/matlab/demToyUnsupervised.m) with `hgplvmSampleDataTr1.mat`.\n", "\n", - "Generate observations from a three-level stack of GPs (top latent \u2192 2D intermediate \u2192 10D data), fit an unsupervised DeepGP with overcomplete ARD widths, and compare recovered signals to stacked PCA. In the paper, only the deep GP recovers latent signals close to the ground truth while learning dimensionalities via ARD.\n" + "Two observation modalities `YA`,`YB` share parent latents; compare DeepGP to stacked PCA and Isomap.\n" ] }, { @@ -32,74 +28,33 @@ "metadata": {}, "outputs": [], "source": [ - "# Install dependencies when running on Google Colab\n", - "import sys\n", - "import subprocess\n", - "\n", + "import sys, subprocess, os, urllib.request\n", "IN_COLAB = \"google.colab\" in sys.modules\n", "if IN_COLAB:\n", - " pkgs = [\"numpy>=1.7\", \"scipy>=0.12\", \"matplotlib\", \"GPy>=1.15\"]\n", - " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", *pkgs])\n", - " subprocess.check_call([\n", - " sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", - " \"git+https://github.com/SheffieldML/PyDeepGP.git@main\",\n", - " ])\n", - " print(\"Installed GPy and PyDeepGP for Colab.\")\n", + " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", + " \"numpy\", \"scipy\", \"matplotlib\", \"GPy>=1.15\", \"scikit-learn\"])\n", + " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", + " \"git+https://github.com/SheffieldML/PyDeepGP.git@examples-damianou2013\"])\n", + " os.makedirs(\"data\", exist_ok=True)\n", + " files = [\n", + " (\"_common.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_common.py\"),\n", + " (\"_matlab_data.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_matlab_data.py\"),\n", + " (\"toy_hierarchy.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/toy_hierarchy.py\"),\n", + " (\"toy_warped_regression.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/toy_warped_regression.py\"),\n", + " (\"usps_digits.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/usps_digits.py\"),\n", + " (\"highfive_shared.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/highfive_shared.py\"),\n", + " (\"data/hgplvmSampleDataTr1.mat\",\n", + " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/hgplvmSampleDataTr1.mat\"),\n", + " (\"data/demHighFiveHgplvm1.mat\",\n", + " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/demHighFiveHgplvm1.mat\"),\n", + " ]\n", + " for dest, url in files:\n", + " if not os.path.isfile(dest):\n", + " print(\"Fetching\", dest)\n", + " urllib.request.urlretrieve(url, dest)\n", + " print(\"Colab setup done.\")\n", "else:\n", - " print(\"Local environment \u2014 ensure PyDeepGP and GPy are importable.\")\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Set QUICK=True for a short smoke run (fewer iterations / smaller models).\n", - "# Set QUICK=False for a paper-scale demonstration (slower).\n", - "QUICK = True\n", - "SEED = 42\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import numpy as np\n", - "import matplotlib.pyplot as plt\n", - "import GPy\n", - "import deepgp\n", - "\n", - "np.random.seed(SEED)\n", - "\n", - "def sample_gp_prior(X, kernel, output_dim, noise=1e-6):\n", - " K = kernel.K(X) + noise * np.eye(X.shape[0])\n", - " L = np.linalg.cholesky(K)\n", - " return L @ np.random.randn(X.shape[0], output_dim)\n", - "\n", - "def rbf_bias(q, ard=True):\n", - " return GPy.kern.RBF(q, ARD=ard) + GPy.kern.Bias(q)\n", - "\n", - "def ard_weights(kern):\n", - " if hasattr(kern, \"rbf\"):\n", - " ls = np.asarray(kern.rbf.lengthscale).ravel()\n", - " else:\n", - " ls = np.asarray(kern.lengthscale).ravel()\n", - " return 1.0 / np.maximum(ls, 1e-12)\n", - "\n", - "def dominant_dims(w, k=2):\n", - " return list(np.argsort(-w)[:k])\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Generative hierarchy\n", - "\n", - "Top latent $Z \\in \\mathbb{R}$ drives a 2D intermediate layer, which drives 10D observations (paper fig. 2a).\n" + " print(\"Local: run from repo; ensure examples/damianou2013 is on sys.path.\")\n" ] }, { @@ -108,31 +63,17 @@ "metadata": {}, "outputs": [], "source": [ - "n = 40 if QUICK else 80\n", - "Z = np.linspace(-1.0, 1.0, n)[:, None]\n", - "X_true = sample_gp_prior(Z, GPy.kern.RBF(1, lengthscale=0.4), output_dim=2)\n", - "X_true += 0.02 * np.random.randn(*X_true.shape)\n", - "Y = sample_gp_prior(X_true, GPy.kern.RBF(2, lengthscale=0.8), output_dim=10)\n", - "Y += 0.05 * np.random.randn(*Y.shape)\n", - "\n", - "fig, ax = plt.subplots(figsize=(7, 3))\n", - "t = np.arange(n)\n", - "ax.plot(t, Z[:, 0], \"k-\", label=\"top $Z$\")\n", - "ax.plot(t, X_true[:, 0], label=\"mid 0\")\n", - "ax.plot(t, X_true[:, 1], label=\"mid 1\")\n", - "ax.set_title(\"Ground-truth hierarchy (fig. 2a style)\")\n", - "ax.legend()\n", - "plt.show()\n", - "print(\"Y\", Y.shape)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Fit DeepGP\n", - "\n", - "Overcomplete latent widths; ARD should down-weight unused dimensions (paper: dimensionalities learnt automatically).\n" + "import os, sys\n", + "from pathlib import Path\n", + "here = Path.cwd().resolve()\n", + "candidates = [here, here.parent, here / \"examples\" / \"damianou2013\",\n", + " here.parent / \"examples\" / \"damianou2013\"]\n", + "for c in candidates:\n", + " if (c / \"toy_hierarchy.py\").is_file():\n", + " sys.path.insert(0, str(c))\n", + " os.chdir(c)\n", + " break\n", + "print(\"cwd:\", os.getcwd())\n" ] }, { @@ -141,41 +82,7 @@ "metadata": {}, "outputs": [], "source": [ - "q_mid, q_top = (4, 3) if QUICK else (5, 4)\n", - "num_inducing = 15 if QUICK else 25\n", - "iters_fixed, iters_free = (50, 150) if QUICK else (200, 1500)\n", - "\n", - "from GPy.core.parameterization.variational import NormalPosterior, VariationalPosterior\n", - "\n", - "def out_var(y):\n", - " if isinstance(y, (NormalPosterior, VariationalPosterior)):\n", - " return float(np.asarray(y.mean).var())\n", - " return float(np.asarray(y).var())\n", - "\n", - "m = deepgp.DeepGP(\n", - " [Y.shape[1], q_mid, q_top],\n", - " Y,\n", - " kernels=[rbf_bias(q_mid), rbf_bias(q_top)],\n", - " num_inducing=num_inducing,\n", - " back_constraint=False,\n", - ")\n", - "for layer in m.layers:\n", - " layer.Gaussian_noise.variance = out_var(layer.Y) * 0.01\n", - " layer.Gaussian_noise.variance.fix()\n", - "m.optimize(max_iters=iters_fixed, messages=False)\n", - "for layer in m.layers:\n", - " layer.Gaussian_noise.variance.unfix()\n", - "m.optimize(max_iters=iters_free, messages=True)\n", - "print(f\"Variational bound: {float(m.log_likelihood()):.3f}\")\n", - "for i, layer in enumerate(m.layers):\n", - " print(f\"Layer {i} ARD:\", np.array2string(ard_weights(layer.kern), precision=3))\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Recovered latents vs stacked PCA\n" + "QUICK = True # set False for demo-scale optimisation\n" ] }, { @@ -184,51 +91,12 @@ "metadata": {}, "outputs": [], "source": [ - "def pca_project(Y, q):\n", - " Yc = Y - Y.mean(0)\n", - " U, S, _ = np.linalg.svd(Yc, full_matrices=False)\n", - " q = min(q, U.shape[1])\n", - " return U[:, :q] * S[:q]\n", - "\n", - "pca_mid = pca_project(Y, q_mid)\n", - "pca_top = pca_project(pca_mid, q_top)\n", - "\n", - "fig, axes = plt.subplots(1, 3, figsize=(11, 3.2))\n", - "axes[0].plot(t, Z[:, 0], \"k-\", label=\"top\")\n", - "axes[0].plot(t, X_true[:, 0], label=\"mid0\")\n", - "axes[0].plot(t, X_true[:, 1], label=\"mid1\")\n", - "axes[0].set_title(\"Ground truth\")\n", - "axes[0].legend(fontsize=8)\n", - "\n", - "for ax, li, title in ((axes[1], 0, \"DeepGP near data\"), (axes[2], 1, \"DeepGP top\")):\n", - " Xm = m.layers[li].X.mean\n", - " for d in dominant_dims(ard_weights(m.layers[li].kern), k=min(2, Xm.shape[1])):\n", - " ax.plot(t, Xm[:, d], label=f\"dim {d}\")\n", - " ax.set_title(title)\n", - " ax.legend(fontsize=8)\n", - "fig.suptitle(\"Fig. 2 style \u2014 DeepGP recovery\")\n", - "fig.tight_layout()\n", - "plt.show()\n", - "\n", - "fig, ax = plt.subplots(figsize=(6, 3))\n", - "ax.plot(t, pca_mid[:, 0], label=\"PCA mid 0\")\n", - "if pca_mid.shape[1] > 1:\n", - " ax.plot(t, pca_mid[:, 1], label=\"PCA mid 1\")\n", - "ax.plot(t, pca_top[:, 0], label=\"PCA top 0\")\n", - "ax.set_title(\"Stacked PCA (greedy)\")\n", - "ax.legend(fontsize=8)\n", - "plt.show()\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## References\n", - "\n", - "- Damianou, A. and Lawrence, N. D. (2013). *Deep Gaussian Processes*. In *AISTATS* (pp. 207\u2013215). \u2014 [PDF](https://proceedings.mlr.press/v31/damianou13a.html)\n", - "- Script counterpart: `examples/damianou2013/toy_hierarchy.py`\n" + "%matplotlib inline\n", + "argv = [\"--quick\"] if QUICK else []\n", + "argv += [\"--outdir\", \"/tmp/dgp2013_nb\"]\n", + "from toy_hierarchy import main\n", + "main(argv)\n" ] } ] -} \ No newline at end of file +} diff --git a/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb b/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb index 302fdc5..8bece00 100644 --- a/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb +++ b/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb @@ -6,10 +6,6 @@ "display_name": "Python 3", "language": "python", "name": "python3" - }, - "language_info": { - "name": "python", - "pygments_lexer": "ipython3" } }, "cells": [ @@ -17,13 +13,13 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Warped toy regression (Damianou & Lawrence 2013, \u00a74.1 / fig. 3)\n", + "# Warped regression \u2014 \u00a74.1 / fig. 3\n", "\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb)\n", "\n", - "**Paper:** [Damianou, A. and Lawrence, N. D. (2013). *Deep Gaussian Processes*. In *AISTATS* (pp. 207\u2013215).](https://proceedings.mlr.press/v31/damianou13a.html)\n", + "**Matlab:** [`demToyRegressionSimple.m`](https://github.com/SheffieldML/deepGP/blob/master/deepGP/matlab/demToyRegressionSimple.m) + `hierGpsNEW` in `demToyDynamicsCreateData.m`.\n", "\n", - "A non-stationary regression problem is created by stacking Linear+RBF \u2192 RBF GPs. Only 25 of 120 points are observed. A standard sparse GP is compared to a DeepGP with **two** hidden layers (one more than the generative depth). The paper reports lower test MSE for the deep model across repeats (fig. 3b).\n" + "Defaults: Ntoy=150, D=15, Ntr=25, H=2, Q=6, K=10. Use `argv += [\"--paper\"]` for N=120, D=10.\n" ] }, { @@ -32,21 +28,33 @@ "metadata": {}, "outputs": [], "source": [ - "# Install dependencies when running on Google Colab\n", - "import sys\n", - "import subprocess\n", - "\n", + "import sys, subprocess, os, urllib.request\n", "IN_COLAB = \"google.colab\" in sys.modules\n", "if IN_COLAB:\n", - " pkgs = [\"numpy>=1.7\", \"scipy>=0.12\", \"matplotlib\", \"GPy>=1.15\"]\n", - " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", *pkgs])\n", - " subprocess.check_call([\n", - " sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", - " \"git+https://github.com/SheffieldML/PyDeepGP.git@main\",\n", - " ])\n", - " print(\"Installed GPy and PyDeepGP for Colab.\")\n", + " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", + " \"numpy\", \"scipy\", \"matplotlib\", \"GPy>=1.15\", \"scikit-learn\"])\n", + " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", + " \"git+https://github.com/SheffieldML/PyDeepGP.git@examples-damianou2013\"])\n", + " os.makedirs(\"data\", exist_ok=True)\n", + " files = [\n", + " (\"_common.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_common.py\"),\n", + " (\"_matlab_data.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_matlab_data.py\"),\n", + " (\"toy_hierarchy.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/toy_hierarchy.py\"),\n", + " (\"toy_warped_regression.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/toy_warped_regression.py\"),\n", + " (\"usps_digits.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/usps_digits.py\"),\n", + " (\"highfive_shared.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/highfive_shared.py\"),\n", + " (\"data/hgplvmSampleDataTr1.mat\",\n", + " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/hgplvmSampleDataTr1.mat\"),\n", + " (\"data/demHighFiveHgplvm1.mat\",\n", + " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/demHighFiveHgplvm1.mat\"),\n", + " ]\n", + " for dest, url in files:\n", + " if not os.path.isfile(dest):\n", + " print(\"Fetching\", dest)\n", + " urllib.request.urlretrieve(url, dest)\n", + " print(\"Colab setup done.\")\n", "else:\n", - " print(\"Local environment \u2014 ensure PyDeepGP and GPy are importable.\")\n" + " print(\"Local: run from repo; ensure examples/damianou2013 is on sys.path.\")\n" ] }, { @@ -55,12 +63,17 @@ "metadata": {}, "outputs": [], "source": [ - "# Set QUICK=True for a short smoke run (fewer iterations / smaller models).\n", - "# Set QUICK=False for a paper-scale demonstration (slower).\n", - "QUICK = True\n", - "SEED = 42\n", - "\n", - "REPEATS = 2 if QUICK else 10\n" + "import os, sys\n", + "from pathlib import Path\n", + "here = Path.cwd().resolve()\n", + "candidates = [here, here.parent, here / \"examples\" / \"damianou2013\",\n", + " here.parent / \"examples\" / \"damianou2013\"]\n", + "for c in candidates:\n", + " if (c / \"toy_hierarchy.py\").is_file():\n", + " sys.path.insert(0, str(c))\n", + " os.chdir(c)\n", + " break\n", + "print(\"cwd:\", os.getcwd())\n" ] }, { @@ -69,39 +82,7 @@ "metadata": {}, "outputs": [], "source": [ - "import numpy as np\n", - "import matplotlib.pyplot as plt\n", - "import GPy\n", - "import deepgp\n", - "\n", - "np.random.seed(SEED)\n", - "\n", - "def sample_gp_prior(X, kernel, output_dim, noise=1e-6):\n", - " K = kernel.K(X) + noise * np.eye(X.shape[0])\n", - " return np.linalg.cholesky(K) @ np.random.randn(X.shape[0], output_dim)\n", - "\n", - "def rbf_bias(q, ard=True):\n", - " return GPy.kern.RBF(q, ARD=ard) + GPy.kern.Bias(q)\n", - "\n", - "def generate_warped(n=120, seed=0):\n", - " rng = np.random.RandomState(seed)\n", - " np.random.seed(seed)\n", - " X = np.linspace(-1.0, 1.0, n)[:, None]\n", - " H = sample_gp_prior(X, GPy.kern.Linear(1) + GPy.kern.RBF(1, lengthscale=0.3), 1)\n", - " H += 0.01 * rng.randn(*H.shape)\n", - " Y = sample_gp_prior(H, GPy.kern.RBF(1, lengthscale=0.25), 10)\n", - " Y += 0.05 * rng.randn(*Y.shape)\n", - " return X, H, Y\n", - "\n", - "def mse(a, b):\n", - " return float(np.mean((np.asarray(a) - np.asarray(b)) ** 2))\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## One draw of the warped process (fig. 3a style)\n" + "QUICK = True\n" ] }, { @@ -110,113 +91,12 @@ "metadata": {}, "outputs": [], "source": [ - "X, H, Y = generate_warped(seed=SEED)\n", - "fig, axes = plt.subplots(2, 1, figsize=(8, 4), sharex=True)\n", - "axes[0].plot(X[:, 0], H[:, 0], \"k-\")\n", - "axes[0].set_ylabel(\"hidden warp\")\n", - "axes[0].set_title(\"Warped generative process\")\n", - "axes[1].plot(X[:, 0], Y[:, 0], label=\"Y[:,0]\")\n", - "axes[1].plot(X[:, 0], Y[:, 1], label=\"Y[:,1]\")\n", - "axes[1].set_xlabel(\"X\")\n", - "axes[1].set_ylabel(\"outputs\")\n", - "axes[1].legend(fontsize=8)\n", - "fig.tight_layout()\n", - "plt.show()\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## GP vs DeepGP over repeats\n", - "\n", - "Training size 25 (paper); DeepGP has two hidden layers between $X$ and $Y$.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from GPy.core.parameterization.variational import NormalPosterior, VariationalPosterior\n", - "\n", - "def out_var(y):\n", - " if isinstance(y, (NormalPosterior, VariationalPosterior)):\n", - " return float(np.asarray(y.mean).var())\n", - " return float(np.asarray(y).var())\n", - "\n", - "def fit_gp(Xtr, Ytr, M):\n", - " m = GPy.models.SparseGPRegression(\n", - " Xtr, Ytr, kernel=GPy.kern.RBF(1) + GPy.kern.Bias(1), num_inducing=M\n", - " )\n", - " m.Gaussian_noise.variance = Ytr.var() * 0.01\n", - " m.Gaussian_noise.variance.fix()\n", - " m.optimize(max_iters=100, messages=False)\n", - " m.Gaussian_noise.variance.unfix()\n", - " m.optimize(max_iters=400, messages=False)\n", - " return m\n", - "\n", - "def fit_dgp(Xtr, Ytr, M, q1, q2, it_f, it_u):\n", - " m = deepgp.DeepGP(\n", - " [Ytr.shape[1], q1, q2, Xtr.shape[1]],\n", - " Ytr,\n", - " X=Xtr,\n", - " kernels=[rbf_bias(q1), rbf_bias(q2), rbf_bias(Xtr.shape[1], ard=False)],\n", - " num_inducing=M,\n", - " back_constraint=False,\n", - " )\n", - " for layer in m.layers:\n", - " layer.Gaussian_noise.variance = out_var(layer.Y) * 0.01\n", - " layer.Gaussian_noise.variance.fix()\n", - " m.optimize(max_iters=it_f, messages=False)\n", - " for layer in m.layers:\n", - " layer.Gaussian_noise.variance.unfix()\n", - " m.optimize(max_iters=it_u, messages=False)\n", - " return m\n", - "\n", - "n_train = 20 if QUICK else 25\n", - "q1, q2 = (3, 2) if QUICK else (4, 3)\n", - "M = 12 if QUICK else 20\n", - "it_f, it_u = (40, 120) if QUICK else (200, 1200)\n", - "\n", - "gp_mses, dgp_mses = [], []\n", - "for r in range(REPEATS):\n", - " seed = SEED + r\n", - " X, _, Y = generate_warped(seed=seed)\n", - " rng = np.random.RandomState(seed + 1000)\n", - " idx = rng.permutation(X.shape[0])\n", - " tr, te = np.sort(idx[:n_train]), np.sort(idx[n_train:])\n", - " Xtr, Ytr, Xte, Yte = X[tr], Y[tr], X[te], Y[te]\n", - " m_gp = fit_gp(Xtr, Ytr, M)\n", - " m_dgp = fit_dgp(Xtr, Ytr, M, q1, q2, it_f, it_u)\n", - " gp_mses.append(mse(m_gp.predict(Xte)[0], Yte))\n", - " dgp_mses.append(mse(m_dgp.predict(Xte)[0], Yte))\n", - " print(f\"Repeat {r+1}/{REPEATS}: GP={gp_mses[-1]:.4f} DeepGP={dgp_mses[-1]:.4f}\")\n", - "\n", - "print(f\"Mean MSE (\u00b1std): GP {np.mean(gp_mses):.4f}\u00b1{np.std(gp_mses):.4f} \"\n", - " f\"DeepGP {np.mean(dgp_mses):.4f}\u00b1{np.std(dgp_mses):.4f}\")\n", - "\n", - "fig, ax = plt.subplots(figsize=(5.5, 3.5))\n", - "xs = np.arange(1, REPEATS + 1)\n", - "ax.plot(xs, gp_mses, \"o-\", label=\"GP\")\n", - "ax.plot(xs, dgp_mses, \"s-\", label=\"DeepGP\")\n", - "ax.set_xlabel(\"experiment #\")\n", - "ax.set_ylabel(\"test MSE\")\n", - "ax.set_title(\"Fig. 3(b) style \u2014 MSE by repeat\")\n", - "ax.legend()\n", - "plt.show()\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## References\n", - "\n", - "- Damianou, A. and Lawrence, N. D. (2013). *Deep Gaussian Processes*. In *AISTATS* (pp. 207\u2013215). \u2014 [PDF](https://proceedings.mlr.press/v31/damianou13a.html)\n", - "- Script counterpart: `examples/damianou2013/toy_warped_regression.py`\n" + "%matplotlib inline\n", + "argv = [\"--quick\"] if QUICK else []\n", + "argv += [\"--outdir\", \"/tmp/dgp2013_nb\"]\n", + "from toy_warped_regression import main\n", + "main(argv)\n" ] } ] -} \ No newline at end of file +} diff --git a/examples/damianou2013/notebooks/03_usps_digits.ipynb b/examples/damianou2013/notebooks/03_usps_digits.ipynb index 8ff4971..16f11b6 100644 --- a/examples/damianou2013/notebooks/03_usps_digits.ipynb +++ b/examples/damianou2013/notebooks/03_usps_digits.ipynb @@ -6,10 +6,6 @@ "display_name": "Python 3", "language": "python", "name": "python3" - }, - "language_info": { - "name": "python", - "pygments_lexer": "ipython3" } }, "cells": [ @@ -17,13 +13,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# USPS digits depth / ARD / 1-NN (Damianou & Lawrence 2013, \u00a74.3 / figs. 6\u20138)\n", + "# USPS digits \u2014 \u00a74.3 / figs. 6\u20138\n", "\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/03_usps_digits.ipynb)\n", "\n", - "**Paper:** [Damianou, A. and Lawrence, N. D. (2013). *Deep Gaussian Processes*. In *AISTATS* (pp. 207\u2013215).](https://proceedings.mlr.press/v31/damianou13a.html)\n", - "\n", - "Subsample 50 examples of each digit $\\{0,1,6\\}$ from USPS ($16\\times16$). Train unsupervised DeepGPs of increasing depth and evaluate (i) the variational bound and (ii) leave-one-out 1-NN error in the top latent features. The paper finds both the bound and nearest-neighbour quality improve with depth, with a 5-layer model nearly separating the classes in a low-dimensional top space.\n" + "**Matlab:** [`demUsps.m`](https://github.com/SheffieldML/deepGP/blob/master/deepGP/matlab/demUsps.m) / [`demDigitsDemonstration.m`](https://github.com/SheffieldML/deepGP/blob/master/deepGP/matlab/demDigitsDemonstration.m) (`vercat2`, H=5, Q={15,12,10,8,6}, K=80).\n" ] }, { @@ -32,36 +26,33 @@ "metadata": {}, "outputs": [], "source": [ - "# Install dependencies when running on Google Colab\n", - "import sys\n", - "import subprocess\n", - "\n", + "import sys, subprocess, os, urllib.request\n", "IN_COLAB = \"google.colab\" in sys.modules\n", "if IN_COLAB:\n", - " pkgs = [\"numpy>=1.7\", \"scipy>=0.12\", \"matplotlib\", \"GPy>=1.15\"]\n", - " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", *pkgs])\n", - " subprocess.check_call([\n", - " sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", - " \"git+https://github.com/SheffieldML/PyDeepGP.git@main\",\n", - " ])\n", - " print(\"Installed GPy and PyDeepGP for Colab.\")\n", + " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", + " \"numpy\", \"scipy\", \"matplotlib\", \"GPy>=1.15\", \"scikit-learn\"])\n", + " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", + " \"git+https://github.com/SheffieldML/PyDeepGP.git@examples-damianou2013\"])\n", + " os.makedirs(\"data\", exist_ok=True)\n", + " files = [\n", + " (\"_common.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_common.py\"),\n", + " (\"_matlab_data.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_matlab_data.py\"),\n", + " (\"toy_hierarchy.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/toy_hierarchy.py\"),\n", + " (\"toy_warped_regression.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/toy_warped_regression.py\"),\n", + " (\"usps_digits.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/usps_digits.py\"),\n", + " (\"highfive_shared.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/highfive_shared.py\"),\n", + " (\"data/hgplvmSampleDataTr1.mat\",\n", + " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/hgplvmSampleDataTr1.mat\"),\n", + " (\"data/demHighFiveHgplvm1.mat\",\n", + " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/demHighFiveHgplvm1.mat\"),\n", + " ]\n", + " for dest, url in files:\n", + " if not os.path.isfile(dest):\n", + " print(\"Fetching\", dest)\n", + " urllib.request.urlretrieve(url, dest)\n", + " print(\"Colab setup done.\")\n", "else:\n", - " print(\"Local environment \u2014 ensure PyDeepGP and GPy are importable.\")\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Set QUICK=True for a short smoke run (fewer iterations / smaller models).\n", - "# Set QUICK=False for a paper-scale demonstration (slower).\n", - "QUICK = True\n", - "SEED = 42\n", - "\n", - "DEPTHS = [1, 2] if QUICK else [1, 2, 3, 4, 5]\n", - "PER_CLASS = 20 if QUICK else 50\n" + " print(\"Local: run from repo; ensure examples/damianou2013 is on sys.path.\")\n" ] }, { @@ -70,67 +61,17 @@ "metadata": {}, "outputs": [], "source": [ - "import gzip\n", - "import os\n", - "import urllib.request\n", - "import numpy as np\n", - "import matplotlib.pyplot as plt\n", - "import GPy\n", - "import deepgp\n", - "\n", - "np.random.seed(SEED)\n", - "DIGITS = (0, 1, 6)\n", - "ESL_URL = \"https://web.stanford.edu/~hastie/ElemStatLearn/datasets/zip.train.gz\"\n", - "CACHE = os.path.join(\"/tmp\", \"pydeepgp_usps_zip.train.gz\")\n", - "\n", - "def download_usps(path=CACHE):\n", - " if os.path.isfile(path) and os.path.getsize(path) > 0:\n", - " return path\n", - " print(\"Downloading USPS zip.train \u2026\")\n", - " req = urllib.request.Request(ESL_URL, headers={\"User-Agent\": \"PyDeepGP-notebooks\"})\n", - " with urllib.request.urlopen(req, timeout=120) as resp, open(path, \"wb\") as out:\n", - " out.write(resp.read())\n", - " return path\n", - "\n", - "def load_usps(path):\n", - " rows = []\n", - " with gzip.open(path, \"rt\") as f:\n", - " for line in f:\n", - " parts = line.strip().split()\n", - " if parts:\n", - " rows.append([float(x) for x in parts])\n", - " data = np.asarray(rows)\n", - " return data[:, 1:], data[:, 0].astype(int)\n", - "\n", - "def subsample(Y, labels, per_class=50, seed=42):\n", - " rng = np.random.RandomState(seed)\n", - " Ys, Ls = [], []\n", - " for d in DIGITS:\n", - " idx = np.where(labels == d)[0]\n", - " choose = rng.choice(idx, size=per_class, replace=False)\n", - " Ys.append(Y[choose]); Ls.append(np.full(per_class, d))\n", - " return np.vstack(Ys), np.concatenate(Ls)\n", - "\n", - "def rbf_bias(q):\n", - " return GPy.kern.RBF(q, ARD=True) + GPy.kern.Bias(q)\n", - "\n", - "def ard_weights(kern):\n", - " ls = np.asarray(kern.rbf.lengthscale).ravel() if hasattr(kern, \"rbf\") else np.asarray(kern.lengthscale).ravel()\n", - " return 1.0 / np.maximum(ls, 1e-12)\n", - "\n", - "def nn_error(X, labels):\n", - " sq = np.sum(X * X, axis=1, keepdims=True)\n", - " d2 = sq + sq.T - 2 * X @ X.T\n", - " np.fill_diagonal(d2, np.inf)\n", - " nn = np.argmin(d2, axis=1)\n", - " mistakes = int(np.sum(labels[nn] != labels))\n", - " return mistakes, mistakes / len(labels)\n", - "\n", - "def depth_dims(depth, data_dim, quick):\n", - " widths = {1: [8], 2: [8, 5], 3: [8, 6, 4]} if quick else {\n", - " 1: [12], 2: [12, 8], 3: [12, 10, 6], 4: [12, 10, 8, 5], 5: [12, 10, 8, 6, 4]\n", - " }\n", - " return [data_dim] + widths[depth]\n" + "import os, sys\n", + "from pathlib import Path\n", + "here = Path.cwd().resolve()\n", + "candidates = [here, here.parent, here / \"examples\" / \"damianou2013\",\n", + " here.parent / \"examples\" / \"damianou2013\"]\n", + "for c in candidates:\n", + " if (c / \"toy_hierarchy.py\").is_file():\n", + " sys.path.insert(0, str(c))\n", + " os.chdir(c)\n", + " break\n", + "print(\"cwd:\", os.getcwd())\n" ] }, { @@ -139,27 +80,7 @@ "metadata": {}, "outputs": [], "source": [ - "path = download_usps()\n", - "Y_all, lab_all = load_usps(path)\n", - "Y, labels = subsample(Y_all, lab_all, per_class=PER_CLASS, seed=SEED)\n", - "Y = (Y - Y.mean(0)) / (Y.std() + 1e-12)\n", - "print(\"Subset\", Y.shape, \"classes\", sorted(set(labels.tolist())))\n", - "\n", - "# Show a few digits\n", - "fig, axes = plt.subplots(1, 6, figsize=(8, 1.6))\n", - "for ax, i in zip(axes, range(6)):\n", - " ax.imshow(Y[i].reshape(16, 16), cmap=\"gray\")\n", - " ax.set_title(str(labels[i]))\n", - " ax.axis(\"off\")\n", - "plt.suptitle(\"USPS subset samples\")\n", - "plt.show()\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Fit depths and score bound / 1-NN\n" + "QUICK = True # depth 1\u20132 only; False runs depths 1\u20135 (slow)\n" ] }, { @@ -168,108 +89,12 @@ "metadata": {}, "outputs": [], "source": [ - "from GPy.core.parameterization.variational import NormalPosterior, VariationalPosterior\n", - "\n", - "def out_var(y):\n", - " if isinstance(y, (NormalPosterior, VariationalPosterior)):\n", - " return float(np.asarray(y.mean).var())\n", - " return float(np.asarray(y).var())\n", - "\n", - "num_inducing = 20 if QUICK else 40\n", - "iters_fixed, iters_free = (40, 100) if QUICK else (200, 2000)\n", - "\n", - "results, models = [], {}\n", - "for depth in DEPTHS:\n", - " print(f\"=== depth {depth} ===\")\n", - " dims = depth_dims(depth, Y.shape[1], QUICK)\n", - " m = deepgp.DeepGP(\n", - " dims, Y,\n", - " kernels=[rbf_bias(q) for q in dims[1:]],\n", - " num_inducing=num_inducing,\n", - " back_constraint=False,\n", - " )\n", - " for layer in m.layers:\n", - " layer.Gaussian_noise.variance = out_var(layer.Y) * 0.01\n", - " layer.Gaussian_noise.variance.fix()\n", - " m.optimize(max_iters=iters_fixed, messages=False)\n", - " for layer in m.layers:\n", - " layer.Gaussian_noise.variance.unfix()\n", - " m.optimize(max_iters=iters_free, messages=False)\n", - " bound = float(m.log_likelihood())\n", - " Xtop = np.asarray(m.layers[-1].X.mean)\n", - " mistakes, rate = nn_error(Xtop, labels)\n", - " print(f\"bound={bound:.3f} 1-NN mistakes={mistakes}/{Y.shape[0]} ({rate:.3%})\")\n", - " results.append((depth, bound, mistakes, rate))\n", - " models[depth] = m\n", - "\n", - "print(\"\\ndepth bound 1-NN errors\")\n", - "for depth, bound, mistakes, rate in results:\n", - " print(f\"{depth:5d} {bound:9.3f} {mistakes:3d} ({rate:.3%})\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Visualisations (figs. 6\u20137 style)\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "deepest = max(DEPTHS)\n", - "m = models[deepest]\n", - "Xtop = np.asarray(m.layers[-1].X.mean)\n", - "w = ard_weights(m.layers[-1].kern)\n", - "dims = list(np.argsort(-w)[:2])\n", - "colors = {0: \"C0\", 1: \"C1\", 6: \"C2\"}\n", - "markers = {0: \"o\", 1: \"s\", 6: \"^\"}\n", - "\n", - "fig, ax = plt.subplots(figsize=(4.5, 4.5))\n", - "for d in DIGITS:\n", - " mask = labels == d\n", - " ax.scatter(Xtop[mask, dims[0]], Xtop[mask, dims[1]],\n", - " c=colors[d], marker=markers[d], s=28, label=str(d))\n", - "ax.set_xlabel(f\"top dim {dims[0]}\")\n", - "ax.set_ylabel(f\"top dim {dims[1]}\")\n", - "ax.set_title(f\"Top latent (depth {deepest})\")\n", - "ax.legend()\n", - "plt.show()\n", - "\n", - "fig, axes = plt.subplots(1, len(m.layers), figsize=(3.0 * len(m.layers), 2.8))\n", - "if len(m.layers) == 1:\n", - " axes = [axes]\n", - "for i, layer in enumerate(m.layers):\n", - " ww = ard_weights(layer.kern)\n", - " axes[i].bar(np.arange(1, len(ww) + 1), ww)\n", - " axes[i].set_title(f\"layer {i} ARD\")\n", - "fig.suptitle(\"ARD weights (1/lengthscale) \u2014 fig. 6 style\")\n", - "fig.tight_layout()\n", - "plt.show()\n", - "\n", - "fig, ax = plt.subplots(figsize=(5.5, 3.5))\n", - "ax.plot([r[0] for r in results], [r[1] for r in results], \"o-\", label=\"variational bound\")\n", - "ax.set_xlabel(\"depth\"); ax.set_ylabel(\"bound\")\n", - "ax2 = ax.twinx()\n", - "ax2.plot([r[0] for r in results], [r[2] for r in results], \"s--\", color=\"C1\", label=\"1-NN mistakes\")\n", - "ax2.set_ylabel(\"1-NN mistakes\")\n", - "ax.set_title(\"Depth selection via bound / 1-NN\")\n", - "plt.show()\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## References\n", - "\n", - "- Damianou, A. and Lawrence, N. D. (2013). *Deep Gaussian Processes*. In *AISTATS* (pp. 207\u2013215). \u2014 [PDF](https://proceedings.mlr.press/v31/damianou13a.html)\n", - "- USPS via ElemStatLearn `zip.train`\n", - "- Script counterpart: `examples/damianou2013/usps_digits.py`\n" + "%matplotlib inline\n", + "argv = [\"--quick\", \"--depths\", \"1\", \"2\"] if QUICK else []\n", + "argv += [\"--outdir\", \"/tmp/dgp2013_nb\"]\n", + "from usps_digits import main\n", + "main(argv)\n" ] } ] -} \ No newline at end of file +} diff --git a/examples/damianou2013/notebooks/04_highfive_mrd.ipynb b/examples/damianou2013/notebooks/04_highfive_mrd.ipynb index 2bddc26..27179af 100644 --- a/examples/damianou2013/notebooks/04_highfive_mrd.ipynb +++ b/examples/damianou2013/notebooks/04_highfive_mrd.ipynb @@ -6,10 +6,6 @@ "display_name": "Python 3", "language": "python", "name": "python3" - }, - "language_info": { - "name": "python", - "pygments_lexer": "ipython3" } }, "cells": [ @@ -17,15 +13,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Shared-view / high-five MRD (Damianou & Lawrence 2013, \u00a74.2 / figs. 4\u20135)\n", + "# High-five MRD \u2014 \u00a74.2 / figs. 4\u20135\n", "\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/04_highfive_mrd.ipynb)\n", "\n", - "**Paper:** [Damianou, A. and Lawrence, N. D. (2013). *Deep Gaussian Processes*. In *AISTATS* (pp. 207\u2013215).](https://proceedings.mlr.press/v31/damianou13a.html)\n", - "\n", - "Two conditionally independent observation views share a parent latent layer (manifold relevance determination / shared GP-LVM style). In the paper this models two CMU MOCAP subjects performing a high-five; ARD on each view reveals shared vs private dimensions (fig. 4).\n", - "\n", - "This notebook uses **synthetic correlated views** by default (reliable in Colab). Optionally try a pods MOCAP stand-in. Multi-view DeepGP requires CIP-0002 MRD support.\n" + "**Matlab:** [`demHighFiveDemonstration.m`](https://github.com/SheffieldML/deepGP/blob/master/deepGP/matlab/demHighFiveDemonstration.m) + `demHighFiveHgplvm1.mat` (YA/YB 78\u00d762).\n" ] }, { @@ -34,21 +26,33 @@ "metadata": {}, "outputs": [], "source": [ - "# Install dependencies when running on Google Colab\n", - "import sys\n", - "import subprocess\n", - "\n", + "import sys, subprocess, os, urllib.request\n", "IN_COLAB = \"google.colab\" in sys.modules\n", "if IN_COLAB:\n", - " pkgs = [\"numpy>=1.7\", \"scipy>=0.12\", \"matplotlib\", \"GPy>=1.15\"]\n", - " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", *pkgs])\n", - " subprocess.check_call([\n", - " sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", - " \"git+https://github.com/SheffieldML/PyDeepGP.git@main\",\n", - " ])\n", - " print(\"Installed GPy and PyDeepGP for Colab.\")\n", + " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", + " \"numpy\", \"scipy\", \"matplotlib\", \"GPy>=1.15\", \"scikit-learn\"])\n", + " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", + " \"git+https://github.com/SheffieldML/PyDeepGP.git@examples-damianou2013\"])\n", + " os.makedirs(\"data\", exist_ok=True)\n", + " files = [\n", + " (\"_common.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_common.py\"),\n", + " (\"_matlab_data.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_matlab_data.py\"),\n", + " (\"toy_hierarchy.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/toy_hierarchy.py\"),\n", + " (\"toy_warped_regression.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/toy_warped_regression.py\"),\n", + " (\"usps_digits.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/usps_digits.py\"),\n", + " (\"highfive_shared.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/highfive_shared.py\"),\n", + " (\"data/hgplvmSampleDataTr1.mat\",\n", + " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/hgplvmSampleDataTr1.mat\"),\n", + " (\"data/demHighFiveHgplvm1.mat\",\n", + " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/demHighFiveHgplvm1.mat\"),\n", + " ]\n", + " for dest, url in files:\n", + " if not os.path.isfile(dest):\n", + " print(\"Fetching\", dest)\n", + " urllib.request.urlretrieve(url, dest)\n", + " print(\"Colab setup done.\")\n", "else:\n", - " print(\"Local environment \u2014 ensure PyDeepGP and GPy are importable.\")\n" + " print(\"Local: run from repo; ensure examples/damianou2013 is on sys.path.\")\n" ] }, { @@ -57,12 +61,17 @@ "metadata": {}, "outputs": [], "source": [ - "# Set QUICK=True for a short smoke run (fewer iterations / smaller models).\n", - "# Set QUICK=False for a paper-scale demonstration (slower).\n", - "QUICK = True\n", - "SEED = 42\n", - "\n", - "USE_PODS = False # set True to try pods CMU walk/jog feature split\n" + "import os, sys\n", + "from pathlib import Path\n", + "here = Path.cwd().resolve()\n", + "candidates = [here, here.parent, here / \"examples\" / \"damianou2013\",\n", + " here.parent / \"examples\" / \"damianou2013\"]\n", + "for c in candidates:\n", + " if (c / \"toy_hierarchy.py\").is_file():\n", + " sys.path.insert(0, str(c))\n", + " os.chdir(c)\n", + " break\n", + "print(\"cwd:\", os.getcwd())\n" ] }, { @@ -71,37 +80,7 @@ "metadata": {}, "outputs": [], "source": [ - "import numpy as np\n", - "import matplotlib.pyplot as plt\n", - "import GPy\n", - "import deepgp\n", - "\n", - "np.random.seed(SEED)\n", - "\n", - "def rbf_bias(q, ard=True):\n", - " return GPy.kern.RBF(q, ARD=ard) + GPy.kern.Bias(q)\n", - "\n", - "def ard_weights(kern):\n", - " ls = np.asarray(kern.rbf.lengthscale).ravel() if hasattr(kern, \"rbf\") else np.asarray(kern.lengthscale).ravel()\n", - " return 1.0 / np.maximum(ls, 1e-12)\n", - "\n", - "def load_views(seed=42, use_pods=False):\n", - " if use_pods:\n", - " try:\n", - " import pods\n", - " data = pods.datasets.cmu_mocap_35_walk_jog()\n", - " Y = np.asarray(data[\"Y\"], dtype=float)[:80]\n", - " mid = Y.shape[1] // 2\n", - " print(\"Using pods walk/jog feature split as a two-view stand-in.\")\n", - " return Y[:, :mid], Y[:, mid:], \"pods\"\n", - " except Exception as exc:\n", - " print(\"pods unavailable:\", exc)\n", - " rng = np.random.RandomState(seed)\n", - " n, q, d = 60, 3, 20\n", - " X = rng.randn(n, q)\n", - " Y1 = X @ rng.randn(q, d) + 0.05 * rng.randn(n, d)\n", - " Y2 = X @ rng.randn(q, d) + 0.05 * rng.randn(n, d)\n", - " return Y1, Y2, \"synthetic\"\n" + "QUICK = True\n" ] }, { @@ -110,115 +89,12 @@ "metadata": {}, "outputs": [], "source": [ - "Y1, Y2, source = load_views(SEED, use_pods=USE_PODS)\n", - "Y1 = (Y1 - Y1.mean(0)) / (Y1.std() + 1e-12)\n", - "Y2 = (Y2 - Y2.mean(0)) / (Y2.std() + 1e-12)\n", - "print(f\"Views {Y1.shape}, {Y2.shape} (source={source})\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Fit shared-hierarchy DeepGP\n", - "\n", - "Observation kernels are a **list of two kernels** (one per view), each with input dimension equal to the shared latent width.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from GPy.core.parameterization.variational import NormalPosterior, VariationalPosterior\n", - "\n", - "def out_var(y):\n", - " if isinstance(y, (NormalPosterior, VariationalPosterior)):\n", - " return float(np.asarray(y.mean).var())\n", - " return float(np.asarray(y).var())\n", - "\n", - "q1, q2 = (4, 3) if QUICK else (6, 4)\n", - "M = 15 if QUICK else 30\n", - "it_f, it_u = (40, 80) if QUICK else (200, 800)\n", - "\n", - "m = deepgp.DeepGP(\n", - " [[Y1.shape[1], Y2.shape[1]], q1, q2],\n", - " [Y1, Y2],\n", - " kernels=[[rbf_bias(q1), rbf_bias(q1)], rbf_bias(q2)],\n", - " num_inducing=M,\n", - " back_constraint=False,\n", - ")\n", - "for layer in m.layers:\n", - " if hasattr(layer, \"views\"):\n", - " for view in layer.views:\n", - " view.Gaussian_noise.variance = out_var(view.Y) * 0.01\n", - " view.Gaussian_noise.variance.fix()\n", - " else:\n", - " layer.Gaussian_noise.variance = out_var(layer.Y) * 0.01\n", - " layer.Gaussian_noise.variance.fix()\n", - "m.optimize(max_iters=it_f, messages=False)\n", - "for layer in m.layers:\n", - " if hasattr(layer, \"views\"):\n", - " for view in layer.views:\n", - " view.Gaussian_noise.variance.unfix()\n", - " else:\n", - " layer.Gaussian_noise.variance.unfix()\n", - "m.optimize(max_iters=it_u, messages=True)\n", - "print(f\"Variational bound: {float(m.log_likelihood()):.3f}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## ARD per view and top layer (fig. 4 style)\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "fig, axes = plt.subplots(1, 3, figsize=(10, 3.2))\n", - "for i, view in enumerate(m.layers[0].views):\n", - " w = ard_weights(view.kern)\n", - " axes[i].bar(np.arange(1, len(w) + 1), w)\n", - " axes[i].set_title(f\"view {i} ARD\")\n", - " axes[i].set_xlabel(\"shared dim\")\n", - " print(f\"View {i} ARD:\", np.array2string(w, precision=3))\n", - "w_top = ard_weights(m.layers[-1].kern)\n", - "axes[2].bar(np.arange(1, len(w_top) + 1), w_top)\n", - "axes[2].set_title(\"top layer ARD\")\n", - "fig.suptitle(\"Shared MRD ARD weights (fig. 4 style)\")\n", - "fig.tight_layout()\n", - "plt.show()\n", - "\n", - "# 2D projection of dominant shared dimensions from view 0's posterior mean\n", - "Xshare = np.asarray(m.layers[0].X.mean)\n", - "w0 = ard_weights(m.layers[0].views[0].kern)\n", - "dims = list(np.argsort(-w0)[:2])\n", - "fig, ax = plt.subplots(figsize=(4.5, 4))\n", - "ax.plot(Xshare[:, dims[0]], Xshare[:, dims[1]], \"o-\", markersize=4)\n", - "ax.set_xlabel(f\"shared dim {dims[0]}\")\n", - "ax.set_ylabel(f\"shared dim {dims[1]}\")\n", - "ax.set_title(\"Shared latent trajectory (fig. 5 style)\")\n", - "plt.show()\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## References\n", - "\n", - "- Damianou, A. and Lawrence, N. D. (2013). *Deep Gaussian Processes*. In *AISTATS* (pp. 207\u2013215). \u2014 [PDF](https://proceedings.mlr.press/v31/damianou13a.html), \u00a74.2\n", - "- Damianou et al. (2012), shared / MRD GP-LVM\n", - "- Lawrence & Moore (2007), hierarchical GP-LVM (high-five MOCAP)\n", - "- Scripts: `examples/damianou2013/highfive_shared.py`, `examples/example_mrd.py`\n", - "- CIP-0002: MRD revival on the maintained DeepGP path\n" + "%matplotlib inline\n", + "argv = [\"--quick\"] if QUICK else []\n", + "argv += [\"--outdir\", \"/tmp/dgp2013_nb\"]\n", + "from highfive_shared import main\n", + "main(argv)\n" ] } ] -} \ No newline at end of file +} diff --git a/examples/damianou2013/notebooks/README.md b/examples/damianou2013/notebooks/README.md index 494d1b3..64962c6 100644 --- a/examples/damianou2013/notebooks/README.md +++ b/examples/damianou2013/notebooks/README.md @@ -1,19 +1,6 @@ -# Damianou & Lawrence 2013 — Colab notebooks +# Colab notebooks (Matlab-aligned) -Interactive recreations of the AISTATS 2013 Deep GP experiments. +Thin wrappers around the Python ports of [SheffieldML/deepGP](https://github.com/SheffieldML/deepGP) demos. +Each notebook installs deps on Colab, fetches scripts / reference `.mat` files, and calls the corresponding module. -| Notebook | Paper | Colab | -| --- | --- | --- | -| [01_toy_hierarchy.ipynb](01_toy_hierarchy.ipynb) | §4.1 fig. 2 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb) | -| [02_toy_warped_regression.ipynb](02_toy_warped_regression.ipynb) | §4.1 fig. 3 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb) | -| [03_usps_digits.ipynb](03_usps_digits.ipynb) | §4.3 figs. 6–8 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/03_usps_digits.ipynb) | -| [04_highfive_mrd.ipynb](04_highfive_mrd.ipynb) | §4.2 figs. 4–5 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/04_highfive_mrd.ipynb) | - -Each notebook: - -1. Shows an **Open in Colab** badge at the top. -2. Installs GPy + PyDeepGP automatically when `google.colab` is detected. -3. Uses `QUICK = True` by default for a short run; set `False` for paper-scale settings. -4. Cites Damianou & Lawrence (AISTATS 2013) and points at the matching `.py` script. - -Paper: https://proceedings.mlr.press/v31/damianou13a.html +See [../SOURCE.md](../SOURCE.md) and [../README.md](../README.md). diff --git a/examples/damianou2013/toy_hierarchy.py b/examples/damianou2013/toy_hierarchy.py index ab5548f..be64949 100644 --- a/examples/damianou2013/toy_hierarchy.py +++ b/examples/damianou2013/toy_hierarchy.py @@ -1,10 +1,9 @@ #!/usr/bin/env python -"""Section 4.1 (fig. 2): unsupervised recovery of a stacked-GP hierarchy. +"""§4.1 / fig. 2 — unsupervised hierarchy (Matlab ``demToyUnsupervised.m``). -Paper: Damianou & Lawrence, Deep Gaussian Processes, AISTATS 2013. - -Generate a three-level stack (top latent → 2D intermediate → 10D observations), -fit a DeepGP with overcomplete ARD widths, and contrast against stacked PCA. +Loads ``hgplvmSampleDataTr1.mat`` (top ``X2`` → intermediates ``XA``,``XB`` → +leaves ``YA``,``YB``), fits a two-view DeepGP with latent widths Q={6,4} +matching the Matlab demo, and compares against stacked PCA and Isomap. """ from __future__ import annotations @@ -15,7 +14,6 @@ import numpy as np -import GPy import deepgp from _common import ( @@ -24,54 +22,41 @@ dominant_dims, ensure_outdir, optimize_staged, - print_ard, rbf_bias, - sample_gp_prior, save_or_show, ) +from _matlab_data import load_toy_hierarchy -def _pca_project(Y: np.ndarray, q: int) -> np.ndarray: - try: - from sklearn.decomposition import PCA - - return PCA(n_components=q).fit_transform(Y) - except Exception: +def stacked_pca(Ya: np.ndarray, Yb: np.ndarray, q1: int, q2: int): + def pca(Y, q): Yc = Y - Y.mean(0) U, S, _ = np.linalg.svd(Yc, full_matrices=False) q = min(q, U.shape[1]) return U[:, :q] * S[:q] + xa, xb = pca(Ya, q1), pca(Yb, q1) + x2 = pca(np.hstack([xa, xb]), q2) + return xa, xb, x2 -def stacked_pca(Y: np.ndarray, dims: list[int]) -> list[np.ndarray]: - """Greedy PCA stack returning latent matrices from bottom to top.""" - spaces = [] - cur = Y - for q in dims: - cur = _pca_project(cur, q) - spaces.append(cur) - return spaces +def stacked_isomap(Ya: np.ndarray, Yb: np.ndarray, q1: int, q2: int, n_neighbors: int = 5): + try: + from sklearn.manifold import Isomap + except ImportError as exc: + raise SystemExit("stacked Isomap requires scikit-learn") from exc -def generate_hierarchy(n: int, seed: int) -> tuple[np.ndarray, np.ndarray, np.ndarray]: - """Return (Z_top, X_mid, Y_obs) for a 1→2→10 stacked GP.""" - rng = np.random.RandomState(seed) - np.random.seed(seed) - - Z = np.linspace(-1.0, 1.0, n)[:, None] - kern_mid = GPy.kern.RBF(1, lengthscale=0.4, variance=1.0) - X = sample_gp_prior(Z, kern_mid, output_dim=2) - X += 0.02 * rng.randn(*X.shape) - kern_obs = GPy.kern.RBF(2, lengthscale=0.8, variance=1.0, ARD=False) - Y = sample_gp_prior(X, kern_obs, output_dim=10) - Y += 0.05 * rng.randn(*Y.shape) - return Z, X, Y + xa = Isomap(n_components=min(q1, Ya.shape[0] - 1), n_neighbors=n_neighbors).fit_transform(Ya) + xb = Isomap(n_components=min(q1, Yb.shape[0] - 1), n_neighbors=n_neighbors).fit_transform(Yb) + x2 = Isomap( + n_components=min(q2, xa.shape[0] - 1), n_neighbors=n_neighbors + ).fit_transform(np.hstack([xa, xb])) + return xa, xb, x2 def build_parser() -> argparse.ArgumentParser: p = argparse.ArgumentParser(description=__doc__) add_common_args(p) - p.add_argument("--n", type=int, default=80, help="Number of observations.") p.add_argument("--num-inducing", type=int, default=25) return p @@ -79,64 +64,111 @@ def build_parser() -> argparse.ArgumentParser: def main(argv: list[str] | None = None) -> int: args = build_parser().parse_args(argv) outdir = ensure_outdir(args.outdir) - n = 40 if args.quick else args.n - q_mid, q_top = (4, 3) if args.quick else (5, 4) - iters_fixed = 50 if args.quick else 200 - iters_free = 150 if args.quick else 1500 - num_inducing = 15 if args.quick else args.num_inducing - Z, X_true, Y = generate_hierarchy(n, args.seed) - print(f"Generated hierarchy: Y={Y.shape}, true mid={X_true.shape}, true top={Z.shape}") + data = load_toy_hierarchy() + YA, YB = data["YA"], data["YB"] + X2, XA, XB = data["X2"], data["XA"], data["XB"] + print( + f"Loaded hgplvmSampleDataTr1: YA={YA.shape} YB={YB.shape} " + f"XA={XA.shape} XB={XB.shape} X2={X2.shape}" + ) + + # Matlab demToyUnsupervised: Q = {6,4} + q_shared, q_top = (4, 3) if args.quick else (6, 4) + num_inducing = 15 if args.quick else args.num_inducing + iters_fixed = 50 if args.quick else 200 + iters_free = 150 if args.quick else 2000 - kernels = [rbf_bias(q_mid, ard=True), rbf_bias(q_top, ard=True)] + kernels = [ + [rbf_bias(q_shared), rbf_bias(q_shared)], + rbf_bias(q_top), + ] m = deepgp.DeepGP( - [Y.shape[1], q_mid, q_top], - Y, + [[YA.shape[1], YB.shape[1]], q_shared, q_top], + [YA, YB], kernels=kernels, num_inducing=num_inducing, back_constraint=False, + inits="PCA", ) optimize_staged(m, iters_fixed, iters_free, messages=args.messages) print(f"DeepGP variational bound: {float(m.log_likelihood()):.3f}") - print_ard(m) + for i, view in enumerate(m.layers[0].views): + print(f"View {i} ARD: {np.array2string(ard_weights(view.kern), precision=3)}") + print(f"Top ARD: {np.array2string(ard_weights(m.layers[-1].kern), precision=3)}") - pca_spaces = stacked_pca(Y, [q_mid, q_top]) + pca_xa, pca_xb, pca_x2 = stacked_pca(YA, YB, q_shared, q_top) + try: + iso_xa, iso_xb, iso_x2 = stacked_isomap(YA, YB, min(2, q_shared), min(2, q_top)) + have_iso = True + except SystemExit: + have_iso = False + print("sklearn not available — skipping Isomap comparison") if not args.no_plot or outdir: import matplotlib.pyplot as plt - fig, axes = plt.subplots(1, 3, figsize=(10, 3.2)) - t = np.arange(n) - axes[0].plot(t, Z[:, 0], "k-", label="top") - axes[0].plot(t, X_true[:, 0], "C0-", label="mid0") - axes[0].plot(t, X_true[:, 1], "C1-", label="mid1") - axes[0].set_title("Ground truth") - axes[0].legend(fontsize=8) - - for ax, layer_idx, title in ( - (axes[1], 0, "DeepGP layer 0 (near data)"), - (axes[2], 1, "DeepGP layer 1 (top)"), - ): - Xmean = m.layers[layer_idx].X.mean - dims = dominant_dims(ard_weights(m.layers[layer_idx].kern), k=min(2, Xmean.shape[1])) - for d in dims: - ax.plot(t, Xmean[:, d], label=f"dim {d}") - ax.set_title(title) - ax.legend(fontsize=8) - - fig.suptitle("Damianou & Lawrence 2013 — toy hierarchy (fig. 2 style)") + t = np.arange(YA.shape[0]) + fig, axes = plt.subplots(3, 2, figsize=(9, 8)) + axes[0, 0].plot(X2) + axes[0, 0].set_title("True top X2") + axes[0, 1].axis("off") + axes[1, 0].plot(XA) + axes[1, 0].set_title("True XA") + axes[1, 1].plot(XB) + axes[1, 1].set_title("True XB") + axes[2, 0].plot(YA) + axes[2, 0].set_title("YA") + axes[2, 1].plot(YB) + axes[2, 1].set_title("YB") + fig.suptitle("Matlab hgplvmSampleDataTr1 ground truth") fig.tight_layout() - save_or_show(fig, outdir, "toy_hierarchy.png", args.no_plot) - - fig2, ax = plt.subplots(figsize=(5, 3)) - ax.plot(t, pca_spaces[0][:, 0], label="PCA mid 0") - if pca_spaces[0].shape[1] > 1: - ax.plot(t, pca_spaces[0][:, 1], label="PCA mid 1") - ax.plot(t, pca_spaces[1][:, 0], label="PCA top 0") - ax.set_title("Stacked PCA (greedy)") - ax.legend(fontsize=8) + save_or_show(fig, outdir, "toy_hierarchy_truth.png", args.no_plot) + + Xshare = np.asarray(m.layers[0].X.mean) + Xtop = np.asarray(m.layers[-1].X.mean) + d_top = dominant_dims(ard_weights(m.layers[-1].kern), k=min(2, Xtop.shape[1])) + w0 = ard_weights(m.layers[0].views[0].kern) + w1 = ard_weights(m.layers[0].views[1].kern) + d_a = dominant_dims(w0, k=min(2, Xshare.shape[1])) + d_b = dominant_dims(w1, k=min(2, Xshare.shape[1])) + + fig2, axes = plt.subplots(2, 2, figsize=(8, 7)) + axes[0, 0].plot(Xtop[:, d_top]) + axes[0, 0].set_title("DeepGP top (dominant dims)") + axes[0, 1].axis("off") + axes[1, 0].plot(Xshare[:, d_a]) + axes[1, 0].set_title("DeepGP shared (view0 ARD dims)") + axes[1, 1].plot(Xshare[:, d_b]) + axes[1, 1].set_title("DeepGP shared (view1 ARD dims)") + fig2.suptitle("demToyUnsupervised-style DeepGP recovery") fig2.tight_layout() - save_or_show(fig2, outdir, "toy_hierarchy_pca.png", args.no_plot) + save_or_show(fig2, outdir, "toy_hierarchy_deepgp.png", args.no_plot) + + fig3, axes = plt.subplots(2, 2, figsize=(8, 7)) + axes[0, 0].plot(pca_x2[:, : min(2, pca_x2.shape[1])]) + axes[0, 0].set_title("Stacked PCA top") + axes[1, 0].plot(pca_xa[:, : min(2, pca_xa.shape[1])]) + axes[1, 0].set_title("Stacked PCA XA") + axes[1, 1].plot(pca_xb[:, : min(2, pca_xb.shape[1])]) + axes[1, 1].set_title("Stacked PCA XB") + axes[0, 1].axis("off") + fig3.suptitle("Stacked PCA") + fig3.tight_layout() + save_or_show(fig3, outdir, "toy_hierarchy_pca.png", args.no_plot) + + if have_iso: + fig4, axes = plt.subplots(2, 2, figsize=(8, 7)) + axes[0, 0].plot(iso_x2) + axes[0, 0].set_title("Stacked Isomap top") + axes[1, 0].plot(iso_xa) + axes[1, 0].set_title("Stacked Isomap XA") + axes[1, 1].plot(iso_xb) + axes[1, 1].set_title("Stacked Isomap XB") + axes[0, 1].axis("off") + fig4.suptitle("Stacked Isomap") + fig4.tight_layout() + save_or_show(fig4, outdir, "toy_hierarchy_isomap.png", args.no_plot) return 0 diff --git a/examples/damianou2013/toy_warped_regression.py b/examples/damianou2013/toy_warped_regression.py index 0ef8a97..e8a8c92 100644 --- a/examples/damianou2013/toy_warped_regression.py +++ b/examples/damianou2013/toy_warped_regression.py @@ -1,11 +1,9 @@ #!/usr/bin/env python -"""Section 4.1 (fig. 3): warped toy regression — DeepGP vs sparse GP. +"""§4.1 / fig. 3 — warped regression (Matlab ``demToyRegressionSimple.m``). -Paper: Damianou & Lawrence, Deep Gaussian Processes, AISTATS 2013. - -Stack Linear+RBF → RBF to make a non-stationary 10D regression problem from a -1D input. Train on 25 points; compare MSE of a standard sparse GP against a -DeepGP that is one hidden layer deeper than the generative process. +Data: ``hierGpsNEW`` in ``demToyDynamicsCreateData.m`` (RBF+Linear warp → RBF). +Defaults follow the Matlab simple demo (Ntoy=150, D=15, Ntr=25, H=2, Q=6, K=10). +Use ``--paper`` for the AISTATS text settings (N=120, D=10). """ from __future__ import annotations @@ -25,55 +23,30 @@ mse, optimize_staged, rbf_bias, - sample_gp_prior, save_or_show, ) +from _matlab_data import generate_hier_gps_new, train_test_split_indices -def generate_warped( - n: int = 120, - seed: int = 0, -) -> tuple[np.ndarray, np.ndarray, np.ndarray]: - """Return (X, H_hidden, Y) for the paper's warped process.""" - rng = np.random.RandomState(seed) - np.random.seed(seed) - X = np.linspace(-1.0, 1.0, n)[:, None] - kern_warp = GPy.kern.Linear(1) + GPy.kern.RBF(1, lengthscale=0.3, variance=1.0) - H = sample_gp_prior(X, kern_warp, output_dim=1) - H += 0.01 * rng.randn(*H.shape) - kern_obs = GPy.kern.RBF(1, lengthscale=0.25, variance=1.0) - Y = sample_gp_prior(H, kern_obs, output_dim=10) - Y += 0.05 * rng.randn(*Y.shape) - return X, H, Y - - -def train_test_split( - X: np.ndarray, - Y: np.ndarray, - n_train: int, - seed: int, -) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]: - rng = np.random.RandomState(seed) - idx = rng.permutation(X.shape[0]) - tr, te = np.sort(idx[:n_train]), np.sort(idx[n_train:]) - return X[tr], Y[tr], X[te], Y[te] - - -def fit_deepgp( - Xtr: np.ndarray, - Ytr: np.ndarray, - q1: int, - q2: int, - num_inducing: int, - iters_fixed: int, - iters_free: int, - messages: bool, -) -> deepgp.DeepGP: - # Two hidden layers between X and Y (paper: one more than the generative depth) - dims = [Ytr.shape[1], q1, q2, Xtr.shape[1]] +def fit_gp(Xtr, Ytr, num_inducing, messages): + kern = GPy.kern.RBF(Xtr.shape[1]) + GPy.kern.Bias(Xtr.shape[1]) + m = GPy.models.SparseGPRegression(Xtr, Ytr, kernel=kern, num_inducing=num_inducing) + m.Gaussian_noise.variance = Ytr.var() * 0.01 + m.Gaussian_noise.variance.fix() + m.optimize(max_iters=100, messages=messages) + m.Gaussian_noise.variance.unfix() + m.optimize(max_iters=400, messages=messages) + return m + + +def fit_dgp(Xtr, Ytr, q, num_inducing, iters_fixed, iters_free, messages): + # H=2 hidden layers between observed X and Y (Matlab simple / paper) + dims = [Ytr.shape[1], q, q, Xtr.shape[1]] kernels = [ - rbf_bias(q1, ard=True), - rbf_bias(q2, ard=True), + rbf_bias(q, ard=True), + rbf_bias(q, ard=True), + # Top mapping sees inputs; Matlab uses lin+white+bias ("dynamics") — + # approximate with RBF+Bias on the observed X layer. rbf_bias(Xtr.shape[1], ard=False), ] m = deepgp.DeepGP( @@ -88,28 +61,15 @@ def fit_deepgp( return m -def fit_gp( - Xtr: np.ndarray, - Ytr: np.ndarray, - num_inducing: int, - messages: bool, -) -> GPy.core.Model: - kern = GPy.kern.RBF(Xtr.shape[1]) + GPy.kern.Bias(Xtr.shape[1]) - m = GPy.models.SparseGPRegression(Xtr, Ytr, kernel=kern, num_inducing=num_inducing) - m.Gaussian_noise.variance = Ytr.var() * 0.01 - m.Gaussian_noise.variance.fix() - m.optimize(max_iters=100, messages=messages) - m.Gaussian_noise.variance.unfix() - m.optimize(max_iters=400, messages=messages) - return m - - def build_parser() -> argparse.ArgumentParser: p = argparse.ArgumentParser(description=__doc__) add_common_args(p) - p.add_argument("--repeats", type=int, default=10, help="Paper uses 10 repeats.") - p.add_argument("--n-train", type=int, default=25) - p.add_argument("--num-inducing", type=int, default=20) + p.add_argument( + "--paper", + action="store_true", + help="Use AISTATS text sizes (N=120, D=10) instead of Matlab simple (150, 15).", + ) + p.add_argument("--repeats", type=int, default=10) return p @@ -117,38 +77,37 @@ def main(argv: list[str] | None = None) -> int: args = build_parser().parse_args(argv) outdir = ensure_outdir(args.outdir) + if args.paper: + n_toy, d_toy = 120, 10 + else: + n_toy, d_toy = 150, 15 + n_tr = 25 + q = 3 if args.quick else 6 + num_inducing = 8 if args.quick else 10 repeats = 2 if args.quick else args.repeats - n_train = 20 if args.quick else args.n_train - q1, q2 = (3, 2) if args.quick else (4, 3) - iters_fixed = 40 if args.quick else 200 - iters_free = 120 if args.quick else 1200 - num_inducing = 12 if args.quick else args.num_inducing + iters_fixed = 40 if args.quick else 500 + iters_free = 120 if args.quick else 2000 - gp_mses = [] - dgp_mses = [] + print( + f"Config: Ntoy={n_toy} D={d_toy} Ntr={n_tr} Q={q} K={num_inducing} " + f"repeats={repeats} ({'paper' if args.paper else 'matlab-simple'})" + ) + gp_mses, dgp_mses = [], [] for r in range(repeats): seed = args.seed + r - X, H, Y = generate_warped(n=120, seed=seed) - Xtr, Ytr, Xte, Yte = train_test_split(X, Y, n_train=n_train, seed=seed + 1000) - - m_gp = fit_gp(Xtr, Ytr, num_inducing=num_inducing, messages=False) - m_dgp = fit_deepgp( - Xtr, - Ytr, - q1=q1, - q2=q2, - num_inducing=num_inducing, - iters_fixed=iters_fixed, - iters_free=iters_free, - messages=args.messages, + X, H, Y = generate_hier_gps_new(n_toy=n_toy, d_toy=d_toy, seed=seed) + ind_tr, ind_ts = train_test_split_indices(n_toy, n_tr, seed=seed + 1000) + Xtr, Ytr = X[ind_tr], Y[ind_tr] + Xte, Yte = X[ind_ts], Y[ind_ts] + + m_gp = fit_gp(Xtr, Ytr, num_inducing, messages=False) + m_dgp = fit_dgp( + Xtr, Ytr, q, num_inducing, iters_fixed, iters_free, messages=args.messages ) - - mse_gp = mse(m_gp.predict(Xte)[0], Yte) - mse_dgp = mse(m_dgp.predict(Xte)[0], Yte) - gp_mses.append(mse_gp) - dgp_mses.append(mse_dgp) - print(f"Repeat {r + 1}/{repeats}: MSE GP={mse_gp:.4f} DeepGP={mse_dgp:.4f}") + gp_mses.append(mse(m_gp.predict(Xte)[0], Yte)) + dgp_mses.append(mse(m_dgp.predict(Xte)[0], Yte)) + print(f"Repeat {r + 1}/{repeats}: MSE GP={gp_mses[-1]:.4f} DeepGP={dgp_mses[-1]:.4f}") print( f"Mean MSE (±std): GP {np.mean(gp_mses):.4f}±{np.std(gp_mses):.4f} " @@ -158,27 +117,25 @@ def main(argv: list[str] | None = None) -> int: if not args.no_plot or outdir: import matplotlib.pyplot as plt - # Illustrate one generative draw - X, H, Y = generate_warped(n=120, seed=args.seed) + X, H, Y = generate_hier_gps_new(n_toy=n_toy, d_toy=d_toy, seed=args.seed) fig, axes = plt.subplots(2, 1, figsize=(7, 4), sharex=True) axes[0].plot(X[:, 0], H[:, 0], "k-") - axes[0].set_ylabel("hidden warp") - axes[0].set_title("Warped process (one draw)") + axes[0].set_ylabel("hidden warp Y1") + axes[0].set_title("hierGpsNEW generative process") axes[1].plot(X[:, 0], Y[:, 0], label="Y[:,0]") axes[1].plot(X[:, 0], Y[:, 1], label="Y[:,1]") - axes[1].set_xlabel("X") - axes[1].set_ylabel("outputs") + axes[1].set_xlabel("X (scaled)") axes[1].legend(fontsize=8) fig.tight_layout() save_or_show(fig, outdir, "toy_warped_process.png", args.no_plot) fig2, ax = plt.subplots(figsize=(5, 3.5)) xs = np.arange(1, repeats + 1) - ax.plot(xs, gp_mses, "o-", label="GP") + ax.plot(xs, gp_mses, "o-", label="sparse GP") ax.plot(xs, dgp_mses, "s-", label="DeepGP") ax.set_xlabel("experiment #") - ax.set_ylabel("MSE") - ax.set_title("Damianou & Lawrence 2013 — fig. 3(b) style") + ax.set_ylabel("test MSE") + ax.set_title("Fig. 3(b) style / demToyRegressionSimple") ax.legend() fig2.tight_layout() save_or_show(fig2, outdir, "toy_warped_mse.png", args.no_plot) diff --git a/examples/damianou2013/usps_digits.py b/examples/damianou2013/usps_digits.py index aab1dd3..67bf247 100644 --- a/examples/damianou2013/usps_digits.py +++ b/examples/damianou2013/usps_digits.py @@ -1,21 +1,17 @@ #!/usr/bin/env python -"""Section 4.3 (figs. 6–8): USPS digits {0,1,6} — depth, ARD, 1-NN in latent space. +"""§4.3 / figs. 6–8 — USPS {0,1,6} (Matlab ``demUsps.m`` / ``demDigitsDemonstration.m``). -Paper: Damianou & Lawrence, Deep Gaussian Processes, AISTATS 2013. - -Downloads the ESL ``zip.train`` USPS split (16×16), keeps 50 examples of each of -digits {0,1,6}, and trains unsupervised DeepGPs of increasing depth. Reports the -variational bound and leave-one-out 1-NN error in the top latent features. +``vercat2``: 50 examples of digits 0, 6, 1. Depth-5 demo uses +Q = {15, 12, 10, 8, 6} and K = 80 inducing points (experimentNo 19 comment). +Reports variational bound and leave-one-out 1-NN error in the top latent space. """ from __future__ import annotations import argparse -import gzip import os import sys -import urllib.request -from typing import List, Sequence, Tuple +from typing import List import numpy as np @@ -32,104 +28,13 @@ rbf_bias, save_or_show, ) +from _matlab_data import USPS_DEPTH_Q, load_usps_vercat2 -ESL_ZIP_TRAIN = "https://web.stanford.edu/~hastie/ElemStatLearn/datasets/zip.train.gz" -DIGITS = (0, 1, 6) - - -def _cache_path(cache_dir: str) -> str: - return os.path.join(cache_dir, "zip.train.gz") - - -def download_usps(cache_dir: str) -> str: - os.makedirs(cache_dir, exist_ok=True) - path = _cache_path(cache_dir) - if os.path.isfile(path) and os.path.getsize(path) > 0: - return path - print(f"Downloading USPS zip.train → {path}") - req = urllib.request.Request(ESL_ZIP_TRAIN, headers={"User-Agent": "PyDeepGP-examples"}) - with urllib.request.urlopen(req, timeout=120) as resp, open(path, "wb") as out: - out.write(resp.read()) - return path - - -def load_usps_matrix(path: str) -> Tuple[np.ndarray, np.ndarray]: - """Return (Y [N,256], labels [N]) with pixels in roughly [-1, 1].""" - rows = [] - with gzip.open(path, "rt") as f: - for line in f: - parts = line.strip().split() - if not parts: - continue - rows.append([float(x) for x in parts]) - data = np.asarray(rows, dtype=float) - labels = data[:, 0].astype(int) - Y = data[:, 1:] - return Y, labels - - -def subsample_digits( - Y: np.ndarray, - labels: np.ndarray, - digits: Sequence[int] = DIGITS, - per_class: int = 50, - seed: int = 42, -) -> Tuple[np.ndarray, np.ndarray]: - rng = np.random.RandomState(seed) - pieces_Y = [] - pieces_L = [] - for d in digits: - idx = np.where(labels == d)[0] - if idx.size < per_class: - raise RuntimeError(f"Digit {d}: need {per_class} examples, found {idx.size}") - choose = rng.choice(idx, size=per_class, replace=False) - pieces_Y.append(Y[choose]) - pieces_L.append(np.full(per_class, d, dtype=int)) - return np.vstack(pieces_Y), np.concatenate(pieces_L) - -def depth_dimensions(depth: int, data_dim: int, quick: bool) -> List[int]: - """Latent widths from data up to the top layer (length = depth + 1 entries for DeepGP). - - ``DeepGP(nDims, Y)`` expects ``nDims = [Y_dim, H1, H2, ..., H_depth]``. - """ - if quick: - # Narrower / shallower budgets for smoke tests - widths = {1: [8], 2: [8, 5], 3: [8, 6, 4]} - else: - # Overcomplete relative to the true manifold; ARD prunes - widths = { - 1: [12], - 2: [12, 8], - 3: [12, 10, 6], - 4: [12, 10, 8, 5], - 5: [12, 10, 8, 6, 4], - } - if depth not in widths: - raise ValueError(f"Unsupported depth {depth}; choose from {sorted(widths)}") - return [data_dim] + widths[depth] - - -def fit_depth( - Y: np.ndarray, - depth: int, - num_inducing: int, - iters_fixed: int, - iters_free: int, - messages: bool, - quick: bool, -) -> deepgp.DeepGP: - dims = depth_dimensions(depth, Y.shape[1], quick=quick) - kernels = [rbf_bias(q, ard=True) for q in dims[1:]] - m = deepgp.DeepGP( - dims, - Y, - kernels=kernels, - num_inducing=num_inducing, - back_constraint=False, - ) - optimize_staged(m, iters_fixed, iters_free, messages=messages) - return m +def depth_dimensions(depth: int, data_dim: int) -> List[int]: + if depth not in USPS_DEPTH_Q: + raise ValueError(f"depth must be in {sorted(USPS_DEPTH_Q)}") + return [data_dim] + list(USPS_DEPTH_Q[depth]) def build_parser() -> argparse.ArgumentParser: @@ -140,16 +45,10 @@ def build_parser() -> argparse.ArgumentParser: type=int, nargs="+", default=None, - help="Hierarchy depths to fit (default: 1..5, or 1..2 with --quick).", + help="Default: 1..5 (or 1..2 with --quick).", ) p.add_argument("--per-class", type=int, default=50) - p.add_argument("--num-inducing", type=int, default=40) - p.add_argument( - "--cache-dir", - type=str, - default=os.path.join(os.path.dirname(__file__), "data"), - help="Directory for zip.train.gz cache.", - ) + p.add_argument("--num-inducing", type=int, default=80) return p @@ -162,33 +61,34 @@ def main(argv: list[str] | None = None) -> int: depths = [1, 2] if args.quick else [1, 2, 3, 4, 5] per_class = 20 if args.quick else args.per_class - num_inducing = 20 if args.quick else args.num_inducing - iters_fixed = 40 if args.quick else 200 - iters_free = 100 if args.quick else 2000 + # Matlab uses K=80; keep a reduced inducing set only for --quick + num_inducing = 25 if args.quick else args.num_inducing + iters_fixed = 40 if args.quick else 500 + iters_free = 100 if args.quick else 3000 - path = download_usps(args.cache_dir) - Y_all, labels_all = load_usps_matrix(path) - Y, labels = subsample_digits(Y_all, labels_all, per_class=per_class, seed=args.seed) - # Zero-mean unit-ish scaling helps optimisation + Y, labels = load_usps_vercat2(per_class=per_class) Y = Y - Y.mean(0) Y = Y / (Y.std() + 1e-12) - print(f"USPS subset: Y={Y.shape}, classes={sorted(set(labels.tolist()))}") + print(f"USPS vercat2-style: Y={Y.shape}, classes={sorted(set(labels.tolist()))}") results = [] models = {} for depth in depths: - print(f"\n=== depth {depth} ===") - m = fit_depth( + print(f"\n=== depth {depth} dims={depth_dimensions(depth, Y.shape[1])} ===") + dims = depth_dimensions(depth, Y.shape[1]) + # For quick runs, shrink Q to keep optimisation tractable + if args.quick: + dims = [Y.shape[1]] + [max(2, q // 2) for q in dims[1:]] + m = deepgp.DeepGP( + dims, Y, - depth=depth, + kernels=[rbf_bias(q) for q in dims[1:]], num_inducing=num_inducing, - iters_fixed=iters_fixed, - iters_free=iters_free, - messages=args.messages, - quick=args.quick, + back_constraint=False, + inits="PCA", ) + optimize_staged(m, iters_fixed, iters_free, messages=args.messages) bound = float(m.log_likelihood()) - # Top latent features = last layer's X Xtop = np.asarray(m.layers[-1].X.mean) mistakes, rate = nearest_neighbour_error(Xtop, labels) print(f"bound={bound:.3f} 1-NN mistakes={mistakes}/{Y.shape[0]} ({rate:.3%})") @@ -196,7 +96,7 @@ def main(argv: list[str] | None = None) -> int: results.append((depth, bound, mistakes, rate)) models[depth] = m - print("\nSummary") + print("\nSummary (Matlab demDigitsDemonstration metrics)") print("depth bound 1-NN errors") for depth, bound, mistakes, rate in results: print(f"{depth:5d} {bound:9.3f} {mistakes:3d} ({rate:.3%})") @@ -207,49 +107,50 @@ def main(argv: list[str] | None = None) -> int: import matplotlib.pyplot as plt Xtop = np.asarray(m.layers[-1].X.mean) - w = ard_weights(m.layers[-1].kern) - dims = dominant_dims(w, k=2) - colors = ["C0", "C1", "C2"] - markers = ["o", "s", "^"] - + dims_plot = dominant_dims(ard_weights(m.layers[-1].kern), k=2) + colors = {0: "C0", 1: "C1", 6: "C2"} + markers = {0: "o", 1: "s", 6: "^"} fig, ax = plt.subplots(figsize=(4.5, 4.5)) - for i, digit in enumerate(DIGITS): - mask = labels == digit + for d in (0, 1, 6): + mask = labels == d ax.scatter( - Xtop[mask, dims[0]], - Xtop[mask, dims[1]], - c=colors[i], - marker=markers[i], + Xtop[mask, dims_plot[0]], + Xtop[mask, dims_plot[1]], + c=colors[d], + marker=markers[d], s=28, - label=str(digit), + label=str(d), ) - ax.set_xlabel(f"top dim {dims[0]}") - ax.set_ylabel(f"top dim {dims[1]}") ax.set_title(f"USPS top latent (depth {deepest})") ax.legend() fig.tight_layout() save_or_show(fig, outdir, "usps_top_latent.png", args.no_plot) - fig2, axes = plt.subplots(1, len(m.layers), figsize=(3.2 * len(m.layers), 3)) + fig2, axes = plt.subplots(1, len(m.layers), figsize=(3.0 * len(m.layers), 3)) if len(m.layers) == 1: axes = [axes] for i, layer in enumerate(m.layers): w = ard_weights(layer.kern) axes[i].bar(np.arange(1, len(w) + 1), w) - axes[i].set_title(f"layer {i} ARD") - axes[i].set_xlabel("dimension") - fig2.suptitle("ARD weights (1/lengthscale)") + axes[i].set_title(f"layer {i}") + fig2.suptitle("ARD (fig. 6 style)") fig2.tight_layout() save_or_show(fig2, outdir, "usps_ard.png", args.no_plot) fig3, ax = plt.subplots(figsize=(5, 3.5)) - ax.plot([r[0] for r in results], [r[1] for r in results], "o-", label="variational bound") + ax.plot([r[0] for r in results], [r[1] for r in results], "o-", label="bound") ax.set_xlabel("depth") - ax.set_ylabel("bound") + ax.set_ylabel("variational bound") ax2 = ax.twinx() - ax2.plot([r[0] for r in results], [r[2] for r in results], "s--", color="C1", label="1-NN mistakes") + ax2.plot( + [r[0] for r in results], + [r[2] for r in results], + "s--", + color="C1", + label="1-NN mistakes", + ) ax2.set_ylabel("1-NN mistakes") - ax.set_title("Depth selection via bound / 1-NN") + ax.set_title("Depth vs bound / 1-NN") fig3.tight_layout() save_or_show(fig3, outdir, "usps_depth.png", args.no_plot) From 639bfa356538f27ae7c42082bc10a6d519ddf716 Mon Sep 17 00:00:00 2001 From: Neil Lawrence Date: Fri, 13 Nov 2026 21:53:16 +0000 Subject: [PATCH 3/3] Rewrite Damianou 2013 notebooks as step-by-step walkthroughs. Replace thin main() wrappers with inlined experiment cells (data, fit, plots); notebooks no longer import the .py entry scripts. --- examples/damianou2013/README.md | 3 +- .../notebooks/01_toy_hierarchy.ipynb | 222 +++++++++++++++--- .../notebooks/02_toy_warped_regression.ipynb | 190 ++++++++++++--- .../notebooks/03_usps_digits.ipynb | 207 +++++++++++++--- .../notebooks/04_highfive_mrd.ipynb | 207 +++++++++++++--- examples/damianou2013/notebooks/README.md | 7 +- 6 files changed, 695 insertions(+), 141 deletions(-) diff --git a/examples/damianou2013/README.md b/examples/damianou2013/README.md index e29aa74..c6c84d7 100644 --- a/examples/damianou2013/README.md +++ b/examples/damianou2013/README.md @@ -19,7 +19,8 @@ Paper: [Damianou & Lawrence, AISTATS 2013](https://proceedings.mlr.press/v31/dam | §4.3 figs. 6–8 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/03_usps_digits.ipynb) [03](notebooks/03_usps_digits.ipynb) | `demUsps.m` / `demDigitsDemonstration.m` | | §4.2 figs. 4–5 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/04_highfive_mrd.ipynb) [04](notebooks/04_highfive_mrd.ipynb) | `demHighFiveDemonstration.m` | -Until this PR is on `main`, open Colab from the branch, e.g. +Notebooks are walkthroughs (data → fit → plots), not wrappers around the +`.py` scripts. Until this lands on `main`, open Colab from the branch, e.g. `…/blob/examples-damianou2013/examples/damianou2013/notebooks/…`. ## Scripts diff --git a/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb b/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb index 2b8741d..a1f4104 100644 --- a/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb +++ b/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb @@ -17,9 +17,18 @@ "\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/01_toy_hierarchy.ipynb)\n", "\n", - "**Matlab:** [`demToyUnsupervised.m`](https://github.com/SheffieldML/deepGP/blob/master/deepGP/matlab/demToyUnsupervised.m) with `hgplvmSampleDataTr1.mat`.\n", + "Port of Matlab [`demToyUnsupervised.m`](https://github.com/SheffieldML/deepGP/blob/master/deepGP/matlab/demToyUnsupervised.m) using\n", + "`hgplvmSampleDataTr1.mat`: two observation views `YA`, `YB` share a parent latent\n", + "hierarchy. We fit a two-view DeepGP (Q={6,4}) and compare stacked PCA / Isomap.\n", "\n", - "Two observation modalities `YA`,`YB` share parent latents; compare DeepGP to stacked PCA and Isomap.\n" + "Paper: [Damianou & Lawrence, AISTATS 2013](https://proceedings.mlr.press/v31/damianou13a.html).\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup\n" ] }, { @@ -29,32 +38,39 @@ "outputs": [], "source": [ "import sys, subprocess, os, urllib.request\n", + "from pathlib import Path\n", + "\n", "IN_COLAB = \"google.colab\" in sys.modules\n", + "ROOT = Path.cwd().resolve()\n", + "# Prefer sibling of notebooks/ (repo layout)\n", + "for c in [ROOT.parent, ROOT, ROOT / \"examples\" / \"damianou2013\"]:\n", + " if (c / \"_common.py\").is_file():\n", + " ROOT = c\n", + " break\n", + "\n", "if IN_COLAB:\n", " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", " \"numpy\", \"scipy\", \"matplotlib\", \"GPy>=1.15\", \"scikit-learn\"])\n", " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", - " \"git+https://github.com/SheffieldML/PyDeepGP.git@examples-damianou2013\"])\n", - " os.makedirs(\"data\", exist_ok=True)\n", - " files = [\n", - " (\"_common.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_common.py\"),\n", - " (\"_matlab_data.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_matlab_data.py\"),\n", - " (\"toy_hierarchy.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/toy_hierarchy.py\"),\n", - " (\"toy_warped_regression.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/toy_warped_regression.py\"),\n", - " (\"usps_digits.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/usps_digits.py\"),\n", - " (\"highfive_shared.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/highfive_shared.py\"),\n", + " f\"git+https://github.com/SheffieldML/PyDeepGP.git@examples-damianou2013\"])\n", + " ROOT = Path.cwd().resolve()\n", + " (ROOT / \"data\").mkdir(exist_ok=True)\n", + " for dest, url in [\n", + " (\"_common.py\", f\"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_common.py\"),\n", + " (\"_matlab_data.py\", f\"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_matlab_data.py\"),\n", " (\"data/hgplvmSampleDataTr1.mat\",\n", " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/hgplvmSampleDataTr1.mat\"),\n", " (\"data/demHighFiveHgplvm1.mat\",\n", " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/demHighFiveHgplvm1.mat\"),\n", - " ]\n", - " for dest, url in files:\n", - " if not os.path.isfile(dest):\n", + " ]:\n", + " p = ROOT / dest\n", + " if not p.is_file():\n", " print(\"Fetching\", dest)\n", - " urllib.request.urlretrieve(url, dest)\n", - " print(\"Colab setup done.\")\n", - "else:\n", - " print(\"Local: run from repo; ensure examples/damianou2013 is on sys.path.\")\n" + " urllib.request.urlretrieve(url, str(p))\n", + "\n", + "os.chdir(ROOT)\n", + "sys.path.insert(0, str(ROOT))\n", + "print(\"Working directory:\", ROOT)\n" ] }, { @@ -63,17 +79,34 @@ "metadata": {}, "outputs": [], "source": [ - "import os, sys\n", - "from pathlib import Path\n", - "here = Path.cwd().resolve()\n", - "candidates = [here, here.parent, here / \"examples\" / \"damianou2013\",\n", - " here.parent / \"examples\" / \"damianou2013\"]\n", - "for c in candidates:\n", - " if (c / \"toy_hierarchy.py\").is_file():\n", - " sys.path.insert(0, str(c))\n", - " os.chdir(c)\n", - " break\n", - "print(\"cwd:\", os.getcwd())\n" + "%matplotlib inline\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "import deepgp\n", + "from _common import (\n", + " ard_weights, dominant_dims, init_noise, unfix_noise, rbf_bias, optimize_staged,\n", + ")\n", + "from _matlab_data import load_toy_hierarchy\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "QUICK = True # set False for Matlab-demo scale (Q={6,4}, longer optimisation)\n", + "q_shared, q_top = (4, 3) if QUICK else (6, 4)\n", + "num_inducing = 15 if QUICK else 25\n", + "iters_fixed = 50 if QUICK else 200\n", + "iters_free = 150 if QUICK else 2000\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Load Matlab ground-truth hierarchy\n" ] }, { @@ -82,7 +115,28 @@ "metadata": {}, "outputs": [], "source": [ - "QUICK = True # set False for demo-scale optimisation\n" + "data = load_toy_hierarchy()\n", + "YA, YB = data[\"YA\"], data[\"YB\"]\n", + "X2, XA, XB = data[\"X2\"], data[\"XA\"], data[\"XB\"]\n", + "print(f\"YA={YA.shape} YB={YB.shape} XA={XA.shape} XB={XB.shape} X2={X2.shape}\")\n", + "\n", + "fig, axes = plt.subplots(3, 2, figsize=(9, 8))\n", + "axes[0, 0].plot(X2); axes[0, 0].set_title(\"True top X2\")\n", + "axes[0, 1].axis(\"off\")\n", + "axes[1, 0].plot(XA); axes[1, 0].set_title(\"True XA\")\n", + "axes[1, 1].plot(XB); axes[1, 1].set_title(\"True XB\")\n", + "axes[2, 0].plot(YA); axes[2, 0].set_title(\"YA\")\n", + "axes[2, 1].plot(YB); axes[2, 1].set_title(\"YB\")\n", + "fig.suptitle(\"Matlab hgplvmSampleDataTr1 ground truth\")\n", + "fig.tight_layout()\n", + "plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Fit two-view DeepGP (Q shared \u2192 top)\n" ] }, { @@ -91,11 +145,109 @@ "metadata": {}, "outputs": [], "source": [ - "%matplotlib inline\n", - "argv = [\"--quick\"] if QUICK else []\n", - "argv += [\"--outdir\", \"/tmp/dgp2013_nb\"]\n", - "from toy_hierarchy import main\n", - "main(argv)\n" + "kernels = [[rbf_bias(q_shared), rbf_bias(q_shared)], rbf_bias(q_top)]\n", + "m = deepgp.DeepGP(\n", + " [[YA.shape[1], YB.shape[1]], q_shared, q_top],\n", + " [YA, YB],\n", + " kernels=kernels,\n", + " num_inducing=num_inducing,\n", + " back_constraint=False,\n", + " inits=\"PCA\",\n", + ")\n", + "optimize_staged(m, iters_fixed, iters_free, messages=False)\n", + "print(f\"variational bound: {float(m.log_likelihood()):.3f}\")\n", + "for i, view in enumerate(m.layers[0].views):\n", + " print(f\"View {i} ARD:\", np.array2string(ard_weights(view.kern), precision=3))\n", + "print(\"Top ARD:\", np.array2string(ard_weights(m.layers[-1].kern), precision=3))\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Recovered latents (dominant ARD dims)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "Xshare = np.asarray(m.layers[0].X.mean)\n", + "Xtop = np.asarray(m.layers[-1].X.mean)\n", + "d_top = dominant_dims(ard_weights(m.layers[-1].kern), k=min(2, Xtop.shape[1]))\n", + "d_a = dominant_dims(ard_weights(m.layers[0].views[0].kern), k=min(2, Xshare.shape[1]))\n", + "d_b = dominant_dims(ard_weights(m.layers[0].views[1].kern), k=min(2, Xshare.shape[1]))\n", + "\n", + "fig, axes = plt.subplots(2, 2, figsize=(8, 7))\n", + "axes[0, 0].plot(Xtop[:, d_top]); axes[0, 0].set_title(\"DeepGP top (dominant dims)\")\n", + "axes[0, 1].axis(\"off\")\n", + "axes[1, 0].plot(Xshare[:, d_a]); axes[1, 0].set_title(\"shared (view0 ARD dims)\")\n", + "axes[1, 1].plot(Xshare[:, d_b]); axes[1, 1].set_title(\"shared (view1 ARD dims)\")\n", + "fig.suptitle(\"demToyUnsupervised-style DeepGP recovery\")\n", + "fig.tight_layout()\n", + "plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Baselines: stacked PCA and Isomap\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def stacked_pca(Ya, Yb, q1, q2):\n", + " def pca(Y, q):\n", + " Yc = Y - Y.mean(0)\n", + " U, S, _ = np.linalg.svd(Yc, full_matrices=False)\n", + " q = min(q, U.shape[1])\n", + " return U[:, :q] * S[:q]\n", + " xa, xb = pca(Ya, q1), pca(Yb, q1)\n", + " return xa, xb, pca(np.hstack([xa, xb]), q2)\n", + "\n", + "pca_xa, pca_xb, pca_x2 = stacked_pca(YA, YB, q_shared, q_top)\n", + "fig, axes = plt.subplots(2, 2, figsize=(8, 7))\n", + "axes[0, 0].plot(pca_x2[:, :min(2, pca_x2.shape[1])]); axes[0, 0].set_title(\"Stacked PCA top\")\n", + "axes[0, 1].axis(\"off\")\n", + "axes[1, 0].plot(pca_xa[:, :min(2, pca_xa.shape[1])]); axes[1, 0].set_title(\"Stacked PCA XA\")\n", + "axes[1, 1].plot(pca_xb[:, :min(2, pca_xb.shape[1])]); axes[1, 1].set_title(\"Stacked PCA XB\")\n", + "fig.suptitle(\"Stacked PCA\")\n", + "fig.tight_layout()\n", + "plt.show()\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from sklearn.manifold import Isomap\n", + "\n", + "def stacked_isomap(Ya, Yb, q1, q2, n_neighbors=5):\n", + " xa = Isomap(n_components=min(q1, Ya.shape[0] - 1), n_neighbors=n_neighbors).fit_transform(Ya)\n", + " xb = Isomap(n_components=min(q1, Yb.shape[0] - 1), n_neighbors=n_neighbors).fit_transform(Yb)\n", + " x2 = Isomap(n_components=min(q2, xa.shape[0] - 1), n_neighbors=n_neighbors).fit_transform(\n", + " np.hstack([xa, xb])\n", + " )\n", + " return xa, xb, x2\n", + "\n", + "iso_xa, iso_xb, iso_x2 = stacked_isomap(YA, YB, min(2, q_shared), min(2, q_top))\n", + "fig, axes = plt.subplots(2, 2, figsize=(8, 7))\n", + "axes[0, 0].plot(iso_x2); axes[0, 0].set_title(\"Stacked Isomap top\")\n", + "axes[0, 1].axis(\"off\")\n", + "axes[1, 0].plot(iso_xa); axes[1, 0].set_title(\"Stacked Isomap XA\")\n", + "axes[1, 1].plot(iso_xb); axes[1, 1].set_title(\"Stacked Isomap XB\")\n", + "fig.suptitle(\"Stacked Isomap\")\n", + "fig.tight_layout()\n", + "plt.show()\n" ] } ] diff --git a/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb b/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb index 8bece00..789c1bc 100644 --- a/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb +++ b/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb @@ -17,9 +17,18 @@ "\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/02_toy_warped_regression.ipynb)\n", "\n", - "**Matlab:** [`demToyRegressionSimple.m`](https://github.com/SheffieldML/deepGP/blob/master/deepGP/matlab/demToyRegressionSimple.m) + `hierGpsNEW` in `demToyDynamicsCreateData.m`.\n", + "Port of Matlab [`demToyRegressionSimple.m`](https://github.com/SheffieldML/deepGP/blob/master/deepGP/matlab/demToyRegressionSimple.m) with\n", + "data from `hierGpsNEW` in `demToyDynamicsCreateData.m`.\n", "\n", - "Defaults: Ntoy=150, D=15, Ntr=25, H=2, Q=6, K=10. Use `argv += [\"--paper\"]` for N=120, D=10.\n" + "Defaults: Ntoy=150, D=15, Ntr=25, H=2, Q=6, K=10 (Matlab simple). Set `PAPER = True`\n", + "for the AISTATS text sizes (N=120, D=10).\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup\n" ] }, { @@ -29,32 +38,39 @@ "outputs": [], "source": [ "import sys, subprocess, os, urllib.request\n", + "from pathlib import Path\n", + "\n", "IN_COLAB = \"google.colab\" in sys.modules\n", + "ROOT = Path.cwd().resolve()\n", + "# Prefer sibling of notebooks/ (repo layout)\n", + "for c in [ROOT.parent, ROOT, ROOT / \"examples\" / \"damianou2013\"]:\n", + " if (c / \"_common.py\").is_file():\n", + " ROOT = c\n", + " break\n", + "\n", "if IN_COLAB:\n", " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", " \"numpy\", \"scipy\", \"matplotlib\", \"GPy>=1.15\", \"scikit-learn\"])\n", " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", - " \"git+https://github.com/SheffieldML/PyDeepGP.git@examples-damianou2013\"])\n", - " os.makedirs(\"data\", exist_ok=True)\n", - " files = [\n", - " (\"_common.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_common.py\"),\n", - " (\"_matlab_data.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_matlab_data.py\"),\n", - " (\"toy_hierarchy.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/toy_hierarchy.py\"),\n", - " (\"toy_warped_regression.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/toy_warped_regression.py\"),\n", - " (\"usps_digits.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/usps_digits.py\"),\n", - " (\"highfive_shared.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/highfive_shared.py\"),\n", + " f\"git+https://github.com/SheffieldML/PyDeepGP.git@examples-damianou2013\"])\n", + " ROOT = Path.cwd().resolve()\n", + " (ROOT / \"data\").mkdir(exist_ok=True)\n", + " for dest, url in [\n", + " (\"_common.py\", f\"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_common.py\"),\n", + " (\"_matlab_data.py\", f\"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_matlab_data.py\"),\n", " (\"data/hgplvmSampleDataTr1.mat\",\n", " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/hgplvmSampleDataTr1.mat\"),\n", " (\"data/demHighFiveHgplvm1.mat\",\n", " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/demHighFiveHgplvm1.mat\"),\n", - " ]\n", - " for dest, url in files:\n", - " if not os.path.isfile(dest):\n", + " ]:\n", + " p = ROOT / dest\n", + " if not p.is_file():\n", " print(\"Fetching\", dest)\n", - " urllib.request.urlretrieve(url, dest)\n", - " print(\"Colab setup done.\")\n", - "else:\n", - " print(\"Local: run from repo; ensure examples/damianou2013 is on sys.path.\")\n" + " urllib.request.urlretrieve(url, str(p))\n", + "\n", + "os.chdir(ROOT)\n", + "sys.path.insert(0, str(ROOT))\n", + "print(\"Working directory:\", ROOT)\n" ] }, { @@ -63,17 +79,13 @@ "metadata": {}, "outputs": [], "source": [ - "import os, sys\n", - "from pathlib import Path\n", - "here = Path.cwd().resolve()\n", - "candidates = [here, here.parent, here / \"examples\" / \"damianou2013\",\n", - " here.parent / \"examples\" / \"damianou2013\"]\n", - "for c in candidates:\n", - " if (c / \"toy_hierarchy.py\").is_file():\n", - " sys.path.insert(0, str(c))\n", - " os.chdir(c)\n", - " break\n", - "print(\"cwd:\", os.getcwd())\n" + "%matplotlib inline\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "import GPy\n", + "import deepgp\n", + "from _common import mse, optimize_staged, rbf_bias\n", + "from _matlab_data import generate_hier_gps_new, train_test_split_indices\n" ] }, { @@ -82,7 +94,24 @@ "metadata": {}, "outputs": [], "source": [ - "QUICK = True\n" + "QUICK = True\n", + "PAPER = False # True \u2192 N=120, D=10\n", + "n_toy, d_toy = (120, 10) if PAPER else (150, 15)\n", + "n_tr = 25\n", + "q = 3 if QUICK else 6\n", + "num_inducing = 8 if QUICK else 10\n", + "repeats = 2 if QUICK else 10\n", + "iters_fixed = 40 if QUICK else 500\n", + "iters_free = 120 if QUICK else 2000\n", + "SEED = 42\n", + "print(f\"Ntoy={n_toy} D={d_toy} Ntr={n_tr} Q={q} K={num_inducing} repeats={repeats}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Generative process (`hierGpsNEW`)\n" ] }, { @@ -91,11 +120,102 @@ "metadata": {}, "outputs": [], "source": [ - "%matplotlib inline\n", - "argv = [\"--quick\"] if QUICK else []\n", - "argv += [\"--outdir\", \"/tmp/dgp2013_nb\"]\n", - "from toy_warped_regression import main\n", - "main(argv)\n" + "X, H, Y = generate_hier_gps_new(n_toy=n_toy, d_toy=d_toy, seed=SEED)\n", + "fig, axes = plt.subplots(2, 1, figsize=(7, 4), sharex=True)\n", + "axes[0].plot(X[:, 0], H[:, 0], \"k-\")\n", + "axes[0].set_ylabel(\"hidden warp\")\n", + "axes[0].set_title(\"hierGpsNEW generative process\")\n", + "axes[1].plot(X[:, 0], Y[:, 0], label=\"Y[:,0]\")\n", + "axes[1].plot(X[:, 0], Y[:, 1], label=\"Y[:,1]\")\n", + "axes[1].set_xlabel(\"X (scaled)\")\n", + "axes[1].legend(fontsize=8)\n", + "fig.tight_layout()\n", + "plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Fit sparse GP and DeepGP across repeats\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def fit_gp(Xtr, Ytr, num_inducing):\n", + " kern = GPy.kern.RBF(Xtr.shape[1]) + GPy.kern.Bias(Xtr.shape[1])\n", + " m = GPy.models.SparseGPRegression(Xtr, Ytr, kernel=kern, num_inducing=num_inducing)\n", + " m.Gaussian_noise.variance = Ytr.var() * 0.01\n", + " m.Gaussian_noise.variance.fix()\n", + " m.optimize(max_iters=100, messages=False)\n", + " m.Gaussian_noise.variance.unfix()\n", + " m.optimize(max_iters=400, messages=False)\n", + " return m\n", + "\n", + "def fit_dgp(Xtr, Ytr, q, num_inducing, iters_fixed, iters_free):\n", + " # H=2 hidden layers between observed X and Y\n", + " dims = [Ytr.shape[1], q, q, Xtr.shape[1]]\n", + " kernels = [rbf_bias(q, ard=True), rbf_bias(q, ard=True), rbf_bias(Xtr.shape[1], ard=False)]\n", + " m = deepgp.DeepGP(\n", + " dims, Ytr, X=Xtr, kernels=kernels,\n", + " num_inducing=num_inducing, back_constraint=False,\n", + " )\n", + " optimize_staged(m, iters_fixed, iters_free, messages=False)\n", + " return m\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "gp_mses, dgp_mses = [], []\n", + "for r in range(repeats):\n", + " seed = SEED + r\n", + " X, H, Y = generate_hier_gps_new(n_toy=n_toy, d_toy=d_toy, seed=seed)\n", + " ind_tr, ind_ts = train_test_split_indices(n_toy, n_tr, seed=seed + 1000)\n", + " Xtr, Ytr = X[ind_tr], Y[ind_tr]\n", + " Xte, Yte = X[ind_ts], Y[ind_ts]\n", + " m_gp = fit_gp(Xtr, Ytr, num_inducing)\n", + " m_dgp = fit_dgp(Xtr, Ytr, q, num_inducing, iters_fixed, iters_free)\n", + " gp_mses.append(mse(m_gp.predict(Xte)[0], Yte))\n", + " dgp_mses.append(mse(m_dgp.predict(Xte)[0], Yte))\n", + " print(f\"Repeat {r + 1}/{repeats}: MSE GP={gp_mses[-1]:.4f} DeepGP={dgp_mses[-1]:.4f}\")\n", + "\n", + "print(\n", + " f\"Mean MSE (\u00b1std): GP {np.mean(gp_mses):.4f}\u00b1{np.std(gp_mses):.4f} \"\n", + " f\"DeepGP {np.mean(dgp_mses):.4f}\u00b1{np.std(dgp_mses):.4f}\"\n", + ")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Fig. 3(b)-style MSE plot\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "fig, ax = plt.subplots(figsize=(5, 3.5))\n", + "xs = np.arange(1, repeats + 1)\n", + "ax.plot(xs, gp_mses, \"o-\", label=\"sparse GP\")\n", + "ax.plot(xs, dgp_mses, \"s-\", label=\"DeepGP\")\n", + "ax.set_xlabel(\"experiment #\")\n", + "ax.set_ylabel(\"test MSE\")\n", + "ax.set_title(\"Fig. 3(b) style / demToyRegressionSimple\")\n", + "ax.legend()\n", + "fig.tight_layout()\n", + "plt.show()\n" ] } ] diff --git a/examples/damianou2013/notebooks/03_usps_digits.ipynb b/examples/damianou2013/notebooks/03_usps_digits.ipynb index 16f11b6..492af9a 100644 --- a/examples/damianou2013/notebooks/03_usps_digits.ipynb +++ b/examples/damianou2013/notebooks/03_usps_digits.ipynb @@ -17,7 +17,18 @@ "\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/03_usps_digits.ipynb)\n", "\n", - "**Matlab:** [`demUsps.m`](https://github.com/SheffieldML/deepGP/blob/master/deepGP/matlab/demUsps.m) / [`demDigitsDemonstration.m`](https://github.com/SheffieldML/deepGP/blob/master/deepGP/matlab/demDigitsDemonstration.m) (`vercat2`, H=5, Q={15,12,10,8,6}, K=80).\n" + "Port of Matlab [`demUsps.m`](https://github.com/SheffieldML/deepGP/blob/master/deepGP/matlab/demUsps.m) /\n", + "[`demDigitsDemonstration.m`](https://github.com/SheffieldML/deepGP/blob/master/deepGP/matlab/demDigitsDemonstration.m):\n", + "`vercat2` (50\u00d7{0,6,1}), depth-5 widths Q={15,12,10,8,6}, K=80.\n", + "\n", + "Metrics: variational bound and leave-one-out 1-NN error in the top latent space.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup\n" ] }, { @@ -27,32 +38,39 @@ "outputs": [], "source": [ "import sys, subprocess, os, urllib.request\n", + "from pathlib import Path\n", + "\n", "IN_COLAB = \"google.colab\" in sys.modules\n", + "ROOT = Path.cwd().resolve()\n", + "# Prefer sibling of notebooks/ (repo layout)\n", + "for c in [ROOT.parent, ROOT, ROOT / \"examples\" / \"damianou2013\"]:\n", + " if (c / \"_common.py\").is_file():\n", + " ROOT = c\n", + " break\n", + "\n", "if IN_COLAB:\n", " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", " \"numpy\", \"scipy\", \"matplotlib\", \"GPy>=1.15\", \"scikit-learn\"])\n", " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", - " \"git+https://github.com/SheffieldML/PyDeepGP.git@examples-damianou2013\"])\n", - " os.makedirs(\"data\", exist_ok=True)\n", - " files = [\n", - " (\"_common.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_common.py\"),\n", - " (\"_matlab_data.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_matlab_data.py\"),\n", - " (\"toy_hierarchy.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/toy_hierarchy.py\"),\n", - " (\"toy_warped_regression.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/toy_warped_regression.py\"),\n", - " (\"usps_digits.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/usps_digits.py\"),\n", - " (\"highfive_shared.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/highfive_shared.py\"),\n", + " f\"git+https://github.com/SheffieldML/PyDeepGP.git@examples-damianou2013\"])\n", + " ROOT = Path.cwd().resolve()\n", + " (ROOT / \"data\").mkdir(exist_ok=True)\n", + " for dest, url in [\n", + " (\"_common.py\", f\"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_common.py\"),\n", + " (\"_matlab_data.py\", f\"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_matlab_data.py\"),\n", " (\"data/hgplvmSampleDataTr1.mat\",\n", " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/hgplvmSampleDataTr1.mat\"),\n", " (\"data/demHighFiveHgplvm1.mat\",\n", " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/demHighFiveHgplvm1.mat\"),\n", - " ]\n", - " for dest, url in files:\n", - " if not os.path.isfile(dest):\n", + " ]:\n", + " p = ROOT / dest\n", + " if not p.is_file():\n", " print(\"Fetching\", dest)\n", - " urllib.request.urlretrieve(url, dest)\n", - " print(\"Colab setup done.\")\n", - "else:\n", - " print(\"Local: run from repo; ensure examples/damianou2013 is on sys.path.\")\n" + " urllib.request.urlretrieve(url, str(p))\n", + "\n", + "os.chdir(ROOT)\n", + "sys.path.insert(0, str(ROOT))\n", + "print(\"Working directory:\", ROOT)\n" ] }, { @@ -61,17 +79,14 @@ "metadata": {}, "outputs": [], "source": [ - "import os, sys\n", - "from pathlib import Path\n", - "here = Path.cwd().resolve()\n", - "candidates = [here, here.parent, here / \"examples\" / \"damianou2013\",\n", - " here.parent / \"examples\" / \"damianou2013\"]\n", - "for c in candidates:\n", - " if (c / \"toy_hierarchy.py\").is_file():\n", - " sys.path.insert(0, str(c))\n", - " os.chdir(c)\n", - " break\n", - "print(\"cwd:\", os.getcwd())\n" + "%matplotlib inline\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "import deepgp\n", + "from _common import (\n", + " ard_weights, dominant_dims, nearest_neighbour_error, optimize_staged, print_ard, rbf_bias,\n", + ")\n", + "from _matlab_data import USPS_DEPTH_Q, load_usps_vercat2\n" ] }, { @@ -80,7 +95,19 @@ "metadata": {}, "outputs": [], "source": [ - "QUICK = True # depth 1\u20132 only; False runs depths 1\u20135 (slow)\n" + "QUICK = True # True \u2192 fewer digits / smaller Q / depths 1\u20132\n", + "DEPTHS = [1, 2] if QUICK else [1, 2, 3, 4, 5]\n", + "per_class = 20 if QUICK else 50\n", + "num_inducing = 25 if QUICK else 80\n", + "iters_fixed = 40 if QUICK else 500\n", + "iters_free = 100 if QUICK else 3000\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Load USPS `vercat2` subset\n" ] }, { @@ -89,11 +116,123 @@ "metadata": {}, "outputs": [], "source": [ - "%matplotlib inline\n", - "argv = [\"--quick\", \"--depths\", \"1\", \"2\"] if QUICK else []\n", - "argv += [\"--outdir\", \"/tmp/dgp2013_nb\"]\n", - "from usps_digits import main\n", - "main(argv)\n" + "Y, labels = load_usps_vercat2(per_class=per_class)\n", + "Y = Y - Y.mean(0)\n", + "Y = Y / (Y.std() + 1e-12)\n", + "print(f\"Y={Y.shape}, classes={sorted(set(labels.tolist()))}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Fit DeepGPs at increasing depth\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def depth_dimensions(depth, data_dim):\n", + " return [data_dim] + list(USPS_DEPTH_Q[depth])\n", + "\n", + "results = []\n", + "models = {}\n", + "for depth in DEPTHS:\n", + " dims = depth_dimensions(depth, Y.shape[1])\n", + " if QUICK:\n", + " dims = [Y.shape[1]] + [max(2, q // 2) for q in dims[1:]]\n", + " print(f\"\\n=== depth {depth} dims={dims} ===\")\n", + " m = deepgp.DeepGP(\n", + " dims, Y,\n", + " kernels=[rbf_bias(q) for q in dims[1:]],\n", + " num_inducing=num_inducing,\n", + " back_constraint=False,\n", + " inits=\"PCA\",\n", + " )\n", + " optimize_staged(m, iters_fixed, iters_free, messages=False)\n", + " bound = float(m.log_likelihood())\n", + " Xtop = np.asarray(m.layers[-1].X.mean)\n", + " mistakes, rate = nearest_neighbour_error(Xtop, labels)\n", + " print(f\"bound={bound:.3f} 1-NN mistakes={mistakes}/{Y.shape[0]} ({rate:.3%})\")\n", + " print_ard(m)\n", + " results.append((depth, bound, mistakes, rate))\n", + " models[depth] = m\n", + "\n", + "print(\"\\ndepth bound 1-NN errors\")\n", + "for depth, bound, mistakes, rate in results:\n", + " print(f\"{depth:5d} {bound:9.3f} {mistakes:3d} ({rate:.3%})\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Top latent scatter and ARD (figs. 6\u20138 style)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "deepest = max(DEPTHS)\n", + "m = models[deepest]\n", + "Xtop = np.asarray(m.layers[-1].X.mean)\n", + "dims_plot = dominant_dims(ard_weights(m.layers[-1].kern), k=2)\n", + "colors = {0: \"C0\", 1: \"C1\", 6: \"C2\"}\n", + "markers = {0: \"o\", 1: \"s\", 6: \"^\"}\n", + "\n", + "fig, ax = plt.subplots(figsize=(4.5, 4.5))\n", + "for d in (0, 1, 6):\n", + " mask = labels == d\n", + " ax.scatter(\n", + " Xtop[mask, dims_plot[0]], Xtop[mask, dims_plot[1]],\n", + " c=colors[d], marker=markers[d], s=28, label=str(d),\n", + " )\n", + "ax.set_title(f\"USPS top latent (depth {deepest})\")\n", + "ax.legend()\n", + "fig.tight_layout()\n", + "plt.show()\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "fig, axes = plt.subplots(1, len(m.layers), figsize=(3.0 * len(m.layers), 3))\n", + "if len(m.layers) == 1:\n", + " axes = [axes]\n", + "for i, layer in enumerate(m.layers):\n", + " w = ard_weights(layer.kern)\n", + " axes[i].bar(np.arange(1, len(w) + 1), w)\n", + " axes[i].set_title(f\"layer {i}\")\n", + "fig.suptitle(\"ARD (fig. 6 style)\")\n", + "fig.tight_layout()\n", + "plt.show()\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "fig, ax = plt.subplots(figsize=(5, 3.5))\n", + "ax.plot([r[0] for r in results], [r[1] for r in results], \"o-\", label=\"bound\")\n", + "ax.set_xlabel(\"depth\")\n", + "ax.set_ylabel(\"variational bound\")\n", + "ax2 = ax.twinx()\n", + "ax2.plot([r[0] for r in results], [r[2] for r in results], \"s--\", color=\"C1\")\n", + "ax2.set_ylabel(\"1-NN mistakes\")\n", + "ax.set_title(\"Depth vs bound / 1-NN\")\n", + "fig.tight_layout()\n", + "plt.show()\n" ] } ] diff --git a/examples/damianou2013/notebooks/04_highfive_mrd.ipynb b/examples/damianou2013/notebooks/04_highfive_mrd.ipynb index 27179af..c8d3002 100644 --- a/examples/damianou2013/notebooks/04_highfive_mrd.ipynb +++ b/examples/damianou2013/notebooks/04_highfive_mrd.ipynb @@ -17,7 +17,16 @@ "\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SheffieldML/PyDeepGP/blob/main/examples/damianou2013/notebooks/04_highfive_mrd.ipynb)\n", "\n", - "**Matlab:** [`demHighFiveDemonstration.m`](https://github.com/SheffieldML/deepGP/blob/master/deepGP/matlab/demHighFiveDemonstration.m) + `demHighFiveHgplvm1.mat` (YA/YB 78\u00d762).\n" + "Port of Matlab [`demHighFiveDemonstration.m`](https://github.com/SheffieldML/deepGP/blob/master/deepGP/matlab/demHighFiveDemonstration.m)\n", + "using `demHighFiveHgplvm1.mat` (subjects A/B, 78\u00d762). Fits a two-view DeepGP and\n", + "compares ARD / latents to the stored hgplvm reference spaces when present.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup\n" ] }, { @@ -27,32 +36,39 @@ "outputs": [], "source": [ "import sys, subprocess, os, urllib.request\n", + "from pathlib import Path\n", + "\n", "IN_COLAB = \"google.colab\" in sys.modules\n", + "ROOT = Path.cwd().resolve()\n", + "# Prefer sibling of notebooks/ (repo layout)\n", + "for c in [ROOT.parent, ROOT, ROOT / \"examples\" / \"damianou2013\"]:\n", + " if (c / \"_common.py\").is_file():\n", + " ROOT = c\n", + " break\n", + "\n", "if IN_COLAB:\n", " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", " \"numpy\", \"scipy\", \"matplotlib\", \"GPy>=1.15\", \"scikit-learn\"])\n", " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n", - " \"git+https://github.com/SheffieldML/PyDeepGP.git@examples-damianou2013\"])\n", - " os.makedirs(\"data\", exist_ok=True)\n", - " files = [\n", - " (\"_common.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_common.py\"),\n", - " (\"_matlab_data.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_matlab_data.py\"),\n", - " (\"toy_hierarchy.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/toy_hierarchy.py\"),\n", - " (\"toy_warped_regression.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/toy_warped_regression.py\"),\n", - " (\"usps_digits.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/usps_digits.py\"),\n", - " (\"highfive_shared.py\", \"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/highfive_shared.py\"),\n", + " f\"git+https://github.com/SheffieldML/PyDeepGP.git@examples-damianou2013\"])\n", + " ROOT = Path.cwd().resolve()\n", + " (ROOT / \"data\").mkdir(exist_ok=True)\n", + " for dest, url in [\n", + " (\"_common.py\", f\"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_common.py\"),\n", + " (\"_matlab_data.py\", f\"https://raw.githubusercontent.com/SheffieldML/PyDeepGP/examples-damianou2013/examples/damianou2013/_matlab_data.py\"),\n", " (\"data/hgplvmSampleDataTr1.mat\",\n", " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/hgplvmSampleDataTr1.mat\"),\n", " (\"data/demHighFiveHgplvm1.mat\",\n", " \"https://raw.githubusercontent.com/SheffieldML/deepGP/master/deepGP/matlab/demHighFiveHgplvm1.mat\"),\n", - " ]\n", - " for dest, url in files:\n", - " if not os.path.isfile(dest):\n", + " ]:\n", + " p = ROOT / dest\n", + " if not p.is_file():\n", " print(\"Fetching\", dest)\n", - " urllib.request.urlretrieve(url, dest)\n", - " print(\"Colab setup done.\")\n", - "else:\n", - " print(\"Local: run from repo; ensure examples/damianou2013 is on sys.path.\")\n" + " urllib.request.urlretrieve(url, str(p))\n", + "\n", + "os.chdir(ROOT)\n", + "sys.path.insert(0, str(ROOT))\n", + "print(\"Working directory:\", ROOT)\n" ] }, { @@ -61,17 +77,34 @@ "metadata": {}, "outputs": [], "source": [ - "import os, sys\n", - "from pathlib import Path\n", - "here = Path.cwd().resolve()\n", - "candidates = [here, here.parent, here / \"examples\" / \"damianou2013\",\n", - " here.parent / \"examples\" / \"damianou2013\"]\n", - "for c in candidates:\n", - " if (c / \"toy_hierarchy.py\").is_file():\n", - " sys.path.insert(0, str(c))\n", - " os.chdir(c)\n", - " break\n", - "print(\"cwd:\", os.getcwd())\n" + "%matplotlib inline\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "import deepgp\n", + "from _common import (\n", + " ard_weights, dominant_dims, init_noise, unfix_noise, rbf_bias,\n", + ")\n", + "from _matlab_data import load_highfive\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "QUICK = True\n", + "q1, q2 = (4, 3) if QUICK else (6, 4)\n", + "num_inducing = 15 if QUICK else 30\n", + "iters_fixed = 40 if QUICK else 200\n", + "iters_free = 80 if QUICK else 1500\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Load high-five views\n" ] }, { @@ -80,7 +113,19 @@ "metadata": {}, "outputs": [], "source": [ - "QUICK = True\n" + "data = load_highfive()\n", + "YA = data[\"YA\"].copy()\n", + "YB = data[\"YB\"].copy()\n", + "YA = (YA - YA.mean(0)) / (YA.std() + 1e-12)\n", + "YB = (YB - YB.mean(0)) / (YB.std() + 1e-12)\n", + "print(f\"YA={YA.shape} YB={YB.shape}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Fit two-view MRD DeepGP\n" ] }, { @@ -89,11 +134,105 @@ "metadata": {}, "outputs": [], "source": [ - "%matplotlib inline\n", - "argv = [\"--quick\"] if QUICK else []\n", - "argv += [\"--outdir\", \"/tmp/dgp2013_nb\"]\n", - "from highfive_shared import main\n", - "main(argv)\n" + "m = deepgp.DeepGP(\n", + " [[YA.shape[1], YB.shape[1]], q1, q2],\n", + " [YA, YB],\n", + " kernels=[[rbf_bias(q1), rbf_bias(q1)], rbf_bias(q2)],\n", + " num_inducing=num_inducing,\n", + " back_constraint=False,\n", + " inits=\"PCA\",\n", + ")\n", + "init_noise(m, fix=True)\n", + "m.optimize(max_iters=iters_fixed, messages=False)\n", + "unfix_noise(m)\n", + "m.optimize(max_iters=iters_free, messages=False)\n", + "print(f\"MRD DeepGP variational bound: {float(m.log_likelihood()):.3f}\")\n", + "for i, view in enumerate(m.layers[0].views):\n", + " print(f\"View {i} ARD:\", np.array2string(ard_weights(view.kern), precision=3))\n", + "print(\"Top ARD:\", np.array2string(ard_weights(m.layers[-1].kern), precision=3))\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## ARD bars (fig. 4 style)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "fig, axes = plt.subplots(1, 3, figsize=(10, 3.2))\n", + "for i, view in enumerate(m.layers[0].views):\n", + " w = ard_weights(view.kern)\n", + " axes[i].bar(np.arange(1, len(w) + 1), w)\n", + " axes[i].set_title(f\"view {i} ARD (fY)\")\n", + " axes[i].set_xlabel(\"shared dim\")\n", + "w_top = ard_weights(m.layers[-1].kern)\n", + "axes[2].bar(np.arange(1, len(w_top) + 1), w_top)\n", + "axes[2].set_title(\"top ARD (fX)\")\n", + "fig.suptitle(\"High-five MRD ARD (fig. 4 style)\")\n", + "fig.tight_layout()\n", + "plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Latent projections (fig. 5 style)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "Xshare = np.asarray(m.layers[0].X.mean)\n", + "Xtop = np.asarray(m.layers[-1].X.mean)\n", + "d_share = dominant_dims(ard_weights(m.layers[0].views[0].kern), k=2)\n", + "d_top = dominant_dims(ard_weights(m.layers[-1].kern), k=2)\n", + "\n", + "fig, axes = plt.subplots(1, 2, figsize=(8, 3.5))\n", + "axes[0].plot(Xshare[:, d_share[0]], Xshare[:, d_share[1]], \".-\")\n", + "axes[0].set_title(\"DeepGP shared (dominant)\")\n", + "axes[1].plot(Xtop[:, d_top[0]], Xtop[:, d_top[1]], \".-\")\n", + "axes[1].set_title(\"DeepGP top (dominant)\")\n", + "fig.suptitle(\"Latent projections (fig. 5 style)\")\n", + "fig.tight_layout()\n", + "plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Reference hgplvm latents from the Matlab mat (if present)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "if \"Xparent\" in data:\n", + " fig, axes = plt.subplots(1, 3, figsize=(10, 3.2))\n", + " axes[0].plot(data[\"Xparent\"][:, 0], data[\"Xparent\"][:, 1], \".-\")\n", + " axes[0].set_title(\"hgplvm Xparent (ref)\")\n", + " axes[1].plot(data[\"Xchild1\"][:, 0], data[\"Xchild1\"][:, 1], \".-\")\n", + " axes[1].set_title(\"hgplvm Xchild1 (ref)\")\n", + " axes[2].plot(data[\"Xchild2\"][:, 0], data[\"Xchild2\"][:, 1], \".-\")\n", + " axes[2].set_title(\"hgplvm Xchild2 (ref)\")\n", + " fig.suptitle(\"Reference latents from demHighFiveHgplvm1.mat\")\n", + " fig.tight_layout()\n", + " plt.show()\n", + "else:\n", + " print(\"No Xparent/Xchild* fields in mat.\")\n" ] } ] diff --git a/examples/damianou2013/notebooks/README.md b/examples/damianou2013/notebooks/README.md index 64962c6..771f63b 100644 --- a/examples/damianou2013/notebooks/README.md +++ b/examples/damianou2013/notebooks/README.md @@ -1,6 +1,9 @@ # Colab notebooks (Matlab-aligned) -Thin wrappers around the Python ports of [SheffieldML/deepGP](https://github.com/SheffieldML/deepGP) demos. -Each notebook installs deps on Colab, fetches scripts / reference `.mat` files, and calls the corresponding module. +Walkthrough notebooks for the Damianou & Lawrence (AISTATS 2013) ports. +Each notebook expands the experiment step by step (data → fit → plots). +Shared helpers live in `../_common.py` and `../_matlab_data.py`. See [../SOURCE.md](../SOURCE.md) and [../README.md](../README.md). + +Matlab demos: [SheffieldML/deepGP](https://github.com/SheffieldML/deepGP).