uncertainty on Dan MacKinlay
https://danmackinlay.name/tags/uncertainty.html
Recent content in uncertainty on Dan MacKinlayHugo -- gohugo.ioen-usThu, 28 Oct 2021 17:48:25 +1100Differentiable PDE solvers
https://danmackinlay.name/notebook/pde_solvers_diff.html
Thu, 28 Oct 2021 17:48:25 +1100https://danmackinlay.name/notebook/pde_solvers_diff.htmlMantaflow/Phiflow jax-cfd DeepXDE JuliaFEM ADCME FEniCS dolfin-adjoint TenFEM Trixi taichi References Suppose we are keen to devise yet another method that will do clever things to augment PDE solvers with ML somehow. To that end it would be nice to have a PDE solver that was not a completely black box but which we could interrogate for useful gradients. Obviously all PDE solvers use gradient information, but only some of them expose that to us as users; e.Machine learning for partial differential equations
https://danmackinlay.name/notebook/ml_pde.html
Thu, 28 Oct 2021 16:31:07 +1100https://danmackinlay.name/notebook/ml_pde.htmlBackground The PINN lineage Deterministic PINN Stochastic PINN Weak formulation Learning a PDE forward operator Fourier neural operator DeepONet GAN approaches Advection-diffusion PDEs in particular Boundary conditions Inverse problems As implicit representations Differentiable solvers References \(\newcommand{\solop}{\mathcal{G}^{\dagger}}\)
Using statistical or machine learning approaches to solve PDEs, and maybe even to perform inference through them. There are many approaches to ML learning of PDEs and I will document on an ad hoc basis as I need them.Probabilistic neural nets
https://danmackinlay.name/notebook/nn_probabilistic.html
Thu, 30 Sep 2021 09:33:20 +1000https://danmackinlay.name/notebook/nn_probabilistic.htmlBackgrounders MC sampling of weights by low-rank Matheron updates Mixture density networks Variational autoencoders Sampling via Monte Carlo Stochastic Gradient Descent as MC inference Laplace approximation Learnable Laplace approximations By stochastic weight averaging. Via random projections In Gaussian process regression Via measure transport Via infinite-width random nets Via NTK Ensemble methods Practicalities References Inferring densities and distributions in a massively parameterised deep learning setting.Bootstrap
https://danmackinlay.name/notebook/bootstrap.html
Sat, 18 Sep 2021 11:45:52 +1000https://danmackinlay.name/notebook/bootstrap.htmlBootstrap bias correction Bootstrap for dependent data Causal bootstrap As a Bayesian method Pedagogic References Resampling your own data to estimate how good your point-estimator is, and to reduce its bias. In general an intuitive technique. However, gets tricky for e.g. dependent data. For a handy crib sheet for bootstrap failure modes, see Thomas Lumley, When the bootstrap doesn’t work.
In the classical mode, this is a frequentist technique without an immediate Bayesian interpretation.Convolutional stochastic processes
https://danmackinlay.name/notebook/stochastic_convolution.html
Mon, 16 Aug 2021 09:28:19 +1000https://danmackinlay.name/notebook/stochastic_convolution.htmlReferences Stochastic processes generated by convolution of white noise with smoothing kernels, which is not unlike kernel smoothing where the “data” is random. Or, to put it another way, these are processes defined as moving averages of some stochastic noise.
For now, I am mostly interested in certain special cases Gaussian convolutions and subordinator convolutions.
C&C Karhunen-Loeve expansion.
References Adler, Robert J. 2010. The Geometry of Random Fields.Uncertainty quantification
https://danmackinlay.name/notebook/uncertainty_quantification.html
Tue, 06 Jul 2021 15:31:29 +1000https://danmackinlay.name/notebook/uncertainty_quantification.htmlTaxonomy DUQ networks Bayes Physical model setting Conformal prediction Chaos expansions Uncertainty Quantification 360 References Using machine learning to make predictions, with a measure of the confidence of those predictions.
Taxonomy Should clarify. TBD. Here is a recent reference on the theme: Kendall and Gal (2017) This extricates aleatoric and epistemic uncertainty. Also to mention, model uncertainty.
DUQ networks Amersfoort et al. (2020); Kendall and Gal (2017)Stochastic processes which represent measures over the reals
https://danmackinlay.name/notebook/measure_priors.html
Mon, 08 Mar 2021 16:44:16 +1100https://danmackinlay.name/notebook/measure_priors.htmlSubordinators Other measure priors References Often I need to have a nonparametric representation for a measure over some non-finite index set. We might want to represent a probability, or mass, or a rate. I might want this representation to be something flexible and low-assumption, like a Gaussian process. If I want a nonparametric representation of functions this is not hard; I can simply use a Gaussian process.Convolutional subordinator processes
https://danmackinlay.name/notebook/subordinator_convolution.html
Mon, 08 Mar 2021 15:29:19 +1100https://danmackinlay.name/notebook/subordinator_convolution.htmlReferences Stochastic processes by convolution of noise with smoothing kernels, where the driving noise is a Lévy subordinator.
Why would we want this? One reason is that this gives us a way to create nonparametric distributions over measures.
References Barndorff-Nielsen, O. E., and J. Schmiegel. 2004. “Lévy-Based Spatial-Temporal Modelling, with Applications to Turbulence.” Russian Mathematical Surveys 59 (1): 65. https://doi.org/10.1070/RM2004v059n01ABEH000701. Çinlar, E. 1979. “On Increasing Continuous Processes.Learning on manifolds
https://danmackinlay.name/notebook/learning_on_manifolds.html
Wed, 03 Mar 2021 12:29:39 +1100https://danmackinlay.name/notebook/learning_on_manifolds.htmlLearning on a given manifold Information Geometry Hamiltonian Monte Carlo Langevin Monte Carlo Natural gradient Homogeneous probability References Abraham Bosse, Moyen vniuersel de pratiquer la perspectiue sur les tableaux, ou surfaces irregulieres : ensemble quelques particularitez concernant cet art, & celuy de la graueure en taille-douce (1653)
A placeholder for learning on curved spaces. Not discussed: learning OF the curvature of spaces.
AFAICT this usually boils down to defining an appropriate stochastic process on a manifold.Stochastic processes on manifolds
https://danmackinlay.name/notebook/stochastic_processes_on_manifolds.html
Mon, 01 Mar 2021 16:13:24 +1100https://danmackinlay.name/notebook/stochastic_processes_on_manifolds.htmlReferences TBD.
References Adler, Robert J. 2010. The Geometry of Random Fields. SIAM ed. Philadelphia: Society for Industrial and Applied Mathematics. Adler, Robert J., and Jonathan E. Taylor. 2007. Random Fields and Geometry. Springer Monographs in Mathematics 115. New York: Springer. https://doi.org/10.1007/978-0-387-48116-6. Adler, Robert J, Jonathan E Taylor, and Keith J Worsley. 2016. Applications of Random Fields and Geometry Draft. https://robert.net.technion.ac.il/files/2016/08/hrf1.pdf. Bhattacharya, Abhishek, and Rabi Bhattacharya.Weighted data in statistics
https://danmackinlay.name/notebook/weighted_data.html
Fri, 06 Nov 2020 08:48:18 +1100https://danmackinlay.name/notebook/weighted_data.htmlThomas Lumley helpfully disambiguates the “three and half distinct uses of the term weights in statistical methodology”.
The three main types of weights are
the ones that show up in the classical theory of weighted least squares. These describe the precision (1/variance) of observations. …. I call these precision weights; Stata calls them analytic weights. the ones that show up in categorical data analysis. These describe cell sizes in a data set, so a weight of 10 means that there are 10 identical observations in the dataset, which have been compressed to a covariate pattern plus a count.Monte Carlo gradient estimation
https://danmackinlay.name/notebook/mc_grad.html
Wed, 30 Sep 2020 10:59:22 +1000https://danmackinlay.name/notebook/mc_grad.htmlReferences Taking gradients through integrals.
See Mohamed et al. (2020) for a roundup.
https://github.com/deepmind/mc_gradients
A common activity for me at the moment is differentiating the integral - for example, through the inverse-CDF lookup.
You see, what I would really like is the derivative of the mass-preserving continuous map \(\phi_{\theta, \tau}\) such that
\[\mathsf{z}\sim F(\cdot;\theta) \Rightarrow \phi_{\theta, \tau}(\mathsf{z})\sim F(\cdot;\tau). \] Now suppose I wish to optimise or otherwise perturb \(\theta\).Nearly sufficient statistics
https://danmackinlay.name/notebook/nearly_sufficient_statistics.html
Mon, 14 Jan 2019 15:10:53 +1100https://danmackinlay.name/notebook/nearly_sufficient_statistics.htmlSufficient statistics in exponential families References 🏗
I’m working through a small realisation, for my own interest, which has been helpful in my understanding of variational Bayes; specifically relating it to non-Bayes variational inference. Also sequential monte carlo.
By starting from the idea of sufficient statistics, we come to the idea of variational inference in a natural way, via some other interesting stopovers.
Consider the Bayes filtering setup.Gaussian Process simulation and circulant embeddings
https://danmackinlay.name/notebook/covariance_simulation.html
Fri, 10 Nov 2017 11:33:22 +1100https://danmackinlay.name/notebook/covariance_simulation.htmlTODO Simulating Gaussian fields with the desired covariance structure The circulant embedding trick Simulating point processes with the desired covariance structure References An converse problem to covariance estimation. Related: phase retrieval. Gaussian process regression.
TODO Discuss in the context of
deterministic covariance expected autocorrelation in covariance Simulating Gaussian fields with the desired covariance structure Following the introduction in (Dietrich and Newsam 1993):Stability (in learning)
https://danmackinlay.name/notebook/stability_in_learning.html
Wed, 05 Oct 2016 18:34:22 +1100https://danmackinlay.name/notebook/stability_in_learning.htmlReferences Your estimate is robust to a deleted data point? it is a stable estimate. This implies generalisability, apparently. The above statements can be made precise, I am told. Making them precise might give us new ideas for risk bounds, model selection, or connections to optimization.
Supposedly there is also connection to differential privacy, but since I don’t yet know anything about differential privacy I can’t take that statement any further except to note I would like to work it out one day.