Curse of Dimensionality
6110572
Approximate nearest neighbors: towards removing the curse of dimensionality
We present two algorithms for the approximate nearest neighbor problem in high dimensional spaces. For data sets of size n living in R d the algorithms require space that is only polynomial in n and d, while achieving query times that are sub linear in n and polynomial in d. We also show applications to other high dimensional geometric problems, such as the approximate minimum spanning tree. The article is based on the material from the authors' STOC'98 and FOCS'01 papers. It unifies, generalizes and simplifies the results from those papers.
STOC '98
Piotr Indyk Rajeev Motwani
54216403
Overcoming the curse of dimensionality in the numerical approximation of semilinear parabolic partial differential equations
For a long time it has been well known that high dimensional linear parabolic partial differential equations (PDEs) can be approximated by Monte Carlo methods with a computational effort which grows polynomially both in the dimension and in the reciprocal of the prescribed accuracy. In other words, linear PDEs do not suffer from the curse of dimensionality. For general semilinear PDEs with Lipschitz coefficients, however, it remained an open question whether these suffer from the curse of dimensionality. In this paper we partially solve this open problem. More precisely, we prove in the case of semilinear heat equations with gradient independent and globally Lipschitz continuous nonlinearities that the computational effort of a variant of the recently introduced multilevel Picard approximations grows at most polynomially both in the dimension and in the reciprocal of the required accuracy.
Proceedings of the Royal Society A
Martin Hutzenthaler Arnulf Jentzen Thomas Kruse Tuan Anh Nguyen Philippe von Wurstemberger
211677439
Overcoming the curse of dimensionality in the numerical approximation of high dimensional semilinear elliptic partial differential equations
Recently, so called full history recursive multilevel Picard (MLP) approximation schemes have been introduced and shown to overcome the curse of dimensionality in the numerical approximation of semilinear parabolic partial differential equations (PDEs) with Lipschitz nonlinearities. The key contribution of this article is to introduce and analyze a new variant of MLP approximation schemes for certain semilinear elliptic PDEs with Lipschitz nonlinearities and to prove that the proposed approximation schemes overcome the curse of dimensionality in the numerical approximation of such semilinear elliptic PDEs.
ArXiv
Christian Beck Lukas Gonon Arnulf Jentzen
196831841
Overcoming the curse of dimensionality in the numerical approximation of Allen Cahn partial differential equations via truncated full history recursive multilevel Picard approximations
Abstract One of the most challenging problems in applied mathematics is the approximate solution of nonlinear partial differential equations (PDEs) in high dimensions. Standard deterministic approximation methods like finite differences or finite elements suffer from the curse of dimensionality in the sense that the computational effort grows exponentially in the dimension. In this work we overcome this difficulty in the case of reaction diffusion type PDEs with a locally Lipschitz continuous coervice nonlinearity (such as Allen Cahn PDEs) by introducing and analyzing truncated variants of the recently introduced full history recursive multilevel Picard approximation schemes.
J. Num. Math.
Christian Beck Fabian Hornung Martin Hutzenthaler Arnulf Jentzen Thomas Kruse
211068735
Curse of Dimensionality on Randomized Smoothing for Certifiable Robustness
Randomized smoothing, using just a simple isotropic Gaussian distribution, has been shown to produce good robustness guarantees against \ell_2$ norm bounded adversaries. In this work, we show that extending the smoothing technique to defend against other attack models can be challenging, especially in the high dimensional regime. In particular, for a vast class of i.i.d.~smoothing distributions, we prove that the largest \ell_p$ radius that can be certified decreases as $O(1/d^\frac{1}{2} \frac{1}{p} with dimension $d$ for $p 2$ Notably, for $p \geq 2$ this dependence on $d$ is no better than that of the \ell_p$ radius that can be certified using isotropic Gaussian smoothing, essentially putting a matching lower bound on the robustness radius. When restricted to \it generalized} Gaussian smoothing, these two bounds can be shown to be within a constant factor of each other in an asymptotic sense, establishing that Gaussian smoothing provides the best possible results, up to a constant factor, when $p \geq 2$ We present experimental results on CIFAR to validate our theory. For other smoothing distributions, such as, a uniform distribution within an \ell_1$ or an \ell_\infty$ norm ball, we show upper bounds of the form $O(1 d) and $O(1 d^{1 \frac{1}{p} respectively, which have an even worse dependence on $d$
ICML
Aounon Kumar Alexander Levine Tom Goldstein Soheil Feizi
53015027
Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality
Deep learning has shown high performances in various types of tasks from visual recognition to natural language processing, which indicates superior flexibility and adaptivity of deep learning. To understand this phenomenon theoretically, we develop a new approximation and estimation error analysis of deep learning with the ReLU activation for functions in a Besov space and its variant with mixed smoothness. The Besov space is a considerably general function space including the Holder space and Sobolev space, and especially can capture spatial inhomogeneity of smoothness. Through the analysis in the Besov space, it is shown that deep learning can achieve the minimax optimal rate and outperform any non adaptive (linear) estimator such as kernel ridge regression, which shows that deep learning has higher adaptivity to the spatial inhomogeneity of the target function than other estimators such as linear ones. In addition to this, it is shown that deep learning can avoid the curse of dimensionality if the target function is in a mixed smooth Besov space. We also show that the dependency of the convergence rate on the dimensionality is tight due to its minimax optimality. These results support high adaptivity of deep learning and its superior ability as a feature extractor.
ICLR
Taiji Suzuki
51846323
On deep learning as a remedy for the curse of dimensionality in nonparametric regression
Assuming that a smoothness condition and a suitable restriction on the structure of the regression function hold, it is shown that least squares estimates based on multilayer feedforward neural networks are able to circumvent the curse of dimensionality in nonparametric regression. The proof is based on new approximation results concerning multilayer feedforward neural networks with bounded weights and a bounded number of hidden neurons. The estimates are compared with various other approaches by using simulated data.
The Annals of Statistics
Benedikt Bauer Michael Kohler
1474026
Breaking the Curse of Dimensionality with Convex Neural Networks
We consider neural networks with a single hidden layer and non decreasing homogeneous activa tion functions like the rectified linear units. By letting the number of hidden units grow unbounded and using classical non Euclidean regularization tools on the output weights, we provide a detailed theoretical analysis of their generalization performance, with a study of both the approximation and the estimation errors. We show in particular that they are adaptive to unknown underlying linear structures, such as the dependence on the projection of the input variables onto a low dimensional subspace. Moreover, when using sparsity inducing norms on the input weights, we show that high dimensional non linear variable selection may be achieved, without any strong assumption regarding the data and with a total number of variables potentially exponential in the number of ob servations. In addition, we provide a simple geometric interpretation to the non convex problem of addition of a new unit, which is the core potentially hard computational element in the framework of learning from continuously many basis functions. We provide simple conditions for convex relaxations to achieve the same generalization error bounds, even when constant factor approxi mations cannot be found (e.g. because it is NP hard such as for the zero homogeneous activation function) We were not able to find strong enough convex relaxations and leave open the existence or non existence of polynomial time algorithms.
J. Mach. Learn. Res.
Francis R Bach
108317166
A proof that rectified deep neural networks overcome the curse of dimensionality in the numerical approximation of semilinear heat equations
Deep neural networks and other deep learning methods have very successfully been applied to the numerical approximation of high dimensional nonlinear parabolic partial differential equations (PDEs) which are widely used in finance, engineering, and natural sciences. In particular, simulations indicate that algorithms based on deep learning overcome the curse of dimensionality in the numerical approximation of solutions of semilinear PDEs. For certain linear PDEs this has also been proved mathematically. The key contribution of this article is to rigorously prove this for the first time for a class of nonlinear PDEs. More precisely, we prove in the case of semilinear heat equations with gradient independent nonlinearities that the numbers of parameters of the employed deep neural networks grow at most polynomially in both the PDE dimension and the reciprocal of the prescribed approximation accuracy. Our proof relies on recently introduced multilevel Picard approximations of semilinear PDEs.
Martin Hutzenthaler Arnulf Jentzen Thomas Kruse Tuan Anh Nguyen
226981498
Addressing the curse of dimensionality in stochastic dynamics: a Wiener path integral variational formulation with free boundaries
A Wiener path integral variational formulation with free boundaries is developed for determining the stochastic response of high dimensional nonlinear dynamical systems in a computationally efficient manner. Specifically, a Wiener path integral representation of a marginal or lower dimensional joint response probability density function is derived. Due to this a priori marginalization, the associated computational cost of the technique becomes independent of the degrees of freedom (d.f. or stochastic dimensions of the system, and thus, the 'curse of dimensionality' in stochastic dynamics is circumvented. Two indicative numerical examples are considered for highlighting the capabilities of the technique. The first relates to marine engineering and pertains to a structure exposed to nonlinear flow induced forces and subjected to non white stochastic excitation. The second relates to nano engineering and pertains to a 100 d.f. stochastically excited nonlinear dynamical system modelling the behaviour of large arrays of coupled nano mechanical oscillators. Comparisons with pertinent Monte Carlo simulation data demonstrate the computational efficiency and accuracy of the developed technique.
Proceedings of the Royal Society A
Ioannis Petromichelakis Ioannis A Kougioumtzoglou