I see that Lipschitz continuity is a common assumption used in optimisation, statistics, machine learning, etc.


Could you point me in the direction of some literature that discusses why Lipschitz continuity is commonly assumed? Does it naturally occur in applications? Contains an important class of functions?


EDIT: I’ve found that it’s central for ODEs because of Picard-Lindelöf theorem but I’d like to have something closer to statistics or optimization.



Reference request: importance of Lipschitz continuity
**In Mathematical/High Dimensional Statistics:**


One fairly amazing result is for $X=(X\_1,\dots,X\_n)$ where the coordinates are i.i.d. standard Gaussians, and $f:R^n \to R$ a $L$-Lipschitz function (w.r.t. Euclidean norm), then the random variable $f(X) - E[f(X)]$ is sub-Gaussian with sub-Gaussian norm $L$, i.e. it satisfies for any $\epsilon \ge 0$:
$$
P(|f(X)- E[f(X)]| \ge \epsilon) \le 2 e^{-\epsilon^2/2 L^2}.
$$
It is somewhat surprising (to me at least) how often this result comes in handy - particularly when working with rademacher/gaussian complexities in mathematical statistics/learning theory and also in random matrix analysis.


To give one concrete example, if $X$ is a $n \times p$ random matrix with i.i.d. standard gaussian entries, and $Y$ is another matrix, then by Weyl's inequality:


$$
\max\_{k=1,\dots, p} |\sigma\_k(X) - \sigma\_k(Y)| \le \|X-Y \|\_F,
$$
where $\sigma\_k$ is the $k$-th largest singular value. This tells us that the $k$-th singular value is a 1-Lipschitz function of the matrix, and so we immediately get
$$
P(|\sigma\_k(X) - E[\sigma\_k(X)]| \ge \epsilon) \le 2e^{-\epsilon^2/2}.
$$


I would suggest the following two books for more on this kind of usage of Lipschitz continuity:


1. High Dimensional Statistics - A non-asymptotic viewpoint by Martin J Wainwright
2. High Dimensional Probability by Roman Vershynin


**In machine learning:**


The concept of Lipschitz continuity is also important in the current machine/deep learning literature where model robustness (which they refer to as adversarial robustness) is an important issue. To understand the robustness of a complicated model (such as a neural network), a lot of work has gone into training networks that define an input-output map with small Lipschitz constant. The intuition here is that if my model is robust, it should not be too affected by perturbations in the input, $f(x+\delta x) \approx f(x)$, and this would be ensured by having $f$ be L-Lipschitz where $L$ is small. See this paper [Training robust neural networks using Lipschitz bounds](https://arxiv.org/pdf/2005.02929.pdf) and their references for more on this.


