A neural network can be considered as a function


$$\mathbf{R}^m\to\mathbf{R}^n\quad
\text{by}\quad x\mapsto w\_N\sigma(h\_{N-1}+w\_{N-1}\sigma(\dotso h\_2+w\_2\sigma(h\_1+w\_1 x)\dotso)),$$


where the $w\_i$ are linear functions (matrices) $\mathbf{R}^{d\_{i-1}}\to\mathbf{R}^{d\_i}$, the $h\_i\in \mathbf{R}^{d\_i}$, with $d\_0=m$ and $d\_N=n$, and $\sigma\colon\mathbf{R}\to\mathbf{R}$ is a non-linear function (e.g. sigmoid), which in the formula above must actually be understood as $\sigma\oplus\dotsb\oplus\sigma$ an appropriate number of times.


There are several studies in the literature (e.g. <https://doi.org/10.1016%2F0893-6080%2891%2990009-T>) proving that the set of such functions (for fixed $N$ and $\{d\_i\}$) is dense in other function spaces, such as measurable-function or continuous-function spaces. Other studies focus on how well functions in this set approximate functions in other sets, according to various measures.


I would like to know, instead, what kind of mathematical structures the set of such functions enjoys, either for fixed or for variable $N$ and $\{d\_i\}$. For example, is it a vector space? (answer seems to be yes if the $\{d\_i\}$ aren't fixed) Is it a ring under function composition or under some other operation? Is it a convex set? – And similar questions.


I'm thankful to anyone who can provide some literature about such questions.


Update – example: this brilliant study by Petersen, Raslan, Voigtlaender <https://arxiv.org/abs/1806.08459> answers the question about convexity.



Structures of the space of neural networks
In information geometry, people study structures of Riemannian manifolds with dual affine connections on sets of neural networks. (The metric measures how close neural networks are in their input-output behaviour.) Riemannian geometry can then be used to study learning algorithms (like gradient descent methods). Here are a couple of references, more can be found by searching for the tags "information geometry" and "neural networks".


* S. Amari. Information geometry of the EM and em algorithms for neural networks. Neural Networks 8 (1995), 1379-1408.
* S. Amari, K. Kurata and H. Nagaoka. Information geometry of Boltzmann machines. IEEE Transactions on Neural Networks, vol. 3, no. 2, (1992), 260-271.
* S. Amari. Information Geometry of Neural Networks — An Overview —. In: Ellacott S.W., Mason J.C., Anderson I.J. (eds) Mathematics of Neural Networks. Operations Research/Computer Science Interfaces Series, vol 8. Springer, Boston, MA, 1997. [(link to Springer website, paywall)](https://link.springer.com/chapter/10.1007/978-1-4615-6099-9_2)


I don't know about recent developments, all the references above are well before the deep learning revolution.


