text:
# Hinterland¶

Enable code autocompletion menu for every keypress in a code cell, instead of only calling it with tab.

The nbextension adds an item to the help menu to turn auto-hinting on and off, and offers some options for configuration:

## Options¶

• hinterland.hint_delay: delay in milliseconds between keypress & hint request. This is used to help ensure that the character from the keypress is added to the CodeMirror editor before the hint request checks the character preceding the cursor against the regexes below.
• hinterland.enable_at_start: Whether to enable hinterland’s continuous hinting when notebook is first opened, or if false, only when selected from the help-menu item.
• hinterland.hint_inside_comments: Whether to request hints while typing code comments. Defaults to false.
• hinterland.exclude_regexp: A regular expression tested against the character before the cursor, which, if a match occurs, prevents autocompletion from being triggered. This is useful, for example, to prevent triggering autocomplete on a colon, which is included by the default Completer.reinvoke pattern. If blank, no test is performed. Note that the regex will be created without any flags, making it case sensitive.
• hinterland.include_regexp: A regular expression tested against the character before the cursor, which must match in order for autocompletion to be triggered. If left blank, the value of the notebook’s Completer.reinvoke_re parameter is used, which can be modified by kernels, but defaults to /[%0-9a-z._/\\:~-]/i. Note that although the Completer.reinvoke_re default is case insensitive by virtue of its /i flag, any regex specified by the user will be created without any flags, making it case sensitive.
• hinterland.tooltip_regexp: A regular expression tested against the character before the cursor, which if it matches, causes a tooltip to be triggered, instead of regular autocompletion. For python, this is useful for example for function calls, so the default regex matches opening parentheses. Note that the regex will be created without any flags, making it case sensitive.


text:
# Change margins for tcolorbox with varwidth environment

In the code posted below, I am using varwidth to shrink my tcolorbox to the text width for short text lines. However, I also have some wraparound text lines and the tcolorbox is too big in that case. How can I retain the shrink-to-fit behavior on short text snippets while changing the maximum margins of the tcolorbox? I would like the maximum margins of the gray area to be slightly less than the regular paragraph margins on both sides, but I also want to keep is shrinking to fit when the text length is smaller than that.

\documentclass[preprint]{revtex4-2}
\usepackage{lipsum}
\usepackage{setspace}
\usepackage{varwidth}
\usepackage{tcolorbox}

\tcbuselibrary{breakable}

\newsavebox{\myboxbox}
\NewDocumentEnvironment{mybox}{+b}% find a better name
{%
\centering
\setlength{\parindent}{0pt}%
% measure the text width
\sbox{\myboxbox}{\begin{varwidth}{\columnwidth}#1\end{varwidth}}%
\begin{tcolorbox}[
breakable,
width=\dimexpr\wd\myboxbox+8mm,
top=3mm,
bottom=4mm,
]
#1
\end{tcolorbox}%
}{}

\begin{document}

\lipsum[1]\\

\begin{mybox}
\begin{singlespace}
\lipsum[1]
\end{singlespace}
\end{mybox}

\begin{mybox}
\begin{singlespace}
Just a few words here.  Just a few words here.
\end{singlespace}
\end{mybox}

\end{document}

Set the varwidth to \dimexpr\columnwidth-8mm to subtract off what you later add back to the width of the tcolorbox.

\documentclass[preprint]{revtex4-2}
\usepackage{lipsum}
\usepackage{setspace}
\usepackage{varwidth}
\usepackage{tcolorbox}

\tcbuselibrary{breakable}

\newsavebox{\myboxbox}
\NewDocumentEnvironment{mybox}{+b}% find a better name
{%
\centering
\setlength{\parindent}{0pt}%
% measure the text width
\sbox{\myboxbox}{\begin{varwidth}{\dimexpr\columnwidth-8mm}#1\end{varwidth}}%
\begin{tcolorbox}[
breakable,
width=\dimexpr\wd\myboxbox+8mm,
top=3mm,
bottom=4mm,
]
#1
\end{tcolorbox}%
}{}

\begin{document}

\lipsum[1]\\

\begin{mybox}
\begin{singlespace}
\lipsum[1]
\end{singlespace}
\end{mybox}

\begin{mybox}
\begin{singlespace}
Just a few words here.  Just a few words here.
\end{singlespace}
\end{mybox}

\end{document}

• Thanks! You are really cool! – hodop smith Mar 25 at 19:31

format: plain


text:
## Tokyo Journal of Mathematics

### A Sufficient Condition That $J(X^*)=J(X)$ Holds for a Banach Space $X$

#### Abstract

It is shown that the James constant of the space $\mathbb{R}^2$ endowed with a $\pi/2$-rotation invariant norm coincides with that of its dual space. As a corollary, we have the same statement on symmetric absolute norms on $\mathbb{R}^2$.

#### Article information

Source
Tokyo J. Math., Volume 41, Number 1 (2018), 219-223.

Dates
First available in Project Euclid: 18 December 2017

Permanent link to this document
https://projecteuclid.org/euclid.tjm/1513566019

Mathematical Reviews number (MathSciNet)
MR3830815

Zentralblatt MATH identifier
06966865

Subjects
Primary: 46B20: Geometry and structure of normed linear spaces

#### Citation

KOMURO, Naoto; SAITO, Kichi-Suke; TANAKA, Ryotaro. A Sufficient Condition That $J(X^*)=J(X)$ Holds for a Banach Space $X$. Tokyo J. Math. 41 (2018), no. 1, 219--223. https://projecteuclid.org/euclid.tjm/1513566019

#### References

• J. Alonso, Uniqueness properties of isosceles orthogonality in normed linear spaces, Ann. Sci. Math. Québec 18 (1994), 25–38.
• J. Alonso, P. Martín and P. L. Papini, Wheeling around von Neumann-Jordan constant in Banach spaces, Studia Math. 188 (2008), 135–150.
• J. Gao and K.-S. Lau, On the geometry of spheres in normed linear spaces, J. Aust. Math. Soc. Ser. A 48 (1990), 101–112.
• M. Kato, L. Maligranda and Y. Takahashi, On James and Jordan-von Neumann constants and the normal structure coefficient of Banach spaces, Studia Math. 144 (2001), 275–295.
• N. Komuro, K.-S. Saito and K.-I. Mitani, Convex property of James and von Neumann-Jordan constant of absolute norms on $\mathbb{R}^2$, Proceedings of the $8$th International Conference on Nonlinear Analysis and Convex Analysis, 301–308, Yokohama Publ., Yokohama, 2015.
• N. Komuro, K.-S. Saito and R. Tanaka, On the class of Banach spaces with James constant $\sqrt{2}$, Math. Nachr. 289 (2016), 1005–1020.
• N. Komuro, K.-S. Saito and R. Tanaka, On the class of Banach spaces with James constant $\sqrt{2}$: Part II, Mediterr. J. Math. 13 (2016), 4039–4061.
• K.-S. Saito, M. Sato and R. Tanaka, When does the equality $J(X^*)=J(X)$ hold for a two-dimensional Banach space $X$?, Acta Math. Sin. (Engl. Ser.) 31 (2015), 1303–1314.
• Y. Takahashi and M. Kato, von Neumann-Jordan constant and uniformly non-square Banach spaces, Nihonkai Math. J. 9 (1998), 155–169.
• F. Wang, On the James and von Neumann-Jordan constants in Banach spaces, Proc. Amer. Math. Soc. 138 (2010), 695–701.
• C. Yang and H. Li, An inequality between Jordan-von Neumann constant and James constant, Appl. Math. Lett. 23 (2010), 277–281.


text:
# Math Help - Homeomorphic finer topology

1. ## Homeomorphic finer topology

Find two different topologies on $\mathbb{R}$ where one is strictly finer, yet the two are homeomorphic

2. Originally Posted by Andreamet
Find two different topologies on $\mathbb{R}$ where one is strictly finer, yet the two are homeomorphic
$(Z, T_1)$ and $(Q, T_2)$ as subspaces of R with a discrete topology.

Let $B_1$ and $B_2$ be bases for topologies $T_1$ and $T_2$.

$T_2$ is strictly finer than $T_1$, since for each x in Z and for each basis element $\{x\} \in B_1$ containing x, we have a basis element $\{x\}$ in $B_2$ that contains x. The converse is not necessarily true.

Since there is a bijection between Z and Q, we have a homeomorphism between $(Z, T_1)$ and $(Q, T_2)$ as subspaces of R with a discrete topology.


text:
## Synthetic topology in Homotopy Type Theory for probabilistic programming

Research output: Working paperResearch

### Documents

The ALEA Coq library formalizes measure theory based on a variant of the Giry monad on the category of sets. This enables the interpretation of a probabilistic programming language with primitives for sampling from discrete distributions. However, continuous distributions have to be discretized because the corresponding measures cannot be defined on all subsets of their carriers. This paper proposes the use of synthetic topology to model continuous distributions for probabilistic computations in type theory. We study the initial $\sigma$-frame and the corresponding induced topology on arbitrary sets. Based on these intrinsic topologies we define valuations and lower integrals on sets, and prove versions of the Riesz and Fubini theorems. We then show how the Lebesgue valuation, and hence continuous distributions, can be constructed.
Original language English ArXiv Published - Dec 2019

### Research areas

• cs.LO, math.CT

Citationformats


text:
# Introduction to Neural Networks

## What Is a Neural Network?

A neural network is inspired by the information processing of neurons. The most common networks learn by example. These networks are taught how to produce correct outputs by training them against large datasets of input-output pairs. It is nearly impossible to reverse engineer exactly how a particular network works. This may not be the case in the future as work is being done to glean insight into these powerful models.

How does one make a neural network? It begins by deciding the architecture. All common architectures include one or more layers, a number of neurons in each layer, what types of neurons are used, and the types of connections between neurons in adjacent layers. The architecture of a network is like the connectome of a brain.

An architecture needs a dataset to train on. A dataset will usually contain desired input and output pairs that the network must learn to correlate. There are cases where pairs are not necessary, the distribution of the data used to learn instead.

A network is trained with an input-output pair by passing the input through it and comparing the result to the output of the pair. The network will produce noise in the beginning. The error between the result and desired output is calculated. For every adjustable parameter in the network, it is calculated. This is repeated many times until the error is reduced to a minimal amount.

## Example Networks

Here are some example networks to give an idea of the kinds of problems neural networks can solve.

## Structure of a Neural Network

A neural network is built out of layers of interconnected neurons. The first layer is called the input layer and receives samples directly from dataset as input, the middle layers are called the hidden layers and take the previous layer’s output as input. The last layer is called the output layer and outputs the final result of the network.

The fact that each layer passes its output to the next makes this a description of a feedforward neural network. Since each layer transforms data from the previous layer, the deeper the layer the more abstract the information at that layer becomes. This is powerful because it allows the network to learn high level features, such as if someone is smiling or frowning in a photograph.

Every network also needs a loss function, which measures the difference between an output and the expected output, and an optimizer that uses the loss to determine how to shift the network’s weights.

### Layer Connections

Layers in a neural network are typically fully connected; each neuron in a layer gets its inputs from every neuron in the previous layer. Sometimes these connections are altered such as in the case of residual networks, where the inputs can be fed to multiple layers.

#### Dropout

Dropout is another method of connecting layers. Dropout will randomly disconnect a given percentage of connections between two layers during training. This may sound like it would break the network, but it actually makes it more robust. The network learns how to arrive at the correct answer even with missing data. Once the network is fully trained and is being applied, dropout is no longer enabled and all the connections are active. The connections are scaled to account for the extra active connections.

Dropout reduces the problem of overfitting, which is when the network is plagiarizing the training data instead of generalizing it. This is like learning by memorization as opposed to learning by understanding.

### Loss Function

A loss function is a measure of how dissimilar two things are. A loss of zero should mean the two things being measured are identical. An example loss function would be the distance between two points in space: If the distance is 0 the points are identical, with increasing distances indicating that the points are less and less similar.

A common loss function is the squared error. You take the difference between two numbers and square it. This has the benefit of keeping the loss positive, and penalizing larger differences.

### Optimizer

An optimizer is a process that uses the loss function and dataset to determine how to change the parameters (weights) of the network.

Stochastic gradient descent (SGD) is the optimization process of running a sample input from the dataset through the network, then using the derivative of the loss with respect to the parameters of the network to nudge those parameters in a direction that lowers that loss. This derivative is called the gradient.

Mini-batch gradient descent is the same process except the gradients from a batch of random samples from the dataset are averaged. This helps ensure the gradient is pointing in a direction that optimizes in a general way for many sample inputs and not just individual inputs.

There are variants of gradient descent such as ones that involve momentum and adaptive learning rates which you can read more about here.

## Anatomy of a Neuron

Artificial neurons act in a similar manner to biological neurons by propagating signals between each other. A typical artificial neuron consists of inputs, weights, a bias, and an activation function.

$\begin{split} a^{l}_{j} =&\ \sigma(z^{l}_{j}) \\ z^{l}_{j} =&\ \sum_{k}w^{l}_{jk}a^{l-1}_{k} + b^{l}_{j} \\ \hline a^{l}_{j} :=&\ \mathrm{Activation\ of\ the}\ j^{\mathrm{th}}\mathrm{\ neuron\ in\ the}\ l^{\mathrm{th}}\mathrm{\ layer.} \\ z^{l}_{j} :=&\ \mathrm{Weighted\ input\ to\ the}\ j^{\mathrm{th}}\mathrm{\ neuron\ in\ the}\ l^{\mathrm{th}}\mathrm{\ layer} \\ w^{l}_{jk} :=&\ \mathrm{Weight\ between\ the}\ j^{\mathrm{th}}\mathrm{\ neuron\ in\ the}\ l^{\mathrm{th}}\mathrm{\ layer} \\ &\ \mathrm{and\ the}\ k^{\mathrm{th}}\mathrm{\ neuron\ in\ the\ previous\ layer.} \\ b^{l}_{j} :=&\ \mathrm{Bias\ of\ the}\ j^{\mathrm{th}}\mathrm{\ neuron\ in\ the}\ l^{\mathrm{th}}\mathrm{\ layer.} \\ \sigma :=&\ \mathrm{The\ activation\ function.} \end{split}$

### Inputs

The input layer’s inputs come from the dataset. Subsequent layers are fed the sum of weighted activations plus bias ($$\sum wa+b$$) from the previous layer.

### Weights

Each weight ($$w$$) scales the respective activation from the previous layer. They are parameters of the network that are learned.

### Bias

The bias ($$b$$) is a parameter simply added to the sum of weighted activations ($$\sum wa$$).

It turns out the bias is conceptually the same as a weight where the corresponding activation is always $$1$$. We can exploit this to prevent the model from becoming more complex than it needs to be. To simulate the bias, an extra input is fed into every neuron. This input is set to a constant value of $$1$$.

### Activation Function

The activation function is a function that defines the output of a neuron. Activation functions are typically chosen to bound the output between a range of values. Popular activation functions are the sigmoid, tanh, RELU, ELU, and softmax functions. The tanh function is nice as it balances the output between $$-1$$ and $$1$$.

This is like a continuous form of a binary choice: A large negative value becomes $$-1$$, and a large positive value becomes $$1$$, with smaller numbers somewhere in-between.

The non-linearity of this function is what makes the whole network non-linear. This non-linearity is what allows neural networks to approximate any function.

The vanishing gradient problem is an issue where a neural network’s training slows down the farther back in the network the gradient is propagated. In the case of the tanh function, its derivative will always be between $$0$$ and $$1$$. This means that for every layer the gradient is propagated through, the gradient is multiplied by some value between $$0$$ and $$1$$. Do this enough times and the gradient becomes smaller and smaller until it “vanishes”. The effect this has is that the farther from the end of the network a layer is, the less it will be updated by the gradient.

Attempts at a solution to the vanishing gradient problem include the residual networks mentioned previously, and recurrent LSTM networks.

## Training a Neural Network

### Backpropagation

Backpropagation is the meat of a neural network’s training algorithm. It relies on using the gradient, the derivative of the loss with respect to the parameters of the network. With this gradient one can update the parameters of the network to reduce the loss, which will make the network perform better.

Before one can use backpropagation, the gradient must be found. An input from the dataset is fed into the network. Then the output of the network is compared to the corresponding expected output using a loss function like the squared error.

Once the loss is computed we can find the derivative of the loss with respect to the weights of the last layer of neurons. Using the notation from earlier, this is $$\frac{\partial C}{\partial w^{L}}$$ where $$C$$ is the loss of the function, and L is the total number of the layers in the network.

This is where backpropagation begins. Remember the chain rule from calculus? A brief reminder: If you multiply two derivatives such that terms cancel out, you get another valid derivative. For example, $$\frac{\ \partial b}{\partial a} \cdot \frac{\partial c}{\partial b} = \frac{\partial c}{\partial a}$$. This can be applied to the neural network too. If you can find $$\frac{\partial w^{L}}{\partial w^{L-1}}$$, you can use the chain rule to easily also find $$\frac{\partial C}{\partial w^{L-1}} = \frac{\partial C}{\partial w^{L}} \cdot \frac{\partial w^{L}}{\partial w^{L-1}}$$.

You repeat this process for each layer until it has gone through the whole network. Then you have the gradient for the entire network, which describes how the loss changes for any particular weight. The weights can then be updated by subtracting a small multiple of the gradient from the weights. This small multiple is the gradient multiplied by the learning rate, which is often set to decrease over training time to allow the network to settle into a local optimum.

## Conclusion

This covers the basics of what one should know about neural networks. There is much more to learn than this but I hope this explanation has given you an idea of how they work. Let me know in the comments if there is anything in this introduction that could be improved upon.


text:
In this notebook, we present a modified version of gradient boosting which uses a reduced number of splits when building the different trees. This algorithm is called “histogram gradient boosting” in scikit-learn.

We previously mentioned that random-forest is an efficient algorithm since each tree of the ensemble can be fitted at the same time independently. Therefore, the algorithm scales efficiently with both the number of cores and the number of samples.

In gradient-boosting, the algorithm is a sequential algorithm. It requires the N-1 trees to have been fit to be able to fit the tree at stage N. Therefore, the algorithm is quite computationally expensive. The most expensive part in this algorithm is the search for the best split in the tree which is a brute-force approach: all possible split are evaluated and the best one is picked. We explained this process in the notebook “tree in depth”, which you can refer to.

To accelerate the gradient-boosting algorithm, one could reduce the number of splits to be evaluated. As a consequence, the generalization performance of such a tree would be reduced. However, since we are combining several trees in a gradient-boosting, we can add more estimators to overcome this issue.

We will make a naive implementation of such algorithm using building blocks from scikit-learn. First, we will load the California housing dataset.

from sklearn.datasets import fetch_california_housing

data, target = fetch_california_housing(return_X_y=True, as_frame=True)
target *= 100  # rescale the target in k$ Note If you want a deeper overview regarding this dataset, you can refer to the Appendix - Datasets description section at the end of this MOOC. We will make a quick benchmark of the original gradient boosting. from sklearn.model_selection import cross_validate from sklearn.ensemble import GradientBoostingRegressor gradient_boosting = GradientBoostingRegressor(n_estimators=200) cv_results_gbdt = cross_validate( gradient_boosting, data, target, scoring="neg_mean_absolute_error", n_jobs=2 )  print("Gradient Boosting Decision Tree") print(f"Mean absolute error via cross-validation: " f"{-cv_results_gbdt['test_score'].mean():.3f} ± " f"{cv_results_gbdt['test_score'].std():.3f} k$")
print(f"Average fit time: "
f"{cv_results_gbdt['fit_time'].mean():.3f} seconds")
print(f"Average score time: "
f"{cv_results_gbdt['score_time'].mean():.3f} seconds")

Gradient Boosting Decision Tree
Mean absolute error via cross-validation: 46.415 ± 2.912 k$Average fit time: 7.824 seconds Average score time: 0.011 seconds  We recall that a way of accelerating the gradient boosting is to reduce the number of split considered within the tree building. One way is to bin the data before to give them into the gradient boosting. A transformer called KBinsDiscretizer is doing such transformation. Thus, we can pipeline this preprocessing with the gradient boosting. We can first demonstrate the transformation done by the KBinsDiscretizer. import numpy as np from sklearn.preprocessing import KBinsDiscretizer discretizer = KBinsDiscretizer( n_bins=256, encode="ordinal", strategy="quantile") data_trans = discretizer.fit_transform(data) data_trans  /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 1 are removed. Consider decreasing the number of bins. warnings.warn( /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 3 are removed. Consider decreasing the number of bins. warnings.warn( /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 6 are removed. Consider decreasing the number of bins. warnings.warn( /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 7 are removed. Consider decreasing the number of bins. warnings.warn(  array([[249., 39., 231., ..., 83., 162., 30.], [248., 19., 203., ..., 28., 161., 30.], [242., 49., 249., ..., 125., 160., 29.], ..., [ 17., 15., 126., ..., 49., 200., 82.], [ 23., 16., 136., ..., 29., 200., 77.], [ 53., 14., 130., ..., 93., 199., 81.]])  Note The code cell above will generate a couple of warnings. Indeed, for some of the features, we requested too much bins in regard of the data dispersion for those features. The smallest bins will be removed. We see that the discretizer transforms the original data into integral values (even though they are encoded using a floating-point representation). Each value represents the bin index when the distribution by quantile is performed. We can check the number of bins per feature. [len(np.unique(col)) for col in data_trans.T]  [256, 50, 256, 253, 256, 256, 207, 235]  After this transformation, we see that we have at most 256 unique values per features. Now, we will use this transformer to discretize data before training the gradient boosting regressor. from sklearn.pipeline import make_pipeline gradient_boosting = make_pipeline( discretizer, GradientBoostingRegressor(n_estimators=200)) cv_results_gbdt = cross_validate( gradient_boosting, data, target, scoring="neg_mean_absolute_error", n_jobs=2, )  /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 1 are removed. Consider decreasing the number of bins. warnings.warn( /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 1 are removed. Consider decreasing the number of bins. warnings.warn( /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 3 are removed. Consider decreasing the number of bins. warnings.warn( /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 3 are removed. Consider decreasing the number of bins. warnings.warn( /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 6 are removed. Consider decreasing the number of bins. warnings.warn( /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 6 are removed. Consider decreasing the number of bins. warnings.warn( /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 7 are removed. Consider decreasing the number of bins. warnings.warn( /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 7 are removed. Consider decreasing the number of bins. warnings.warn(  /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 1 are removed. Consider decreasing the number of bins. warnings.warn( /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 1 are removed. Consider decreasing the number of bins. warnings.warn( /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 3 are removed. Consider decreasing the number of bins. warnings.warn( /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 3 are removed. Consider decreasing the number of bins. warnings.warn( /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 6 are removed. Consider decreasing the number of bins. warnings.warn( /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 6 are removed. Consider decreasing the number of bins. warnings.warn( /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 7 are removed. Consider decreasing the number of bins. warnings.warn( /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 7 are removed. Consider decreasing the number of bins. warnings.warn(  /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 1 are removed. Consider decreasing the number of bins. warnings.warn( /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 3 are removed. Consider decreasing the number of bins. warnings.warn( /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 6 are removed. Consider decreasing the number of bins. warnings.warn( /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:279: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 7 are removed. Consider decreasing the number of bins. warnings.warn(  print("Gradient Boosting Decision Tree with KBinsDiscretizer") print(f"Mean absolute error via cross-validation: " f"{-cv_results_gbdt['test_score'].mean():.3f} ± " f"{cv_results_gbdt['test_score'].std():.3f} k$")
print(f"Average fit time: "
f"{cv_results_gbdt['fit_time'].mean():.3f} seconds")
print(f"Average score time: "
f"{cv_results_gbdt['score_time'].mean():.3f} seconds")

Gradient Boosting Decision Tree with KBinsDiscretizer
Mean absolute error via cross-validation: 45.781 ± 2.026 k$Average fit time: 4.634 seconds Average score time: 0.015 seconds  Here, we see that the fit time has been reduced but that the generalization performance of the model is identical. Scikit-learn provides specific classes which are even more optimized for large dataset, called HistGradientBoostingClassifier and HistGradientBoostingRegressor. Each feature in the dataset data is first binned by computing histograms, which are later used to evaluate the potential splits. The number of splits to evaluate is then much smaller. This algorithm becomes much more efficient than gradient boosting when the dataset has over 10,000 samples. Below we will give an example for a large dataset and we will compare computation times with the experiment of the previous section. from sklearn.ensemble import HistGradientBoostingRegressor histogram_gradient_boosting = HistGradientBoostingRegressor( max_iter=200, random_state=0) cv_results_hgbdt = cross_validate( histogram_gradient_boosting, data, target, scoring="neg_mean_absolute_error", n_jobs=2, )  print("Histogram Gradient Boosting Decision Tree") print(f"Mean absolute error via cross-validation: " f"{-cv_results_hgbdt['test_score'].mean():.3f} ± " f"{cv_results_hgbdt['test_score'].std():.3f} k$")
print(f"Average fit time: "
f"{cv_results_hgbdt['fit_time'].mean():.3f} seconds")
print(f"Average score time: "
f"{cv_results_hgbdt['score_time'].mean():.3f} seconds")

Histogram Gradient Boosting Decision Tree
Mean absolute error via cross-validation: 43.758 ± 2.694 k\$
Average fit time: 1.114 seconds
Average score time: 0.084 seconds

The histogram gradient-boosting is the best algorithm in terms of score. It will also scale when the number of samples increases, while the normal gradient-boosting will not.


text:
# 10.1. Stock Market Prediction¶

A recent post on Towards Data Science (TDS) demonstrated the use of ARIMA models to predict stock market data with raw statsmodels. This post addresses the same problem using pmdarima’s auto-ARIMA, and ends up achieving a different result with an even lower error rate.

For this example, all we’ll need is Numpy, Pandas and pmdarima. Matplotlib is optional, but highly encouraged in order to qualitatively validate the results of the model fit. To run this example, you’ll need pmdarima version 1.5.2 or greater. If you’re running this in a notebook, make sure to include %matplotlib inline, or the plots will not show up under your cells!

Keep in mind you’ll need Python 3.6+ to install pmdarima.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# %matplotlib inline

import pmdarima as pm
print(f"Using pmdarima {pm.__version__}")
# Using pmdarima 1.5.2

The pmdarima module conveniently includes the dataset we’ll be using as an internal utility. Rather than carting around .csv files, you can simply load the data from the package:

from pmdarima.datasets.stocks import load_msft

 Date Open High Low Close Volume OpenInt 1986-03-13 0.06720 0.07533 0.06720 0.07533 1371330506 0 1986-03-14 0.07533 0.07533 0.07533 0.07533 409569463 0 1986-03-17 0.07533 0.07533 0.07533 0.07533 176995245 0 1986-03-18 0.07533 0.07533 0.07533 0.07533 90067008 0 1986-03-19 0.07533 0.07533 0.07533 0.07533 63655515 0

## 10.1.2. Data splitting¶

As with all statistical and ML modeling, we need to make sure we’ve split our data so we can evaluate model performance on a hold-out set. However, unlike other traditional supervised learning, time series models intrinsically introduce endogenous temporality, meaning that the values at any given point $$y_{t}$$ in our time series likely have some effect on some future value, $$y_{t+n}$$. Therefore, we cannot simply split our data randomly; we must make a clean split in our time series (and exogenous variables, if present).

As in the TDS example, we’ll use $$0.8 * dataSize$$ as our training sample.

## 10.1.3. Pre-modeling analysis¶

As you may know (if not, venture over to Tips to using auto_arima before continuing), an ARIMA model has 3 core hyper-parameters, known as “order”:

• $$p$$: The order of the auto-regressive (AR) model (i.e., the number of lag observations)
• $$d$$: The degree of differencing.
• $$q$$: The order of the moving average (MA) model. This is essentially the size of the “window” function over your time series data.

Part of the science behind the auto-arima approach is intelligently finding the proper combination of p, d, q such that you achieve the best fit. The TDS article took the approach of fixing the $$p$$ parameter at 5 after examining auto-correlations with lag plots. A lag plot can provide clues about the underlying structure of your data [1]:

• A linear shape to the plot suggests that an autoregressive model is probably a better choice.
• An elliptical plot suggests that the data comes from a single-cycle sinusoidal model.
from pandas.plotting import lag_plot

fig, axes = plt.subplots(3, 2, figsize=(8, 12))
plt.title('MSFT Autocorrelation plot')

# The axis coordinates for the plots
ax_idcs = [
(0, 0),
(0, 1),
(1, 0),
(1, 1),
(2, 0),
(2, 1)
]

for lag, ax_coords in enumerate(ax_idcs, 1):
ax_row, ax_col = ax_coords
axis = axes[ax_row][ax_col]
lag_plot(df['Open'], lag=lag, ax=axis)
axis.set_title(f"Lag={lag}")

plt.show()

As you can see, all the lags look fairly linear, so it’s a good indicator that an auto-regressive model is a good choice. But since we don’t want to allow simple visual bias to impact our decision here, we’ll allow the auto_arima to select the proper lag term for us.

### 10.1.3.1. Estimating the differencing term¶

The TDS article selected $$d=1$$ as the differencing term. But how did they make that choice? With pmdarima, we can run several differencing tests against the time series to select the best number of differences such that the time series will be stationary.

Here, we’ll use the KPSS test and ADF test, selecting the maximum value between the two to be conservative. Fortunately, in this case, both tests indicated that $$d=1$$ was the best answer, but in the case where they disagreed, we could try both or allow auto_arima to auto-select the d term.

from pmdarima.arima import ndiffs

kpss_diffs = ndiffs(y_train, alpha=0.05, test='kpss', max_d=6)

print(f"Estimated differencing term: {n_diffs}")
# Estimated differencing term: 1

Therefore, we will use $$d=1$$.

## 10.1.4. Fitting our model¶

Now it’s time to let the auto_arima method do its magic:

auto = pm.auto_arima(y_train, d=n_diffs, seasonal=False, stepwise=True,
suppress_warnings=True, error_action="ignore", max_p=6,
max_order=None, trace=True)

Notice that we preset d=n_diffs, since we’ve already settled on a value for d. However, we’re allowing our ARIMA models explore various values of p and q. After a few seconds, we arrive at the following solution:

print(auto.order)
# (0, 1, 0)

Where the TDS model was of order (5, 1, 0), we ended up selecting a significantly more simple model. But how does it perform?

## 10.1.5. Updating the model¶

Now that the heavy lifting of selecting model hyper-parameters has been performed, we can update our model by simulating days passing with our test set. For each new observation, we’ll let our model progress for several more iterations, allowing MLE to update its discovered parameters and shifting the latest observed value. Then we can measure the error on the forecasts:

from sklearn.metrics import mean_squared_error
from pmdarima.metrics import smape

model = auto  # seeded from the model we've already fit

def forecast_one_step():
fc, conf_int = model.predict(n_periods=1, return_conf_int=True)
return (
fc.tolist()[0],
np.asarray(conf_int).tolist()[0])

forecasts = []
confidence_intervals = []

for new_ob in y_test:
fc, conf = forecast_one_step()
forecasts.append(fc)
confidence_intervals.append(conf)

# Updates the existing model with a small number of MLE steps
model.update(new_ob)

print(f"Mean squared error: {mean_squared_error(y_test, forecasts)}")
print(f"SMAPE: {smape(y_test, forecasts)}")
# Mean squared error: 0.34238951346274243
# SMAPE: 0.9825490519101439

In the end, our model ended up way out-performing the TDS model!

 Source MSE SMAPE pmdarima 0.342 0.983 (!!) TDS article 0.343 40.776

## 10.1.6. Viewing forecasts¶

Let’s take a look at the forecasts our model produces overlaid on the actuals (in the first plot), and the confidence intervals of the forecasts (in the second plot):

fig, axes = plt.subplots(2, 1, figsize=(12, 12))

# --------------------- Actual vs. Predicted --------------------------
axes[0].plot(y_train, color='blue', label='Training Data')
axes[0].plot(test_data.index, forecasts, color='green', marker='o',
label='Predicted Price')

axes[0].plot(test_data.index, y_test, color='red', label='Actual Price')
axes[0].set_title('Microsoft Prices Prediction')
axes[0].set_xlabel('Dates')
axes[0].set_ylabel('Prices')

axes[0].set_xticks(np.arange(0, 7982, 1300).tolist(), df['Date'][0:7982:1300].tolist())
axes[0].legend()

# ------------------ Predicted with confidence intervals ----------------
axes[1].plot(y_train, color='blue', label='Training Data')
axes[1].plot(test_data.index, forecasts, color='green',
label='Predicted Price')

axes[1].set_title('Prices Predictions & Confidence Intervals')
axes[1].set_xlabel('Dates')
axes[1].set_ylabel('Prices')

conf_int = np.asarray(confidence_intervals)
axes[1].fill_between(test_data.index,
conf_int[:, 0], conf_int[:, 1],
alpha=0.9, color='orange',
label="Confidence Intervals")

axes[1].set_xticks(np.arange(0, 7982, 1300).tolist(), df['Date'][0:7982:1300].tolist())
axes[1].legend()

## 10.1.7. Conclusion¶

The TDS article provided an awesome example of how to use ARIMAs to predict stocks. Our hope in this example was to show how using pmdarima can simplify and enhance the models you build. If you’d like to run the already-setup notebook for yourself, head on over to the project’s Git page and grab the example notebook.