text:
# Import and export¶

## Import¶

To import data, go to the “Import” tab in phpMyAdmin. To import data into a specific database or table, open the database or table before going to the “Import” tab.

In addition to the standard Import and Export tab, you can also import an SQL file directly by dragging and dropping it from your local file manager to the phpMyAdmin interface in your web browser.

If you are having troubles importing big files, please consult 1.16 I cannot upload big dump files (memory, HTTP or timeout problems)..

You can import using following methods:

Form based upload

Can be used with any supported format, also (b|g)zipped files, e.g., mydump.sql.gz .

Form based SQL Query

Can be used with valid SQL dumps.

Using upload directory

### CodeGen¶

NHibernate file format. Planned versions: Java, Hibernate, PHP PDO, JSON, etc. So the preliminary name is codegen.

### CSV¶

Comma separated values format which is often used by spreadsheets or various other programs for export/import.

### CSV for Microsoft Excel¶

This is just preconfigured version of CSV export which can be imported into most English versions of Microsoft Excel. Some localised versions (like “Danish”) are expecting ”;” instead of ”,” as field separator.

### Microsoft Word 2000¶

If you’re using Microsoft Word 2000 or newer (or compatible such as OpenOffice.org), you can use this export.

### JSON¶

JSON (JavaScript Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write and it is easy for machines to parse and generate.

Changed in version 4.7.0: The generated JSON structure has been changed in phpMyAdmin 4.7.0 to produce valid JSON data.

The generated JSON is list of objects with following attributes:

type

Type of given object, can be one of:

header
Export header containing comment and phpMyAdmin version.
database
Start of a database marker, containing name of database.
table
Table data export.
version

Used in header type and indicates phpMyAdmin version.

comment

Optional textual comment.

name

Object name - either table or database based on type.

database

Database name for table type.

data

Table content for table type.

Sample output:

[
{
"comment": "Export to JSON plugin for PHPMyAdmin",
"type": "header",
"version": "4.7.0-dev"
},
{
"name": "cars",
"type": "database"
},
{
"data": [
{
"car_id": "1",
"description": "Green Chrysler 300",
"make_id": "5",
"mileage": "113688",
"price": "13545.00",
"transmission": "automatic",
"yearmade": "2007"
}
],
"database": "cars",
"name": "cars",
"type": "table"
},
{
"data": [
{
"make": "Chrysler",
"make_id": "5"
}
],
"database": "cars",
"name": "makes",
"type": "table"
}
]

### LaTeX¶

If you want to embed table data or structure in LaTeX, this is right choice for you.

LaTeX is a typesetting system that is very suitable for producing scientific and mathematical documents of high typographical quality. It is also suitable for producing all sorts of other documents, from simple letters to complete books. LaTeX uses TeX as its formatting engine. Learn more about TeX and LaTeX on the Comprehensive TeX Archive Network also see the short description od TeX.

The output needs to be embedded into a LaTeX document before it can be rendered, for example in following document:

\documentclass{article}
\title{phpMyAdmin SQL output}
\author{}
\usepackage{longtable,lscape}
\date{}
\setlength{\parindent}{0pt}
\usepackage[left=2cm,top=2cm,right=2cm,nohead,nofoot]{geometry}
\pdfpagewidth 210mm
\pdfpageheight 297mm
\begin{document}
\maketitle

% insert phpMyAdmin LaTeX Dump here

\end{document}

### MediaWiki¶

Both tables and databases can be exported in the MediaWiki format, which is used by Wikipedia to display tables. It can export structure, data or both, including table names or headers.

### OpenDocument Spreadsheet¶

Open standard for spreadsheet data, which is being widely adopted. Many recent spreadsheet programs, such as LibreOffice, OpenOffice, Microsoft Office or Google Docs can handle this format.

### OpenDocument Text¶

New standard for text data which is being widely addopted. Most recent word processors (such as LibreOffice, OpenOffice, Microsoft Word, AbiWord or KWord) can handle this.

### PDF¶

For presentation purposes, non editable PDF might be best choice for you.

### PHP Array¶

You can generate a php file which will declare a multidimensional array with the contents of the selected table or database.

### SQL¶

Export in SQL can be used to restore your database, thus it is useful for backing up.

The option ‘Maximal length of created query’ seems to be undocumented. But experiments has shown that it splits large extended INSERTS so each one is no bigger than the given number of bytes (or characters?). Thus when importing the file, for large tables you avoid the error “Got a packet bigger than ‘max_allowed_packet’ bytes”.

#### Data Options¶

Complete inserts adds the column names to the SQL dump. This parameter improves the readability and reliability of the dump. Adding the column names increases the size of the dump, but when combined with Extended inserts it’s negligible.

Extended inserts combines multiple rows of data into a single INSERT query. This will significantly decrease filesize for large SQL dumps, increases the INSERT speed when imported, and is generally recommended.

### Texy!¶

Texy! markup format. You can see example on Texy! demo.

### XML¶

Easily parsable export for use with custom scripts.

Changed in version 3.3.0: The XML schema used has changed as of version 3.3.0

### YAML¶

YAML is a data serialization format which is both human readable and computationally powerful ( <https://yaml.org> ).


text:
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.

# Math help from the Learning Centre

This guide provides useful resources for a wide variety of math topics. It is targeted at students enrolled in a math course or any other Centennial course that requires math knowledge and skills.

## Definitions

Cost per Unit

The cost per unit (often denoted cost/unit) is the cost of one unit of whatever is being purchased. This could be:

• the cost of one apple (cost per one apple or “cost/apple”) ,
• the cost of one pound of flour (cost per pound or “cost/pound”), or
• the cost of one kg of dates (cost per kg or "cost/kg")

We can use the following formula to calculate cost per unit:

$$Cost/Unit=As-Purchased \ Cost \div \ total \ number \ of \ units$$,

where As-Purchased Cost (APC) is the cost paid to the supplier for the as-purchased quantity (APQ) of products you purchase. To review APQ, visit our Yield Percent page.

Total Cost

When we have calculated the Cost/Unit, we can use it to determine the Total Cost using the general formula:

$$Total \ Cost = Number \ of \ Units \times Cost/Unit$$

Note: The units used in 'Number of Units' and 'Cost/Unit' must be the same. (For example, we can use the above formula to directly calculate the total cost of 4lbs of flour at $10/lb, but cannot use it for 4kgs of flour at$10/lb).

Edible Portion Cost (EPC)

The edible portion cost (EPC) is the cost per unit of the edible portion quantity (EPQ) of products purchased. To review EPQ, visit our Yield Percent page. To calculate EPC, we use the formula:

$$Edible \ Portion \ Cost = \frac{APC}{Yield \ Percentage}$$,

where $$Yield \ Percentage=\frac{APQ}{EPQ}$$ is expressed as a decimal.

Note: EPC is always equal to or greater than APC. If the purchased product from the supplier is used as is (e.g. flour, sugar, etc.), then $$APC=EPC$$. Otherwise, if further processing or cleaning is required (e.g. vegetables, meats, etc.), then $$APC<EPC$$.

## Examples

Cost per Unit

Example: If you are looking for corn in a weekly grocery flyer and it shows that the price for two corns is $4.00, what is the cost per corn? Solution: We see that if two corns cost$4.00, then one corn should cost half of that amount, $2.00. In this case, one unit is one corn, so using the cost per unit formula, we have the calculation: $$Cost/Corn = 4.00 \div 2 \ corn= 2.00$$ So, the cost is$2.00 per corn (or $2.00/corn). Example: You are going to bake bread and need to buy flour. If one bag of flour is sold for$12.50 and each bag weighs 10 pounds, what is the cost per pound?

Solution:

From the question, we know that the APC=$12.50 (per bag or 10 pounds of flour) and the total number of units is 10 (pounds). In this case, one unit is one pound, so substituting these values into the cost per unit formula gives: $$Cost/Pound = 12.50 \div 10 \ pounds = 1.25$$ So, the cost is$1.25 per pound of flour (or $1.25/pound). Total Cost ExampleYou purchase two bags of flour from the grocery store. If the cost of one bag is$12.00, what is the total cost of the flour you bought?

Solution:

From the question, we know that the number of units purchased is 2 (bags) and the cost/unit is $12.00/bag. Since both values are in the same units (bag), we can directly calculate the total cost using the general formula as follows: $$Total \ Cost = 2 \ bags \times 12.00/bag = 24$$ So, the total cost for both bags of flour is$24.00.

Example: You bought a box of oranges for $9.00. If each box contains 6 oranges, what is the total cost for 4 oranges? Solution: From the question, we know that the number of units being purchased is 4 (oranges), but do not know the cost per unit (1 orange). Instead, we know that a box of 6 oranges costs$9.00, so we first determine the cost per unit using this information:

$$Cost/Orange = 9.00 \div 6 \ oranges = 1.50$$

Since both the number of units and the cost/unit are in the same units (oranges), we can directly calculate the total cost using the general formula as follows:

$$Total \ Cost = 4 \ oranges \times 1.50/orange = 6$$

So, the total cost for 4 oranges is $6.00. ExampleTurmeric powder can be purchased in 250g containers for$12.00 each. How much would 1 kg of turmeric powder cost?

Solution:

In this case, the units in the given problem are the not the same: cost per unit is in grams (g), while the number of units purchased is in kilograms (kg). So before using the total cost formula, we need to convert from one unit into the other:

$$1kg = 1000g$$

We also need to find the cost per unit:

$$Cost \ per \ unit = 12 \div 250g = 0.048/g$$

Finally, substituting the values we've found into the total cost formula gives:

$$Total \ Cost = 1000g \times 0.048/g = 48$$

So, the total cost for 1kg (or 1000g) of turmeric powder costs $48.00. Example: Apples are on sale at a price of$3.50 per 1kg. How much would 5 pounds of apples cost?

Solution:

In this case, the units in the given problem are the not the same: cost per unit is in kilograms (kg), while the number of units purchased is in pounds (lbs). So before using the total cost formula, we need to convert from one unit into the other:

$$1kg = 2.20lbs$$

Substituting the information we've found and from the original question into the total cost formula gives:

$$Total \ Cost = 5\cancel{lbs} \times 3.50/2.20\cancel{lbs} = 7.95$$

(the pounds units cancel each other out)

So, the total cost for 5 pounds of apples costs $7.95. Edible Portion Cost (EPC) Example: You purchased 25 pounds of corn for$35.00. After cleaning and peeling, if you have 15 pounds of corn remaining, calculate the edible portion cost.

Solution:

We must first determine the the yield percentage using the following steps:

1. Identify the EPQ and APQ: From the question, we know that $$EPQ=15 \ pounds$$ and $$APQ=25 \ pounds$$.
2. Check the EPQ and APQ units: Both the APQ and EPQ are in pounds. If not, we would have to convert either one of the values to the same unit as the other.
3. Substitute EPQ and APQ values into the yield percentage formula:

$$Yield \ Percentage = \frac{15 \ pounds}{25 \ pounds}\times 100=60\%$$

Now, we can find the edible portion cost substituting the yield percentage of 0.6 (expressed as decimal) into the formula:

$$Edible \ Portion \ Cost: \frac{35.00}{0.6}=58.33$$

So, the edible portion cost of the corn is \$58.33.

format: plain


text:
Originally Posted by rdgeorge
Anything that takes more than one action is a project so Yes - have a project for every file, even if the count is 100 or more. In Omnifocus, you can group similar projects into folders which may help. Also, you can add deadlines for some actions (a closing, for example). Also, you can view Next Actions by project and also by Context, which is helpful.

rdgeorge
Thank you - the folders is the key. Otherwise, I was having a hard time imagining how I could filter out those files when I needed to see all the other balls in the air.


text:
Under the auspices of the Computational Complexity Foundation (CCF)

REPORTS > DETAIL:

Revision(s):

Revision #1 to TR18-164 | 28th June 2019 15:16

On the symmetries of and equivalence test for design polynomials

Revision #1
Authors: Nikhil Gupta, Chandan Saha
Accepted on: 28th June 2019 15:16
Downloads: 26
Keywords:

Abstract:

In a Nisan-Wigderson design polynomial (in short, a design polynomial), every pair of monomials share a few common variables. A useful example of such a polynomial, is the following:
$$\text{NW}_{d,k}(\mathbf{x}) = \sum_{h \in \mathbb{F}_d[z], ~\deg(h) \leq k}{~~~~\prod_{i = 0}^{d-1}{x_{i, h(i)}}},$$
where $d$ is a prime, $\mathbb{F}_d$ is the finite field with $d$ elements, and $k \ll d$. The degree of the gcd of every pair of monomials in $\text{NW}_{d,k}$ is at most $k$. For concreteness, we fix $k = \lceil \sqrt{d} \rceil$. The family of polynomials $\mathcal{NW} :=$ {$\text{NW}_{d,k} : d \text{ is a prime}$} and close variants of it have been used as hard explicit polynomial families in several recent arithmetic circuit lower bound proofs. But, unlike the permanent, very little is known about the various structural and algorithmic/complexity aspects of $\mathcal{NW}$ beyond the fact that $\mathcal{NW} \in \text{VNP}$. Is $\text{NW}_{d,k}$ characterized by its symmetries? Is it circuit-testable, i.e., given a circuit $\text{C}$ can we check efficiently if $\text{C}$ computes $\text{NW}_{d,k}(\mathbf{x})$? What is the complexity of equivalence test for $\mathcal{NW}$, i.e., given black-box access to a $f \in \mathbb{F}[\mathbf{x}]$, can we check efficiently if there exists an invertible linear transformation $A$ such that $f = \text{NW}_{d,k}(A \cdot \mathbf{x})$? Characterization of polynomials by their symmetries plays a central role in the geometric complexity theory program. Here, we answer the first two questions and partially answer the third.

We show that $\text{NW}_{d,k}(\mathbf{x})$ is characterized by its group of symmetries over $\mathbb{C}$, but not over $\mathbb{R}$. We also show that $\text{NW}_{d,k}(\mathbf{x})$ is characterized by circuit identities which implies that $\text{NW}_{d,k}(\mathbf{x})$ is circuit-testable in randomized polynomial time. As another application of this characterization, we obtain the flip theorem'' for $\mathcal{NW}$: Suppose $\text{NW}_{d,k}(\mathbf{x})$ is not computable by circuits of size $s$. Then, there exist $\text{poly}(s)$ many points $\mathbf{a}_{1},\ldots,\mathbf{a}_{m}$ such that for every circuit $\text{C}$ of size $s$, there is an $\ell \in [m]$ satisfying $\text{C}(\mathbf{a}_{\ell}) \neq \text{NW}_{d,k}(\mathbf{a}_{\ell})$. These points can be computed deterministically in $\text{poly}(s)$ time if black-box polynomial identity testing for size-$s$ circuits can be derandomized in $\text{poly}(s)$ time. It is well-known that the permanent polynomial has the above-mentioned features.

We give an efficient equivalence test for $\mathcal{NW}$ in the case where the transformation $A$ is a block-diagonal permutation-scaling matrix. The design of this algorithm is facilitated by an almost complete understanding of the group of symmetries of $\text{NW}_{d,k}(\mathbf{x})$: We show that if $A$ is in the group of symmetries of $\text{NW}_{d,k}(\mathbf{x})$ then $A = D \cdot P$, where $D$ and $P$ are diagonal and permutation matrices respectively. This is proved by completely characterizing the Lie algebra of $\text{NW}_{d,k}(\mathbf{x})$, and using an interplay between the Hessian of $\text{NW}_{d,k}(\mathbf{x})$ and the evaluation dimension.

Changes to previous version:

We have added a special case of the equivalence test for design polynomials.

Paper:

TR18-164 | 18th September 2018 18:41

On the symmetries of design polynomials

TR18-164
Authors: Nikhil Gupta, Chandan Saha
Publication: 18th September 2018 19:36
Downloads: 301
Keywords:

Abstract:

In a Nisan-Wigderson design polynomial (in short, a design polynomial), the gcd of every pair of monomials has a low degree. A useful example of such a polynomial is the following:
$$\text{NW}_{d,k}(\mathbf{x}) = \sum_{h \in \mathbb{F}_d[z], ~\deg(h) \leq k}{~~~~\prod_{i = 0}^{d-1}{x_{i, h(i)}}},$$
where $d$ is a prime, $\mathbb{F}_d$ is the finite field with $d$ elements, and $k \ll d$. The degree of the gcd of every pair of monomials in $\text{NW}_{d,k}$ is at most $k$. For concreteness, let us fix $k = \lceil \sqrt{d} \rceil$. The family of polynomials $\mathcal{NW} :=$ {$\text{NW}_{d,k} : d \text{ is a prime}$} and close variants of it have been used as hard explicit polynomial families in several recent arithmetic circuit lower bound proofs. But, unlike the permanent, very little is known about the various complexity and structural aspects of $\mathcal{NW}$ beyond the fact that $\mathcal{NW} \in \text{VNP}$. Is $\mathcal{NW}$ $\text{VNP}$-complete? Is $\text{NW}_{d,k}$ characterized by its symmetries? Is it circuit-testable, i.e., given a circuit $\text{C}$ can we check efficiently if $\text{C}$ computes $\text{NW}_{d,k}$? Characterization of polynomials by their symmetries plays a central role in the geometric complexity theory program. Here, we answer the last two questions.

We show that $\text{NW}_{d,k}$ is characterized by its group of symmetries over $\mathbb{C}$. We also show that $\text{NW}_{d,k}$ is characterized by circuit identities which implies that $\text{NW}_{d,k}$ is circuit-testable in randomized polynomial time. As another implication, we obtain the flip theorem'' for $\mathcal{NW}$: Suppose $\text{NW}_{d,k}$ is not computable by circuits of size $s$. Then, there exist $\text{poly}(s)$ many points $\mathbf{a}_1, \ldots, \mathbf{a}_m$ such that for every circuit $\text{C}$ of size $s$, there is an $\ell \in [m]$ satisfying $\text{C} (\mathbf{a}_\ell) \neq \text{NW}_{d,k}(\mathbf{a}_\ell)$. These points can be computed deterministically in $\text{poly}(s)$ time if black-box polynomial identity testing for size-$s$ circuits can be derandomized in $\text{poly}(s)$ time. It is well-known that the permanent polynomial has the above-mentioned features.

We also show a structural similarity between the group of symmetries of the permanent and that of $\text{NW}_{d,k}$: If $A$ is in the group of symmetries of $\text{NW}_{d,k}$ then $A = D \cdot P$, where $D$ and $P$ are diagonal and permutation matrices respectively. This is proved by completely characterizing the Lie algebra of $\text{NW}_{d,k}$, and using an interplay between the Hessian of $\text{NW}_{d,k}$ and the evaluation dimension.

ISSN 1433-8092 | Imprint


text:
Suggested languages for you:

Q. 10 - Excercises And Problems

Expert-verified
Found in: Page 1082

### Physics for Scientists and Engineers: A Strategic Approach with Modern Physics

Book edition 4th
Author(s) Randall D. Knight
Pages 1240 pages
ISBN 9780133942651

# An electron in a cathode-ray beam passes between long parallel-plate electrodes that are apart. A , wide magnetic field is perpendicular to the electric field between the plates. The electron passes through the electrodes without being deflected if the potential difference between the plates is .a. What is the electron’s speed?b. If the potential difference between the plates is set to zero, what is the electron’s radius of curvature in the magnetic field?

(a) Speed of electron

(b) Electrons radius of curvature in magnetic field.

See the step by step solution

## Part(a) Step 1: Given information

parallel plate electrode

length of electrode:

width(d): ;

magnetic field(B)= (perpendicular to electric field)

No deflection when

## Part(a) Step 2: Finding velocity of electron

Using formula:

When electron pass without deflection then (where v is speed of electron, E is electric field)

Hence, electron's speed is

## Part(b) step 1: Given information

parallel plate electrode

mass of electron(m)=

charge on electron(q)=

magnetic field(B)= (perpendicular to electric field)

velocity of electron=(from part (a))

## Part(b) Step 2: Finding radius of curvature

In presence of magnetic field only, we know that

where R is radius, m is mass of electron, v is its velocity, q is charge, B is magnetic field present;

Hence, electrons radius of curvature in magnetic field is


text:
# Answer the following question: Find the co-ordinates of the foot of the perpendicular drawn from the point P(−1, 3) the line 3x − 4y − 16 = 0 - Mathematics and Statistics

Diagram
Sum

Find the co-ordinates of the foot of the perpendicular drawn from the point P(−1, 3) the line 3x − 4y − 16 = 0

#### Solution

Let M be the foot of perpendicular drawn from P(–1, 3) to the line

3x – 4y – 16 = 0   ...(i)

Slope of the line 3x – 4y – 16 = 0 is

(-3)/(-4) = 3/4

Since PM ⊥ to line (i),

slope of PM = (-4)/3

∴ Equation of PM is

y – 3 = (-4)/3(x + 1)

∴ 3(y – 3) = –4(x + 1)

∴ 3y – 9 = – 4x – 4

∴ 4x +  3y – 5 = 0  ...(ii)

The foot of perpendicular i.e., point M, is the point of intersection of equation (i) and (ii).

By (i) x 3 + (ii) x 4, we get

25x = 68

∴ x = 68/25

Substituting x = 68/25 in (ii), we get

4(68/25) + 3y - 5 = 0

∴ 3y = 5 - 4(68/25) = (125 - 272)/25 = (-147)/25

∴ y = (-49)/25

∴ The co-ordinates of the foot of perpendicular M are (68/25, (-49)/25).

Is there an error in this question or solution?

#### APPEARS IN

Balbharati Mathematics and Statistics 1 (Arts and Science) 11th Standard Maharashtra State Board
Chapter 5 Straight Line
Miscellaneous Exercise 5 | Q II. (26) | Page 126


text:
35 views
Consider an IPV4 addressing system,where at same time two multicast groups are ongoing,by choosing their multicast group addresses at random. The probability they interfere each other i.e. choosing same multicast address is ?

i think it should be

(1/2^28)*(1/2^28)

but it has given (1/2^28) in made easy solutions

1
2


text:
# Math Help - complex anaysis help

1. ## complex anaysis help

Let $w,z \in \mathbb{C}$.
If $|w|<1$ and $|z| \leq 1$, show that $|\frac{z-w}{1-\bar wz}| \leq 1$. When does equality hold?

2. Originally Posted by dori1123
Let $w,z \in \mathbb{C}$.
If $|w|<1$ and $|z| \leq 1$, show that $|\frac{z-w}{1-\bar wz}| \leq 1$. When does equality hold?
Rudin, huh? What have you tried?

3. I know $|\frac{z-w}{1- \bar wz}|=\frac{|z-w|}{|1-\bar wz|}$ and $|z-w| \leq |z|+|w| \leq 1+1 =2$, not sure what to do with $|1 - \bar wz|$, I'm trying to work this way, not sure if it works... $|1 - \bar wz|=|-(\bar wz - 1)|=|(- \bar w)(z - \frac{1}{\bar w})|=|(- \bar w)(z - \frac{w}{w \bar w})|=|\bar w||z - \frac{w}{w \bar w}|=|\bar w||z - \frac{w}{|w|^2}|$=...

4. Originally Posted by dori1123
Let $w,z \in \mathbb{C}$.
If $|w|<1$ and $|z| \leq 1$, show that $\Bigl|\frac{z-w}{1-\bar wz}\Bigr| \leq 1$. When does equality hold?
You want to show that $|z-w|\leqslant|1-\bar wz|$. Square both sides and use the fact that $|z|^2 = \bar zz$. Then what you are trying to prove is equivalent to $(z-w)(\bar z -\bar w) \leqslant (1-\bar wz)(1-w\bar z)$. Multiply out both sides and simplify ... .


text:
# necessary and sufficient conditions for an isolated dynamical system which can approach thermal equilibrium automatically

Given an isolated $N$-particle system with only two body interaction, that is $$H=\sum_{i=1}^N\frac{\mathbf{p}_i^2}{2m}+\sum_{i<j}V(\mathbf{r}_i-\mathbf{r}_j)$$

In the thermodynamic limit, that is $N\gg 1$ and $N/V=$constant, it seems that not all two body interaction can make system approach thermal equilibrium automatically. For example, if the interaction is inverse square attractive force, we know the system cannot approach thermal equilibrium.

Although there is Boltzmann's H-theorem to derive the second law of thermodynamics, it relies on the Boltzmann equation which is derived from Liouville's equation in approximation of low density and short range interaction.

My question:

1. Does it mean that any isolated system with low density and short range interaction can approach thermal equilibrium automatically? If not, what's the counterexample?

2. For long range interaction or high density isolated system, what's the necessary and sufficient conditions for such system can approach thermal equilibrium automatically? What's about coulomb interaction of plasma(i.e. same number of positive and negative charge)?

3. How to prove rigorously that a pure self-gravitational system cannot approach equilibrium? I only heard the hand-waving argument that gravity has the effect of clot, but I never see the rigorous proof.

I know there is maximal entropy postulate in microscopic ensemble.I just want to find the range of application of this postulate of equilibrium statistical mechanics. I'm always curious about the above questions but I never saw the discussion in any textbook of statistical mechanics. You can also cite the literature in which I can find the answer.

1. Does it mean that any isolated system with low density and short range interaction can approach thermal equilibrium automatically? If not, what's the counterexample?

No, you cannot guarantee equilibrium is always reached. Example: the ideal gas of point particles in a perfectly rigid container. As discussed in the question How does a gas of particles with uniform speed reach the Maxwell-Boltzmann distribution?, you need, e.g., non-ideal walls or finite particles to reach equilibrium in this system.

In general, a system can reach thermodynamical equilibrium when it's ergodic or mixing, so that the "fast degrees of freedom" can be averaged out, and the systems can be described by the thermodynamical quantities alone. To know when a system exhibits these properties, check Are there necessary and sufficient conditions for ergodicity?.

1. (a) For long range interaction or (b) high density isolated system, what's the necessary and sufficient conditions for such system can approach thermal equilibrium automatically? (c) What's about coulomb interaction of plasma(i.e. same number of positive and negative charge)?

a) Rigorously, systems with long-range interactions don't reach equilibrium. More specifically: Relaxation towards equilibrium is extremely slow and the relaxation time diverges with the number of particles.

These systems can actually have two phases of relaxation. A fast phase (sometimes called "violent" in astrophysics), followed by a second, (divergingly) slow one, which can also present quasi-stationary states. Other unusual features include non-additivity and regions of negative specific heat. For details, you might check, e.g.,

b) In high-density systems, short-range interactions tend to dominate and the answer to the previous question (namely, mixing leads to equilibrium) applies.

c) In a plasma the charge is shielded and the interactions are not truly (Coulombian) long-range.

1. How to prove rigorously that a pure self-gravitational system cannot approach equilibrium? I only heard the hand-waving argument that gravity has the effect of clot, but I never see the rigorous proof.

Since gravity is a long-range interaction, this last question is a particular case of the previous one. At any rate, a recent reference on the subject is Melkikh's Can we use thermodynamics in the systems with gravity? (e-print).

Regardless of the equilibrium question, the subject is very relevant, especially in the context of quantum gravity. See, e.g., Brown et al.'s Thermodynamic ensembles and gravitation (e-print) and Martinez' The postulates of gravitational thermodynamics (arXiv).

As for the "effect of clot", that might refer to dissipative (such as protoplanetary) systems.


text:
# How do you derive this? SImple question

Derive for time!
D=Vit+1/2at^2

THANKS for you help

$$v(t) = v_i + a t$$
and between the vertical lines $t_1 = 0$ and $t_2 = t$. The geometric figure is a right trapezoid (trapzium outside U.S.) and the area is:
$$D = \frac{1}{2} \left[ v_i + (v_i + a \, t) \right] \cdot t = v_i \, t + \frac{a \, t^2}{2}$$


text:
# The real line equipped with the distance $p(x,y)= |\arctan (x)-\arctan (y)|$ is an incomplete metric space.

Prove that the real line equipped with the distance $p(x,y)= |\arctan (x)-\arctan (y)|$ is an incomplete metric space.

## My attempt

Duplicate but I want to ask about my specific proof and also how to prove that it is a metric space

A metric space is called complete if every cauchy sequence of points in $M$ converges in $M$.

A sequence $x=(x_1,x_2,...x_n)$ is cauchy if for all $\epsilon>0$ there exists an $N \in \mathbb{N}$ such that $m,n \geq N$ implies $d(x_n,x_m)< \epsilon$.

The $\arctan$ function is well-defined from $-\pi/2$ to $\pi/2$.

Fix $\epsilon >0$. Let $X_n=n$. $$d(n,\pi/2)<\epsilon$$

$$|\arctan(n)-\arctan(\pi/2)| < \epsilon$$

Not sure where to go from here. I think I also need to prove that this is a metric

• Is the sequence $n \mapsto n$ Cauchy? Does it converge? – copper.hat May 1 '17 at 1:55
• The domain of $\arctan$ is all of $\mathbb{R}$. Its range is $\left(-\frac{\pi}{2},\frac{\pi}{2}\right)$. – kccu May 1 '17 at 1:55
• Note that $\arctan(x)-\arctan(y) = \arctan{\frac{x-y}{1+xy}}$. – marty cohen May 1 '17 at 4:49
• Ah, yes, this problem comes from the most recent Putnam exam, doesn't it? – Feryll May 1 '17 at 8:03
• $\arctan(x)$ is well-defined on all of $\mathbb{R}$, with values in $(-\frac{\pi}{2}, \frac{\pi}{2})$. The precise function is irrelevant for proving it to be a metric, it only needs to 1-1. – Henno Brandsma May 1 '17 at 8:50

(I). If $f:\mathbb R\to \mathbb R$ is injective then $d(x,y)=|f(x)-f(y)|$ is a metric on the set $\mathbb R:$

(i). $x\ne y\implies d(x,y)=|f(x)-f(y)|\ne 0$ because $f$ is $1$-to-$1$.

(ii). $d(x,y)=d(y,x).$ Obvious.

(iii). $d(x,z)=|f(x)-f(z)|=|(f(x)-f(y))+(f(y)-f(z)|\leq$ $\leq |f(x)-f(y)|+|f(y)-f(z)|=d(x,y)+d(y,z).$

The derivative of $\arctan x$ is $1/(1+x^2) ,$ which is always positive. So $\arctan x$ is strictly increasing, and therefore is $1$-to-$1.$

(II). We have $\lim_{n\to \infty}\arctan n=\pi /2,$ so $(\arctan n)_n$ is a Cauchy sequence, that is, $$\lim_{n\to \infty}\sup_{a>b>n}|\arctan a-\arctan b|=0.$$ So with respect to the metric $d(x,y)=|\arctan x-\arctan y|,$ the sequence $(n)_n$ is a $d$-Cauchy sequence.

But there is no $z\in \mathbb R$ such that $\lim_{n\to \infty}d(n,z)=0$, as this would require $\lim_{n\to \infty}|\arctan (n)-\arctan (z)|=0,$ which implies $\arctan z=\lim_{n\to \infty}\arctan n=\pi /2,$ and there is no such $z\in \mathbb R.$


text:
# Permutation group with a nice lattice of block systems

Let $$X$$ be a finite set and $$G$$ be a transitive subgroup of the symmetric group on $$X.$$ Recall that a (complete) block system for this action is a partition of $$X = B_1 \cup \cdots \cup B_k$$ into equally-sized blocks $$B_1 , \ldots , B_k,$$ which is preserved in the sense that blocks are always mapped to blocks under the group action. The block systems for a particular action form a lattice whose maximum is $$\{ X \}$$ and whose minimum is the partition of $$X$$ into singletons.

In the lattice of block systems, we may associate to any chain a sequence of integers: starting from the maximum element of that chain, divide the size of a block of the given block system by the size of a block of its predecessor.

Question: Is there a name for group actions or permutation groups satisfying the condition that the aforementioned sequences associated to every maximal chain determine the same multiset?

Example 1: Let $$G=X=A_4$$ acting on itself by multiplication. The lattice of block systems is essentially the lattice of subgroups (blocks $$=$$ cosets.) There are two incomparable chains of subgroups whose associated sequences are $$(3, 2, 2)$$ and $$(4, 3),$$ so the condition is not satisfied.

Example 2: $$G = A_4$$ acting as usual on $$X= \{ 1,2,3,4 \}$$ is primitive, so the only sequence is $$(4)$$ and the condition is trivially satisfied.

Example 3: For $$D_8$$ acting on itself by multiplication, the sequence is always $$(2, 2, 2),$$ so the condition is satisfied.

Example 4: For $$\mathbb{Z}_6$$ acting on itself by multiplication, the sequences are $$(3,2)$$ and $$(2,3),$$ so the condition is satisfied.

Example 5: For the wreath product $$S_n \wr S_m$$ acting as usual on $$[n] \times [m],$$ the lattice of block systems is a chain of length $$2.$$ The only sequence is $$(m, n),$$ and hence the condition is satisfied.

This condition seems natural to me from the point of view of Ritt's polynomial decomposition theorems and related work. In that context, $$G$$ is a Galois group acting on the fibers of some polynomial or rational map of finite degree, and we want to "factor" that map as a composition of "irreducible" maps. The condition should then imply that the set of degrees of the irreducible elements are independent of the decomposition. For instance, corresponding to Example 4, the decompositions $$(z^3)^2 = (z^2)^3$$ have the same set of associated degrees.

Rather than characterizing such actions (although that would most certainly be welcome!), I am mostly just wondering if this condition or some close cousin already has an established name in the literature.

• Could you give a more detailed (and perhaps more formal) definition of the sequence of integers associated with a chain of blocks systems?. I cannot understand it at the moment. In examples 1 - 4, the product of the integers in the sequence is the order of the group, but that is not the case in Example 5. Mar 15 at 19:29
• @DerekHolt Indeed, I made mistakes writing down the sequences for both Examples 2 and 5. In general, the product of the sequence should be the cardinality of $X.$
– tim
Mar 15 at 19:41
• Note that the lattice of block systems is isomorphic to the lattice of subgroups of $G$ containing a particular vertex-stabiliser $G_x$. (This is well known and not hard to prove.) The integers you have in mind are then the successive indices of subgroups in a chain. So you're asking for group-subgroups pairs, such that, in every maximal refinement as a subgroup chain, the multiset of indices is the same. (Maybe you knew this already, but just in case.) (I'm ignoring the detail of whether the point-stabiliser is core-free, because it doesn't really matter.) Mar 16 at 5:31
• I believe that I could prove without too much difficulty that the property holds for all actions of a finite group $G$ if and only if $G$ is supersolvable (in which case all numbers in the integer sequences are prime). Mar 16 at 7:27
• Also a technical query about the definition: would you consider the sequences $(4,4,2)$ and $(4,2,2,2)$ to be equivalent? They have the same underlying set (so I think they are equivalent by your definition) but with different multiplicities. Mar 16 at 7:29


text:
## New Paper

### New Cosmology group paper

The PAU Survey: Measurements of the 4000 Å spectral break with narrow-band photometry ” by members of the Cosmology Group and PIC has been uploaded to arXiv.

One of the most important features in the visible part of galaxy spectra is the D4000 break, generated around 4000 \AA\ by the absorption of several ionised metallic elements, as well as the latest lines in the Balmer series. D4000 is one of the most commonly proxy to estimate the stellar age of galaxies allowing the trace of the star formation history of galaxies. The direct measurement of the D4000 is generally reserved for spectroscopic surveys. For the first time, we present a general method to directly measure the D4000 spectral break index with narrow-band photometry.

This method reduces the bias in D4000 estimations on simulated galaxies by a factor of two in comparison to the previous method (Stothert at el. 2018). Together with SED fitting approach, we applied direct method to observed PAUS galaxies in the VIPERS field ($i_{\rm AB} < 22.5$, $0.5 < z < 1$). PAUS direct method shows an excellent agreement (>0.99) with spectroscopic measurements with Gaussian noise up to i_{AB}< 21, i.e. two magnitudes below PAUS completeness limit, keeping the performance on the level of ~0.9 for the full sample. Our results show that narrow-band photometry is a good proxy to recreate accurately spectral breaks, which is not achievable with broad-band photometry. Moreover, the direct narrow-band D4000 estimations might be used as a powerful classification tool to separate blue and red galaxies without the necessity of spectroscopic information or model-dependent SED-based derivable.

The relation of D4000 on stellar mass and star formation rates obtained with narrow-band photometry is in excellent agreement with the one derived from spectroscopy (within 0.5-1sigma) allowing us to study galaxy properties on a wide field with accuracy similar to the one provided by spectroscopy. Thus, the narrow-band photometry might be considered as the next step in resolving the nature vs nurture dilemma in galaxy evolution studies.

## Latest Group News

Conference

IFAE at the European Astronomical Society Annual Meeting (EAS 2022)

July 13, 2022

New publication

New DESI paper

May 18, 2022

Overview of the Instrumentation for the Dark Energy Spectroscopic Instrument ” has been uploadd to arXiv. Several IFAE members are co-authors of the paper.

New publication

New DESI paper by the cosmology group

May 18, 2022

The effect of quasar redshift errors on Lyman-α forest correlation functions ” by A. Font-Ribera, Ignasi Pérez-Ràfols, César Ramírez-Pérez and collaborators has been uploaded to arXiv.


text:
## anonymous 3 years ago Range of$f(x)=\sqrt[3]{x-1}+\sqrt[3]{3-x}$

1. anonymous

????

2. anonymous

made a stupid error, just a sec.

3. anonymous

|dw:1347772271638:dw|

4. anonymous

yeah range is limited...i dont how to evaluate the exact interval for range :(

5. anonymous

*know

6. anonymous

Can you plot the two parts of the function? then combine them.|dw:1347772792991:dw| For analyzing the final result : If it's a diagonal line then yes both will go from -infinity to +infinity, IF HOWEVER it is a VERTICAL or HORIZONTAL line then your domain or range will be limited.

7. anonymous

i want to show it algebrically if its possible !!

8. anonymous

idea: seperate f(x) into 2 functions, find the domain of the inverse of each one of the function, find the intersection of the 2 domains, that gives the range of f(x)

9. nipunmalhotra93

Range is ]0,2]. f(x) is symmetrical about x=2. I evaluated the first and second derivatives. for x<1<2 f'(x)>0 and f''(x)>0 for x<1, f'(x)>0 and f''(x)<0. behavior on other side of 2 is symmetrical. Also, evaluating the limit at -infinity, we get 0. So the graph would be something like this.|dw:1347780606499:dw| That's it. Correct me if I'm wrong.

10. nipunmalhotra93

Another small test is that f(x) can't be <0 because solving it gives -1<-3 which is a contradiction.

11. anonymous

the range of f(x) is ( 0,2]

12. anonymous

to show it algebrically you need to assume $\sqrt[3]{x-1}= \alpha$ $\sqrt[3]{3-x}= \beta$ and after that you can obtain the required equations

13. anonymous

as $f(x)=y=\alpha +\beta$ so find out x in form of y and you would get the answer if you want more detailed solution then i can provide you that too

Find more explanations on OpenStudy


text:
# Symmetric vs. Asymmetric Synapses in Visual Cortex

A montage of 100 synapses illustrating differences in opacity of the postsynaptic density (PSD) and shape of synaptic vesicles. The synapses are displayed in approximate order of decreasing opacity. For synapses 1-56, it is impossible to discern an exact order since the PSDs appear similar. For image 57 and higher, the density decreases in thickness and the vesicles appear more elongate by image 83. For purposes of classification, Colonnier considered images 1-79 to represent asymmetric synapses whereas synapses 83-100 were identified as symmetric. He concluded that these two classifications were sufficient since a proportion of synapses failed to appear morphologically "transitional" between the asymmetric and symmetric categories (x 17000).

Reproduced from Colonnier (1968).


text:
# Lower Bound of Central Binomial Coefficients

I would like to prove by induction the following inequality:

$\frac{4^n}{n+1} < \binom{2n}{n}$, for all natural numbers n > 1.

Any hints?

-

You know that $$\binom{2(n+1)}{n+1} = \frac{2(n+1)(2n+1)}{(n+1)^2}\binom{2n}{n}.$$

The main step of the induction proof is therefore

$$\frac{4^n}{n+1} \frac{2(n+1)(2n+1)}{(n+1)^2} \ge \frac{4^{n+1}}{n+2}.$$

This inequality is easy to show after simplification.

-
Is there any book reference for this result? Thank you. –  Michael Dec 1 '12 at 14:12
One has (see this question and this one) the identity $$4^n=\sum_{i=0}^n\binom{2i}i\binom{2(n-i)}{n-i}$$ On the right we have $n+1$ terms, if we can show that each one of them is at most $\binom{2n}n$, and at least one is strictly less, then we will have shown $4^n<(n+1)\binom{2n}n$. Now $\binom{2n}n$ counts the lattice paths across a $n\times n$ square, while $\binom{2i}i\binom{2(n-i)}{n-i}$ counts such paths that pass through the point $(i,i)$, so the required inequalities are obvious. I'm sorry I didn't use induction.


text:
# The Essential Supremum as a Limit

Let $(X, \mathcal F, \mu)$ be a finite measure space and let $f\in L^\infty(X, \mu)$. Define $\alpha_n=\int_X |f|^n\ d\mu$. Then $$\lim_{n\to \infty}\frac{\alpha_{n+1}}{\alpha_n}=\|f\|_\infty$$

Thanks.

• HINT I think this may help. Note that $$\frac{\alpha_{n+1}}{\alpha_n}=\frac{\|f\|_{n+1}^{n+1}}{\|f\|_{n}^{n}} = \left( \frac{\|f\|_{n+1}}{\|f\|_{n}} \right)^n \|f\|_{n+1}$$ Now, use the fact that when the space has finite measure, $\lim_{p\to \infty }\| f\|_p=\|f\|_\infty$, which has already been proven here math.stackexchange.com/questions/242779/limit-of-lp-norm – Alonso Delfín Apr 21 '15 at 13:43

Since $f\in L^\infty(X,\mu)$ and $\mu(X) < \infty$, then $f\in L^n(X,\mu)$ for all $n \ge 1$. Since

$$\alpha_{n+1} \le \|f\|_\infty \int_X |f|^n\, d\mu = \|f\|_\infty \alpha_n,$$

we have $$\varlimsup_{n\to \infty} \frac{\alpha_{n+1}}{\alpha_n} \le \|f\|_\infty.$$

On the other hand, since $\alpha_n^{1/n} = \|f\|_n \to \|f\|_\infty$ as $n\to \infty$,

$$\varliminf_{n\to \infty} \frac{\alpha_{n+1}}{\alpha_n} \ge \varliminf_{n\to \infty} \alpha_n^{1/n} = \|f\|_\infty.$$

Therefore $$\lim_{n\to \infty} \frac{\alpha_{n+1}}{\alpha_n} = \|f\|_\infty.$$

Note. We can get $\varliminf_{n\to \infty} \frac{\alpha_{n+1}}{\alpha_n} \ge \|f\|_\infty$ alternatively by applying Holder's inequality with conjugate exponents $n+1$ and $(n+1)/n$. Doing so, we obtain $\alpha_n \le \mu(X)^{1/(n+1)}\alpha_{n+1}^{n/(n+1)}$. Thus

$$\varliminf_{n\to \infty} \frac{\alpha_{n+1}}{\alpha_n} \ge \varliminf_{n\to \infty} \mu(X)^{-1/(n+1)} \alpha_{n+1}^{1/(n+1)} = \lim_{n\to \infty} \|f\|_{n+1} = \|f\|_\infty.$$

• That's great. Thanks! – caffeinemachine Apr 21 '15 at 14:11


text:
# Prove this limit, I am really stuck

Prove that $$\lim_{x\to 0}\frac1{x^2+1}=1.$$

I know after adding common denominators that we get $(1-x^2)/(x^2+1)$. But I don't know of a way to bound $x$ here from bounding my $\delta$. Can I get some help please? Exam is tomorrow. Use and epsilon-delta proof

-
Since the denominator converges to $1$, there is actually nothing to prove. –  Daniel Robert-Nicoud Mar 2 '14 at 1:34
Your function is continuous at $f(0)$, therefore $\lim _{x\rightarrow 0} f(x) = f(0)$ –  Astrum Mar 2 '14 at 1:39
I see, I mis-added, thats embarrassing –  Jack Armstrong Mar 2 '14 at 1:40

Let $\varepsilon > 0$ and choose $\delta = \sqrt{\varepsilon}$, then $|x-0|<\delta$ gives us

$$\left|\frac{1}{x^2+1}-1\right| = \left|\frac{1-(x^2+1)}{x^2+1}\right| = \frac{x^2}{x^2+1}<\frac{x^2}{1}<\delta^2=(\sqrt{\varepsilon})^2=\varepsilon.$$

-
I mis-added, thats embarrassing –  Jack Armstrong Mar 2 '14 at 1:41

Well let's get started by bounding $\left|{f(x) - 1}\right|$:

$$\left|{\frac{1}{x^2 + 1} - 1}\right| = \left|{\frac{x^2}{x^2 + 1}}\right| \lt x^2$$

The last inequality was due to teh fact that the denominator is always greater than $1$.

Therefore given any $\epsilon \gt 0$ there exists $\delta (=\epsilon ^{\frac 1 2}) \gt 0$ such that $\left|{x - 0}\right| \lt \epsilon ^{\frac 1 2} \implies \left|{\frac{1}{x^2 + 1} - 1}\right| \lt \epsilon$.

Q.E.D.

Taking the square root here is not a problem since we are dealing with strictly positive quantities. And you should also convince yourself that $a \lt b \iff \sqrt a \lt \sqrt b$ for positive values $a$ and $b$

-


text:
# Homework Help: Classical problem: projectile exploding into two parts with one part going down

1. Sep 28, 2006

### knightpraetor

So my problem is: a mass M that is fired at 45 degrees with KE E_0, at the top of the trajectory, projectile explodes with additional energy E_0 into two parts, the first fragment travels straight down , what is the velocity of the the first and the velocity and direction of the second part.

Also, what is the ratio of m1/m2 when m2 is maximized.

so far i've played with it for awhile, but basically using conservation of energy and conservation of momentum in the X and Y directions..gives me 3 equations for 3 unknowns (theta [angle the second particle fires off at], and k1 and k2, where

v1 = -k1*v, and v2 = k2*v

, that i'm to write in terms of other unknowns that it is assumed i will know when calculating for actual values i guess ( ie mass of the first and second particles, and initial velocity of the big mass M and the starting energy)

Anyways, I'm wondering if i'm doing this right, i start with a bunch of equations

$$4E_0 = m_1 (k_1 v)^2 + m_2 (k_2 v)^2$$

$$-m_1 k_1 v = m_2 k_2 v sin(\theta)$$

$$(m_1+m_2) v = m_2 k_2 v cos(\theta)$$

are these the right equations? ...and so to solve these qquation i wish to solve this system of equations for equations involving k1, k2, and theta...i assume it's easiest to solve for k2, plug it into k1, then plug both those equations into the energy one and solve for theta....is taht right? and i have no clue how you would even begin to go about the second part

2. Sep 29, 2006

### Astronuc

Staff Emeritus
What if one determines the initial energy/momentum of both at the time of the explosion. At the top of the arc, the vertical velocity is zero, but each mass has a horizontal velocity. Determine horizontal and vertical velocity in terms of E0. Neglecting air resistance, the combined mass would be moving in a parabolic trajectory.

Then determine the initial velocities of each mass at the explosion (with respect to CM). Note that one accelerates upward and the other downward, then there is gravity acting on each mass.

If the bottom mass goes straight down, then the horizontal component of it's velocity must be equal to it's initial horizontal velocity.


text:
# Machine learning for financial prediction: experimentation with David Aronson’s latest work – part 2

Posted on May 10, 2016 by
43 comments
7,232 Views

## Introduction

My first post on using machine learning for financial prediction took an in-depth look at various feature selection methods as a data pre-processing step in the quest to mine financial data for profitable patterns. I looked at various methods to identify predictive features including Maximal Information Coefficient (MIC), Recursive Feature Elimination (RFE), algorithms with built-in feature selection, selection via exhaustive search of possible generalized linear models, and the Boruta feature selection algorithm. I personally found the Boruta algorithm to be the most intuitive and elegant approach, but regardless of the method chosen, the same features seemed to keep on turning up in the results.
In this post, I will take this analysis further and use these features to build predictive models that could form the basis of autonomous trading systems. Firstly, I’ll provide an overview of the algorithms that I have found to generally perform well on this type of machine learning problem as well as those algorithms recommended by David Aronson (2013) in Statistically Sound Machine Learning for Algorithmic Trading of Financial Instruments (SSML). I’ll also discuss a framework for measuring the performance of various models to facilitate robust comparison and model selection. Finally, I will discuss methods for combining predictions to produce ensembles that perform better than any of the constituent models alone.
Without further ado, let’s dive in and discuss some machine learning algorithms.

## Algorithm selection

Anyone familiar with machine learning can tell you that the quantity of algorithms available to the practitioner these days is staggering. With the rise of open source packages like R, used widely by industry, academics and hobbyists to collaborate on and share machine learning research, some of the most cutting edge statistical learning frameworks are literally at our fingertips.
Its an exciting time, but also somewhat daunting.
At last count there were 81 machine learning packages listed on CRAN’s Machine Learning Task View, and many of those packages provide access to numerous individual algorithms! It’s impractical to perform an exhaustive search for the ‘best’ algorithm for a particular task, but it is certainly possible to arrive at some guidelines around what tends to work better for particular use cases.
Aronson and Masters (2013) prefer linear and quadratic regression, boosted trees, and general regression neural networks. They state that “a single decision tree’s utility is debatable for financial data”, as are bagged ensembles of trees such as random forests, however boosted trees may be more appropriate. Personally, I found that neural networks initialized using stacked autoencoders were more promising than the general regression networks preferred by Aronson and Masters. I also found particularly useful models based on Friedman’s gradient boosting machine (Friedman, 2001). In addition, I was able to obtain surprisingly decent results from a simple k-Nearest Neighbors algorithm, however I had less success with bagging methods like random forests. Like Aronson and Masters, I avoided using single decision trees – for the the number of variables used in the investigation (see next section), there seems little point. The table below shows the algorithms that I investigated and highlights those that showed the most promise for this particular use case.
The table with ID 2 not exists.

Even with the small selection listed here, we clearly have some decisions to make. I’ll present some results from these models below.

## Choosing combinations of variables

For the purpose of this exercise, I chose the six features from the previous post that I feel are most likely to convey predictive information about the target variable. There are numerous combinations of features that we could use to build individual models, for example, various combinations of 2, 3, etc variables. Assuming we only build models based on at least two variables, we have 57 possible unique combinations from a pool of six features. Aronson and Masters (2013) caution against using too many variables due to the risk of overfitting the data, stating that two or three variables is usually the practical limit. In this analysis, I’ll take their advice and explore models with two or three features only. This limits the number of unique combinations to 35.
The following features were chosen (refer to the previous post for calculation details):

1. 3-period momentum
2. Delta bollinger width
3. 10-period velocity
4. 10:100 period ATR ratio
5. 10:20 period ATR ratio
6. 7-period ATR

Each feature was normalized to the range $+1 : -1$ using a rolling 50-period window. I generally find that this method gives superior model performance to normalizing over the entire data set.
Download the raw data I used in this study via this link: EUvarsD1. In addition, the code below uses Zorro to calculate the features and target variables listed above and outputs them to a CSV file like the one in the download link. Using this code, you can generate your own feature sets for use in the modelling framework described in this post. You can also  use it to change any other parameters, such as the normalization period and the amount of historical data. Using Zorro, this can easily be adapted to other markets as well. Click the box below to expand the code.

## A framework for measuring and comparing performance

We have 35 possible variable combinations and 7 algorithms with which to construct predictive models. The subset of variables was constrained based on the feature selection process discussed in the last post. I’ve constrained the list of algorithms by attempting to maximize their diversity. For example, I’ve chosen a simple nearest neighbor algorithm, a bagging algorithm, boosting algorithms, tree-based models, neural networks and so on. Clearly, I’ve constrained my universe of models to only a fraction of what is possible. Still, there is a lot of choice. We could randomly choose various models in the hope of landing on something profitable, however since with today’s computing power we very much have the means to implement it, I much prefer the idea of a systematic, comprehensive assessment. I’ll describe my framework for such an assessment below.
In this example, I will train each model to maximize the return of trading the model’s predictions normalized to the recent volatility measured by the 100-period ATR. Simple enough, but how would we objectively assess the performance of each model against this metric? And more importantly, how do we account for overfitting? Ideally, we would measure the out-of-sample performance of each model, but of course we have a finite amount of data and we need to maximize its utility. This is where cross validation comes in. There are plenty of great sources on the internet for detailed descriptions of cross validation, so I will only describe the procedure briefly.
Cross validation involves dividing the training data into k portions, training a model on k-1 portions and testing it on the portion that was held out. The performance on the hold out set is a first estimate of the true out of sample performance. The procedure is repeated using each subsample as the hold out test set and the results finally aggregated. This is known as k-fold cross validation. The estimate can be made more robust (usually) by randomly resampling the data into new portions and repeating the procedure. This is known as repeated k-fold cross validation. Other variations include bootstrapped cross validation (where resampling is undertaken with replacement – that is, individual observations can appear more than once in any subsample) and leave-one-out cross validation (where one observation at a time is held out, the remainder of the data used to train a model, and the performance of the model estimated by predicting the held out observation, repeated for every observation in the training set).
Cross validation is a very useful procedure for estimating the true out of sample performance while maximizing the utility of the training data. This sounds fantastic, and it is for most data sets, but time series data presents some unique challenges. Consider a data set with no temporal dimension and in which the observations are independent and identically distributed. In any predictive modelling task for such a data set, we can never have too much data (within the practical constraints of computing resources of course). This is not necessarily the case with time series data.
If we use too much of the time series to train our model, we risk including irrelevant and outdated patterns which of course by definition are unlikely to show up in the next instances of the time series. If we use too little data, we run the risk of under-fitting the model and missing the predictive information we hope to capture. How much data do we need then? I don’t know, but I intend to find out.
Rob Hyndman describes a method for cross-validating time series data which is extremely useful for algorithmic system development as it actually mirrors a procedure that can be used for live trading. Also known as “rolling origin forecast evaluation” and “forward chaining”, it is the best method I have found for quickly estimating the future performance of a model. The procedure is implemented thus:

1. Fit the model to a window of sequential data of length $t$: $x_1, x_2, …, x_t$
2. Predict the next value in the sequence, $x_{t+1}$, and compute the forecast error by comparing the prediction with the observed value.
3. Shift the origin of the window forward by one and repeat steps 1 and 2.
4. Repeat the process until $x_1=x_{n-t}$ where $n$ is the length of the series (ie until we run out of data for creating a window of length $t$)
5. Aggregate the forecast error for an estimate of the out of sample performance of the model

For readers familiar with walk forward analysis, time series cross validation is equivalent to walk forward analysis with the test set being a single period.

Wait! Before you keep reading....

Learn why Algo Trading is the only trading that will make you profitable long term — and where to start.

Discover exactly what algo trading is, the skills used by 99% of profitable traders, and how to escape your demo account to trade profitably.

## Is there an optimal window length?

As mentioned above, I want to investigate the existence or otherwise of an optimal amount of data to include in the rolling window of training data. How much data is too much? Too little? At what point do we incorporate old and outdated information into a model to its detriment? At what point do we underfit the model due to lack of data?
For this experiment, I’ll model the EUR/USD exchange rate using a gradient boosting machine, a neural network and a k-nearest neighbors algorithm using various window lengths in the cross-validation procedure. My hypothesis is that there exists an optimal amount of data that maximizes the performance of a model for this particular time series. I am choosing three different algorithms in order to test the sensitivity of the optimal window length to the choice of algorithm. I expect that the length of the window itself will be an optimization parameter that is different for each market and that may itself change with time.
The figures below show the Sharpe ratios and directional accuracy by window length for the cross-validated model predictions. Returns of the analogous trading system were calculated as follows:
for all $y > 0$, go long at the close and hold the position for 1 period
for all $y < 0$, go short at the close and hold the position for 1 period
where $y$ is the prediction of the next period’s return normalized to the 100-period ATR. Results are exclusive of trading costs.

The Sharpe ratio shows an obvious increase with increasing window length. This came as a surprise to me, as I suspected that there would be a point where data was too far removed from current market conditions to be of use in a predictive trading model. If such a point exists, it is clearly greater than 1,000 days. In hindsight, normalizing each feature using a rolling 50-period normalization window very likely ensures that the model dynamically adapts to changing conditions, but I must admit that I stumbled upon this more by accident than by design.
Setting aside the obvious trend in the Sharpe ratio results, the other thing that jumps out at me is that the cross-validated performance of the k-Nearest Neighbors algorithm is at least on par with, and at times significantly better than the much more complex gradient boosting machines and neural networks. It is interesting that such a simple algorithm can hold its own against the more complex models.
In my last post, I talked about the concept of a “prediction threshold”, being a method of filtering trades by entering the market only when the magnitude of the predicted next-period return exceeds a certain value. My hypothesis for doing so is that there is merit in filtering the small moves, which would be affected by randomness, in favor of the larger moves. Said differently, it makes sense (depending on the objectives of the trader) to target those returns in the tails of the distribution. Testing this hypothesis, I plotted a heatmap of Sharpe ratios by window length and prediction threshold for the neural network model:

In general, there appears to be merit to this hypothesis with Sharpe ratios generally increasing for increasing prediction threshold.
Here’s the code I used for the neural network models (the models based on k-nearest neighbors and gradient boosting machines used a similar procedure):

## Comprehensive model comparison

Now that I know that, in general, a longer rolling window helps a predictive model achieve a larger Sharpe ratio, my next experiment will be a comprehensive comparison of the 245 possible models from our 7 algorithms and 35 variable combinations.
The results below show the cross-validated Sharpe ratios for each combination of algorithm and variable subset investigated in this study using a rolling window of 1,000 days of history as the training data. In each case, the optimal model was found by tuning the individual algorithm’s hyperparameters across a sensible subset of possible values. No prediction threshold was used to filter trades (that is, each model took a long or short position every day). Once again, transaction costs are excluded.

The neural network initialized using a stacked autoencoder is the clear out-performer of this group of algorithms. This model seems to be able to learn the underlying predictive patterns better than any other algorithm used in this study, and it does so consistently regardless of the combination of variables used. Initialization with a stacked autoencoder essentially forces the network to recreate the input data in the pre-training phase, which results in a network that tends to learn the features that form a good representation of its input, reducing the noise component of the input data. With this in mind, it may be possible to use this technique with a larger number of features, potentially foregoing an extensive feature selection process. This is an interesting idea, and something to pursue at another time.
We can also see that the boosting algorithms performed relatively well, the multi-adaptive regressive spline models were consistent, and that k-nearest neighbors performed well for such a simple model. The Cubist models were less consistent and the random forest models performed worst of all.

The best Sharpe ratio of all the models was 1.72.
The following R code shows how I implemented this framework for efficiently testing and comparing the algorithms used in this study:

## Accounting for data mining bias

Data mining bias refers to the unfortunate selection of a trading model based on randomly good performance. For instance, a system with no basis in economic or financial reality has a profit expectancy of exactly zero, excluding transaction costs. However, due to the finite sample size of a backtest, sometimes such a system will show a backtested performance that can lead us to believe it is better (or worse) than random. As the number of samples grow in live trading, the worthlessness of such a system becomes apparent.
Data mining bias shows up often, but it is of particular concern when we harness computing power to systematically test numerous potential trading models, exactly as I have done in this post. White (2000) describes a method for accounting for this data mining bias, referred to as White’s Reality Check or the Bootstrap Reality Check. David Aronson covers it in detail in “Evidence Based Technical Anlysis.” I will only describe the procedure briefly here, but here is the link to the original paper.
White’s Reality Check requires that we keep a record of all variants of the trading model that were tested during the development process and produce a zero-mean returns series for each. We then randomize these returns series using bootstrap resampling and note the total return of the best performer. This process is repeated several thousand times, and the median best return corresponds to the data mining bias introduced by the development process. We can then observe where the originally selected best model fits into the distribution of bootstrapped results to obtain a confidence level relating to its possession or otherwise of an actual positive expectancy. Saddest of all, and I think this is why White used the term “reality check”, is that the expected performance of the selected strategy in real trading is only the performance measured in its backtest MINUS the median performance of the bootstrapped returns series.
Here is the histogram of Sharpe ratios from implementing White’s Reality Check on the 245 models under comparison for 5,000 bootstrap iterations:

The best Sharpe ratio I obtained is approximately equivalent to the median bootstrapped best Sharpe ratio, implying that its expectancy is actually close to zero. However, I have clearly mis-estimated the data mining bias since I excluded the models discarded during the hyperparameter tuning phase of model construction. In addition, this method is known to have a bias towards Type II errors. In other words, this method tends to reject systems that do have an edge. In any event, following is the R code that implements the reality check. I leave it to the interested reader to implement the reality check across the full set of models including those rejected during hyperparameter tuning.

## Ensembles and hybrid methods

Ensembling is the practice of aggregating the predictions of multiple models in order to achieve a prediction accuracy that exceeds any individual model. It is analogous to using a committee of experts to reach a consensus. There are numerous ways to create an ensemble, including:

• Bagging: aggregate models based on bootstrapped training data, for example the random forest algorithm.
• Boosting: models developed sequentially where each additional model aims to improve performance on the least accurate part of the feature space of the previous model. An example is the boosted trees used previously in this study.
• Stacking: model predictions are combined using a “meta-model”. In my experience tends to result in over-fit models.
• Aggregating models based on random subsets of the input features.
• Aggregating models based on random subsets of the training data, which is equivalent to bagging without replacement as opposed to bootstrapping.
• Aggregating models based on different algorithms.

We have seen above that the models that incorporate boosting fared better than the random forest models, which use the bagging technique. Aronson and Masters (2013) advocate the latter three methods listed above as a protection against overfiitting. They argue that even if the component models are over-fit, if they are trained on different training and/or feature sets, they will be exposed to different noise patterns while real patterns will tend to be represented across the various training sets. The idea is that the noise patterns cancel out, while the real patterns are reinforced. Ensembles appear to be most effective when the component models each have a positive expectancy, but whose predictions are uncorrelated.

In this study, I will focus on combining different algorithms and different subsets of the feature set and I will once again advocate a systematic approach. Following is my approach for selecting component models for an ensemble:
1. Build a model for each variable and algorithm combination (already done, above)
2. Exclude any model with a cross-validated Sharpe ratio < $x$, say 1.25
3. Examine a correlation matrix of the predictions of the remaining models. Retain a subset of models such that all correlations are < 0.75
4. Combine the remaining models by either averaging the predictions or forming a majority vote on the direction

The figure below shows the equity curves of the best model and the two ensembles formed by averaging and majority vote described above (trading position sizes on EUR/USD relative to the 100-period ATR and excluding trading costs).

In this case, there is little overall difference between the best model and either of the ensembles. There may be other possible ensembles that perform better; I haven’t performed an exhaustive search. Note that in the absence of an out of sample test following the aggregation of component models into an ensemble, this step has introduced a measure of selection bias into the results. Here’s the code to reproduce these results:

Finally, I’ll try combining the predictions using a linear regression model as a “meta-model”. This approach has the potential to curve-fit the results, so I will test it using the time series cross validation approach used above:

This gives a Sharpe ratio of only 0.94, significantly less than the simpler ensembles and indeed the best of the component models.

## Next Steps

This post and the last provide a framework for systematically comparing predictive models based on machine learning algorithms as the basis of trading systems. Despite the length of the posts, they have barely scratched the surface of what is possible. The following are ideas for future research that I intend to pursue. I would love to hear from people interested in collaborating on any of these ideas and I offer my work thus far as a basis from which to proceed:

• Regime-based split linear models. The idea is that the training data is split based on different market regimes on the basis of a particular variable or condition for determining where the split occurs. Regimes characterized by extreme volatility are inherently difficult to predict, and it may be useful to simply avoid trading during these periods. A regime-based approach could potentially verify this hypothesis and provide clues as to its practical application. I have briefly investigated this approach and did not find a measurable increase in performance, but I haven’t investigated closely enough to rule out this idea completely.
• Another regime-based approach is to dynamically adjust the relative weights on individual component models based on each model’s performance in the current market conditions. This is appealing in that it does not require classification of the market regime. The weight adjustment is essentially regime-agnostic in the sense that it would not care about terms such as “trending” or “range bound”, whose identification of course carries substantial lag. By taking its cues from real-time performance, a dynamically adaptive weight allocation approach would minimize this lag to the extent possible.
• So far I have simply aggregated models into ensembles through averaging predictions, combining directional votes and using simple linear regression as a “meta-model”. My feeling is that combining predictions using non-linear stacking methods will lead to over-fitting. However, it would be useful to verify this feeling on hard data.
• I have also read about ‘sequential prediction” which, like boosting, involves building an ensemble from a series of models. If the first model is able to learn a dominant pattern, its residuals are then used as input to a second model under the assumptions that these residuals contain the noise terms as well as more subtle patterns. The first model’s predictions are then considered a measure of the dominant pattern and the second model’s predictions an estimate of the deviation of the first model’s prediction from the target.
• A related approach is to attempt to capture linear relationships in the data using classical time series methods such as ARIMA/GARCH and then use the residuals of these models as input to an algorithm capable of capturing more complex non-linear relationships, such as a neural network.
• Finally, the work presented here has only considered the most recent values of individual features in model construction. There may be benefit in incorporating recent past values as features, for example the 1-period return for each of the last 3 periods.

## A Note on the Practicalities of Trading Systems Research

If I can offer one piece of advice based on my experiences, I would caution systems researchers (especially those with a penchant for the theoretical) against trying to build the “perfect” predictive model, if such a thing even exists. Admittedly it is an interesting exercise, but an approach much more grounded in the realities of extracting profits from the markets is to focus on building a model that meets one’s needs. If I could turn back the clock to the tune of about twelve months, I would go live with my machine learning models much earlier (as soon as they were fit for purpose) and then refined them as time went on. I made the mistake of repeatedly putting off the go-live date until I had improved the models performance just a bit more. In doing so, I missed some perfectly achievable profits. In addition, I would have uncovered implementation and execution issues much more efficiently.

## Conclusions

You have seen in this post and the last that mining financial data for profitable patterns in a statistically sound manner is incredibly time consuming and effort-intensive. What you’ve read in these last two posts is simply a distillation of a significant research effort and one that I hope provides a framework for others interested in this field to take it further.
We have seen that neural networks initialized with stacked autoencoders have a great deal of potential in predictive models for trading systems, but that incredibly simple models like the k-nearest neighbors algorithm can also perform adequately. I have also touched on the specter of data mining bias and explored one possible method for accounting for it. Finally, we explored ensembles of component models, but didn’t get a significant boost to model performance in this case.
I intend to pursue the ideas listed in the Next Steps section, likely beginning with the sequential prediction approach. If people are interested in collaborating on any of these ideas, I would very much like to hear from you.

Wait! Before you keep reading....

Learn why Algo Trading is the only trading that will make you profitable long term — and where to start.

Discover exactly what algo trading is, the skills used by 99% of profitable traders, and how to escape your demo account to trade profitably.

#### (43) Comments

May 10, 2016 at 9:46 pm

Thanks,Very interesting
Can you share the data set files?

May 11, 2016 at 8:10 am

Hi Lev
Apologies for that oversight. You can now access the raw data via a download link in the post (under the heading “Choosing combinations of variables”). I’ve also added the source code (written in Lite-C, compatible with Zorro) that I used to generate this data. This code can be modified to generate custom feature sets for any market or time period for which you have price data. The code will output a CSV file compatible with the framework in the post.
Kris

May 13, 2016 at 6:38 pm

Hi Kris
Thanks for the source and csv.
I tried to reproduce your “heatmap of Sharpe ratios by window length and prediction threshold” and I get different results , All my predictions was in range (-0.3 – 0.3)
Lev

May 13, 2016 at 9:03 pm

That result has really got me scratching my head. I suspect something is not quite right with the implementation on your end because with so many models being compared, you would expect to see much more variation in the Sharpes beyond that range through random variation alone.
If anyone else gets this result, please let me know in the comments.

May 13, 2016 at 10:33 pm

I mean predictions not shapes.
The problem is only in “heatmap of Sharpe ratios by window length and prediction threshold” sample
modellist.nnet[[j]]$pred$pred in range of (-0.3 – 0.3)

May 14, 2016 at 2:46 pm

OK, I understand now. That’s actually not an unexpected result and indicates that in this case the model is having difficulty predicting the observations in the tails of the returns/atr distribution. Here’s one of my results, for comparison:
summary(as.numeric(modellist.nnet[[9]]$pred$pred))
Min. 1st Qu. Median Mean 3rd Qu. Max.
-0.51530 -0.10380 -0.01300 -0.01491 0.07690 0.40590
Check out what happens with the directional accuracy of these predictions though. They are actually good enough to give the model a decent cross-validated equity curve. They also hint at some potential improvements that can be made. For example, would a classification approach be better than the regression approach presented here? What about intelligent subsampling of the input to each iteration of the time series cross-validation procedure? I actually don’t know off the top of my head if such approaches would make a difference, but the results you see hint that there are improvements that could be made.

May 11, 2016 at 4:53 am

Fantastic tutorial series! Had never come across a neural network initialised with stacked autoencoders before. Looking forward to playing about with this technique.

May 11, 2016 at 8:17 am

Thanks Gekko! They are a relatively recent addition to my toolkit as well. The unsupervised reconstruction of the input appears to help the network effectively clarify a pattern if it does exist. The Zorro guys have done some work in this area too, using a wider feature set and a deeper architecture than the ones I’ve used here. They achieved a directional accuracy in the order of ~60%. My own work corroborates this. SAEs have enormous potential.

May 12, 2016 at 12:13 am

This grabbed my attention as well. I have not had much success with NNs of various architectures for regression problems. Classification of course is quite good, although my lack of patience for fiddling with feature scaling and data shape often just lead me to use a GBM. I wonder if there is any literature out there explaining why it would help with regression specifically. There is quite a bit that explains why boosted trees and NNs don’t fare well in regression. If unsupervised pre-training really changes the game for regression that’s very encouraging, as there are alot of those techniques already out there and more on the horizon. I am going to start testing toute suite.

May 12, 2016 at 9:36 am

Hi Shane. In my experience, neural nets fare extremely well on complex regression tasks. They have been referred to as “universal approximators” for good reason, having the ability to approximate any linear or non-linear function (compare this with for example a linear time series model like ARIMA). Being powerful learners, they also have the ability to very easily over-fit the training data. Therefore it makes sense to apply appropriate guards against over-fitting: regularization techniques (L1, L2), random dropouts, unsupervised pre-training and early stopping can be effective ways to get the most out of a neural network while preventing its power from getting out of control.
Also, if you are more comfortable with classification, you can simply re-frame the regression problem into a classification one using something like:
if (next.day.return > 0) target.variable = 1
else target.variable = 0

May 15, 2016 at 1:59 am

Thanks for the reply (and the great blog material in general). I actually have no discomfort with regression, and wrt finance/trading, continuous or countable targets are usually my choice. I simply just have not been able to surpass a MARS model or a custom nonparametric approach using a probabilistic programming library with any NN (and still haven’t after a few tests with deepnet BTW). However, this may have a great deal to do with my feature space and my choice of target. One observation I have made is that these R NN libraries have done a great job of setting some defaults on the myriad hyperparameters and providing some quick automated ways to iterate through them.
One particular architecture that I note is glaringly lacking is Recurrent Neural Networks. LSTMs, GRUs and other time series/sequence aware NN architectures have achieved the best performance for me wrt finance/trading. Obviously, there ability to model non-stationary data is desirable. So hopefully someone is generous enough to write some libraries for those in R that become part of the caret ecosystem. In the meantime, I’ll keep at it with Torch, Keras, Blocks etc.
Again, you have motivated me to spend some more time with these models and the caret ecosystem in particular. Look forward to future posts.

May 17, 2016 at 11:48 am

Hi Shane
Investigating recurrent networks is a priority for me too. These guys use recurrent architectures in many of the topics discussed in this publication and report good results in the FX markets. I know of no library in R that incorporates RNNs just yet. Time to make a foray into the world of R library creation perhaps?

[…] Machine learning for financial prediction: experimentation with Aronson s latest work – part 2… My first post on using machine learning for financial prediction took an in-depth look at various feature selection methods as a data pre-processing step in the quest to mine financial data for profitable patterns. I looked at various methods to identify predictive features including Maximal Information Coefficient (MIC), Recursive Feature Elimination (RFE), algorithms with built-in feature […]

May 11, 2016 at 9:49 pm

HI Kris
Great Article and thanks for posting!
I am interested in the Stacked Auto encoder approach you are using.
If you are using this to compress the data, can you not use more than 2/3 features to create inputs to your NN. Can i also ask how you go about choosing the optimal number of layers and the number of nodes in the Auto-encoder.
Tom

May 12, 2016 at 9:17 am

Hi Thomas
Indeed it does make sense to use more features with the stacked autoencoder approach. The unsupervised reconstruction of the input assists the network to detect any predictive patterns that may be present. This means that redundant or noisy features will have less of an impact on the output. I would however caution against a brute force approach where you throw everything you’ve got at such a network. You will quickly run into practical computation issues, particularly if you are simultaneously exploring various network architectures. Some intelligent feature engineering goes a long way.
In terms of choosing an optimal architecture, unfortunately there isn’t a simple formula that I’m aware of. Yoshua Bengio at the University of Montreal published some very useful guidelines for practical training of deep architectures which I use to guide my starting point. From there, I’ll train and cross-validate several architectures and hyperparameter sets. Max Kuhn’s caret package in R is my go-to toolkit for doing this as efficiently as possible. Here’s a link to Bengio’s paper: https://arxiv.org/pdf/1206.5533v2.pdf and Max’s website: https://topepo.github.io/caret/index.html
One word of caution, if I may: It’s easy to get bogged down in the search for an “optimal” model even having discovered several models that will serve the intended purpose very well. The search for an “optimal” model could go on indefinitely if we allowed it. Focus on a defined objective for your model and let that guide your efforts.
Thanks for commenting!

June 4, 2016 at 1:20 am

Thanks for the response Kris. Much appreciated.

May 12, 2016 at 4:56 am

Kris, that’s another great post ! 
I love that “Comprehensive model comparison” heatmap.
I like using the K-Nearest Neighbors algorithm for my “exploration” (i.e. exploring my feature space) because it is MUCH faster than most other types of algorithms.
I find that KNN usually gives reasonable results, thus it allows me to find interesting features much more quickly ….
Let’s say I have 1000 features;
choose(1000, 2) + choose(1000, 3) = 166,666,500
choose(1000, 2) + choose(1000, 3) + choose(1000, 4) = 41,583,791,250
That’s a lot of combinations !
With KNN I can explore ~100,000 random combinations over a 24 hour period. That’s perhaps a 100x speedup compared with say SVM.
Best, Nick

May 12, 2016 at 9:23 am

Thanks Nick!
I agree, the heatmaps reveal a lot of useful information quickly. Plus they brighten up the blog a little!
That’s a lot of features! An exhaustive search of even the 2- and 3-variable combinations at a rate of 100,000/day would require approximately 4.5 years to complete! But of course you are doing random search of the feature space which is obviously a lot more efficient. Still, without seeing your data set, I would speculate that out of the 1,000 features, there would be many highly correlated, and therefore redundant features which could be removed with little to no detriment of the final model. Again, without seeing your data, I suspect that an intelligent feature selection phase would assist greatly.
Your approach got me thinking and I started googling to see whether others were using also this approach. Turns out there is some activity in this area. Have you seen this paper: https://papers.nips.cc/paper/2848-nearest-neighbor-based-feature-selection-for-regression-and-its-application-to-neural-activity.pdf?
Have you experimented with such a feature-weighted version of k-NN?

May 12, 2016 at 9:48 am

Yes, a lot of them will be correlated to some degree and an intelligent feature selection process does help.
It’s easy to have 1000 features these days. Even starting with the standard technical indicators in TTR (~50 I think), you could add Super Smoother filters, take differences, apply PCA …. Then we have macro, fundamental and sentiment data….
.
A lot of people won’t like the data-driven approach – whatever works for them I suppose ….
Cheers Kris.

May 12, 2016 at 9:54 am

Indeed! There is certainly an air of dismissiveness to the data-driven approach in the quant finance community. Personally, I see it as just another weapon in the arsenal and a useful source of portfolio diversification in addition to the classical approaches.
Cheers

May 16, 2016 at 5:08 pm

Hi Kris,
Do you choose the best network structure for “live” trading after the optimization
nnetGrid <- expand.grid(.layer1 = c(2:3), .layer2 = c(0:2), .layer3=0, .hidden_dropout=c(0, 0.1, 0.2), .visible_dropout=c(0, 0.1, 0.2))
Because I understand that for each model you choose the result with best porfit with summery function?

May 17, 2016 at 12:13 pm

Hi Lev
That line simply specifies the grid of hyperparameters that are tested. The caret train function works as follows: for each hyperparameter combination, build a model on each iteration of the cross-validation procedure and calculate the average performance across each cross-validation iteration. Once this is done, the ‘best’ combination of hyperparameters is selected and a model fitted on the entire training data set. The specific case of time series cross validation is very applicable to trading since it mimics the process we would go through in order to trade the markets using these methods, with one important caveat: if at any point in time we picked the ‘best’ model based on this hyperparameter tuning process for live trading, we introduce selection bias and therefore the cross-validated performance is an over-estimate of actual performance. An unbiased estimate would only be possible with another out of sample validation data set. White’s Reality Check is another method of dealing with this.
In practice, I find that a further out of sample validation set is generally a more accurate predictor of future performance, but the utility of this is milted by the finite nature of our data set. I am happy enough to go live with a model selected in this way, but I temper my expectations of its future performance using White’s Reality Check.

May 16, 2016 at 6:16 pm

[…] Machine learning for financial prediction: experimentation with Aronson’s latest work – part… […]

June 11, 2016 at 5:03 pm

Interesting post – many thanks. Re your second point under “Next Steps” were you thinking along the lines of a variant of Zorro’s “Equity Curve Trading”, only rather than turning models on/off adjusting their relative position sizes (though some might be reduced to zero)?
Was also wondering if you had considered using equity curve plus other performance stats (e.g. rolling Sharpe/Sortino/K-Ratio) as predictors for a standalone overlay model that would automatically control individual model position size (weighting)?

June 12, 2016 at 5:20 pm

Hi Andy. I actually had in mind something like your second point: using a rolling performance measure as a means of controlling individual model weighting. However there are of course many approaches and I don’t favour one over the other. What I am really trying to achieve is to reduce the selection bias in my system by including as many models as possible. The downside of my approach is that there is always a lag in calculating the “optimal” portfolio weights.

June 13, 2016 at 5:52 pm

Thanks. A long while back I had a similar lag issue with calculating the weights of a portfolio of non-ML models. One technique that seemed to work (occasionally quite well) and minimised the lag was to use a Kalman filter on the equity curves and calculate various difference measures based on that. I tried using all Kalman values, the filtered values, the smooths and the predictions. As I recall, the predictions worked best, which I didn’t expect.

July 22, 2016 at 3:37 am

Hi – thanks for the fascinating and informative post!
Did you try holding some data out and seeing how one of your deep neural nets performed on it? When I follow along your code and attempt doing this, the predictions on out of sample data have very little variation, suggesting an input scaling problem or that the network has been overfit…
Thanks!

August 2, 2016 at 6:57 pm

Hey Jim
Yes I did try holding out some data and testing the performance of the various neural nets. My out of sample predictions do show a sensible amount of variation. I get a greater variation between the neural nets trained on smaller training windows and those trained on larger windows, as one would expect. Further, while directional out of sample predictions tend to converge as the training window increases, I still see variation in the absolute out of sample predictions.
I doubt the problem you are having is related to input scaling if you are using the same data that I used, since the inputs are all scaled to the range -1 to +1 (you can verify this with the R command “summary(eu)”. Of course, if you are using your own data, you would need to ensure that you pre-processed your input features appropriately.
Thanks for checking out my blog!

December 22, 2016 at 4:48 am

Hi Kris,
Great posts! Many thanks for sharing.
Have you ever put real money to trade using your machine learning systems? Some people say these systems always mysteriously failed, while others say GS, JP Morgan and DE Shaw etc all use machine learning system to trade and made big bucks, what’s your comment on this?

December 22, 2016 at 10:55 pm

Hi Jeff
No problem, glad you found it useful. Yes, I am currently allocating to my ML strategies. I don’t think there is any great mystery about trading systems failing – it happens all the time. I have little idea what those guys you mentioned do to make money, but if they aren’t using machine learning, they are foregoing a very useful tool.

December 28, 2016 at 4:30 am

Hi Kris,
Thank you so much for your comment. What do your average profit factors look like for your machine learning trading systems ? Which factor you think leads to a successful trading system, indicators or algorithms, if you’re asked to name only one?
Happy Holidays
Jeff

January 3, 2017 at 11:31 pm

Hi Jeff
If I had to pick one, I’d pick execution as the most important determinant of algo trading success. That’s probably a very boring response, but one that I think is justified!
Happy holidays to you too.

February 15, 2017 at 12:14 am

Kris,
Thanks so much for this really insightful tutorial, I think this is one of the best examples of nnet application to risk and return data in the public domain.
I have a basic question on the absretSummary function you’re using to assess the learner’s performance. Are you using the observed risk-adjusted returns in the line rather than geometric or logn scaled returns?
trades <- positions*data[, "obs"]
Thanks & Regards,

February 15, 2017 at 1:43 am

Matt, thanks for the kind words, great to hear that the article was useful.
You’re correct – I’m using the observed risk adjusted returns in my objective function: the last close minus the prior close divided by the 100-period average true range.
Cheers

March 23, 2017 at 1:33 pm

… [Trackback]
[…] Read More Infos here: robotwealth.com/machine-learning-for-financial-prediction-experimentation-with-david-aronsons-latest-work-part-2/ […]

May 3, 2017 at 2:16 am

Someone pointed out that there is no sense to do a variable analysis because with different data-window or different time at all, the variables would be different.
I see that if you choose a specific time ( as the years of the strategy ) and data-window ( as the period to generate an indicator ) everything will be different. Then a SMA cross over strategy will be winning using 20-period in 2009 and 15-period over 2015. The fact that the period changes does not mean that the analysis is wrong, as the fact that the variable importance may change does not mean that the analysis is wrong.
I see two solutions for the variable importance change depending on data-window etc:
1- We use different data-windows and years to do the analysis and we pick up only the variables that remain as importance in all the different scenarios. Then the discussion is closed because the variables choosen are the best in each scenario.
2- We perform a WFO where the variables used are changed to train the model as the parameters of the model are tuned in every WFO cycle. In that way “variable tunning” should be added to the analysis meaning that if we choose that the model is retrainned the first trading day of each month, then the variable analysis should be as well done and be part of that training. does it make sense?

May 3, 2017 at 10:04 pm

For sure, you raise really good points. Both your solutions are sensible; however in my experience you’ll have more success with the second one. Including the variable selection step in the model re-training process is not uncommon in machine learning applications to time series data. Another option is to create models with the best subset of variables over different lookback periods and ensemble their predictions.

August 8, 2017 at 8:53 pm

Great content on your site!
I’m trying to get my head around stacked autoencoders and dropout.
Quora: “Though the fundamental principle is same, making sure that large no of the parameters do not over-fit the data. But on a closer look they work differently. While denoising work on  input layer only, dropout work on all layers (but output).  So I guess when u have a deep network and you also want the hidden units of some higher layer to avoid over-fitting you might want to choose dropout over denoising. For shallow models both should work the same.”
The following quote is from the following article: https://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf
The idea of adding noise to the states of units has previously been used in the context of Denoising Autoencoders (DAEs) by Vincent et al. (2008, 2010) where noise is added to the input units of an autoencoder and the network is trained to reconstruct the noise-free input. Our work extends this idea by showing that dropout can be eﬀectively applied in the hidden layers as well and that it can be interpreted as a form of model averaging
Seems like autoencoders and dropout are two ways of achieving the same thing, although dropout can be used on more layers. Any thoughts on this would be most welcome!

August 25, 2017 at 10:31 am

I guess you could say that, although I’m not sure about the mathematical equivalence. In my experience, dropout has been the most effective way to control overfitting in my applications. As I understand denoising autoencoders, we apply noise only on the input layer, while of course dropout extends through the network. I was heavily into autoencoders a few years ago, but have moved away from them in the deep learning research I’m currently doing.

September 2, 2017 at 11:30 pm

Hi Kris and thank you so much, all your work are fantastic!
I have a question
why  modellist.nnet[[j]]$pred$pred value are different from predict( modellist.nnet[[j]]) ?

Thanks & Regards

December 5, 2017 at 4:30 pm

Hi Bob, you’re most welcome. It’s admittedly been a while since I used caret, but from memory the caret model object includes predictions from the training/cross-validation procedure whereas predict is used to present new observations to your model. Check out the caret documentation, which is a great resource in general, for more info.

December 1, 2017 at 7:20 am

Hi, I’m trying to execute the Rolling Forecast Origin Framework but it is giving to me an error (well, it is giving lots of errors, but I guess that the other ones are because of this first one):

December 5, 2017 at 4:37 pm

Yep, I’ve seen this one too. It is a recent phenomenon arising due to some changes to the way caret interfaces with the parallel backend. I know that Max (caret’s author) is aware of the problem and has written a fix. At the time of writing, this fix existed in the dev version on GitHub, but it hadn’t yet been pushed to CRAN. My advice: try updating to the latest version of caret, and if the problem persists, then you’ll need the dev version from GitHub. In that case, first, do install.packages('devtools') then install from GitHub via devtools::install_github('topepo/caret/pkg/caret')

FREE VIDEO COURSE

MASTER ZORRO IN 90 MINUTES:

CRACK ALGO TRADING'S SECRET WEAPON

• Cut hours off connecting to brokers and FXCM
• Run accurate and reliable backtests with less code
• Manage, clean and convert most types of data
• Follow along: code and optimize a simple strategy