text:
On why a few points suffice to describe spatiotemporal large-scale brain dynamics

# On why a few points suffice to describe spatiotemporal large-scale brain dynamics

Ignacio Cifre    Mahdi Zarepour    Silvina G Horovitz    Sergio Cannas    Dante R Chialvo Facultat de Psicologia, Ciències de l’educació i de l’Esport, Blanquerna, Universitat Ramon Llull, Barcelona, Spain Center for Complex Systems Brain Sciences (CEMSC), Universidad Nacional de San Martín, 25 de Mayo 1169, San Martín, (1650), Buenos Aires, Argentina. National Institute of Neurological Disorders and Stroke, National Institutes of Health, Bethesda, MD, USA Instituto de Física Enrique Gaviola (IFEG), Facultad de Matemática, Astronomía y Física, Universidad Nacional de Córdoba, Ciudad Universitaria, (5000), Córdoba, Argentina Consejo Nacional de Investigaciones Científicas y Tecnológicas (CONICET), Godoy Cruz 2290, Buenos Aires, Argentina
July 13, 2019
###### Abstract

An heuristic signal processing scheme recently introduced shows how brain signals can be efficiently represented by a sparse spatiotemporal point process. The approach has been validated already for different relevant conditions demonstrating that preserves and compress a surprisingly large fraction of the signal information. In this paper the conditions for such compression to succeed are investigated as well as the underlying reasons for such good performance. The results show that the key lies in the correlation properties of the time series under consideration. It is found that signals with long range correlations are particularly suitable for this type of compression, where inflection points contain most of the information. Since this type of correlation is ubiquitous in signals trough out nature including music, weather patterns, biological signals, etc., we expect that this type of approach to be an useful tool for their analysis.

###### pacs:

In the analysis of complex spatiotemporal patterns, such as large scale brain dynamics, an important challenge is the adequate coarse graining of the data. In the case of brain imaging the dataset is composed of several thousand time series, of the so called BOLD (“blood oxygenated level dependent”) signal, covering the entire brain. The usual question in this analysis revolves around the detection of burst of correlated activity across certain regions, which requires extensive computations, in part due to the usually humongous size of the data sets.

Recently it was uncoveredT1 (); T2 (); T3 (); caballero (); allan () that these type of problems can be efficiently analyzed using only the timings of the peak amplitude signal events, i.e., a point process (PP). Subsequent work using similar approachesli (); liu1 (); liu2 (); chen (); jiang (); Amico (); Wu () further confirmed that the method entails a large compression of the original signals. Overall these findings not only suggest a way to speed up computations, but most importantly highlight the need to clarify which aspects or features of the brain imaging signals contain the most relevant information.

The present work is dedicated to clarify the reasons underlying the effectiveness of this approach. The results show that the key lies in the correlation properties of the time series under consideration. In synthesis, it is found that signals with long range correlations are particularly suitable for this type of compression, where inflection points contains most of the information. The results applied as well to other signals from any origin as long as their correlation features are similar.

Figure 1 summarizes the basic process that has been used inT1 (); T2 (); T3 () to define the point process in brain signals. The data consists in time series representing the activity of one of many thousands small brain regions, recorded from the brain using functional magnetic resonance imaging (fMRI). This imaging technique measures in each small region a “blood oxygenated level dependent” signal (i.e., “BOLD”), that is an estimation of the blood’ saturation of oxygen, which itself is proportional to the local neuronal activity. As shown in the figure, time points are selected at the upward threshold (here at unity) crossings of the signal (filled circles). The point process can be constructed also by selecting the local peaks of the BOLD time series. The temporal co-occurrence of the points defines the co-activation matrix (bottom graphs) which can be further averaged to estimate the correlation matrix of the system under study.

It has been established already, in different circumstancesT1 (); T2 (); T3 (), that the co-activation matrix obtained with the PP methods is very similar to the correlation matrix computed from the full (i.e., continuous) BOLD signal. Since this implies a large compression, the question is why a few points are enough to compute results similar to those obtained with the full signal. Figure 2 shows an example constructed from BOLD time series from an experiment in which the subject is resting T3 (). The results demonstrate that as few as 4 points are already sufficient to define clusters of co-activation, as demonstrated previously in T1 (); T2 (); T3 (). In addition, the results here show how de-activations (i.e., blueish colors) are also evidenced by the PP approach.

A simple visual inspection of the BOLD traces reveals that the type of signals we are dealing with are temporally correlated. This is very well known, the neuronal activity is temporally and spatially correlated, and furthermore the activity is convoluted by the hemodynamic transfer function which in itself introduce additional temporal correlations. Therefore, for any time series with that properties, it seems natural to think that the most informative points are those in which its derivative changes sign. The rest of the points are redundant, since they can be predicted, up to a degree, by a linear estimator.

This is illustrated in Figure 3 using as an example two minutes of BOLD recording (normalized by its standard deviation (S.D)). After setting a threshold the inflection points larger than a given value are identified. These points constitutes the marked point process in question. Now we ask how much of the raw signal is left out if these points are used to extrapolate a piece-wise linear time series. To answer that we analyze BOLD time series from the brain of a subject during an experiment in which fMRI data is collected at restT2 (). We proceed to compute the linear correlation between the two time series, the raw and the piece-wise linear one. In panels B and C are shown the results for different values of threshold (in units of S.D.) as well as for the correlation of the time series, estimated by the value of the first autocorrelation coefficient . Panel D shows that as the BOLD signal’ autocorrelation increases the similarity between the piece-wise linear and the raw signals increases, evaluated in two ways: by the error and the linear correlation between both time series. As expected, the raising of the threshold from zero (i.e, less information from the signal is considered) it is followed by a monotonic increase of the and a decrease of the values (see Panel E).

According with the present hypothesis, the functional dependences shown by the BOLD signals in Panel D and E shall be replicated by using synthetic signals with similar autocorrelation properties. For that we generate artificial time series with autocorrelation values identical to those of the BOLD signals using the routine _alpha_gaussian.m }rom MATLAB. Panels B and C show that the dependence with the threshold and exhibit very similar behavior. The results show that the key lies in the correlation properties of the time series under consideration. In synthesis, it is found that signals with long range correlations are particularly suitable for this type of compression, where inflection points contains most of the information. The results shall apply as well to other signals from any origin as long as their autocorrelation features are similar.

In summary, the success and the merits of the PP approach to represent spatiotemporal dynamics are related to a very trivial fact: in the case of autocorrelated signals the only informative points are those with zero derivative (inflection points); remaining ones are more or less straight lines which can be in principle, and for certain applications, ignored. Applications of these ideas to a diversity of fields should be expected.

## References

• (1) Tagliazucchi E, Balenzuela P, Fraiman D, Montoya P, Chialvo DR (2011). Spontaneous BOLD event triggered averages for estimating functional connectivity at resting state. Neurosci. Lett., 488(2), 158–163.
• (2) Tagliazucchi E, Balenzuela P, Fraiman D, Chialvo DR. (2012). Criticality in large-scale brain fMRI dynamics unveiled by a novel point process analysis. Front. Physiol., 3:15 https://doi.org/10.3389/fphys.2012.00015
• (3) Tagliazucchi E, Siniatchkin M, Helmut Laufs H, Chialvo DR.(2016). The voxel-wise functional connectome can be efficiently derived from co-activations in a sparse spatio-temporal point-process. Front. Neurosci. /doi.org/10.3389/fnins.2016.00381
• (4) Petridou N, Gaudes CC, Dryden IL, Francis ST, Gowland PA. (2013). Periods of rest in fMRI contain individual spontaneous events which are related to slowly fluctuating spontaneous activity. Hum. Brain. Mapp., 34(6), 1319–1329.
• (5) Allan TW, Francis ST, Caballero-Gaudes C, Morris PG, Liddle EB, Liddle PF, Brookes MJ, & Gowland PA (2015). Functional connectivity in MRI is driven by spontaneous BOLD events. PloS ONE, 10 (4).
• (6) Li W, Li Y, Hu C, Chen X, Dai H. (2014). Point process analysis in brain networks of patients with diabetes. Neurocomputing, 145, 182–189.
• (7) Liu X, Duyn JH. (2013). Time-varying functional network information extracted from brief instances of spontaneous brain activity. Proc. Natl. Acad. Sci. USA, 110(11), 4392–4397.
• (8) Liu X, Chang C, Duyn JH. (2013). Decomposition of spontaneous brain activity into distinct fMRI co-activation patterns. Front. Sys. Neurosci., 7:101
• (9) Chen JE, Chang C, Greicius MD, Glover GH. (2015). Introducing co-activation pattern metrics to quantify spontaneous brain network dynamics. NeuroImage doi:10.1016/j.neuroimage.2015.01.057.
• (10) Jiang X, Lv J, Zhu D, Zhang T, Hu X, Guo L, Liu T. (2014). Integrating group-wise functional brain activities via point processes. Biomedical Imaging (ISBI), 2014 IEEE 11th International Symposium on (pp. 669–672). IEEE.
• (11) Amico E., Gomez F., Di Perri C., Vanhaudenhuyse A., Lesenfants D., Boveroux P., et al. . (2014). Posterior cingulate cortex-related co-activation patterns: a resting state fMRI study in propofol-induced loss of consciousness. PLoS ONE 9:e100012. 10.1371/journal.pone.0100012
• (12) Wu G. R., Liao W., Stramaglia S., Ding J. R., Chen H., Marinazzo D. (2013). A blind deconvolution approach to recover effective connectivity brain networks from resting state fMRI data. Med. Image Anal. 17, 365–374. 10.1016/j.media.2013.01.003
• (13) Cordes D, Haughton V, Carew J. D., Arfanakis K., Maravilla K. (2002). Hierarchical clustering to measure connectivity in fMRI resting-state data. Magn. Reson. Imaging 20, 305-317. 10.1016/S0730-725X(02)00503-9
You are adding the first comment!
How to quickly get a good reply:
• Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
• Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
• Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters


text:
The UvA-LINKER will give you a range of other options to find the full text of a publication (including a direct link to the full-text if it is located on another database on the internet).
De UvA-LINKER biedt mogelijkheden om een publicatie elders te vinden (inclusief een directe link naar de publicatie online als deze beschikbaar is in een database op het internet).

# Search results

Query: faculty: "FNWI" and publication year: "2007"

 Authors G. Derks, A. Doelman, S.A. van Gils, H. Susanto Title Stability Analysis of pi-Kinks in a 0-pi Josephson Junction Journal SIAM Journal on Applied Dynamical Systems Volume 6 Year 2007 Issue 1 Pages 99-141 ISSN 15360040 Faculty Faculty of Science Institute/dept. FNWI: Korteweg-de Vries Institute for Mathematics (KdVI) Abstract Abstract We consider a spatially nonautonomous discrete sine-Gordon equation with constant forcing and its continuum limit(s) to model a 0-pi Josephson junction with an applied bias current. The continuum limits correspond to the strong coupling limit of the discrete system. The nonautonomous character is due to the presence of a discontinuity point, namely, a jump of pi in the sine- Gordon phase. The continuum model admits static solitary waves which are called pi-kinks and are attached to the discontinuity point. For small forcing, there are three types of pi-kinks. We show that one of the kinks is stable and the others are unstable. There is a critical value of the forcing beyond which all static pi-kinks fail to exist. Up to this value, the (in)stability of the pi-kinks can be established analytically in the strong coupling limits. Applying a forcing above the critical value causes the nucleation of $2\pi$-kinks and -antikinks. Besides a pi-kink, the unforced system also admits a static $3\pi$-kink. This state is unstable in the continuum models. By combining analytical and numerical methods in the discrete model, it is shown that the stable pi-kink remains stable and that the unstable pi-kinks cannot be stabilized by decreasing the coupling. The $3\pi$-kink does become stable in the discrete model when the coupling is sufficiently weak. Keywords: 0-pi Josephson junction; 0-pi sine-Gordon equation; semifluxon; pi-kink Document type Article Document finder

format: plain


text:
A cubical block of side 7 cm is surmounted by a hemisphere.
Question:

A cubical block of side 7 cm is surmounted by a hemisphere. What is the greatest diameter the hemisphere can have? Find the surface area of the solid.

Solution:

On 7 cm × 7 cm base of the cubical block, we can mount hemisphere having greatest diameter equal to 7 cm.

Here, the radius of the hemisphere = 3.5 cm.

Now, the surface area of the solid made in figure.

= The surface area of the cube + The curved surface area of the hemisphere – The area of the base of the hemisphere.

$=\left\{6 \times(7)^{2}+2 \pi \times(3.5)^{2}-\pi \times(3.5)^{2}\right\} \mathrm{cm}^{2}$

( $\because$ the part of the top of the cubical part which is covered by the hemisphere is not visible outside)

$=\left\{6 \times 49+\frac{22}{7} \times \frac{35}{10} \times \frac{35}{10}\right\} \mathrm{cm}^{2}$

$=\left\{\mathbf{2 9 4}+\mathbf{1 1} \times \frac{\mathbf{3 5}}{\mathbf{1 0}}\right\} \mathrm{cm}^{2}=332.5 \mathrm{~cm}^{2}$


text:
# Pervasive Parallelism (PPar)

Response to 2014/15 survey feedback.

I have taken note of the feedback provided through the Student Online Survey. I have the following responses/clarifications:

• Thanks for this feedback, and also for that gathered during my own survey at the end of the course.


text:
MathSciNet bibliographic data MR361759 46E35 Rivière, N. M.; Sagher, Y. Interpolation between $L\sp{\infty }$$L\sp{\infty }$ and $H\sp{1}$$H\sp{1}$, the real method. J. Functional Analysis 14 (1973), 401–409. Links to the journal or article are not yet available

For users without a MathSciNet license , Relay Station allows linking from MR numbers in online mathematical literature directly to electronic journals and original articles. Subscribers receive the added value of full MathSciNet reviews.

Username/Password Subscribers access MathSciNet here

AMS Home Page

 American Mathematical Society 201 Charles Street Providence, RI 02904-6248 USA
© Copyright 2016, American Mathematical Society
Privacy Statement


text:
Search
Question: MA plot DiffBind in ggplot2
0
13 months ago by
reubenmcgregor880 wrote:

Hi again, sorry lots of questions at the moment,

I would like to replicate the following function from diffBind

> dba.plotMA(tamoxifen, bXY=TRUE)

However I don't know how to retrieve this data, in Deseq2 I believe it would be the baseMean column. Or is the "Conc" column output by "dab.report" the values used for the plot ma?

I know one option would be:

> DBA$contrasts[[n]]$DESeq2\$DEdata, bReduceObjects=F

Where I assume DBA is the DBA object, but what is "n"??

Also, on a separate note, is the "Fold" column output by "dba.report" as default the log2 fold values?

Thanks

modified 13 months ago by Rory Stark2.6k • written 13 months ago by reubenmcgregor880
2
13 months ago by
Rory Stark2.6k
CRUK, Cambridge, UK
Rory Stark2.6k wrote:

Yes, the MA plot data is the same as reported by dba.report(). The X axis is the "Conc" column, and the Y axis is the "Fold" column. These are all reported in log2 form, so that Fold is simply Conc1-Conc2.

I'm not sure what you mean in the bReduceObject=F line of code?

-Rory

Thanks Rory,

Related question, in the dba.report() output, what are the column headers named after the conditions (i.e. Conc_condition1, Conc_condition2). I assume these are what is used in the dba.plotMA the bXY=TRUE and are (as you explained above) log2 transformed values, but form where are these values derived? I ask as I have some negative values from peaks which where identified as differentially expressed.

Am I interpreting that right if I say that they are very low expressing (below 1 in the non log2 values) peaks, but which where consistently (in all replicates) increased or decreased, hence being identified as significantly differentially expressed?

0
13 months ago by
Rory Stark2.6k
CRUK, Cambridge, UK
Rory Stark2.6k wrote:

This is an artifact of reporting log values of normalized data. After normalization, some of the values may be less than one and hence will have negative log2 scores. So you are correct that if the normalized read counts for a peak in the replicates of one condition are very small (negative log values), add the counts for the replicates of the other condition are consistently higher, the peak can be identified as being differentially bound (with low FDR).

-Rory


text:
# scipy.special.jv¶

scipy.special.jv(v, z) = <ufunc 'jv'>

Bessel function of the first kind of real order and complex argument.

Parameters
varray_like

Order (float).

zarray_like

Argument (float or complex).

Returns
Jndarray

Value of the Bessel function, $$J_v(z)$$.

jve

$$J_v$$ with leading exponential behavior stripped off.

spherical_jn

spherical Bessel functions.

Notes

For positive v values, the computation is carried out using the AMOS [1] zbesj routine, which exploits the connection to the modified Bessel function $$I_v$$,

\begin{align}\begin{aligned}J_v(z) = \exp(v\pi\imath/2) I_v(-\imath z)\qquad (\Im z > 0)\\J_v(z) = \exp(-v\pi\imath/2) I_v(\imath z)\qquad (\Im z < 0)\end{aligned}\end{align}

For negative v values the formula,

$J_{-v}(z) = J_v(z) \cos(\pi v) - Y_v(z) \sin(\pi v)$

is used, where $$Y_v(z)$$ is the Bessel function of the second kind, computed using the AMOS routine zbesy. Note that the second term is exactly zero for integer v; to improve accuracy the second term is explicitly omitted for v values such that v = floor(v).

Not to be confused with the spherical Bessel functions (see spherical_jn).

References

1

Donald E. Amos, “AMOS, A Portable Package for Bessel Functions of a Complex Argument and Nonnegative Order”, http://netlib.org/amos/

#### Previous topic

scipy.special.ellipeinc

#### Next topic

scipy.special.jve


text:
# E ^ x-y

Wolfram|Alpha brings expert-level knowledge and capabilities to the broadest possible range of people—spanning all professions and education levels.

I. De nition: Var(XjY) = E (X E[XjY]) 2. jY = E X. 2. E[XjY] 2. jY .

01.05.2008 Wolfram|Alpha brings expert-level knowledge and capabilities to the broadest possible range of people—spanning all professions and education levels. Get the answer to Integral of x*e^x with the Cymath math problem solver - a free math equation solver and math solving app for calculus and algebra. логическое произведение x∙y∙z∙… равно 1, т.е. выражение является истинным, только тогда, когда все сомножители равны 1 (а в остальных случаях равно 0) E X Y T, Worldwide. 7,137 likes. E X Y T is a global entertainment company founded in London, UK and specialising in touring, artist development, licensing and media. 3 + E 3(x,y) = 3y+ 6xy+6x2y− 9 2 y 3 + E 3(x,y) A second way to get the same result exploits the single variable Taylor expansions ex = 1+x+ 1 2!

## 30 ноя 2018 Ответ: y'=[-y(e^xy)-2xsin(x²+y²)] /[x(e^xy)+2ysin(x²+y²)] Пошаговое объяснение: Дана неявно заданная функцияe^xy-cos(x²+y²)

The expected values of the powers of X are called the moments of X; the moments about the mean of X are expected values of powers of X − E[X]. Compute answers using Wolfram's breakthrough technology & knowledgebase, relied on by millions of students & professionals. For math, science, nutrition, history e^x just means multiply e with itself x times.

### e x + e y = e (x + y) Dividing both sides by e (x + y), e (− y) + e (− x) = 1. Differentiating both sides with respect to x, (− y 1) (e (− y)) + (− 1) (e (− x)) = 0. y 1 (e (− y)) = − e (− x) Dividing both sides with e (− y), y 1 = − e (y − x) Hence Proved. Also note that this is my first time writing a math answer so I

выражение является истинным, только тогда, когда все сомножители равны 1 (а в остальных случаях равно 0) E X Y T, Worldwide. 7,137 likes. E X Y T is a global entertainment company founded in London, UK and specialising in touring, artist development, licensing and media. 3 + E 3(x,y) = 3y+ 6xy+6x2y− 9 2 y 3 + E 3(x,y) A second way to get the same result exploits the single variable Taylor expansions ex = 1+x+ 1 2! x2 + 1 3!

We have LECTURE 12 Conditional expectations • Readings: Section 4.3; • Given the value y of a r.v. Y: parts of Section 4.5 E[X | Y = y]= xpno!

- Slader. Какая производная у функции e^xy. Ответить. Голосование за лучший ответ. 1 / 2. F J, 10 лет назад. Мыслитель.

Focado em rappers americanos, divulguem a page. Facebook is showing information to help you better understand the purpose of a Page. Once again, we apply the inverse function ex to both sides. We could use the identity e 2 ln y = (e ln y)2 or we could handle the coeﬃcient of 2 as shown below. 2 ln y = ln(y + 1) + x ln y 2 = ln(y + 1) + x eln y 2 = eln(y+1) · ex y 2 = (y + 1) · e x y 2 − e x · y − e x = 0 E is the symbol representing the base of the natural logarithm Log.It is also known as Euler's number and can be input as \[ExponentialE].

y = k Proof. As we know, X and Y are independent if and only if fX;Y(x;y) = fX(x)fY(y) or, equiva-lently, fXjY(xjy)= fX(x). But then E(XjY =y)=åx xfXjY(xjy)=åx xfX(x)=E(X). 2 2. E[E(g(X)jY)]=E(g(X)) Proof. Set Z = g(X). Statement (i) of Theorem 1 applies to any two r.v.’s.

Hence, applying it to Z and Y we obtain E[E(ZjY)]= E(Z) which is the same Stack Exchange network consists of 176 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share … Stack Exchange network consists of 176 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share … The basic answer is yes, this is simply the multiplicative rule for indices. For a number $a$, the general rule is $a^x \cdot a^y = a^{x+y}$. Intuitively for non-negative integers we can identify these symbols as "[math]a[/ma In probability theory, the expected value of a random variable, denoted () or [], is a generalization of the weighted average, and is intuitively the arithmetic mean of a large number of independent realizations of .The expected value is also known as the expectation, mathematical expectation, mean, average, or first moment.Expected value is a key concept in economics, finance, and many other E(X + Y) = Sum(z P(X + Y = z)) where the sum extends over all possible values of z. Thus you are looking at all possible combinations of values of X and Y that add to z.

i bitcoin inr
krypto daňový formulář 8949
transakce blokování geneze ethereum
průvodce ověřením stavu id
seznam tvůrců trhu cboe
jak nakupovat cenu bitcoinu aud

### f(x, y) = e−xy. For the interior of the region, we find the critical points: fx = −ye− xy, fy = −xe−xy, so the only critical point is (0,0), and f(0,0) = 1.

I. De nition: Var(XjY) = E (X E[XjY]) 2. jY = E X. 2. E[XjY] 2. jY . I. Var(XjY) is a random variable that depends on Y. It is the variance of X in the conditional distribution for X given Y. Sal finds dy/dx for e^(xy²)=x-y using implicit differentiation.

## Y (t), then Z = X + Y has the moment generating function, M Z(t) = M X(t)M Y (t). 2. Find a variance of the random variables in Example 1. Finally, we can also deﬁne the conditional expectation, E(X | Y), and conditional variance, E[(X− µ X)2 | Y)], of a random variable X given another random variable Y. The expectation is over the

Note that the purple branch in the 1.

Deﬁnition: The correlation coeﬃcient (ì :) between X and Y, denoted by ρ xy, is deﬁned as: ρ В ответе напишите буквы x, y, z в том порядке, в котором идут соответствующие им столбцы (сначала – буква, соответствующая первому столбцу, затем – буква, соответствующая второму столбцу, и т. д.) Буквы в ответе пишите Compute answers using Wolfram's breakthrough technology & knowledgebase, relied on by millions of students & professionals. For math, science, nutrition, history LECTURE 12 Conditional expectations • Readings: Section 4.3; • Given the value y of a r.v. Y: parts of Section 4.5 E[X | Y = y]= xpno! X|Y (x y) (mean and variance only; transforms) x (integral in continuous case) Lecture outline • Stick example: stick of length! break at uniformly chosen point Y • Conditional expectation break again at uniformly chosen point X Get an answer for 'What is the double integral of:f(x,y)=e^(x+y) when R is the area bounded by y=x+1, y=x-1, y=1-x, y=-1-x? How to find R?' and find homework help for other Math questions at eNotes 15.07.2008 1) e0 =1, a0 =1 2) e x+y=exe y, a =axa 3) e−x = 1 ex, a −x = 1 ax 4) ex y =e xy, ax y =a 5) d dx e x=e , d dx eg(x) =g′(x)eg(x), d dx ax =(lna)a 6) R ex dx=ex +C, R eax dx= 1 a e ax +C ifa6=0 7) lim x→∞ ex =∞, lim x→−∞ ex =0 lim x→∞ ax =∞, lim x→−∞ ax =0ifa>1 lim x→∞ ax =0, lim x→−∞ ax =∞ if0


text:
# A constant function!

• July 6th 2009, 12:00 PM
NonCommAlg
A constant function!
True or False:

$x \ + \ \sin x \ + \ \sin(x \ + \ \sin x) \ + \ \sin(x \ + \ \sin x \ + \ \sin(x \ + \ \sin x)) \ + \ \cdots=\pi$ for all $0 < x < 2\pi.$
• July 6th 2009, 05:08 PM
halbard
Let $f(x)=x+\sin x$ for $0. Then $f'(x)=1+\cos x\geq0$, so $f(x)$ is one-one, mapping $I=(0,2\pi)$ to itself.

Define $f_0(x)=f(x)$, $f_n(x)=f(f_{n-1}(x))$. The given function is $G(x)=\lim_{n\to\infty}f_n(x)$, where the limit exists, and $G:I\to I$.

It follows that $f(G(x))=G(x)$, and so $G(x)+\sin G(x)=G(x)$.

Therefore $\sin G(x)=0$, which implies that $G(x)=\pi$ on $I$.

As for the limit, we know that $0<\frac{\sin x}x\leq 1$ for $-\pi, and so $0\leq 1-\frac{\sin(\pi-x)}{\pi-x}=1-\frac{\sin x}{\pi-x}<1$ for $x\in I$.

Given $\epsilon>0$ we may choose $k<1$ depending on $\epsilon$ such that $1-\frac{\sin x}{\pi-x} for all $x\in[\epsilon,2\pi-\epsilon]$.

It follows that $|\pi-x-\sin x| for all $x\in[\epsilon,2\pi-\epsilon]$.

This means that $|\pi-f(x)| in this interval. Also if $f$ maps $[\epsilon,2\pi-\epsilon]$ to itself, since $f(x)$ is closer to $\pi$ than $x$ is. Thus

$|\pi-f_n(x)|. It follows that $G(x)=\lim_{n\to\infty}f_n(x)=\pi$ for all $x$ in the interval $[\epsilon,2\pi-\epsilon]$.

Since $\epsilon$ is arbitrary (get this!) we have $G(x)=\pi$ for all $x\in I$.

If this works I will definitely LOL.


text:
# Invariant Distances and Metrics in Complex Analysis

###### Marek Jarnicki and Peter Pflug
Publisher:
Walter de Gruyter
Publication Date:
2013
Number of Pages:
861
Format:
Hardcover
Edition:
2
Series:
Expositions in Mathematics 9
Price:
224.00
ISBN:
9783110253863
Category:
Monograph
[Reviewed by
Steven G. Krantz
, on
11/26/2013
]

It is a hallowed and much cited result in the literature of several complex variables that the unit ball and the unit polydisk in $\mathbb{C}^n$ are not biholomorphically equivalent. This 1906 result of Henri Poincaré has had an enormous influence on the development of the subject for the past 100 years. Knowing that there is no Riemann mapping theorem in the several variable setting, we seek substitute results. We also search for tools that will help us understand the situation.

One of the major developments that has come to the rescue is the creation of various invariant metrics. Again it was Poincaré who first invented a conformally invariant metric (the so-called Poincaré metric) on the disc. In the 1920s and 1930s, Stefan Bergman gave a construction for a biholomorphically invariant metric that works on any bounded domain. Later on, Carathéodory and Kobayashi gave metric constructions that are particularly appealing because they are based on the classical proof of the Riemann mapping theorem in one complex variable.

There are a number of other invariant metrics — due to N. Sibony, K. T. Hahn, and others. But it is safe to say the the Bergman, Carathéodory, and Kobayashi metrics are the most prominent and most widely used. There has been a flood of results on these three metrics in the past fifty years. C. Fefferman’s Fields-Medal-winning work on the boundary behavior of biholomorphic mappings was based on a remarkable study of the Bergman metric, and that gave this whole enterprise a shot in the arm.

The book under review is an encyclopedic treatment of invariant metrics and their applications in one and several complex variables. It is an ambitious project, and a successful one. Certainly there are other books that treat invariant metrics to a greater or lesser extent (including some by this reviewer), but none that is nearly so comprehensive as the present one. Jarnicki and Pflug are prominent authorities in the field, and well placed to write a book like this one. Both authors have published extensively on invariant metrics, and both are quintessentially knowledgeable.

The book has many charming and useful features. The Bibliography alone — with more than 500 entries — is a valuable resource. The Table of Notation is unusually detailed and very helpful. Following the inspiration of Bourbaki’s famous “curvy road” sign, these authors label some passages in their book with a sign to indicate that this is an open problem that the reader should think about, and other passages with a sign to tell the reader to pick up the pencil and calculate.

The book is more than 800 pages long. It would not work well as a textbook, partly because it is so extensive and partly because it is so dense. It is also not the sort of book that one just sits down and reads straight through. But it is a wonderful resource, and is written in such a way that it is very natural for the reader to dip into various portions as the need arises.

The writing in the book is exceptionally clear, and the level of detail is usually just about right. The authors are quite skilled at providing context and motivation, and they give plenty of interesting examples and ancillary results. Each chapter ends with a brief list of (research) problems, and these will give students and faculty alike much food for thought.

Certainly the book of Jarnicki and Pflug will occupy a prominent spot on my bookshelf, and will soon show the well-worn characteristics of a tome that is both respected and admired.

Steven G. Krantz has taught at UCLA, Princeton University, Penn State, and Washington University in St. Louis. He was Chair of the latter department for five years. He has won the Chauvenet Prize, the Beckenbach Book Award, and the Kemper Prize. He is currently the editor of the Notices of the American Mathematical Society.


text:
# Re: Relation of cpu_*_costs?

 От: SZŰCS Gábor Re: Relation of cpu_*_costs? 7 июня 2004 г. , 17:18:32 0a0101c44cb3$89d3c2b0$0403a8c0@fejleszt4 (см: обсуждение, исходный текст) Relation of cpu_*_costs?  ("SZŰCS Gábor") pgsql-performance

Dear Tom,

Thanks for your response.

----- Original Message -----
From: "Tom Lane" <>
Sent: Monday, June 07, 2004 3:51 PM

> That's pretty hard to believe; particularly on modern machines, I'd
> think that moving it down would make more sense than moving it up.
> You're essentially asserting that the CPU time to process one tuple
> is almost half of the time needed to bring a page in from disk.

That is exactly what I had in mind. We found that 5x10krpm HW RAID 5 array
blazing fast, while we were really disappointed about CPU. E.g.
* tar'ing 600MB took seconds; gzip'ing it took minutes.
* initdb ran so fast that I didn't have time to hit Ctrl+C because
I forgot a switch ;)
* dumping the DB in or out was far faster than adddepend between 7.2 and 7.3
* iirc index scans returning ~26k rows of ~64k were faster than seq scan.
(most suspicious case of disk cache)

But whatever is the case with my hardware -- could you tell me something
(even a search keyword ;) ) about my theoretical question: i.e. relation of
cpu_*_costs?

> I suspect that your test cases were toy cases small enough to be
> fully cached and thus not incur any actual I/O ...

Dunno. The server has 1GB RAM; full DB is ~100MB; largest query was ~7k
which moved at least 2 tables of >200k rows and several smaller ones. If it
is a "toy case" for such hw, I humbly accept your opinion.

BTW its runtime improved from 53 to 48 sec -- all due to changing cpu tuple
cost. I ran the query at different costs, in fast succession:

run    cost    sec
#1    0.01    53
#2    0.4     50
#3    1.0     48
#4    1.0     48
#5    0.4     48
#6    0.01    53

For the second result, I'd say disk cache, yes-- but what about the last
result? It's all the same as the first one. Must have been some plan change
(I can send the exp-ana results if you wish)

G.
%----------------------- cut here -----------------------%
\end

В списке pgsql-performance по дате сообщения:

От: Josué Maldonado
Дата:
Сообщение: Re: Join slow on "large" tables
От: Josh Berkus
Дата:
Сообщение: Re: Join slow on "large" tables


text:
In programming languages, we find logical operators for and and or. In fact, Python uses the actual words and and or for these operators.

# Python via the reticulate package
x = True
y = False
x and y
#> False
x or y
#> True

In Javascript, we see && for and and || for or instead.

// Javascript via engine = "node" in knitr
let x = true;
let y = false;
console.log(x && y);
console.log(x || y);
// false
// true

In R, we have two versions of logical and (& and &&) and logical or (| and ||). What’s going on?

Documentation in help("Logic", package = "base") provides the following:

& and && indicate logical AND and | and || indicate logical OR. The shorter form performs elementwise comparisons in much the same way as arithmetic operators. The longer form evaluates left to right examining only the first element of each vector. Evaluation proceeds only until the result is determined. The longer form is appropriate for programming control-flow and typically preferred in if clauses.

Let’s unpack this paragraph

## The shorter operators are vectorized

The crucial difference is that the shorter versions (&, |) are vectorized. Given two vectors, they will apply logical and/or on pairs of elements from each vector. In the example below, ttff[1] is and-ed with tftf[1], ttff[2] is and-ed with tftf[2], and so on. This vectorization is a pretty important feature, for example, when we are comparing columns in a dataframe in order to filter rows.

# Returns something of length four
ttff <- c(TRUE,  TRUE, FALSE, FALSE)
tftf <- c(TRUE, FALSE,  TRUE, FALSE)
ttff & tftf
#> [1]  TRUE FALSE FALSE FALSE
ttff | tftf
#> [1]  TRUE  TRUE  TRUE FALSE

In contrast, && and || only work on scalars (length-one values). They return just one element. In this example, they look at ttff[1] and tftf[1].

# Returns something of length one
ttff && tftf
#> [1] TRUE
ttff || tftf
#> [1] TRUE

To help remember the distinction, think of the longer versions (&&, ||) as stricter forms of the logical operators. They don’t just care about truthiness or falsiness, but they also care about length. The extra and/or characters are there because the operators are extra, if you will.

Okay, that was the point of this post: to describe the difference between the short and long operators and introduce the intuition that the longer forms are stricter. The rest of this post will dig into some other oddities and notes about and and or.

## Short circuit evaluation

You may have noticed a strange or unclear detail in the documentation.

The longer form evaluates left to right examining only the first element of each vector. Evaluation proceeds only until the result is determined.

This part is describing the semantics of short-circuit evaluation. Here are two facts about and and or:

• x and y is false when either x or y is false. Therefore, if x is false, we don’t need to look at y at all.
• x or y is true when either x or y is true. Therefore, if x is true, we don’t need to look at y at all.

R will not evaluate the second operand for && and || if it can learn the answer from the first operand. Thus, short-circuit evaluation will ignore the stop() calls in the examples below.

FALSE && stop("this is an error")
#> [1] FALSE
TRUE || stop("this is an error")
#> [1] TRUE

# Short-circuiting doesn't apply. Need the second operand.
TRUE && stop("this is an error")
#> Error in eval(expr, envir, enclos): this is an error
FALSE || stop("this is an error")
#> Error in eval(expr, envir, enclos): this is an error

### The NULL-or-default pattern

Logically, the short-circuit evaluation for || is equivalent to a particular kind of if statement:

if_x_then_x_else_y <- function(x, y) {
if (x) x else y
}
if_x_then_x_else_y(TRUE, FALSE)
#> [1] TRUE
# short-circuited
if_x_then_x_else_y(TRUE, stop("this is an error"))
#> [1] TRUE

In languages that treat undefined values as falsy and defined values as truthy, this if (x) x else y behavior is sometimes used as an idiom to set a default, backup value for a variable. In the Javascript code below, the undefined variable name is treated as falsy, the string "I don't know your name" is treated as truthy, so the or returns the second string. In other words, “the first truthy value is returned”.

let name;
let fallback = name || "I don't know your name!";
console.log(name);
console.log(fallback);
// undefined
// I don't know your name!

(This pattern, incidentally, appears to have earned its own operator in Javascript with the “nullish coalescing operator” ??.)

Why do I mention this programming idiom from Javascript? Because setting a default for missing values is pretty useful, and this syntax is pretty nice. The tidyverse provides a null coalescing operator, inspired by Ruby’s || operator.

library(purrr)
1 %||% 2
#> [1] 1
NULL %||% 2
#> [1] 2

# Not exactly or-like. It just cares about NULL-ness.
FALSE %||% 2
#> [1] FALSE

# See the source code
%||%
#> function (x, y)
#> {
#>     if (is_null(x))
#>         y
#>     else x
#> }
#> <bytecode: 0x000000001f07d660>
#> <environment: namespace:rlang>

## if() statements want the stricter operators

Recall the following from the documentation:

The longer form is appropriate for programming control-flow and typically preferred in if clauses.

if() statements are not vectorized. (See ifelse() instead.) if() statements complain when they see a vector:

if (ttff | tftf) {
"We are going to get a warning."
}
#> Warning in if (ttff | tftf) {: the condition has length > 1 and only the first
#> element will be used
#> [1] "We are going to get a warning."

The idea behind the documentation is that because & and | return vectors and because if() only likes scalars, we should not use the shorter forms in if() statements. They provide the wrong output for if(). But that does not mean the following code with the stricter or operator is correct.

if (ttff || tftf) {
"No warning but this code is not right."
}
#> [1] "No warning but this code is not right."

Although the code here does not raise any warnings, it reduces all of the information in ttff and tftf into just ttff[1] and tftf[1]. Those values likely will not be appropriate for the programming task at hand, so we should provide the scalars ourselves.

### all() and any() can apply and and or down a vector

Because I am talking about and and or and about creating logical scalars, I want to advertise two particular functions that can reduce a logical vector into a scalar. all() is TRUE when all of the elements are TRUE. For a vector, it would be like replacing the commas in c(TRUE, TRUE, FALSE) with &s. any() provides the analogous down-vector behavior for |. any() is TRUE when any of the elements in the vector are TRUE (and not NA—more on that later).

all(c(TRUE, TRUE, TRUE))
#> [1] TRUE
c(TRUE & TRUE & TRUE)
#> [1] TRUE

all(c(TRUE, FALSE, TRUE))
#> [1] FALSE
c(TRUE & FALSE & TRUE)
#> [1] FALSE

c(TRUE | FALSE | TRUE)
#> [1] TRUE

# The input can be scalars or vectors
all(TRUE, TRUE, TRUE, c(TRUE, FALSE))
#> [1] FALSE
any(FALSE, FALSE, FALSE, c(FALSE, TRUE))
#> [1] TRUE

# These appear not to short circuit
any(TRUE, stop("this is an error"))
#> Error in eval(expr, envir, enclos): this is an error
all(FALSE, stop("this is an error"))
#> Error in eval(expr, envir, enclos): this is an error

### We can make the strict operators even stricter

Recall the following unsettling example where just ttff[1] and tftf[1] are considered in the if() statement.

if (ttff || tftf) {
"No warning but this code is not right."
}
#> [1] "No warning but this code is not right."

It would be nice to rule out this behavior outright and make this behavior illegal. In fact, we can make our code stricter by setting the system environment variable _R_CHECK_LENGTH_1_LOGIC2_. Once set, using the strict forms on inputs longer than 1 will throw an error.

Sys.setenv("_R_CHECK_LENGTH_1_LOGIC2_" = "TRUE")
ttff && tftf
#> Error in ttff && tftf: 'length(x) = 4 > 1' in coercion to 'logical(1)'
ttff || tftf
#> Error in ttff || tftf: 'length(x) = 4 > 1' in coercion to 'logical(1)'

# Default behavior
Sys.unsetenv("_R_CHECK_LENGTH_1_LOGIC2_")
ttff && tftf
#> [1] TRUE
ttff || tftf
#> [1] TRUE

A related environment variable is _R_CHECK_LENGTH_1_CONDITION_ that turn vectors inside of if() into errors instead of warnings.

Sys.setenv("_R_CHECK_LENGTH_1_CONDITION_" = "TRUE")
if (ttff | tftf) {
"This code will not even be seen."
}
#> Error in if (ttff | tftf) {: the condition has length > 1

# Default behavior
Sys.unsetenv("_R_CHECK_LENGTH_1_CONDITION_")
if (ttff | tftf) {
"We are going to get a warning."
}
#> Warning in if (ttff | tftf) {: the condition has length > 1 and only the first
#> element will be used
#> [1] "We are going to get a warning."

To make project code more robust, one might consider setting these inside of a .Renviron file or using them with dotenv. (But if I remember correctly, these checks apply to package code so you get errors for legal-but-dodgy R code in other people’s packages.)

## NAs infect other values

All of the above examples conveniently avoided NAs. These are missing values that infect other logical values, turning them into NAs. For this section, I want to briefly highlight some behaviors of NAs and some functions that can help us work around them.

For and, an NA with any non-FALSE value is NA. For or, an NA with any non-TRUE value is NA. That’s a funny sentence, but it reflects the case where we can infer the answer without seeing the NA value. TRUE | NA (and NA | TRUE) returns TRUE because it would return TRUE if the NA was actually a TRUE or FALSE. The same holds for FALSE & NA (and NA & FALSE) returning FALSE. If we un-missing-ed the NA into TRUE or FALSE, the statement would still be FALSE.

tfnn <- c(TRUE, FALSE, NA, NA)
nntf <- c(NA, NA, TRUE, FALSE)
tfnn & nntf
#> [1]    NA FALSE    NA FALSE
tfnn | nntf
#> [1] TRUE   NA TRUE   NA

# Infecting in all() and any()
TRUE && NA
#> [1] NA
FALSE || NA
#> [1] NA
all(TRUE, TRUE, NA)
#> [1] NA
any(FALSE, FALSE, NA)
#> [1] NA

# These return FALSE and TRUE because they would return TRUE and FALSE
# regardless of whether the NA was a TRUE or a FALSE.
FALSE && NA
#> [1] FALSE
TRUE || NA
#> [1] TRUE
all(NA, FALSE, FALSE)
#> [1] FALSE
any(TRUE, FALSE, NA)
#> [1] TRUE

This uhh complicates things, so how do I check if what I have is TRUE or FALSE? isTRUE() and isFALSE() provide direct tests of whether the input is the scalar TRUE or the scalar FALSE.

c(isTRUE(TRUE), isTRUE(FALSE), isTRUE(NA))
#> [1]  TRUE FALSE FALSE
c(isFALSE(TRUE), isFALSE(FALSE), isFALSE(NA))
#> [1] FALSE  TRUE FALSE

The documentation notes that

if(isTRUE(cond)) may be preferable to if(cond) because of NAs

so isTRUE() is something we run into packaged code.

isTRUE() and isFALSE() are not vectorized, but we can check elements in a vector are TRUE and only TRUE in a few different ways.

# Make a new function
Vectorize(FUN = isTRUE)(tfnn)
#> [1]  TRUE FALSE FALSE FALSE

# Apply the function on a vector
vapply(X = tfnn, FUN = isTRUE, FUN.VALUE = logical(1))
#> [1]  TRUE FALSE FALSE FALSE

# Use table-lookup or set operations
tfnn %in% TRUE
#> [1]  TRUE FALSE FALSE FALSE
is.element(el = tfnn, set = TRUE)
#> [1]  TRUE FALSE FALSE FALSE

I think that’s just about every useful thing I want to say about and and or in R. But just remember, && and || are longer operators because they are stricter.

Last knitted on 2021-07-02. Source code on GitHub.1

1. sessioninfo::session_info()
#> - Session info ---------------------------------------------------------------
#>  setting  value
#>  version  R version 4.1.0 (2021-05-18)
#>  os       Windows 10 x64
#>  system   x86_64, mingw32
#>  ui       RTerm
#>  language (EN)
#>  collate  English_United States.1252
#>  ctype    English_United States.1252
#>  tz       America/Chicago
#>  date     2021-07-02
#>
#> - Packages -------------------------------------------------------------------
#>  package     * version date       lib source
#>  cli           3.0.0   2021-06-30 [1] CRAN (R 4.1.0)
#>  evaluate      0.14    2019-05-28 [1] CRAN (R 4.1.0)
#>  git2r         0.28.0  2021-01-10 [1] CRAN (R 4.1.0)
#>  here          1.0.1   2020-12-13 [1] CRAN (R 4.1.0)
#>  jsonlite      1.7.2   2020-12-09 [1] CRAN (R 4.1.0)
#>  knitr       * 1.33    2021-04-24 [1] CRAN (R 4.1.0)
#>  lattice       0.20-44 2021-05-02 [1] CRAN (R 4.1.0)
#>  magrittr      2.0.1   2020-11-17 [1] CRAN (R 4.1.0)
#>  Matrix        1.3-4   2021-06-01 [1] CRAN (R 4.1.0)
#>  png           0.1-7   2013-12-03 [1] CRAN (R 4.1.0)
#>  purrr       * 0.3.4   2020-04-17 [1] CRAN (R 4.1.0)
#>  ragg          1.1.3   2021-06-09 [1] CRAN (R 4.1.0)
#>  rappdirs      0.3.3   2021-01-31 [1] CRAN (R 4.1.0)
#>  Rcpp          1.0.6   2021-01-15 [1] CRAN (R 4.1.0)
#>  reticulate    1.20    2021-05-03 [1] CRAN (R 4.1.0)
#>  rlang         0.4.11  2021-04-30 [1] CRAN (R 4.1.0)
#>  rprojroot     2.0.2   2020-11-15 [1] CRAN (R 4.1.0)
#>  sessioninfo   1.1.1   2018-11-05 [1] CRAN (R 4.1.0)
#>  stringi       1.6.2   2021-05-17 [1] CRAN (R 4.1.0)
#>  stringr       1.4.0   2019-02-10 [1] CRAN (R 4.1.0)
#>  systemfonts   1.0.2   2021-05-11 [1] CRAN (R 4.1.0)
#>  textshaping   0.3.5   2021-06-09 [1] CRAN (R 4.1.0)
#>  withr         2.4.2   2021-04-18 [1] CRAN (R 4.1.0)
#>  xfun          0.24    2021-06-15 [1] CRAN (R 4.1.0)
#>
#> [1] C:/Users/Tristan/Documents/R/win-library/4.1
#> [2] C:/Program Files/R/R-4.1.0/library

Tags:

Posted:


text:
# Polls and Sampling Errors in the Presidental Debate Results

My biggest pet peeve is press coverage of statistics. As someone who is mathematically literate, I’m constantly infuriated by it. Basic statistics isn’t that hard, but people can’t be bothered to actually learn a tiny bit in order to understand the meaning of the things they’re covering.

My twitter feed has been exploding with a particularly egregious example of this. After monday night’s presidential debate, there’s been a ton of polling about who “won” the debate. One conservative radio host named Bill Mitchell has been on a rampage about those polls. Here’s a sample of his tweets:

Statistical analysis has a very simple point. We’re interested in understanding the properties of a large population of things. For whatever reason, we can’t measure the properties of every object in that population.

The exact reason can vary. In political polling, we can’t ask every single person in the country who they’re going to vote for. (Even if we could, we simply don’t know who’s actually going to show up and vote!) For a very different example, my first exposure to statistics was through my father, who worked in semiconductor manufacturing. They’d produce a run of 10,000 chips for use in Satellites. They needed to know when, on average, a chip would fail from exposure to radiation. If they measured that in every chip, they’d end up with nothing to sell.)

Anyway: you can’t measure every element of the population, but you still want to take measurements. So what you do is randomly select a collection of representative elements from the population, and you measure those. Then you can say that with a certain probability, the result of analyzing that representative subset will match the result that you’d get if you measured the entire population.

How close can you get? If you’ve really selected a random sample of the population, then the answer depends on the size of the sample. We measure that using something called the “margin of error”. “Margin of error” is actually a terrible name for it, and that’s the root cause of one of the most common problems in reporting about statistics. The margin of error is a probability measurement that says “there is an $N$% probability that the value for the full population lies within the margin of error of the measured value of the sample.”.

Right away, there’s a huge problem with that. What is that variable doing in there? The margin of error measures the probability that the full population value is within a confidence interval around the measured sample value. If you don’t say what the confidence interval is, the margin of error is worthless. Most of the time – but not all of the time – we’re talking about a 95% confidence interval.

But there are several subtler issues with the margin of error, both due to the name.

1. The “true” value for the full population is not guaranteed to be within the margin of error of the sampled value. It’s just a probability. There is no hard bound on the size of the error: just a high probability of it being within the margin..
2. The margin of error only includes errors due to sample size. It does not incorporate any other factor – and there are many! – that may have affected the result.
3. The margin of error is deeply dependent on the way that the underlying sample was taken. It’s only meaningful for a random sample. That randomness is critically important: all of sampled statistics is built around the idea that you’ve got a randomly selected subset of your target population.

Let’s get back to our friend the radio host, and his first tweet, because he’s doing a great job of illustrating some of these errors.

The quality of a sampled statistic is entirely dependent on how well the sample matches the population. The sample is critical. It doesn’t matter how big the sample size is if it’s not random. A non-random sample cannot be treated as a representative sample.

So: an internet poll, where a group of people has to deliberately choose to exert the effort to participate cannot be a valid sample for statistical purposes. It’s not random.

It’s true that the set of people who show up to vote isn’t a random sample. But that’s fine: the purpose of an election isn’t to try to divine what the full population thinks. It’s to count what the people who chose to vote think. It’s deliberately measuring a full population: the population of people who chose to vote.

But if you’re trying to statistically measure something about the population of people who will go and vote, you need to take a randomly selected sample of people who will go to vote. The set of voters is the full population; you need to select a representative sample of that population.

Internet polls do not do that. At best, they measure a different population of people. (At worst, with ballot stuffing, they measure absolutely nothing, but we’ll give them this much benefit of the doubt.) So you can’t take much of anything about the sample population and use it to reason about the full population.

And you can’t say anything about the margin of error, either. Because the margin of error is only meaningful for a representative sample. You cannot compute a meaningful margin of error for a non-representative sample, because there is no way of knowing how that sampled population compares to the true full target population.

And that brings us to the second tweet. A properly sampled random population of 500 people can produce a high quality result with a roughly 5% margin of error and a 95% confidence interval. (I’m doing a back-of-the-envelope calculation here, so that’s not precise.) That means that if the population were randomly sampled, we could say there is in 19 out of 20 polls of that size, the full population value would be within +/- 4% of value measured by the poll. For a non-randomly selected sample of 10 million people, the margin of error cannot be measured, because it’s meaningless. The random sample of 500 people tells us a reasonable estimate based on data; the non-random sample of 10 million people tells us nothing.

And with that, on to the third tweet!

In a poll like this, the margin of error only tells us one thing: what’s the probability that the sampled population will respond to the poll in the same way that the full population would?

There are many, many things that can affect a poll beyond the sample size. Even with a truly random and representative sample, there are many things that can affect the outcome. For a couple of examples:

How, exactly, is the question phrased? For example, if you ask people “Should police shoot first and ask questions later?”, you’ll get a very different answer from “Should police shoot dangerous criminal suspects if they feel threatened?” – but both of those questions are trying to measure very similar things. But the phrasing of the questions dramatically affects the outcome.

What context is the question asked in? Is this the only question asked? Or is it asked after some other set of questions? The preceding questions can bias the answers. If you ask a bunch of questions about how each candidate did with respect to particular issues before you ask who won, those preceding questions will bias the answers.

When you’re looking at a collection of polls that asked different questions in different ways, you expect a significant variation between them. That doesn’t mean that there’s anything wrong with any of them. They can all be correct even though their results vary by much more than their margins of error, because the margin of error has nothing to do with how you compare their results: they used different samples, and measured different things.

The problem with the reporting is the same things I mentioned up above. The press treats the margin of error as an absolute bound on the error in the computed sample statistics (which it isn’t); and the press pretends that all of the polls are measuring exactly the same thing, when they’re actually measuring different (but similar) things. They don’t tell us what the polls are really measuring; they don’t tell us what the sampling methodology was; and they don’t tell us the confidence interval.

Which leads to exactly the kind of errors that Mr. Mitchell made.

And one bonus. Mr. Mitchell repeatedly rants about how many polls show a “bias” by “over-sampling< democratic party supporters. This is a classic mistake by people who don't understand statistics. As I keep repeating, for a sample to be meaningful, it must be random. You can report on all sorts of measurements of the sample, but you cannot change it.

If you’re randomly selecting phone numbers and polling the respondents, you cannot screen the responders based on their self-reported party affiliation. If you do, you are biasing your sample. Mr. Mitchell may not like the results, but that doesn’t make them invalid. People report what they report.

In the last presidential election, we saw exactly this notion in the idea of “unskewing” polls, where a group of conservative folks decided that the polls were all biased in favor of the democrats for exactly the reasons cited by Mr. Mitchell. They recomputed the poll results based on shifting the samples to represent what they believed to be the “correct” breakdown of party affiliation in the voting population. The results? The actual election results closely tracked the supposedly “skewed” polls, and the unskewers came off looking like idiots.

We also saw exactly this phenomenon going on in the Republican primaries this year. Randomly sampled polls consistently showed Donald Trump crushing his opponents. But the political press could not believe that Donald Trump would actually win – and so they kept finding ways to claim that the poll samples were off: things like they were off because they used land-lines which oversampled older people, and if you corrected for that sampling error, Trump wasn’t actually winning. Nope: the randomly sampled polls were correct, and Donald Trump is the republican nominee.

If you want to use statistics, you must work with random samples. If you don’t, you’re going to screw up the results, and make yourself look stupid.

When a friend asks me to write about something, I try do it. Yesterday, a friend of mine from my Google days, Daniel Martin, sent me a link, and asked to write about it. Daniel isn’t just a former coworker of mine, but he’s a math geek with the same sort of warped sense of humor as me. He knew my blog before we worked at Google, and on my first Halloween at Google, he came to introduce himself to me. He was wearing a purple shirt with his train ticket on a cord around his neck. For those who know any abstract algebra, get ready to groan: he was purple, and he commuted. He was dressed as an Abelian grape.

The real subject of the article involves a recent twitter-storm around a professor at Boston University. This professor tweeted some about racism and history, and she did it in very blunt, not-entirely-professional terms. The details of what she did isn’t something I want to discuss here. (Briefly, I think it wasn’t a smart thing to tweet like that, but plenty of white people get away with worse every day; the only reason that she’s getting as much grief as she is is because she dared to be a black woman saying bad things about white people, and the assholes at Breitbart used that to fuel the insatiable anger and hatred of their followers.)

But I don’t want to go into the details of that here. Lots of people have written interesting things about it, from all sides. Just by posting about this, I’m probably opening myself up to yet another wave of abuse, but I’d prefer to avoid and much of that as I can. Instead, I’m just going to rip out the introduction to this article, because it makes a kind of incredibly stupid mathematical argument that requires correction. Here are the first and second paragraphs:

There aren’t too many African Americans in higher education.

In fact, black folks only make up about 4 percent of all full time tenured college faculty in America. To put that in context, only 14 out of the 321—that’s about 4 percent—of U.S. astronauts have been African American. So in America, if you’re black, you’ve got about as good a chance of being shot into space as you do getting a job as a college professor.

Statistics and probability can be a difficult field of study. But… a lot of its everyday uses are really quite easy. If you’re going to open your mouth and make public statements involving probabilities, you probably should make sure that you at least understand the first chapter of “probability for dummies”.

This author doesn’t appear to have done that.

The most basic fact of understanding how to compare pretty much anything numeric in the real world is that you can only compare quantities that have the same units. You can’t compare 4 kilograms to 5 pounds, and conclude that 5 pounds is bigger than 4 kilograms because 5 is bigger than four.

That principle applies to probabilities and statistics: you need to make sure that you’re comparing apples to apples. If you compare an apple to a grapefruit, you’re not going to get a meaningful result.

The proportion of astronauts who are black is 14/321, or a bit over 4%. That means that out of every 100 astronauts, you’d expect to find four black ones.

The proportion of college professors who are black is also a bit over 4%. That means that out of every 100 randomly selected college professors, you’d expect 4 to be black.

So far, so good.

But from there, our intrepid author takes a leap, and says “if you’re black, you’ve got about as good a chance of being shot into space as you do getting a job as a college professor”.

Nothing in the quoted statistic in any way tells us anything about anyone’s chances to become an astronaut. Nothing at all.

This is a classic statistical error which is very easy to avoid. It’s a unit error: he’s comparing two things with different units. The short version of the problem is: he’s comparing black/astronaut with astronaut/black.

You can’t derive anything about the probability of a black person becoming an astronaut from the ratio of black astronauts to astronauts.

Let’s pull out some numbers to demonstrate the problem. These are completely made up, to make the calculations easy – I’m not using real data here.

Suppose that:

• the US population is 300,000,000;
• black people are 40% of the population, which means that there are are 120,000,000 black people.
• there are 1000 universities in America, and there are 50 faculty per university, so there are 50,000 university professors.
• there are 50 astronauts in the US.
• If 4% of astronauts and 4% of college professors are black, that means that there are 2,000 black college professors, and 2 black astronauts.

In this scenario, as in reality, the percentage of black college professors and the percentage of black astronauts are equal. What about the probability of a given black person being a professor or an astronaut?

The probability of a black person being a professor is 2,000/120,000,000 – or 1 in 60,000. The probability of a black person becoming an astronaut is just 2/120,000,000 – or 1 in 60 million. Even though the probability of a random astronaut being black is the same as a the probability of a random college professor being black, the probability of a given black person becoming a college professor is 10,000 times higher that the probability of a given black person becoming an astronaut.

This kind of thing isn’t rocket science. My 11 year old son has done enough statistics in school to understand this problem! It’s simple: you need to compare like to like. If you can’t understand that, if you can’t understand your statistics enough to understand their units, you should probably try to avoid making public statements about statistics. Otherwise, you’ll wind up doing something stupid, and make yourself look like an idiot.

(In the interests of disclosure: an earlier version of this post used the comparison of apples to watermelons. But given the racial issues discussed in the post, that had unfortunate unintended connotations. When someone pointed that out to me, I changed it. To anyone who was offended: I am sorry. I did not intend to say anything associated with the racist slurs; I simply never thought of it. I should have, and I shouldn’t have needed someone to point it out to me. I’ll try to be more careful in the future.)

# Back to an old topic: Bad Vaccine Math

The very first Good Math/Bad Math post ever was about an idiotic bit of antivaccine rubbish. I haven’t dealt with antivaccine stuff much since then, because the bulk of the antivaccine idiocy has nothing to do with math. But the other day, a reader sent me a really interesting link from what my friend Orac calls a “wretched hive of scum and quackery”, naturalnews.com, in which they try to argue that the whooping cough vaccine is an epic failure:

(NaturalNews) The utter failure of the whooping cough (pertussis) vaccine to provide any real protection against disease is once again on display for the world to see, as yet another major outbreak of the condition has spread primarily throughout the vaccinated community. As it turns out, 90 percent of those affected by an ongoing whooping cough epidemic that was officially declared in the state of Vermont on December 13, 2012, were vaccinated against the condition — and some of these were vaccinated two or more times in accordance with official government recommendations.

As reported by the Burlington Free Press, at least 522 cases of whooping cough were confirmed by Vermont authorities last month, which was about 10 times the normal amount from previous years. Since that time, nearly 100 more cases have been confirmed, bringing the official total as of January 15, 2013, to 612 cases. The majority of those affected, according to Vermont state epidemiologist Patsy Kelso, are in the 10-14-year-old age group, and 90 percent of those confirmed have already been vaccinated one or more times for pertussis.

Even so, Kelso and others are still urging both adults and children to get a free pertussis shot at one of the free clinics set up throughout the state, insisting that both the vaccine and the Tdap booster for adults “are 80 to 90 percent effective.” Clearly this is not the case, as evidenced by the fact that those most affected in the outbreak have already been vaccinated, but officials are apparently hoping that the public is too naive or disengaged to notice this glaring disparity between what is being said and what is actually occurring.

It continues in that vein. The gist of the argument is:

1. We say everyone needs to be vaccinated, which will protect them from getting the whooping cough.
2. The whooping cough vaccine is, allagedly, 80 to 90% effective.
3. 90% of the people who caught whooping cough were properly vaccinated.
4. Therefore the vaccine can’t possibly work.

What they want you to do is look at that 80 to 90 percent effective rate, and see that only 10-20% of vaccinated people should be succeptible to the whooping cough, and compare that 10-20% to the 90% of actual infected people that were vaccinated. 20% (the upper bound of the succeptible portion of vaccinated people according to the quoted statistic) is clearly much smaller than 90% – therefore it’s obvious that the vaccine doesn’t work.

Of course, this is rubbish. It’s a classic apple to orange-grove comparison. You’re comparing percentages, when those percentages are measuring different groups – groups with wildly difference sizes.

Take a pool of 1000 people, and suppose that 95% are properly vaccinated (the current DTAP vaccination rate in the US is around 95%). That gives you 950 vaccinated people and 50 unvaccinated people who are unvaccinated.

In the vaccinated pool, let’s assume that the vaccine was fully effective on 90% of them (that’s the highest estimate of effectiveness, which will result in the lowest number of succeptible vaccinated – aka the best possible scenario for the anti-vaxers). That gives us 95 vaccinated people who are succeptible to the whooping cough.

There’s the root of the problem. Using numbers that are ridiculously friendly to the anti-vaxers, we’ve still got a population of twice as many succeptible vaccinated people as unvaccinated. so we’d expect, right out of the box, that better than 2/3rds of the cases of whooping cough would be among the vaccinated people.

In reality, the numbers are much worse for the antivax case. The percentage of people who were ever vaccinated is around 95%, because you need the vaccination to go to school. But that’s just the childhood dose. DTAP is a vaccination that needs to be periodically boosted or the immunity wanes. And the percentage of people who’ve had boosters is extremely low. Among adolescents, according to the CDC, only a bit more than half have had DTAP boosters; among adults, less that 10% have had a booster within the last 5 years.

What’s your succeptibility if you’ve gone more than 5 years without vaccination? Somewhere 40% of people who didn’t have boosters in the last five years are succeptible.

So let’s just play with those numbers a bit. Assume, for simplicity, than 50% of the people are adults, and 50% children, and assume that all of the children are fully up-to-date on the vaccine. Then you’ve got 10% of the children (10% of 475), 10% of the adults that are up-to-date (10% of 10% of 475), and 40% of the adults that aren’t up-to-date (40% of 90% of 475) is the succeptible population. That works out to 266 succeptible people among the vaccinated, which is 85%: so you’d expect 85% of the actual cases of whooping cough to be among people who’d been vaccinated. Suddenly, the antivaxers case doesn’t look so good, does it?

Consider, for a moment, what you’d expect among a non-vaccinated population. Pertussis is highly contagious. If someone in your household has pertussis, and you’re succeptible, you’ve got a better than 90% chance of catching it. It’s that contagious. Routine exposure – not sharing a household, but going to work, to the store, etc., with people who are infected still gives you about a 50% chance of infection if you’re succeptible.

In the state of Vermont, where NaturalNews is claiming that the evidence shows that the vaccine doesn’t work, how many cases of Pertussis have they seen? Around 600, out of a state population of 600,000 – an infection rate of one tenth of one percent. 0.1 percent, from a virulently contagious disease.

That’s the highest level of Pertussis that we’ve seen in the US in a long time. But at the same time, it’s really a very low number for something so contagious. To compare for a moment: there’s been a huge outbreak of Norovirus in the UK this year. Overall, more than one million people have caught it so far this winter, out of a total population of 62 million, for a rate of about 1.6% or sixteen times the rate of infection of pertussis.

Why is the rate of infection with this virulently contagious disease so different from the rate of infection with that other virulently contagious disease? Vaccines are a big part of it.

# Big Number Bogosity from a Christian College Kid

I know that I just posted a link to a stupid religious argument, but I was sent a link to another one, which I can’t resist mocking.

As I’ve written about quite often, we humans really stink at understanding big numbers, and how things scale. This is an example of that. We’ve got a jerk who’s about to graduate from a dinky christian college, who believes that there must be something special about the moral atmosphere at his college, because in his four years at the school, there hasn’t been a single murder.

Yeah, seriously. He really believes that his school is special, because it’s gone four whole years without a murder:

Considering that the USA Today calculated 857 college student deaths from 2000 to 2005, how does one school manage to escape unscathed? It’s certainly not chance or luck. For Patrick Henry College, it’s in our Christian culture.

Critics mock us for our strict rules – like no dancing or drinking on campus, no members of the opposite sex permitted in your dorm room, nightly curfew hours – and the lack of a social atmosphere it creates. We have been the subject of books (God’s Harvard), television shows, op-eds, and countless blogs who rant against our brand of overbearing right-wing Christianity that poisons society’s freedom.

Yet, what is the cost of students being able to “express” themselves? Is that freedom worth the cost of drunk driving deaths, drug related violence, and love affairs turned fatal?

There were 857 college student deaths in the five-year period from 2000 to 2005! Therefore, any college where there weren’t any murders in that period must be something really special. That christian culture must be making a really big difference, right?

Well, no.

According to Google Answers, the US Census Department reports that there are 2363 four year colleges in the US. So, assuming the widest possible distribution of student deaths, there were 1506 colleges with no student deaths in a five-year period. Or, put another way, more than 60% of colleges in the US went that five-year period without any violent student deaths.

Or, let’s try looking at it another way. According to the census, there are 15.9 million people currently enrolled in college. The school that, according to the author, is so remarkable for going without any murders in the last four years? It has 325 students. Not 325 per class – 325 total.

In other words, among a group making up less than 2/1000ths of one percent of the college population, there were no murders. Assuming that the distribution of violent deaths is perfectly uniform (which it obviously isn’t; but let’s just keep things simple), given that there were 857 violent deaths in the student population as a whole, how many violent deaths would you expect among the student body at his dinky christian college?

That would be a big, fat zero.

The fact that there were no violent deaths at his school isn’t remarkable, not at all. But to a twit who’s incapable of actually understanding what numbers mean, that’s not the conclusion to be drawn. It’s also not that the violent death among college students is actually remarkably rare. Nor is it that most college students will go through college without any violent deaths on campus. No – according to a twit, with 857 violent campus deaths over five years, the only reasonable conclusion is that there must be something special about the ridiculous religious rules at his college that prevented the great rampaging plague of violence from touching the students at his school.

I actually spent five years as an undergraduate at Rutgers University in NJ. During that time, there were no violent student deaths. (There was one death by alchohol poisoning; and there was one drunk driving accident that killed four students.) But zero violent deaths. Gosh, Rutgers must have been an absolutely amazingly moral university! And gosh, we had all of those horrible sinful things, like dancing, and co-ed dorms! How did we manage to go all that time with no violence?

It must have been the prayers of the very nice Rabbi at the Chabad house on campus. Yeah, that must be it! Couldn’t just be random chance, right?

Ok, now let me stop being quite so pettily snide for a moment.

What’s going on here is really simple. We hear a whole lot about violence on campus. And when you hear about eight-hundred and some-odd violent deaths on campus, it sounds like a lot. So, intuitively, it sure seems like there must be a whole lot of violence on campus, and it must be really common. So if you can go through your whole time in college without having any violence occur on campus, it seems like it must be unusual.

That’s because, as usual, we really suck at understanding big numbers and scale. 800 sounds like a lot. The idea that there are nearly sixteen million college students is just not something that we understand on an intuitive level. The idea that nearly a thousand deaths could be a tiny drop in the bucket – that it really amounts to just one death per 100,000 students per year – it just doesn’t make sense to us. A number like 800 is, just barely, intuitively meaningful to us. One million isn’t. Fifteen million isn’t. And a ratio with a number that we can’t really grasp intuitively on the bottom? That’s not going to be meaningful either.

Bozo-boy is making an extremely common mistake. He’s just simply failing to comprehend how numbers scale; he’s not understanding what big numbers really mean.

# Big Numbers and Air Travel

As you’ve surely heard by now, on christmas day, some idiot attempted to
blow up an airplane by stuffing his underwear full of explosives and then
lighting his crotch on fire. There’s been a ton of coverage of this – most of
which takes the form of people running around wetting their pants in terror.

One thing which I’ve noticed, though, is that one aspect of this whole mess
ties in to one of my personal obsessions: scale. We humans are really,
really lousy at dealing with big numbers. We just absolutely
have a piss-poor ability to really comprehend numbers, or to take what we
know, and put it together in a quantitative way.

# I am the antichrist. No, really!

I normally try to ignore things like this, but this is just too funny.

In general, I find arguments like this to be extremely silly. This is, basically, like
playing with gematria – only instead of doing real gematria (which can be quite silly enough),
it’s like our friend “Gotcha” – mixing systems and screwing things up until you get the results
you want.

Lots of the particularly crazy strain of Christians really, desperately want to believe
that Barack Obama is the antichrist. They want an explanation for how this black man with
a muslim name could possible have actually been elected – they don’t believe it could possibly
have happened honestly. And their doctrine requires the antichrist to come soon. Combine
those two, and you’ve got what, for them, is a sort of perfect storm.

Which gives us things like this. For more mockery, see beneath the fold.

# Moronic Probability and Stupid Physics

Via the Bad Astronomer comes one of the most pathetic abuses of
probability that I’ve ever seen. I’m simply amazed that this idiot was willing
to go on television and say this.

 The Daily Show With Jon Stewart M – Th 11p / 10c Large Hadron Collider thedailyshow.com

The crank in question is Walter Wagner, the moron who tried to use a lawsuit
to stop the LHC from being activated. (Just that much, already, is amazingly silly;
he sued in Hawaii, but the LHC is in Geneva, Switzerland. How does a Hawaiian court
have any jurisdiction?)

Anyway… Wagner claims that the LHC could destroy the earth. See, there’s a tiny theoretical chance that the right collision in the LHC could create a microscopic black hole. According to Wagner, if that happens, the black hole will swallow the entire earth.

That claim is, itself, based on some pretty bad math. The only theory that predicts
that it’s possible to create a microscopic black hole also predicts that such a black
hole would evaporate – that is, would completely disappear in a burst of energy – immediately. The
exact same math that predicts that you could create a black hole in a high-energy collision also
predicts that the hole would be destroyed before it had time to do any damage. If you tweak it so that the black hole lasts longer, the energy requirements change so that it’s no longer possible to create it in the LHC. To make the black hole last a microsecond is absolutely beyond the
energy of any collider that we could ever build on the earth.

But let’s skip that – demonstrating that is pretty complicated. To get an idea of
the level of understanding of the guy who claims that there’s a real danger, let’s just
take a look at what he says.

When asked what the probability of the LHC destroying the earth is, he says 50%. Why?
Because either it could happen, or it couldn’t – therefore, there’s a 50% chance of it happening.

You could argue that that’s naive Bayesian reasoning – but if you did, you’d be an idiot. Classic Bayesian arguments about stuff like this would say that you use 50/50 as an initial prior in the absence of any other information; then you adjust that based on whatever
other information you have available. For Mr. Wagner’s stupid argument, it’s based on
a complex physical theory – a complex physical theory which provides lots of information
which you can use to update your probability estimate.

Mr. Wagner’s 50/50 claim is based on the fact that he’s absolutely clueless about how any of
this stuff works. He clearly doesn’t understand probability, and he clearly doesn’t understand
physics.

But he’s awfully funny.

# Lottery Probabilities and Clueless Reporters

A simple, silly, but entertaining example of mathematical illiteracy by way of the Associated Press:

OMAHA, Neb. (AP) — The odds are against something this odd. But a Nebraska Lottery official says there was no mistake: The same three numbers in Nebraska’s Pick 3 lottery were drawn two nights in a row this week.

Lottery spokesman Brian Rockey said one of two lottery computers that randomly generate numbers produced the numbers 1, 9 and 6 — in that order — for Monday night’s Pick 3 drawing. Rockey says the next night, the lottery’s other computer produced the same three numbers in the same sequence.

The odds of such an occurrence? One in a million.

Close… Only off by three orders of magnitude…

# If you measure the wrong thing, you get the wrong answer: Down's syndrome in Britain

One of the blogs I read regularly is Ben Goldacre’s “Bad Science”. I recommend
it highly. (Which reminds me that I really need to find some time to update my blogroll!) In saturday’s entry, he discussed a BBC Radio documentary that described how Britain is becoming a much more welcoming place for Down’s syndrome babies.

Ben did a good job of shredding it. But I also wanted to take a stab, focusing on
the mathematical problem that underlies it, because it’s a great example of two very
common errors – first, the familiar confusing correlation and causation, and
second, using incorrect metrics.

# Astrology and the Olympics

It’s a classic kind of crackpot silliness, which I’ve described
in numerous articles before. It’s yet another example of pareidolia – that is, seeing patterns where there aren’t any.

When we look at large quantities of data, there are bound
to be things that look like patterns. In fact, it would be
surprising if there weren’t apparent parents for us to find. That’s
just the nature of large quantities of data.

In this case, it’s an astrologer claiming to have found
astrological correlations in who wins olympic competitions:

Something fishy is happening at the Olympic Games in Beijing. Put it all down to the stars.

Forget training, dedication and determination. An athlete’s star sign could be the secret to Olympic gold.

After comparing the birthdates of every Olympic winner since the modern Games began in 1896, British statistician Kenneth Mitchell discovered gold medals really are written in the stars.

He found athletes born in certain months were more likely to thrive in particular events.

Mitchell dubbed the phenomenon “The Pisces Effect” (pisces is Latin for fish) after finding that athletes born under the sign received around 30 percent more medals than any other star sign in events like swimming and water polo.


text:
# Image Encoding

Alex Egg,

Bandwidth resources are limited. Digital media information (audio, images, video) is information dense. There is a high value in conservation of bandwidth while serving digital media. Digital media information has a high degree of redundancy. Through exploitation of these redundancies we can gain advantages bandwidth and storage conservation. This paper explores some basic concepts in digital media representation within computers. First, we look at two popular color representation schemes: RGB and YCbCr. Then, we look at simple schemes called upscaling and downscaling, which are used to reduce the storage requirements of digital media. While not compression proper, these methods are the foundation for digital media representation. The goals of the paper are:

• Learn how to read and display images in Matlab.
• Learn RGB and YcrCb color spaces.
• Learn subsampling, upsampling, interpolation, replication methods and MSE computation in Matlab.

## Digital Color

Our first task was to read the jpeg image into Matlab. Matlab represents an image as an array of pixels where each pixel represents an R, G or B value. Therefore we get a wh3 array (where w is the width of the image and h is the height), or a 3D array. To display each color band, we simply replace the other band values with zeros (Figure 1). We then explore the YCbCr color space which can be used w/ the rgb2ycbcr command (Figure 2). This will give us three different bands: Y, Cb and Cr which operate in a way similar to the RGB color space but exploit information about human vision to reduce redundancy.

Figure 1: RGB color bands

Figure 2: YCbCr color bands

## Sampling

We then move to the part of the lab where we experiment with sampling. First, we sub-sample the chroma bands of the YCbCr image. We use a ratio of 4:2:0 which removes every other row and column from the Cb and Cr pixels. We then use two different upsampling schemes to restore the image. The first technique, replacement, simply takes an adjacent color and copies it in place of the pixel removed in the sub-sampling step. This is a simple technique, so it can be improved upon by using the second technique: linear interpolation. Using linear interpolation, we take the average of the adjacent pixels and replace the missing pixel with that value. When we compare the two upsampled images with the original, it is clear that linear interpolation provides smoother color gradients in the reconstruction.

Figure 3: subsampled cb and cr bands

We then perform some analysis on the reconstructed images. It should be noted that the Cr and Cb bands upsampled images have half as much pixel information than the original, however, they are still almost indistinguishable (Figure 5). This is because the implementation of the YCbCr color space exploits the weaknesses in human vision. Human vision is most sensitive to changes in intensity, not color which RGB is based on. So, even though 4:2:0 means 1/2 resolution in both horizontal and vertical directions, the difference is imperceptible even on the naive upscaling method of simple color replacement.

## Results

The differences between the original and upscaled versions can be quantified by the mean squared error equation:

Figure 4: MSE equation for an image

y  | 0.000000
Cb | 0.291327
Cr | 25.119344

Table 1: MSE across ycbcr bands

As displayed in Table 1, there is no error between the yband of the original and the 4:2:0 upscaled because we did not subsample anything from that band. We can see that there is a relatively large error on the cr band, however, that is within our allowable margin of error. This is because the overall effect the cr band has on the human eye’s perception of the image is negligible compared to the y and cr bands. See the below figures for an example:

Figure 5: The difference between the interpolation methods is almost imperceptible.

We then compare the compression ratio of the 4:2:0 image w/ the original. This is simply the ratio of pixel count in the original to the upsampled :

original/upsampled = 2

or a 1:2 compression ratio, a modest compression and essentially no quality loss.
Conclusion

It is clear from this presentation that by use of naive compression methods such as interpolated upsampling we can achieve a reasonable degree of data compression with minimal quality loss. Given, this fact it would be prudent to explore more aggressive lossy compression methods in which a higher degree of compression is achieved for a higher loss of quality.

See the 2nd part of this series on JPEG compression: http://www.eggie5.com/47-jpeg-compression


text:
SEARCH HOME
 Math Central Quandaries & Queries
 Question from teresa, a student: Driving a rental car x miles cost y= 0.25x + 25 dollars. (a) How much would it cost to rent the car but not drive it? (b) How much does it cost to drive the car 1 additional mile? (c) What is the y-intercept of y= 0.25x + 25? What does it represent? (d) What is the slope of y= 0.25x + 25? What does it represent?

Hi Teresa,

Let me look at a similar question. I am wondering how many years I should keep my car before I consider selling it and a dealer told me that an approximate value of my car $y$ is given by

$y = \20~000 - \3~600 x$

where x is the number of years I have owned that car.

What does this function tell me?

When I bought the car $x = 0$ and then $y = \20~000 - \3~600 \times 0 = \20~000$ and hence its value then (hopefully what I paid for it) was $\$20 000.$What was its value one year later? That is when$x = 1$and hence $y = \20~000 - \3~600 \times 1 = \20~000 - \3~600 = \16~400$ and hence it has dropped$\$3~600$ in value in the first year.

What was its value one year later, that is when $x = 2?$ Looking at the function $y = \2~ 000 - \3~ 600 x$ you can see that it will drop another $\$3 600$in value. The slope of the line $y = \20~000 - \3~600 x$ is$-3~600\$ and the units are dollars per year and it tells me how much the value of my car decreases every year.

Try your problem now and if you get stuck, write back and tell us what you did and we will try to help,
Penny

Math Central is supported by the University of Regina and The Pacific Institute for the Mathematical Sciences.