text:
# Sur la distribution des nombres premiers

@article{KochSurLD,
title={Sur la distribution des nombres premiers},
author={Helge von Koch},
journal={Acta Mathematica},
volume={24},
pages={159-182}
}
• H. V. Koch
• Published 1 December 1901
• Mathematics
• Acta Mathematica
316 Citations
New estimates for some integrals of functions defined over primes
In this paper we give new estimates for integrals involving some arithmetic functions defined over prime numbers. The main focus here is on the prime counting function π(x) and the Chebyshev
A Simple and Accurate Relation Between the Logarithm Integral Li(x) and the Primes Counting Function π(x) is Derived Making use of the O.E.I.S. Prime Numbers “Sequences”
• P. Ascarelli
• Mathematics
Journal of Mathematics and Statistics
• 2021
Email: ascarelli.p@gmail.com Abstract: Today the prime numbers π(x) contained under the number x appears to be somewhat overestimated by the logarithm integral function Li(x) and underestimated by
The Elementary Proof of the Riemann's Hypothesis
This research paper aims to explicate the complex issue of the Riemann's Hypothesis and ultimately presents its elementary proof. The method implements one of the binomial coefficients, to
On the Asymptotic Density of Prime k-tuples and a Conjecture of Hardy and Littlewood
• L. T'oth
• Mathematics, Computer Science
Computational Methods in Science and Technology
• 2019
"Skewes numbers" are found for 8 more prime k-tuples and provided numerical data in support of the Hardy-Littlewood conjecture and several algorithms to compute such numbers are presented.
Effective estimates for some functions defined over primes
. In this paper we give eﬀective estimates for some classical arithmetic functions deﬁned over prime numbers. First we ﬁnd the smallest real number x 0 so that some inequality involving Chebyshev’s ϑ
An analytic approach to the Riemann hypothesis
• Mathematics
• 2019
In this work we consider a new functional equation for the Riemann zeta-function in the critical half-strip. With the help of this equation we prove that finding non-trivial zeros of the Riemann
New Estimates for Some Functions Defined Over Primes
New explicit estimates for Chebyshev's $\vartheta$-function are established and new upper and lower bounds for some functions defined over the prime numbers are derived, for instance the prime counting function $\pi(x)$, which improve the currently best ones.
Estimates for π(x) for Large Values of x and Ramanujan's Prime Counting Inequality
In this paper we use refined approximations for Chebyshev's $\vartheta$-function to establish new explicit estimates for the prime counting function $\pi(x)$, which improve the current best estimates
Explicit results on primes
Let p be a prime number. In this thesis we prove the following theorems. Theorem -1.0.1. Let x0 ≥ 4·1018 be a fixed constant and let x > x0. Then there exists at least one prime p such that (1 −∆−1)x

format: plain


text:
# The algebra of convergent series.

Theorem 9.2.1   Let and be two convergent series. Then the series is convergent and its sum is equal to .

Theorem 9.2.2   Let be a convergent series and . Then the series is convergent and its sum is equal to .

Together these two theorems mean that the set of convergent series of real numbers is a real vector space.

Remark 9.2.3   Adding or deleting terms from a series does not change the convergence/divergence. For a convergent series, it changes the value of the sum.

Example 9.2.4

Noah Dana-Picard 2007-12-28


text:
Expii

# Integrating Powers and Polynomials - Expii

We summarize how to integrate powers ($$x^n$$) and polynomials.


text:
Hardcover | Out of Print | 312 pp. | 6 x 9 in | 4 line drawings, 6 graphs, 2 figures, 1 table | February 2008 | ISBN: 9780262182652
Paperback | $5.75 Short | £4.95 | 312 pp. | 6 x 9 in | 4 line drawings, 6 graphs, 2 figures, 1 table | January 2012 | ISBN: 9780262517584 eBook |$14.00 Short | January 2012 | ISBN: 9780262253406

## Midbrain Mutiny

The Picoeconomics and Neuroeconomics of Disordered Gambling: Economic Theory and Cognitive Science

## Overview

The explanatory power of economic theory is tested by the phenomenon of irrational consumption, examples of which include such addictive behaviors as disordered and pathological gambling. Midbrain Mutiny examines different economic models of disordered gambling, using the frameworks of neuroeconomics (which analyzes decision making in the brain) and picoeconomics (which analyzes patterns of consumption behavior), and drawing on empirical evidence about behavior and the brain.

The book describes addiction in neuroeconomic terms as chronic disruption of the balance between the midbrain dopamine system and the prefrontal and frontal serotonergic system, and reviews recent evidence from trials testing the effectiveness of antiaddiction drugs. The authors argue that the best way to understand disordered and addictive gambling is with a hybrid picoeconomic-neuroeconomic model.

Don Ross is Professor of Economics and Dean of Commerce at the University of Cape Town, and Research Fellow in the Center for Economic Analysis of Risk at Georgia State University. He is the author of Economic Theory and Cognitive Science: Microexplanation (MIT Press, 2005), companion volume to Midbrain Mutiny.

Carla Sharp is Associate Professor of Psychology at the University of Houston.

Rudy E. Vuchinich was formerly Professor of Psychology at the University of Alabama at Birmingham.

David Spurrett is Professor of Philosophy and Director of the Cognitive Science Program at the Howard College Campus of the University of KwaZulu-Natal, South Africa.

## Endorsements

“Relying on findings from the burgeoning area of Neuroeconomics, Midbrain Mutiny makes the case that the neural basis of addiction's core is in sight. It is so convincing, in large part, because it simultaneously illuminates the very behavioral findings that have led others to doubt the feasibility of such a reduction. This is sure-footed pragmatism and, indeed, scientific unification at its most thrilling!”
John Monterosso, University of California, Los Angeles
“In this bold synthesis of contemporary research and theory bringing together advances in behavioral- and neuro- economics, Ross and colleagues provide a novel understanding of addiction and apply it to the growing international challenge of pathological gambling. This book is a must read for anyone interested in the science of addiction and addictive gambling and I predict will soon become the standard work for understanding the neuroscientific basis of gambling disorders.”
Warren K. Bickel, Mills Chair of Alcoholism and Drug Abuse Prevention, and Director, Center for Addiction Research, University of Arkansas for Medical Sciences
Midbrain Mutiny gathers together all current, significant research and theory on the psychology and neurology and economics of gambling. In this truly excellent book the authors organize that material, reconcile apparent differences, and make the compelling argument that we are on the verge of a significant breakthrough in treatment of disordered gambling.”
Howard Rachlin, Psychology Department, State University of New York, Stony Brook
“Researchers who want to study impulsive or irrational behavior are well advised to study gambling. It shares with substance addictions a concrete consumption pattern and physiological adaptation, and shares with purely emotional impulses the absence of an ingested agent and an allure for people of all levels of sophistication. Midbrain Mutiny integrates a wide range of research findings on impulsive motivation in general, and on gambling in particular, to develop a parsimonious model of disordered gambling that can also illuminate many other problem behaviors. The result is a superb platform on which neurophysiologists, imaging researchers, psychologists, economists, and philosophers of mind will be able to meet and connect their findings.”
George Ainslie, Department of Veterans Affairs Medical Center, and Temple Medical College


text:
diannesalazar
9

# solve. x/3 - 12 = 24

$\frac{x}{3} - 12 = 24 \\ \\ \frac{x}{3} = 24 + 12 \\ \\ \frac{x}{3} = 36 \\ \\ x = 36 \times 3 \\ \\ x = 108 \\ \\$ The final result is: x = 108.


text:
# [Tex/LaTex] How to write “LaTeX” in LaTeX with the first letter bolded

boldfontsformattingsymbols

So I know to write \LaTeX to get the symbol, but is there a way to do it letter by letter so as to make the first letter bold?

Note: There's a similar question and answer here: Color up each letter of \LaTeX word

You can find the definitions of \TeX, \LaTeX and \LaTeXe in latex.ltx (which is really, really surprising ;-) )

Easiest way is to use the definitions and save them as new commands, say \BLaTeX for Bold \LaTeX ;-)

However, the kerning is wrong initially, so I changed it slightly -- please provide own values for the kerning if the given example does not really fit yet regarding personal taste.

\documentclass{article}

\usepackage{xcolor}
\makeatletter
\newcommand{\Lletterstyle}[1]{%
\textcolor{red}{\textbf{#1}}%
}
\DeclareRobustCommand{\BLaTeX}{\Lletterstyle{L}\kern-.35em%
{\sbox\z@ T%
\vbox to\ht\z@{\hbox{\check@mathfonts
\fontsize\sf@size\z@
\math@fontsfalse\selectfont
A}%
\vss}%
}%
\kern-.15em%
\TeX%
}
\DeclareRobustCommand{\BLaTeXe}{\mbox{\m@th
\if b\expandafter\@car\f@series\@nil\boldmath\fi
\BLaTeX\kern.15em2$_{\textstyle\varepsilon}$}}
\makeatother

\begin{document}

\BLaTeX\ or \LaTeX

\BLaTeXe\ or \LaTeXe

Huge:\huge

\BLaTeX\ or \LaTeX

\BLaTeXe\ or \LaTeXe

\end{document}


text:
The ratio of the spacing of two consecutive frets is {\displaystyle {\sqrt[{12}]{2}}} (twelfth root of two). In practice, luthiers determine fret positions using the constant 17.817—an approximation to 1/(1-1/ {\displaystyle {\sqrt[{12}]{2}}} ). If the nth fret is a distance x from the bridge, then the distance from the (n+1)th fret to the bridge is x-(x/17.817).[15] Frets are available in several different gauges and can be fitted according to player preference. Among these are "jumbo" frets, which have much thicker gauge, allowing for use of a slight vibrato technique from pushing the string down harder and softer. "Scalloped" fretboards, where the wood of the fretboard itself is "scooped out" between the frets, allow a dramatic vibrato effect. Fine frets, much flatter, allow a very low string-action, but require that other conditions, such as curvature of the neck, be well-maintained to prevent buzz.
Some piezo-equipped guitars use a hexaphonic pickup. "Hex" is a prefix meaning six. A hexaphonic pickup produces a separate output for each string, usually from a discrete piezoelectric or magnetic pickup for each string. This arrangement lets on-board or external electronics process the strings individually for modeling or Musical Instrument Digital Interface (MIDI) conversion. Roland makes hexaphonic pickups for guitar and bass, and a line of guitar modeling and synthesis products. Line 6's hexaphonic-equipped Variax guitars use on-board electronics to model the sound after various vintage instruments, and vary pitch on individual strings.
Electric guitars and bass guitars have to be used with a guitar amplifier and loudspeaker or a bass amplifier and speaker, respectively, in order to make enough sound to be heard by the performer and audience. Electric guitars and bass guitars almost always use magnetic pickups, which generate an electric signal when the musician plucks, strums or otherwise plays the instrument. The amplifier and speaker strengthen this signal using a power amplifier and a loudspeaker. Acoustic guitars that are equipped with a piezoelectric pickup or microphone can also be plugged into an instrument amplifier, acoustic guitar amp or PA system to make them louder. With electric guitar and bass, the amplifier and speaker are not just used to make the instrument louder; by adjusting the equalizer controls, the preamplifier, and any onboard effects units (reverb, distortion/overdrive, etc.) the player can also modify the tone (aka timbre or "colour") and sound of the instrument. Acoustic guitar players can also use the amp to change the sound of their instrument, but in general, acoustic guitar amps are used to make the natural acoustic sound of the instrument louder without changing its sound that much.
Another factor to consider are the alloys used to make the string. Acoustic strings may be phosphor bronze or an 80/20 bronze/zinc alloy, electric strings may be pure steel or a steel/nickel alloy, and the outer winding on the thicker strings may be either round wound or flat wound, which is typically used by jazz guitarists for smooth fingering with less fretting noise. Acoustic and electric strings are both now available with special super-thin coatings to protect them from sweat and corrosion. Coated strings cost more, but generally last much longer. Find out what type of string your favorite player uses, try guitars with different types of strings at your local guitar store, then try a few different sets on your guitar to see what feels and sounds best for your playing style. Major manufacturers include Ernie Ball, Martin, GHS, D'Addario, and Elixir, among others. Make sure you have a string winder to make installing the strings go faster, and some pliers with a fine wire cutter to clip the excess string from the tuning post, a well as some guitar polish - changing strings is a good time to perform some routine maintenance. Keep a clean cloth in your case to wipe down the strings after each use, along with extra bridge pins for your acoustic guitar. Regular string changes are the best way to keep your tone crisp and clear, so grab a pack and tune 'em up!
###### Bolt-on necks, though they are historically associated with cheaper instruments, do offer greater flexibility in the guitar's set-up, and allow easier access for neck joint maintenance and repairs. Another type of neck, only available for solid body electric guitars, is the neck-through-body construction. These are designed so that everything from the machine heads down to the bridge are located on the same piece of wood. The sides (also known as wings) of the guitar are then glued to this central piece. Some luthiers prefer this method of construction as they claim it allows better sustain of each note. Some instruments may not have a neck joint at all, having the neck and sides built as one piece and the body built around it.

There are two basic types of electric guitars: solidbody and hollowbody. Today, the electric guitar still features in all types of music Ð rock, blues, jazz and big bands Ð and is played by men and women, young and old, throughout the world. Some well-known electric guitarists include Muddy Waters, Jimi Hendrix, Eric Clapton, Jimmy Page, Eddie Van Halen, Steve Vai, Joe Satriani, Pat Metheny, Wes Montgomery, Chrissie Hynde and Liz Phair.
If you are brand new to the guitar, we recommend you start with the Beginner Guitar Quick-Start Series. It’s a collection of 12 step-by-step videos that will give you a solid foundation for all future lessons. You’ll learn how to play guitar, how to hold the guitar, how to tune your guitar, how to strum the guitar, how to play your first chords, and even how to play your first song on guitar.
Once you know the basic chords, it might be easier to think of them the way the function inside a key. For example, when in the key of E, the E (I) is called the Tonic. It's what all the other chords want to get to—which is what helps give western music its sense of motion. The A (IV) in the key of E functions as the Subdominant—it's sort of a passive in-between, just as happy to continue forward, as to relax back to the Tonic. The Dominant is just what it sounds like: it leads you where it wants to go. In the key of E, that role is filled by the B (V), and will definitely make your brain want to get back to the Tonic! When you get more familiar with the chords, and want to sketch out a tune, try writing it as I-IV-V (or variations of that) instead of E-A-B. It will make it much easier to transpose when you find out your singer cannot sing in the original key!
Electric guitars and bass guitars have to be used with a guitar amplifier and loudspeaker or a bass amplifier and speaker, respectively, in order to make enough sound to be heard by the performer and audience. Electric guitars and bass guitars almost always use magnetic pickups, which generate an electric signal when the musician plucks, strums or otherwise plays the instrument. The amplifier and speaker strengthen this signal using a power amplifier and a loudspeaker. Acoustic guitars that are equipped with a piezoelectric pickup or microphone can also be plugged into an instrument amplifier, acoustic guitar amp or PA system to make them louder. With electric guitar and bass, the amplifier and speaker are not just used to make the instrument louder; by adjusting the equalizer controls, the preamplifier, and any onboard effects units (reverb, distortion/overdrive, etc.) the player can also modify the tone (aka timbre or "colour") and sound of the instrument. Acoustic guitar players can also use the amp to change the sound of their instrument, but in general, acoustic guitar amps are used to make the natural acoustic sound of the instrument louder without changing its sound that much.
Ask any veteran musician, and they'll tell you that the early stages of learning a musical instrument go by a lot smoother when you're having fun. For this reason, Guitar Center strives their hardest to ensure every guitar lesson they offer is fully-engaging and an absolute blast for everyone involved. Whether you're into the warm, natural sound of an acoustic guitar or have aspirations of blowing out ear drums on the biggest stages in town, GC's guitar lessons are designed so that players of all ages, skill levels and tastes learn the chords and scales they need to know (and want to know) in a comfortable environment.
So, what should you look for in a set of strings? There's no one answer. To go back to those same analogies: every woodwind player has a preferred strength and shape of reed, and every drummer likes a set of sticks with particular balance and tip shape. It's the same with strings: you've got different gauges and different materials to choose from, and ultimately the right ones for you are a matter of preference.
The intervals between the notes of a chromatic scale are listed in a table, in which only the emboldened intervals are discussed in this article's section on fundamental chords; those intervals and other seventh-intervals are discussed in the section on intermediate chords. The unison and octave intervals have perfect consonance. Octave intervals were popularized by the jazz playing of Wes Montgomery. The perfect-fifth interval is highly consonant, which means that the successive playing of the two notes from the perfect fifth sounds harmonious.
This unique degree program combines contemporary music performance training with the rigorous study of traditional disciplines including sight-reading, music history, arranging, conducting, recording and more. The comprehensive program instructs students to demonstrate stylistic depth, creative maturity and professional competence in preparation for careers in today’s music industry.


text:
### Home > CALC > Chapter 7 > Lesson 7.1.5 > Problem7-52

7-52.

As Khalid inflates a spherical balloon, Kareem wonders about its different rates. He knows that the rate at which Khalid blows is equal to the rate at which the volume changes $( \frac { d V } { d t } )$. As the balloon inflates, other aspects are changing as well, such as the radius and the surface area.

1. If $\frac { d V } { d t }= 10\frac {\text{cm} ^ { 3 } } {\text{sec} }$, find the rate of change of the radius, $\frac { d r } { d t }$, when $r = 3$ cm.

Using just geometry (no Calculus), find the equation for a generic spherical balloon.
$V=\frac{4}{3}\pi r^{3}$

Substitute the given information and solve.

2. If $\frac { d V } { d t }= 12\frac {\text{cm} ^ { 3 } } {\text{sec} }$, find the rate of change of the surface area, $\frac { d A } { d t }$, when $r = 5$ cm.

Find $\frac{dr}{dt}$ , you will need that information later.

Using just geometry, write the equation for the Surface Area of a generic spherical balloon: $A = 4πr^²$

3. Describe what happens to the balloon when $\frac { d V } { d t }$ is negative.

What does a negative rate represent?


text:
# inferring heavy-tail distribution from finite sample of histogram data

I have some data in the form of bins and counts. Here is one complete non-truncated example:

bins: 60-100, 100-200, 200-300, 300-500, 500-1000, 1000-2000, 2000-3000, >3000
counts: 275, 320, 112, 65, 53, 44, 16, 15

Counts is the number of objects with sizes given by corresponding bin (60-100, 100-200 etc). If we construct the probability density function (dividing counts by bin size and scaling to total counts), we get some decreasing exp-like dependence. I have a hypothesis (and the nature of the objects implies so), that this may be the right-hand tail of a skewed Gaussian with its center somewhere out of measurements scope (for bins around 10 or less). But there is another idea, that this function may come from a long-tailed (or heavy-tailed) statistics.

(Update in clarification). Under "heavy-tailed" the stable distribution law is meant (in Levy sense) different from Gaussian, which in its intuitive concept involves: infinite dispersion or higher moments and asymptotics as x^-n instead of exp(-x^2).

Is it possible on the basis of these data only (I have several other such objects) to discriminate between exponential decrease and long-tail behaviour? Perhaps, with stating some confidence and probability of such hypothesis. What are tests which could answer this question and how they can be implemented preferably in R and/or in Mathematica? (other languages suit as well, if the solution exists therein).

In favour of this hypothesis is that the amount of matter for each bin (that is \int f(x)*x dx) increases with increased bin number, which seems to be not the case, if we had purely Gaussian tail.

If the sample is too sparse, and no conclusion can be drawn with reasonable confidence, how can I estimate the amount of needed data (number of bins and/or bin sizing) which are sufficient for this inference?

• The skew normal (or skew Gaussian) is one kind of long tailed distribution. Exponential decrease is another kind of long tailed distribution. The data are clearly long tail; do you want to test one long-tail distribution vs. another? If so, which two? – Peter Flom Aug 20 '17 at 11:25
• The first thing is to clearly show the discrimination between long-tailed (~x^n) and short-tailed exponential (exp(-x^n)). As next thing, maybe to try to infer the class of long tails (n in x^n asymptotics). But I wonder, whether exponential decrease belongs to long-tailed? Perhaps, there is a terminology mismatch. I learned that "long-tailed" refer to distributions with infinite moments and x^(-n) assymptotics. As opposed to "short-tailed", to which Gaussian belongs - see, e.g. ufn.ru/en/articles/2003/8/c – astrsk Aug 20 '17 at 11:41
• To me, "long tailed" just means ... well ... having a long tail! It needn't have infinite moments. But that terminology difference would explain my confusion. – Peter Flom Aug 20 '17 at 12:18
• @Peter There's an explicit definition for "long tailed" here --- Heavy-tailed distribution: Definition of long-tailed distribution (though you can sometimes find other definitions as well as it being used in a more general sense). It's not clear that the OP intends this particular one though -- it should be made explicit in the question – Glen_b Aug 20 '17 at 12:40
• I did not distinguish "long-tailed" and "heavy-tailed" in the context as in the above article. So maybe right way to broaden the wording and to speak about "heavy-tailed". For me ever "heavy" (or "long-"?) meant stable distributions (in Levi sense), but different from the Gaussian. This necessarily implies infinite moments and x^-n asymptotics. I never came into such details as this "long-tail" subclass – astrsk Aug 20 '17 at 14:00

There are lots of ways to define "heavy tails" depending on which specific property is important, but a common definition is when the tails are heavy enough for the variance to be infinite. (This meaning is relevant in probability theory because it vitiates application of the central limit theorem.) In any case, it is possible to use the data to get an idea of the likely shape of the tails, though this always requires extrapolation beyond the data range.

Heavy-tailed distributions (infinite variance): Consider a density function $$f$$ with tails that decay according to the power-law:

$$f(x) \rightarrow c x^{-\omega-1} \quad \quad \text{as } x \rightarrow \infty.$$

From this form is it easy to show that:

$$\int \limits_x^\infty (r-\mu)^2 f(r) dr = \mathcal{O}(x^{2-\omega}),$$

so the upper-end of the integral converges to a finite value only if $$\omega>2$$. Hence, the variance of the distribution will be finite so long as both tails decay at least as fast as a power-law greater than cubic decay. It is important to note that heavy-tailed distributions only arise when the variable of interest is unbounded. If the variable of interest is bounded to some finite region then the variance must be finite and so no problem arises.

Looking at tail behaviour in the data: We can use the observed sample values in the tail of the distribution to see if this appears to be the case for our data. From the form of a density function with power-law decay, it is easy to show that the log-tail-probability and log-mean deviation are related in the tail by:

$$\ln \mathbb{P}(X>x) \rightarrow \text{const} - \omega \ln(x-\mu) \quad \quad \text{as } x \rightarrow \infty.$$

Letting $$x_{(1)} \geqslant ... \geqslant x_{(n)}$$ be the ordered sample values we can estimate the log-tail-probability by $$\ln \hat{\mathbb{P}}(X>x_{(i)}) = \ln(2i-1)- \ln(2n)$$. For the values occurring in the tails of the distribution we should therefore expect the values to follow the relationships:

$$\begin{matrix} \text{Right tail } & & & & \quad \quad \quad \ln(2i-1) \approx \text{const} - \omega \ln|x_{(i)}-\bar{x}_n|, \\ \text{Left tail} \quad & & & & \text{ } \ln(2(n-i)-1) \approx \text{const} - \omega \ln|x_{(i)}-\bar{x}_n|. \\ \end{matrix}$$

(Note that we are taking logarithms so we only look at the values on the appropriate side of the mean. This is reasonable since we are only interested in the tail values in each tail.) In order to diagnose whether there is evidence of heavy tails in the distribution we can construct tail plots which show the logarithmic terms in these approximate equations, and we can look at the slope of the relationship to estimate the value $$\omega$$ for each tail. The values in the tail-plots are given by:

$$\begin{matrix} \text{Vertical axis (Right tail) } & & & & \quad \quad \quad \ln(2i-1) \\ \text{Vertical axis (Left tail)} \quad & & & & \text{ } \ln(2(n-i)-1) \\ \text{Horizontal axis } \quad \quad \quad \quad & & & & \quad \quad \ln|x_{(i)}-\bar{x}_n| \end{matrix}$$

Here is an example of this kind of plot for some data from a distribution with lower-bounded support but no upper bound:

From these tail plots we see that both tails appear to be decaying faster than cubic decay. For the left-tail we already know it is not heavy-tailed (since it is bounded), but it is comforting that this is reflected in the plot. For the right-tail we can see that it is fairly close to cubic decay, but it does appear to be faster. In interpreting these plots we concentrate on the values that are further into the tails, which are the values closer to the right-hand-side of each plot. It is important to note that a larger sample would show more values in the tails which might show a different rate of decay.

• Not sure how "common" the "infinite variance" definition of heavy-tailedness is, but it is too limited a definition to be of practical use for many situations, just like the "exponential tails" definition is too limited. Heavy-tailed distributions are used to model processes that produce occasional rare, extreme values. Such processes can easily have bounded support; see my example above. – BigBendRegion Oct 6 '18 at 12:00
• I find that the commonness of the definitions depends on the commonness of use of the theorems that depend on those properties. Since finite variance is a requirement of the standard version of the CLT, it is quite commonly applied, and thus, its conditions are quite commonly used. In that context the relevant meaning of "heavy tails" is the one above. I think we agree that in other contexts a different meaning is used. – Ben Oct 6 '18 at 23:19
• Agreed, I am after a notion of "heavy tails" that is useful in practice to model real processes that produce occasional, extreme values. Others will have different notions. My main concern is that people believe what they see on the internet, and therefore buy into narrow definitions provided, eg., on Wikipedia, as if they are definitive and comprehensive. Then they repeat these notions as if they are firmly established and generally accepted. Perhaps we need a different term, like "models for processes that produce occasional extreme values," rather than "heavy-tailed" models. – BigBendRegion Oct 8 '18 at 13:08
• Then presumably they will read my first sentence: "There are lots of ways to define "heavy tails" depending on which specific property is important...". – Ben Oct 8 '18 at 23:23
• Yes, that's a good caveat, my apologies. Unfortunately, the current Wikipedia entry on "Heavy-Tailed Distribution" has no such caveat and is quite dogmatic in its statement of precisely what "heavy-tailed" means (and, by implication, what it does not mean). It is problematic because people (even Ph.D.s who post here) think that the Wikipedia entry gives the precise, accepted definition. – BigBendRegion Oct 9 '18 at 10:44

There are no agreed-upon definitions for long- and heavy-tailed. The power law definitions are often given, but they are too restrictive for practical purposes. The whole point of using such distributions is to model processes that produce occasional, extreme observations far in excess of what you would expect using the normal distribution. Such processes need not obey power laws or even have infinite support. For example, mix a U(-1,1) with a U(-10000, 10000), with mixing p=.0001 on the latter: This distribution has finite support, no power law, but is extremely heavy tailed. But using the power law definition, it is arguably "light-tailed."

Kurtosis measures (moment- and quantile-based) are useful alternative definitions of heavy-tailedness that do not have the limitations of the power law definitions.

Thus, as regards to the OP's data analysis question, I would suggest fitting the lognormal distribution using interval censored maximum likelihood, and then use the kurtosis of that fitted distribution as a measure of its heavy-tailedness. As far as the first category, you might question whether the interval is really 60-100 or 0-100. If 60 is really a hard-and-fast lower bound, then you need to modify the distribution. Otherwise, use 0-100 when fitting the lognormal.


text: Title Search for new physics in lepton+photon+X events with 929 $pb^{-1}$ of $p\overline{p}$ collisions at $\sqrt{s}$=1.96 TeV Author Abulencia, A. Van Remortel, N. et al. Faculty/Department Faculty of Sciences. Physics Publication type article Publication 2007 Lancaster, Pa , 2007 Subject Physics Source (journal) Physical review : D : particles and fields. - Lancaster, Pa, 1970 - 2003 Volume/pages 75(2007) :11 , p. 112001,1-112001,15 ISSN 0556-2821 ISI 000247625300004 Carrier E Target language English (eng) Full text (Publishers DOI) Affiliation University of Antwerp Abstract We present results of a search at CDF in 929±56  pb-1 of pp̅ collisions at 1.96 TeV for the anomalous production of events containing a high-transverse momentum charged lepton (ℓ, either e or μ) and photon (γ), accompanied by missing transverse energy (E̸T), and/or additional leptons and photons, and jets (X). We use the same selection criteria as in a previous CDF Run I search, but with an order-magnitude larger data set, a higher pp̅ collision energy, and the CDF II detector. We find 163 ℓγE̸T+X events, compared to an expectation of 150.6±13.0 events. We observe 74 ℓℓγ+X events, compared to an expectation of 65.1±7.7 events. We find no events similar to the Run I eeγγE̸T event. Full text (open access) https://repository.uantwerpen.be/docman/irua/40074d/01fa6c47.pdf E-info http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcApp=PARTNER_APP&SrcAuth=LinksAMR&KeyUT=WOS:000247625300004&DestLinkType=RelatedRecords&DestApp=ALL_WOS&UsrCustomerID=ef845e08c439e550330acc77c7d2d848 http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcApp=PARTNER_APP&SrcAuth=LinksAMR&KeyUT=WOS:000247625300004&DestLinkType=FullRecord&DestApp=ALL_WOS&UsrCustomerID=ef845e08c439e550330acc77c7d2d848 http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcApp=PARTNER_APP&SrcAuth=LinksAMR&KeyUT=WOS:000247625300004&DestLinkType=CitingArticles&DestApp=ALL_WOS&UsrCustomerID=ef845e08c439e550330acc77c7d2d848 Handle


text:
# Talk:Impedance bridging

WikiProject Electronics (Rated Start-class, Low-importance)
This article is part of WikiProject Electronics, an attempt to provide a standard approach to writing articles about electronics on Wikipedia. If you would like to participate, you can choose to edit the article attached to this page, or visit the project page, where you can join the project and see a list of open tasks. Leave messages at the project talk page
Start  This article has been rated as Start-Class on the project's quality scale.
Low  This article has been rated as Low-importance on the project's importance scale.

## Complex Impedance

For the general case of complex impedance, the load voltage is maximal at ${\displaystyle Z_{L}={\frac {2Z_{S}{\bar {Z_{S}}}}{Z_{S}-{\bar {Z_{S}}}}}}$. I believe this must be mentioned. Leokor (talk) 02:57, 12 February 2013 (UTC)

## Article name

I moved this to impedance bridging since it gives this idea that it is the "opposite" of impedance matching, but the term "voltage bridging" i think is more common.

Also they are not opposites really. One maximizes power and one maximizes voltage. Not opposites.

Also to differentiate it from a bridged amplifier, in which two positive outputs are inverted from each other and used to drive a load in a floating type configuration. not the same thing.

Also I am going to keep editing these. - Omegatron 20:54, Jun 23, 2004 (UTC)

### Voltage bridging

Moved from Talk:Impedance matchingOmegatron 19:51, 13 April 2007 (UTC)

Is this an acccepted term in the industry?--Light current 03:29, 16 December 2006 (UTC)

No. We kind of made it up, from the common term "bridging". Do you know of a better one? — Omegatron 04:08, 5 January 2007 (UTC)
No, but I didnt think neologisms or original research (terminology) were allowed 8-)--Light current 17:48, 5 January 2007 (UTC)
The only terms I can think of is 'parallel connection' or 'tapping' as in phone tapping--Light current 18:53, 5 January 2007 (UTC)

It's not actually made up, but it's not commonly referred to by name:

• "Calculation the damping of impedance bridging or matching an interface connecting Zout and Zin""[1]
• "If the load impedance is 10 times or more the source impedance, it is called a "bridging" impedance. Bridging results in maximum VOLTAGE transfer from the source to the load."[2]
• This might be related?
• "High-impedance bridging input does not load signal source"[3]
• "High impedance/bridging" inputs [4]
• "High-impedance bridging inputs allow connection from either high or low impedance sources."[5]
• "Actual output impedance is 100 ohms (47 ohms unbalanced) and the TB-6 "Mic-All" amplifiers will drive virtually any line load from 600 ohms to high impedance bridging!"[6]

"High-impedance bridging" might actually be a better term? — Omegatron 19:14, 5 January 2007 (UTC)

Since the term 'high impedance bridging' is in the literature, I think we could use that. Voltage bridging merely redirects to impedance bridging, so its just a matter of deleting the voltage bridging page. —The preceding unsigned comment was added by Light current (talkcontribs) 20:11, 5 January 2007 (UTC).
What literature?
Why would you delete the redirect? — Omegatron 21:21, 5 January 2007 (UTC)

Here is a book that mentions "bridging" vs "matching".

http://books.google.com/books?id=wBlRtAlKPFsC&dq=%22impedance+bridging%22&q=bridging#searchOmegatron 19:50, 13 April 2007 (UTC)

## "Second Device" vs. "Previous Device"

"It is a bridging connection if the second device does not appreciably load the previous device." Which is the "second" device? The load or the source? How is "previous" defined here? I'm pretty sure that this means that the source does not appreciably load the load device, but this language could be clearer. Steve carlson 23:27, 9 March 2007 (UTC)

## Amp-versus-speaker not relevant at all!

The connection between an amplifier and speaker is not an instance of impedance bridging. Amplifiers are current sources.

The reason why a low output impedance is desired is for two reasons: damping, which is an electro-mechanical effect, and maximum power transfer to the speaker.

The purpose of this connection is not to pass along a signal to the speaker with minimal voltage drop, but to drop current into the speaker while dissipating as little heat as possible in the amplifier and controlling the speaker's motion. — Preceding unsigned comment added by 192.139.122.42 (talk) 23:46, 2 June 2011 (UTC)

I too saw this section and came here to comment that it's not relevant in an article about impedance bridging. The whole section should be deleted or moved somewhere else. — Brianonn (talk) 07:39, 12 September 2011 (UTC)
This section seems to be VERY important. There is too much lack of understanding.
Amplifiers are voltage sources. A voltage source allows maximum power transfer to the speaker. And .. it is often said the amplifiers has 8 Ohm , but the source impedance of the amplifier will be like 0.1 Ohm. Edited to show the peculiarities of this special bridging.
--AK45500 (talk) 12:56, 5 September 2018 (UTC)

## Reference 404's

I'm just pointing out that the Rod Elliot reference now 404's. I found this article. http://sound.whsites.net/impedanc.htm but am not sure if it's equivalent. — Preceding unsigned comment added by Petergcook (talkcontribs) 00:52, 28 November 2016 (UTC)

seems to work --AK45500 (talk) 13:08, 5 September 2018 (UTC)


text:
Discuss New Concept,New Technic,New Tools, Including EAI,BPM,SOA,Tibco,IBM MQ,Tuxedo, Cloud,Hadoop,NoSQL,J2EE,Ruby,Scala,Python, Performance,Scalability,Distributed,HA, Social Network,Machine Learning.

November 16, 2012  Tagged with: , , , ,

### Purpose

This guide provides an overview of the HDFS High Availability (HA) feature and how to configure and manage an HA HDFS cluster.

This document assumes that the reader has a general understanding of general components and node types in an HDFS cluster. Please refer to the HDFS Architecture guide for details.

### Background

Prior to Hadoop 2.0.0, the NameNode was a single point of failure (SPOF) in an HDFS cluster. Each cluster had a single NameNode, and if that machine or process became unavailable, the cluster as a whole would be unavailable until the NameNode was either restarted or brought up on a separate machine.

This impacted the total availability of the HDFS cluster in two major ways:

• In the case of an unplanned event such as a machine crash, the cluster would be unavailable until an operator restarted the NameNode.
• Planned maintenance events such as software or hardware upgrades on the NameNode machine would result in windows of cluster downtime.

The HDFS High Availability feature addresses the above problems by providing the option of running two redundant NameNodes in the same cluster in an Active/Passive configuration with a hot standby. This allows a fast failover to a new NameNode in the case that a machine crashes, or a graceful administrator-initiated failover for the purpose of planned maintenance.

### Architecture

In a typical HA cluster, two separate machines are configured as NameNodes. At any point in time, exactly one of the NameNodes is in an Active state, and the other is in a Standby state. The Active NameNode is responsible for all client operations in the cluster, while the Standby is simply acting as a slave, maintaining enough state to provide a fast failover if necessary.

In order for the Standby node to keep its state synchronized with the Active node, the current implementation requires that the two nodes both have access to a directory on a shared storage device (eg an NFS mount from a NAS). This restriction will likely be relaxed in future versions.

When any namespace modification is performed by the Active node, it durably logs a record of the modification to an edit log file stored in the shared directory. The Standby node is constantly watching this directory for edits, and as it sees the edits, it applies them to its own namespace. In the event of a failover, the Standby will ensure that it has read all of the edits from the shared storage before promoting itself to the Active state. This ensures that the namespace state is fully synchronized before a failover occurs.

In order to provide a fast failover, it is also necessary that the Standby node have up-to-date information regarding the location of blocks in the cluster. In order to achieve this, the DataNodes are configured with the location of both NameNodes, and send block location information and heartbeats to both.

It is vital for the correct operation of an HA cluster that only one of the NameNodes be Active at a time. Otherwise, the namespace state would quickly diverge between the two, risking data loss or other incorrect results. In order to ensure this property and prevent the so-called “split-brain scenario,” the administrator must configure at least one fencing method for the shared storage. During a failover, if it cannot be verified that the previous Active node has relinquished its Active state, the fencing process is responsible for cutting off the previous Active’s access to the shared edits storage. This prevents it from making any further edits to the namespace, allowing the new Active to safely proceed with failover.

### Hardware resources

In order to deploy an HA cluster, you should prepare the following:

• NameNode machines – the machines on which you run the Active and Standby NameNodes should have equivalent hardware to each other, and equivalent hardware to what would be used in a non-HA cluster.
• Shared storage – you will need to have a shared directory which both NameNode machines can have read/write access to. Typically this is a remote filer which supports NFS and is mounted on each of the NameNode machines. Currently only a single shared edits directory is supported. Thus, the availability of the system is limited by the availability of this shared edits directory, and therefore in order to remove all single points of failure there needs to be redundancy for the shared edits directory. Specifically, multiple network paths to the storage, and redundancy in the storage itself (disk, network, and power). Beacuse of this, it is recommended that the shared storage server be a high-quality dedicated NAS appliance rather than a simple Linux server.

Note that, in an HA cluster, the Standby NameNode also performs checkpoints of the namespace state, and thus it is not necessary to run a Secondary NameNode, CheckpointNode, or BackupNode in an HA cluster. In fact, to do so would be an error. This also allows one who is reconfiguring a non-HA-enabled HDFS cluster to be HA-enabled to reuse the hardware which they had previously dedicated to the Secondary NameNode.

### Deployment

#### Configuration overview

Similar to Federation configuration, HA configuration is backward compatible and allows existing single NameNode configurations to work without change. The new configuration is designed such that all the nodes in the cluster may have the same configuration without the need for deploying different configuration files to different machines based on the type of the node.

Like HDFS Federation, HA clusters reuse the nameservice ID to identify a single HDFS instance that may in fact consist of multiple HA NameNodes. In addition, a new abstraction called NameNode ID is added with HA. Each distinct NameNode in the cluster has a different NameNode ID to distinguish it. To support a single configuration file for all of the NameNodes, the relevant configuration parameters are suffixed with the nameservice ID as well as the NameNode ID.

#### Configuration details

To configure HA NameNodes, you must add several configuration options to your hdfs-site.xml configuration file.

The order in which you set these configurations is unimportant, but the values you choose for dfs.nameservices and dfs.ha.namenodes.[nameservice ID] will determine the keys of those that follow. Thus, you should decide on these values before setting the rest of the configuration options.

• dfs.nameservices- the logical name for this new nameserviceChoose a logical name for this nameservice, for example “mycluster”, and use this logical name for the value of this config option. The name you choose is arbitrary. It will be used both for configuration and as the authority component of absolute HDFS paths in the cluster.

Note: If you are also using HDFS Federation, this configuration setting should also include the list of other nameservices, HA or otherwise, as a comma-separated list.

<property>
<name>dfs.nameservices</name>
<value>mycluster</value>
</property>
• dfs.ha.namenodes.[nameservice ID]- unique identifiers for each NameNode in the nameserviceConfigure with a list of comma-separated NameNode IDs. This will be used by DataNodes to determine all the NameNodes in the cluster. For example, if you used “mycluster” as the nameservice ID previously, and you wanted to use “nn1″ and “nn2″ as the individual IDs of the NameNodes, you would configure this as such:
<property>
<name>dfs.ha.namenodes.mycluster</name>
<value>nn1,nn2</value>
</property>

Note: Currently, only a maximum of two NameNodes may be configured per nameservice.

• dfs.namenode.rpc-address.[nameservice ID].[name node ID]- the fully-qualified RPC address for each NameNode to listen onFor both of the previously-configured NameNode IDs, set the full address and IPC port of the NameNode processs. Note that this results in two separate configuration options. For example:
<property>
<value>machine1.example.com:8020</value>
</property>
<property>
<value>machine2.example.com:8020</value>
</property>

Note: You may similarly configure the “servicerpc-address” setting if you so desire.

• dfs.namenode.http-address.[nameservice ID].[name node ID]- the fully-qualified HTTP address for each NameNode to listen onSimilarly to rpc-address above, set the addresses for both NameNodes’ HTTP servers to listen on. For example:
<property>
<value>machine1.example.com:50070</value>
</property>
<property>
<value>machine2.example.com:50070</value>
</property>

Note: If you have Hadoop’s security features enabled, you should also set the https-address similarly for each NameNode.

• dfs.namenode.shared.edits.dir- the location of the shared storage directoryThis is where one configures the path to the remote shared edits directory which the Standby NameNode uses to stay up-to-date with all the file system changes the Active NameNode makes. You should only configure one of these directories. This directory should be mounted r/w on both NameNode machines. The value of this setting should be the absolute path to this directory on the NameNode machines. For example:
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>file:///mnt/filer1/dfs/ha-name-dir-shared</value>
</property>
• dfs.client.failover.proxy.provider.[nameservice ID]- the Java class that HDFS clients use to contact the Active NameNodeConfigure the name of the Java class which will be used by the DFS Client to determine which NameNode is the current Active, and therefore which NameNode is currently serving client requests. The only implementation which currently ships with Hadoop is the ConfiguredFailoverProxyProvider, so use this unless you are using a custom one. For example:
<property>
<name>dfs.client.failover.proxy.provider.mycluster</name>
</property>
• dfs.ha.fencing.methods- a list of scripts or Java classes which will be used to fence the Active NameNode during a failoverIt is critical for correctness of the system that only one NameNode be in the Active state at any given time. Thus, during a failover, we first ensure that the Active NameNode is either in the Standby state, or the process has terminated, before transitioning the other NameNode to the Active state. In order to do this, you must configure at least one fencing method. These are configured as a carriage-return-separated list, which will be attempted in order until one indicates that fencing has succeeded. There are two methods which ship with Hadoop: shell and sshfence. For information on implementing your own custom fencing method, see the org.apache.hadoop.ha.NodeFencer class.
• sshfence- SSH to the Active NameNode and kill the processThe sshfence option SSHes to the target node and uses fuser to kill the process listening on the service’s TCP port. In order for this fencing option to work, it must be able to SSH to the target node without providing a passphrase. Thus, one must also configure the dfs.ha.fencing.ssh.private-key-files option, which is a comma-separated list of SSH private key files. For example:
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>

<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/home/exampleuser/.ssh/id_rsa</value>
</property>

Optionally, one may configure a non-standard username or port to perform the SSH. One may also configure a timeout, in milliseconds, for the SSH, after which this fencing method will be considered to have failed. It may be configured like so:

<property>
<name>dfs.ha.fencing.methods</name>
</property>
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>
</property>
• shell- run an arbitrary shell command to fence the Active NameNodeThe shell fencing method runs an arbitrary shell command. It may be configured like so:
<property>
<name>dfs.ha.fencing.methods</name>
<value>shell(/path/to/my/script.sh arg1 arg2 ...)</value>
</property>

The string between ‘(‘ and ‘)’ is passed directly to a bash shell and may not include any closing parentheses.

The shell command will be run with an environment set up to contain all of the current Hadoop configuration variables, with the ‘_’ character replacing any ‘.’ characters in the configuration keys. The configuration used has already had any namenode-specific configurations promoted to their generic forms — for example dfs_namenode_rpc-address will contain the RPC address of the target node, even though the configuration may specify that variable as dfs.namenode.rpc-address.ns1.nn1.

Additionally, the following variables referring to the target node to be fenced are also available:

 $target_host hostname of the node to be fenced$target_port IPC port of the node to be fenced $target_address the above two, combined as host:port$target_nameserviceid the nameservice ID of the NN to be fenced $target_namenodeid the namenode ID of the NN to be fenced These environment variables may also be used as substitutions in the shell command itself. For example: <property> <name>dfs.ha.fencing.methods</name> <value>shell(/path/to/my/script.sh --nameservice=$target_nameserviceid $target_host:$target_port)</value>
</property>

If the shell command returns an exit code of 0, the fencing is determined to be successful. If it returns any other exit code, the fencing was not successful and the next fencing method in the list will be attempted.

Note: This fencing method does not implement any timeout. If timeouts are necessary, they should be implemented in the shell script itself (eg by forking a subshell to kill its parent in some number of seconds).

• fs.defaultFS- the default path prefix used by the Hadoop FS client when none is givenOptionally, you may now configure the default path for Hadoop clients to use the new HA-enabled logical URI. If you used “mycluster” as the nameservice ID earlier, this will be the value of the authority portion of all of your HDFS paths. This may be configured like so, in your core-site.xml file:
<property>
<name>fs.defaultFS</name>
<value>hdfs://mycluster</value>
</property>

#### Deployment details

After all of the necessary configuration options have been set, one must initially synchronize the two HA NameNodes’ on-disk metadata. If you are setting up a fresh HDFS cluster, you should first run the format command (hdfs namenode -format) on one of NameNodes. If you have already formatted the NameNode, or are converting a non-HA-enabled cluster to be HA-enabled, you should now copy over the contents of your NameNode metadata directories to the other, unformatted NameNode using scp or a similar utility. The location of the directories containing the NameNode metadata are configured via the configuration options dfs.namenode.name.dir and/or dfs.namenode.edits.dir. At this time, you should also ensure that the shared edits dir (as configured by dfs.namenode.shared.edits.dir) includes all recent edits files which are in your NameNode metadata directories.

At this point you may start both of your HA NameNodes as you normally would start a NameNode.

You can visit each of the NameNodes’ web pages separately by browsing to their configured HTTP addresses. You should notice that next to the configured address will be the HA state of the NameNode (either “standby” or “active”.) Whenever an HA NameNode starts, it is initially in the Standby state.

Now that your HA NameNodes are configured and started, you will have access to some additional commands to administer your HA HDFS cluster. Specifically, you should familiarize yourself with all of the subcommands of the “hdfs haadmin” command. Running this command without any additional arguments will display the following usage information:

Usage: DFSHAAdmin [-ns <nameserviceId>]
[-transitionToActive <serviceId>]
[-transitionToStandby <serviceId>]
[-failover [--forcefence] [--forceactive] <serviceId> <serviceId>]
[-getServiceState <serviceId>]
[-checkHealth <serviceId>]
[-help <command>]

This guide describes high-level uses of each of these subcommands. For specific usage information of each subcommand, you should run “hdfs haadmin -help <command>”.

• transitionToActive and transitionToStandby- transition the state of the given NameNode to Active or StandbyThese subcommands cause a given NameNode to transition to the Active or Standby state, respectively. These commands do not attempt to perform any fencing, and thus should rarely be used. Instead, one should almost always prefer to use the “hdfs haadmin -failover” subcommand.
• failover- initiate a failover between two NameNodesThis subcommand causes a failover from the first provided NameNode to the second. If the first NameNode is in the Standby state, this command simply transitions the second to the Active state without error. If the first NameNode is in the Active state, an attempt will be made to gracefully transition it to the Standby state. If this fails, the fencing methods (as configured by dfs.ha.fencing.methods) will be attempted in order until one succeeds. Only after this process will the second NameNode be transitioned to the Active state. If no fencing method succeeds, the second NameNode will not be transitioned to the Active state, and an error will be returned.
• getServiceState- determine whether the given NameNode is Active or StandbyConnect to the provided NameNode to determine its current state, printing either “standby” or “active” to STDOUT appropriately. This subcommand might be used by cron jobs or monitoring scripts which need to behave differently based on whether the NameNode is currently Active or Standby.
• checkHealth- check the health of the given NameNodeConnect to the provided NameNode to check its health. The NameNode is capable of performing some diagnostics on itself, including checking if internal services are running as expected. This command will return 0 if the NameNode is healthy, non-zero otherwise. One might use this command for monitoring purposes.

Note: This is not yet implemented, and at present will always return success, unless the given NameNode is completely down.

### Automatic Failover

#### Introduction

The above sections describe how to configure manual failover. In that mode, the system will not automatically trigger a failover from the active to the standby NameNode, even if the active node has failed. This section describes how to configure and deploy automatic failover.

#### Components

Automatic failover adds two new components to an HDFS deployment: a ZooKeeper quorum, and the ZKFailoverController process (abbreviated as ZKFC).

Apache ZooKeeper is a highly available service for maintaining small amounts of coordination data, notifying clients of changes in that data, and monitoring clients for failures. The implementation of automatic HDFS failover relies on ZooKeeper for the following things:

• Failure detection – each of the NameNode machines in the cluster maintains a persistent session in ZooKeeper. If the machine crashes, the ZooKeeper session will expire, notifying the other NameNode that a failover should be triggered.
• Active NameNode election – ZooKeeper provides a simple mechanism to exclusively elect a node as active. If the current active NameNode crashes, another node may take a special exclusive lock in ZooKeeper indicating that it should become the next active.

The ZKFailoverController (ZKFC) is a new component which is a ZooKeeper client which also monitors and manages the state of the NameNode. Each of the machines which runs a NameNode also runs a ZKFC, and that ZKFC is responsible for:

• Health monitoring – the ZKFC pings its local NameNode on a periodic basis with a health-check command. So long as the NameNode responds in a timely fashion with a healthy status, the ZKFC considers the node healthy. If the node has crashed, frozen, or otherwise entered an unhealthy state, the health monitor will mark it as unhealthy.
• ZooKeeper session management – when the local NameNode is healthy, the ZKFC holds a session open in ZooKeeper. If the local NameNode is active, it also holds a special “lock” znode. This lock uses ZooKeeper’s support for “ephemeral” nodes; if the session expires, the lock node will be automatically deleted.
• ZooKeeper-based election – if the local NameNode is healthy, and the ZKFC sees that no other node currently holds the lock znode, it will itself try to acquire the lock. If it succeeds, then it has “won the election”, and is responsible for running a failover to make its local NameNode active. The failover process is similar to the manual failover described above: first, the previous active is fenced if necessary, and then the local NameNode transitions to active state.

For more details on the design of automatic failover, refer to the design document attached to HDFS-2185 on the Apache HDFS JIRA.

#### Deploying ZooKeeper

In a typical deployment, ZooKeeper daemons are configured to run on three or five nodes. Since ZooKeeper itself has light resource requirements, it is acceptable to collocate the ZooKeeper nodes on the same hardware as the HDFS NameNode and Standby Node. Many operators choose to deploy the third ZooKeeper process on the same node as the YARN ResourceManager. It is advisable to configure the ZooKeeper nodes to store their data on separate disk drives from the HDFS metadata for best performance and isolation.

The setup of ZooKeeper is out of scope for this document. We will assume that you have set up a ZooKeeper cluster running on three or more nodes, and have verified its correct operation by connecting using the ZK CLI.

#### Before you begin

Before you begin configuring automatic failover, you should shut down your cluster. It is not currently possible to transition from a manual failover setup to an automatic failover setup while the cluster is running.

#### Configuring automatic failover

The configuration of automatic failover requires the addition of two new parameters to your configuration. In your hdfs-site.xml file, add:

 <property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>

This specifies that the cluster should be set up for automatic failover. In your core-site.xml file, add:

 <property>
<name>ha.zookeeper.quorum</name>
<value>zk1.example.com:2181,zk2.example.com:2181,zk3.example.com:2181</value>
</property>

This lists the host-port pairs running the ZooKeeper service.

As with the parameters described earlier in the document, these settings may be configured on a per-nameservice basis by suffixing the configuration key with the nameservice ID. For example, in a cluster with federation enabled, you can explicitly enable automatic failover for only one of the nameservices by setting dfs.ha.automatic-failover.enabled.my-nameservice-id.

There are also several other configuration parameters which may be set to control the behavior of automatic failover; however, they are not necessary for most installations. Please refer to the configuration key specific documentation for details.

#### Initializing HA state in ZooKeeper

After the configuration keys have been added, the next step is to initialize required state in ZooKeeper. You can do so by running the following command from one of the NameNode hosts.

$hdfs zkfc -formatZK This will create a znode in ZooKeeper inside of which the automatic failover system stores its data. #### Starting the cluster with start-dfs.sh Since automatic failover has been enabled in the configuration, the start-dfs.sh script will now automatically start a ZKFC daemon on any machine that runs a NameNode. When the ZKFCs start, they will automatically select one of the NameNodes to become active. #### Starting the cluster manually If you manually manage the services on your cluster, you will need to manually start the zkfc daemon on each of the machines that runs a NameNode. You can start the daemon by running: $ hadoop-daemon.sh start zkfc

If you are running a secure cluster, you will likely want to ensure that the information stored in ZooKeeper is also secured. This prevents malicious clients from modifying the metadata in ZooKeeper or potentially triggering a false failover.

In order to secure the information in ZooKeeper, first add the following to your core-site.xml file:

 <property>
<name>ha.zookeeper.auth</name>
<value>@/path/to/zk-auth.txt</value>
</property>
<property>
<name>ha.zookeeper.acl</name>
<value>@/path/to/zk-acl.txt</value>
</property>

Please note the ‘@’ character in these values — this specifies that the configurations are not inline, but rather point to a file on disk.

The first configured file specifies a list of ZooKeeper authentications, in the same format as used by the ZK CLI. For example, you may specify something like:

digest:hdfs-zkfcs:mypassword

…where hdfs-zkfcs is a unique username for ZooKeeper, and mypassword is some unique string used as a password.

Next, generate a ZooKeeper ACL that corresponds to this authentication, using a command like the following:

$java -cp$ZK_HOME/lib/*:$ZK_HOME/zookeeper-3.4.2.jar org.apache.zookeeper.server.auth.DigestAuthenticationProvider hdfs-zkfcs:mypassword output: hdfs-zkfcs:mypassword->hdfs-zkfcs:P/OQvnYyU/nF/mGYvB/xurX8dYs= Copy and paste the section of this output after the ‘->’ string into the file zk-acls.txt, prefixed by the string “digest:“. For example: digest:hdfs-zkfcs:vlUvLnd8MlacsE80rDuu6ONESbM=:rwcda In order for these ACLs to take effect, you should then rerun the zkfc -formatZK command as described above. After doing so, you may verify the ACLs from the ZK CLI as follows: [zk: localhost:2181(CONNECTED) 1] getAcl /hadoop-ha 'digest,'hdfs-zkfcs:vlUvLnd8MlacsE80rDuu6ONESbM= : cdrwa #### Verifying automatic failover Once automatic failover has been set up, you should test its operation. To do so, first locate the active NameNode. You can tell which node is active by visiting the NameNode web interfaces — each node reports its HA state at the top of the page. Once you have located your active NameNode, you may cause a failure on that node. For example, you can use kill -9 <pid of NN> to simulate a JVM crash. Or, you could power cycle the machine or unplug its network interface to simulate a different kind of outage. After triggering the outage you wish to test, the other NameNode should automatically become active within several seconds. The amount of time required to detect a failure and trigger a fail-over depends on the configuration of ha.zookeeper.session-timeout.ms, but defaults to 5 seconds. If the test does not succeed, you may have a misconfiguration. Check the logs for the zkfc daemons as well as the NameNode daemons in order to further diagnose the issue. ### Automatic Failover FAQ • Is it important that I start the ZKFC and NameNode daemons in any particular order?No. On any given node you may start the ZKFC before or after its corresponding NameNode. • What additional monitoring should I put in place?You should add monitoring on each host that runs a NameNode to ensure that the ZKFC remains running. In some types of ZooKeeper failures, for example, the ZKFC may unexpectedly exit, and should be restarted to ensure that the system is ready for automatic failover. Additionally, you should monitor each of the servers in the ZooKeeper quorum. If ZooKeeper crashes, then automatic failover will not function. • What happens if ZooKeeper goes down?If the ZooKeeper cluster crashes, no automatic failovers will be triggered. However, HDFS will continue to run without any impact. When ZooKeeper is restarted, HDFS will reconnect with no issues. • Can I designate one of my NameNodes as primary/preferred?No. Currently, this is not supported. Whichever NameNode is started first will become active. You may choose to start the cluster in a specific order such that your preferred node starts first. • How can I initiate a manual failover when automatic failover is configured?Even if automatic failover is configured, you may initiate a manual failover using the same hdfs haadmin command. It will perform a coordinated failover. ### BookKeeper as a Shared storage (EXPERIMENTAL) One option for shared storage for the NameNode is BookKeeper. BookKeeper achieves high availability and strong durability guarantees by replicating edit log entries across multiple storage nodes. The edit log can be striped across the storage nodes for high performance. Fencing is supported in the protocol, i.e, BookKeeper will not allow two writers to write the single edit log. The meta data for BookKeeper is stored in ZooKeeper. In current HA architecture, a Zookeeper cluster is required for ZKFC. The same cluster can be for BookKeeper metadata. For more details on building a BookKeeper cluster, please refer to the BookKeeper documentation The BookKeeperJournalManager is an implementation of the HDFS JournalManager interface, which allows custom write ahead logging implementations to be plugged into the HDFS NameNode. • BookKeeper Journal ManagerTo use BookKeeperJournalManager, add the following to hdfs-site.xml.  <property> <name>dfs.namenode.shared.edits.dir</name> <value>bookkeeper://zk1:2181;zk2:2181;zk3:2181/hdfsjournal</value> </property> <property> <name>dfs.namenode.edits.journal-plugin.bookkeeper</name> <value>org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager</value> </property> The URI format for bookkeeper is bookkeeper://[zkEnsemble]/[rootZnode] [zookkeeper ensemble] is a list of semi-colon separated, zookeeper host:port pairs. In the example above there are 3 servers, in the ensemble, zk1, zk2 & zk3, each one listening on port 2181. [root znode] is the path of the zookeeper znode, under which the edit log information will be stored. The class specified for the journal-plugin must be available in the NameNode’s classpath. We explain how to generate a jar file with the journal manager and its dependencies, and how to put it into the classpath below. • More configuration options • dfs.namenode.bookkeeperjournal.output-buffer-size- Number of bytes a bookkeeper journal stream will buffer before forcing a flush. Default is 1024.  <property> <name>dfs.namenode.bookkeeperjournal.output-buffer-size</name> <value>1024</value> </property> • dfs.namenode.bookkeeperjournal.ensemble-size- Number of bookkeeper servers in edit log ensembles. This is the number of bookkeeper servers which need to be available for the edit log to be writable. Default is 3.  <property> <name>dfs.namenode.bookkeeperjournal.ensemble-size</name> <value>3</value> </property> • dfs.namenode.bookkeeperjournal.quorum-size- Number of bookkeeper servers in the write quorum. This is the number of bookkeeper servers which must have acknowledged the write of an entry before it is considered written. Default is 2.  <property> <name>dfs.namenode.bookkeeperjournal.quorum-size</name> <value>2</value> </property> • dfs.namenode.bookkeeperjournal.digestPw- Password to use when creating edit log segments.  <property> <name>dfs.namenode.bookkeeperjournal.digestPw</name> <value>myPassword</value> </property> • dfs.namenode.bookkeeperjournal.zk.session.timeout- Session timeout for Zookeeper client from BookKeeper Journal Manager. Hadoop recommends that this value should be less than the ZKFC session timeout value. Default value is 3000.  <property> <name>dfs.namenode.bookkeeperjournal.zk.session.timeout</name> <value>3000</value> </property> • Building BookKeeper Journal Manager plugin jarTo generate the distribution packages for BK journal, do the following.$ mvn clean package -Pdist

This will generate a jar with the BookKeeperJournalManager, all the dependencies needed by the journal manager, hadoop-hdfs/src/contrib/bkjournal/target/hadoop-hdfs-bkjournal-VERSION.jar

Note that the -Pdist part of the build command is important, as otherwise the dependencies would not be packaged in the jar. The dependencies included in the jar are shaded to avoid conflicts with other dependencies of the NameNode.

• Putting the BookKeeperJournalManager in the NameNode classpathTo run a HDFS namenode using BookKeeper as a backend, copy the bkjournal jar, generated above, into the lib directory of hdfs. In the standard distribution of HDFS, this is at $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/ cp hadoop-hdfs/src/contrib/bkjournal/target/hadoop-hdfs-bkjournal-VERSION.jar$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/

• Current limitations1) NameNode format command will not format the BookKeeper data automatically. We have to clean the data manually from BookKeeper cluster and create the /ledgers/available path in Zookeeper. —- $zkCli.sh create /ledgers 0$ zkCli.sh create /ledgers/available 0 —- Note: bookkeeper://zk1:2181;zk2:2181;zk3:2181/hdfsjournal The final part /hdfsjournal specifies the znode in zookeeper where ledger metadata will be stored. Administrators may set this to anything they wish.

2) Security in BookKeeper. BookKeeper does not support SASL nor SSL for connections between the NameNode and BookKeeper storage nodes.

3) Auto-Recovery of storage node failures. Work inprogress BOOKKEEPER-237. Currently we have the tools to manually recover the data from failed storage nodes.


text:
# Printing bibliography with biblatex in Tufte-handout fullwidth environment

I'm trying to print the bibliography (using biblatex) inside the fullwidth environment. But when I place the \printbibliography command inside the fullwidth environment, every line other than the very first gets indented.

As far I as can tell, biblatex prints bibliographies as follows:

1. The first line of each item is flushed left.

2. Any other line of each item is indented using \bibhang.

If I simply place \printbibliography at the end of my file, I get the expected behavior:

If instead I place it inside a fulldiwth environment, every line in the bibliography other than the first one gets indented the same amount.

Here's a minimal example

\documentclass[nobib]{tufte-handout}
\usepackage[style=authoryear-comp]{biblatex}
\begin{filecontents*}{mybib5.bib}
@MISC{Volkmann,
author = {Volkmann, Albert},
title = {My Title},
year = {2005},
sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.}
}

@MISC{Caesar,
author = {Caesar, Gaius J.},
title = {My long title},
year = {2005},
sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.}
}

\end{filecontents*}
\title{Test}
\author{Me}
\bibliography{mybib5}
\begin{document}
\maketitle
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut  aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. \cite{Volkmann,Caesar}

\begin{fullwidth}
\printbibliography
\end{fullwidth}
\end{document}

The output looks like so:

If I set \parindent to 0pt right before \printbibliography, I get the same result. If instead I set \bibhang to 0pt right before \printbibliography, so that

\begin{fullwidth}
\printbibliography
\end{bibliography}

is replaced by:

\begin{fullwidth}
\bibhang=0pt
\printbibliography
\end{bibliography}

no line in the bibliography is indented:

What I can't get is the default behavior (first line of each item flushed left, every additional line indented) within the fullwidth environment.

Any thoughts?

UPDATE: I'm using version 3.5.2 of tufte-common.def.

-
My biblatex and natbib hate each other, so I can't even compile this, sorry. :) –  Frg Mar 1 '12 at 0:04
Might want to add that you're using a revised copy of tufe-common.def. What class layout options are you actually using? The twoside and symmetric settings complicate the problem. –  Audrey Mar 2 '12 at 17:27
Good point, @Audrey. Yeah, I'm using v. 3.5.2. I was not using any options; this was the behavior after compiling the files above with a clean install of v. 3.5.2. –  apc Mar 2 '12 at 22:54

With the default tufte-latex layout, you can just extend the right margin inside the bibliography environment.

\documentclass[nobib]{tufte-handout}
\usepackage{hyphenat}
\usepackage[american]{babel}
\usepackage{csquotes}
\usepackage[style=authoryear-comp,autocite=footnote]{biblatex}

\defbibenvironment{bibliography}
{\list{}{%
\leftmargin\bibhang
\itemindent-\leftmargin
\itemsep\bibitemsep
\parsep\bibparsep}}
{\endlist}
{\item}

\begin{document}
Filler text \autocite{companion,ctan,markey,aristotle:poetics,aristotle:rhetoric}.
\printbibliography
\end{document}

With the layout settings twoside,symmetric something like tufte-latex's fullwidth environment is needed, but it won't work if the bibliography breaks across pages. For a single bibliography contained in one page, the following should give you proper indentation inside the fullwidth environment.

\defbibenvironment{bibliography}
{\list{}{%
\leftmargin\bibhang
\itemindent-\leftmargin
\listparindent\itemindent
\parsep\bibparsep}}
{\endlist}
{\item}

The more general case can be handled using the mdframed package and some spacing trickery with \AtEveryBibitem.

\documentclass[nobib,twoside,symmetric]{tufte-handout}
\usepackage{hyphenat}
\usepackage[american]{babel}
\usepackage{csquotes}
\usepackage[style=authoryear-comp]{biblatex}
\usepackage{mdframed}

\newmdenv[skipabove=3.5ex plus 1ex minus .2ex,% Equal to section title spacing
innerleftmargin=0pt,innerrightmargin=0pt,%
innerbottommargin=0pt,innertopmargin=0pt,linewidth=0pt,innermargin=0pt,%
outermargin=\dimexpr-\marginparwidth-\marginparsep\relax]{mdfullwidth}

\defbibenvironment{bibliography}
{\list{}{\parsep\bibparsep}}
{\endlist}
{\item}

\AtEveryBibitem{\hskip-\bibhang}

\begin{document}
\null\vfill
One of the most prominent and distinctive features of this style is the
extensive use of sidenotes.\footcite{companion,ctan,markey,aristotle:rhetoric} There is a wide margin to provide ample room for sidenotes and small figures. Any footnotes will automatically be converted to sidenotes.
\begin{mdfullwidth}
\printbibliography[type=online]
\end{mdfullwidth}
\pagebreak
One of the most prominent and distinctive features of this style is the
extensive use of sidenotes.\footcite{companion,ctan,markey,aristotle:poetics} There is a wide margin to provide ample room for sidenotes and small figures. Any footnotes will automatically be converted to sidenotes.
\begin{mdfullwidth}
\printbibliography[nottype=online]
\end{mdfullwidth}
\end{document}

-
Works like a charm. Thanks! –  apc Mar 6 '12 at 17:04