## Twenty Tips For Interpreting Scientific Claims

It’s science!

Title of today’s post is taken from article of the same name in Nature by William Sutherland, David Spiegelhalter, Mark Burgman. Several readers asked me to comment.

I’ll assume you’ve read the original. I kept the same order and wording as their points, and try not to repeat any of their good points.

Differences and chance cause variation. Chance can’t and doesn’t cause anything. Chance isn’t a thing, therefore it can never be a cause. Differences don’t cause things per se: things do (sizes of differences can certainly change rates of change). We cannot always identify causal agents, just correlates of change.

No measurement is exact. Well, not quite, but I take their point. Measurement error is vastly more prevalent than acknowledged and almost never accounted for. Leading to…can you guess? Over-certainty.

Bias is rife. Amen and amen. But, just like admonishing the public by reminding them they look ugly in jeans, they always think it’s the other guy and not them. Yes, you, even you, are biased. Even you. And you. Even if you’re part of a team that won prizes.

Bigger is usually better for sample size. Indeed, except for cost and the possibility of being overwhelmed or misled by errata, bigger is always better.

Correlation does not imply causation. But the opposite is true: causation causes correlation. People often forget the distinction between ontology and epistemology. This is also my fault for not making this distinction clearer more often. Most probability models are epistemological, meaning they say what are the changes in probability of some outcome given changes in input variables. The problem comes when people interpret the changing probabilities of the outcome as being caused by the input variables, which is usually not true.

Regression to the mean can mislead. See this on the so-called Sports Illustrated curse.

Extrapolating beyond the data is risky. The reason is probability models are usually not causative and even when they are few check them for accuracy (everybody checks them for fit, via p-values, posteriors and the like).

Beware the base-rate fallacy. Think of it this way. If you’re forecasting “No rain” for Tucson each day, you’re likely to be right most of the time. But your boast carries little importance. Try the same forecast for Norman, Oklahoma and you’re accuracy heads south. This is why we should speak of skill—the improvement over naive predictions—instead of accuracy rates. Right, climatologists?

Controls are important. The only thing wrong with this point is that important should read everything. The more you can control, the closer your model comes to causality. Problem is that controlling “everything” in human interactions or in anything contingent is impossible. It will always—as in always—be possible that something other than what we thought caused the outcome.

Randomization avoids bias. No. Randomization is not a property. It “gives” nothing to your results. “Randomization” belongs to the old days of magical thinking. Rather, assigning control of an experiment to persons without a financial or emotional interest reduces but cannot avoid bias. That residual bias exists is why there are always calls for replication.

Seek replication, not pseudoreplication. And speaking of replication… Listen up sociologists, psychologists, and so on: It is not a replication unless the experiment is repeated in exactly the same way where they only differences are those things you could not control in the first experiment. “More or less” the same way is not exactly the same way and is therefore not a replication. A mass of published literature on the same subject is only a weak indicator of truth. Who remembers frontal lobotomies, etc., etc., etc.?

Scientists are human. And because they are typically in positions commanding money and people, they fall prey more often to the standard sins.

Significance is significant. No, it is not, or at least not necessarily. “Significance” means attaining a wee p-value, one less than the magic number. And this result may not have and usually does not have practical bearing on questions of interest about the thing at hand. Finding a wee p-value is child’s play. Finding something useful to say is far harder.

Separate no effect from non-significance. Here I must quote: “The lack of a statistically significant result (say a P-value > 0.05) does not mean that there was no underlying effect: it means that no effect was detected.” This is only partially true. Lack of a wee p-value might mean the effect was there but undetected. On the other hand, the effect might be there and detectable, too. It’s just the p-values are terrible at discovering which situation we’re in. An effect without a wee p-value may still be important. If instead we looked at probability models as they should be looked at, as predictive statements, we could say more.

Effect size matters. Wee p-values alone mean nothing. Repeat that until you get sick of repeating it. This is another call for predictive analytics.

Study relevance limits generalizations. It’s funny how many reporters never read the papers they report on.

Feelings influence risk perception. And this is because feelings are part of what we risk! Money, after all, is only a crude device to measure our feelings. And just because you hate fat people eating transfats does not mean the risk of disease from eating transfats is high. And just because you hate smoking does not mean that “second-hand” smoke is perilously dangerous, etc., etc.

Dependencies change the risks. Try not to look at anything in isolation, unless the thing is amenable to isolation. Dice come to mind. The changes that await us when global warming finally strikes (soon, soon) do not.

Data can be dredged or cherry picked. “Big data” anyone? One thing Big Data guarantees is shocked looks on the face of managers who were certain sure they picked up a “significant” signal in their gleaming, massive datasets.

Extreme measurements may mislead. Like I always say, any survey or result is true conditional on the set of premises belonging to the experiment. Vary any of these premises, the result no longer holds. The more premises, i.e. conditions, there are, the greater the chance the results are not meaningful beyond the realm of this single experiment. Journalists often change this premises in their reporting; but to be fair, so do many scientists when summarizing their work. Memorize this.

## Friedrich A. Hayek’s Lecture “The Pretense of Knowledge”

Somehow Hayek wrote his multitudinous works wearing a suit.

This was reprinted in the Wall Street Journal over the weekend. I’ve chopped it into parts for commenting.

To act on the belief that we possess the knowledge and the power which enable us to shape the processes of society entirely to our liking, knowledge which in fact we do not possess, is likely to make us do much harm. In the physical sciences there may be little objection to trying to do the impossible; one might even feel that one ought not to discourage the over-confident because their experiments may after all produce some new insights. But in the social field the erroneous belief that the exercise of some power would have beneficial consequences is likely to lead to a new power to coerce other men being conferred on some authority.

If a physics experiment goes south, all that happens is some money is wasted but with the reasonable chance something is learned. Not always, of course. Think of the “cold” fusion hiccup twenty years ago. But when an, as Mill put it, “experiment in living” sours people suffer. And nothing is learned.

The love of theory is far too strong in economics, sociology, education and the like for observation to wound belief. In physics, what has gone wrong is usually identifiable, but in the “nudging” sciences the evidence is always somewhat ambiguous. There is aways wiggle room in whatever happens that the theory which drove the experiment might be true. And this is enough.

Human society is so unfathomably complex that all experiments should be approached with trepidation and fear and with an “out,” a way to revert, if at all possible, to the old ways. The essence of at least one definition of conservatism it that “change we can believe in” just for the sake of change is more likely to lead to grief than to happiness. There is a vast amount of wisdom packed into tradition which should not be overthrown lightly.

Even if such power is not in itself bad, its exercise is likely to impede the functioning of those spontaneous ordering forces by which, without understanding them, man is in fact so largely assisted in the pursuit of his aims. We are only beginning to understand on how subtle a communication system the functioning of an advanced industrial society is based — a communications system which we call the market and which turns out to be a more efficient mechanism for digesting dispersed information than any that man has deliberately designed.

The (highly degreed, well-placed) bureaucrat looks at the market and says to himself, “There’s no way to understand this, therefore it is not understandable. Therefore I must direct it, then I will understand it.” The conclusion has elements of truth—it is easier to claim to understand what one directs—but it does not follow from the premise. And anyway, we’re right back at the beginning. The directions (regulations, laws, taxes) cause changes nobody could have foreseen. The urge to perfection is never humiliated by history.

If man is not to do more harm than good in his efforts to improve the social order, he will have to learn that in this, as in all other fields where essential complexity of an organized kind prevails, he cannot acquire the full knowledge which would make mastery of the events possible. He will therefore have to use what knowledge he can achieve, not to shape the results as the craftsman shapes his handiwork, but rather to cultivate a growth by providing the appropriate environment, in the manner in which the gardener does this for his plants.

Hayek follows Burke and says change should be small, incremental, with the acknowledgement, like in gardening, that despite our best efforts, a poor harvest is more likely to result from enthusiasm than from conservatism.

There is danger in the exuberant feeling of ever growing power which the advance of the physical sciences has engendered and which tempts man to try, “dizzy with success,” to use a characteristic phrase of early communism, to subject not only our natural but also our human environment to the control of a human will. The recognition of the insuperable limits to his knowledge ought indeed to teach the student of society a lesson of humility which should guard him against becoming an accomplice in men’s fatal striving to control society — a striving which makes him not only a tyrant over his fellows, but which may well make him the destroyer of a civilization which no brain has designed but which has grown from the free efforts of millions of individuals.

That cannot be said better, but just as a for instance, note that the Expertism embraced by all modern liberal democratic societies has gone so far that government social services feels it can judge the mental health of tourists, as in Britain, where an Italian woman having a panic attack was forcibly sedated and cut open, her baby removed from her womb, taken away, and not given back. The experts feel they would do a better job raising this non-citizen’s baby than its mother. Link.

## On Dressing Well

Make that, “On Dressing Not So Well.” Here is a picture from a mall in Santa Monica, first appearing in the Los Angeles Times.

Tis always the season for looking poorly.

The article which surrounded this picture is titled “The season of excess begins”, and the caption of the photo aptly begins “Black Friday shoppers are seen…”

Truly it was a black day, and not just because of the fights, riots, and nauseating avarice displayed by many shoppers. No. It was the visual assault that cast dark shadows.

Let’s engage in some amateur photo reconnaissance. The location? A mall with top-end stores, meaning that to shop in them one has to have piles of the green stuff at arm’s reach. These are not poor people. Further proof: nobody looks like they’re going hungry. There are no beggars. The accoutrement of the mall—the tree, the giant present—are of superior quality. The buildings themselves are clean, sharp.

So money isn’t a problem. What is?

What first struck me were the two gentleman in the center. One is wearing what looks like a modified logo of an oil company and a hat advertising his preference in sports, which is evidently something he wished others to know about, the depth of his ardency so strong that he put it where everyone must see it. His neighbor wears a hipster hat and blue rumpled t-shirt on which is imprinted a message thankfully too small for us to read, but one which this man thought of such important that he should emblazon it on his chest.

Both men are wearing jeans, which is no story. Therefore, that they look sloppy is a given. The jeans are too long and gather in folds around their feet, which are shod with tennis shoes the shape of which would have been familiar to Frankenstein’s monster. We can bet that, if asked, both would say they are “comfortable.”

Next are the two older women leading them on either side. Both are wearing jeans. The pair of the leftmost lady’s are faded, as if she could not afford a newer pair. The rightmost lady has done herself no favor and has chosen a pair which are about as unflattering as is possible. But then jeans only flatter bodies which don’t need praise. Otherwise, they are ugly and make the wearers ugly. Yes, that means you, too. (Yes, even you, who thought the “you” of the previous sentence wasn’t really you. It was.)

The blouse of the rightmost lady resembles one of those hidden art pictures which if stared at reveal a rocketship or a man riding a horse. The blouse of the other lady looks to be a smock, such as one might wear while casting pottery.

There is no need to continue. Every person, without exception, looks terrible. No one picked something which fit his or her body. Everybody’s clothes are ugly, sad, unkempt. Everybody did their damnedest to appear like an unthinking high school sophomore. In this and only this did they succeed.

The clothes are the visual equivalent of popular, which is to say, bad music. There is no pleasure to be had in looking at anybody.

## What To Do On Black Friday

This, one of our most sacred days of the year, can be exhausting, especially in the choices one must make. Should one wake at 3am or should one even sleep? Should one take part in a riot, tumbling headlong into a store to be the first to secure the new iWhatsit, which is rumored to be 0.00132″ slimmer than last month’s model? Or should one circle the mall’s outer limits for hours spying for a spot to park?

Just what form should the joy of the upcoming and unnamed and soon to be unnameable holiday take? Well, here are two “black” ideas.

1. Go see a priest. Confessions heard daily.

Truly, confession is good for the soul.

2. Read Wordsworth, the old curmudgeon. Nothing black here, but the sheer absence of it can put one in mind of it.

England, 1802

O FRIEND! I know not which way I must look
For comfort, being, as I am, opprest,
To think that now our life is only drest
For show; mean handy-work of craftsman, cook,
Or groom!—We must run glittering like a brook
In the open sunshine, or we are unblest:
The wealthiest man among us is the best:
No grandeur now in nature or in book
Delights us. Rapine, avarice, expense,
This is idolatry; and these we adore:
Plain living and high thinking are no more:
The homely beauty of the good old cause
Is gone; our peace, our fearful innocence,
And pure religion breathing household laws.

— William Wordsworth. 1770-1850

3. Or chat with a nun! Develop the habit.

Calling all souls!

Update Our first story. Violence flares as shoppers slug it out for the best Black Friday deals.

## Old Lodge Skins’ Prayer Of Thanksgiving

A real Little Big Man existed. Bottom row, second from right.

From Little Big Man, Thomas Berger. Eschew the movie, which shares only the title and the names of a few characters in the book, which is the moral and historical opposite of the (more or less) politically correct film.

Also highly recommended (as orientation) is The Fighting Cheyennes by George Bird Grinnell, who was born in 1849 and who wrote the book in 1915 (it’s still in print). It is a non-patronizing, non-romantic look at the battles the Cheyenne fought in, as much as possible, their own words.

The Cheyenne are Human Beings. They call themselves that because they are and were, and because they act and acted just like human beings everywhere, including those white and black versions with which they had and have many strange interactions. Little Big Man was born white and named Jack Crabbe, but through a series of curious incidents was raised by the Cheyenne during the time in which that great nation was going into eclipse.

Berger wrote Little Big Man at a time (1964) when white boys still wanted to run off and be Indians. (Nearly twenty years later, Grizzly Adams fulfilled the same function.) Some call Berger’s work “comic”, which is the most inapt description which could be imagined.

Old Lodge Skins was Little Big Man’s adoptive grandfather. The scene takes place shortly after the Battle of Little Big Horn slash Battle at the Greasy Grass. There is much in this prayer that still works.

Then he commenced to pray to the Everywhere Spirit in the same stentorian voice, never sniveling but bold and free.

“Thank you for making me a Human Being! Thank you for helping me become a warrior! Thank you for all my victories and for all my defeats. Thank you for my vision, and for the blindness in which I saw further.

“I have killed many men and loved many women and eaten much meat. I have also been hungry, and I thank you for that and for the added sweetness that food has when you receive it after such a time.

“You make all things and direct them in their ways, O Grandfather, and now you have decided that the Human Beings will soon have to walk a new road. Thank you for letting us win once before that happened. Even if my people must eventually pass from the face of the earth, they will live on in whatever men are fierce and strong. So that when women see a man who is proud and brave and vengeful, even if he has a white face, they will cry: ‘That is a Human Being!’…”

I stood there in awe and Old Lodge Skins started to sing, and when the cloud arrived overhead, the rain started to patter across his uplifted face, mixing with the tears of joy there.

It might have been ten minutes or an hour, and when it stopped and the sun’s setting rays cut through, he give his final thanks and last request.

“Take care of my son here,” he says, “and see that he does not go crazy.”

He laid down then on the damp rocks and died right away. I descended to the treeline, fetched back some poles, and built him a scaffold. Wrapped him in the red blanket and laid him thereon. Then after a while I started down the mountain in the fading light.

Posted in Fun

## What Random Means In Random Number Generation

Dice throws appear random because their outcomes cannot be predicted naively. But knowing the physics and initial conditions, they can be predicted, and so are not truly random.

It’s simple, really. A “random” number generator spits out a string of numbers or characters from some set, say c1, c2,…, cp, where each of these is a character out of p total. Example: c1 = ‘a’, c2 = ‘b’, etc.

Before the generator starts, using only the premises that the generator can produce only these p characters, we deduce the probability that the first character is ci (for any i in 1 to p) as 1/p.

Start the generator. The characters roll out one at a time. To be be considered “random”, this condition must hold:

Pr( Ct = ci | C1, C2, Ct-1, generator premises) = 1/p for all t and all i,

where I mean by this notation that the probability the next character (Ct) will be one of the set is 1/p no matter how far along in the series we are (short of infinity) and no matter what character we consider. The “generator premises” are those which state only the characters c1 through cp are available, etc.

In other words, if there is any information in the series to date that allows us to calculate any probability other than 1/p for Ct, then the series is not “random.” Be clear that “allows us” means “logically possible” and not necessarily practical or in-practice possible. There may be some existence proof which says something like “Given this generator and an output series like this-and-such, the probability of Ct is x, which does not equal 1/p”. We may never see the this-and-such series, but still if the output series is possible, then the output is not “random.”

Now don’t start going all frequentist on me and say, “Look here, Briggs, you fool. I’ve ran the generator a finite number of times and the relative frequencies of the observed Ct don’t match 1/p.” I’d reply, “Get back to me when you reach an infinite number of times.”

Run the generator for just one character. The observed relative frequencies will be 0, 0, …, 1, …, 0, where it’s 0s for all ci except the 1 character which showed. What does this prove? Next to nothing. Probability just isn’t relative frequency, but relative frequency can match the probability. Probabilities predict relative frequencies. In that spirit, we know the series is not “random” if we can do a better job predicting the series than saying the probability of the next character is 1/p (for any character).

But I see the idea of relative frequency is still alluring. Perhaps this is why. There is in math the idea of a “normal” number, unfortunately named because “normal” in probability means something else. A normal string or number is one in which the digits/characters repeat equally often, yea even unto infinity. Examples: 0.111222333444555666777888999000111222… and ‘abcabcabcabc…’ (we know the numbers are limited to digits 0-9, but here I limit the characters to a,b,c).

These normal numbers are in no sense “random”, because if you know where you are you know with certainty what the next digit or character is. Plus, there are some technical ideas, where a number may be normal in one base (say 10 or 2) but not normal in another base. Here is an example of a number, Stoneham’s constant, which is normal in one base but not another. So “normal” does not imply random, but we have the sense that “random” implies “normal”, which it does.

Truly “random” numbers are probably (I don’t believe there is a proof) normal in any base. Another way to put it, in terms on information, is to say that the number cannot be compressed (in any base); that is, speaking loosely, that it takes just as many characters to compress the number as to display it. The above-linked article gives some hints on the “randomness” of π, which appears normal in many (all?) bases and which cannot be compressed. So where do the digits of π and other transcendental numbers come from? Only God knows.

Last point: many “random” number generators—where by now we see that “random” merely means unknown or unpredictable in some sense—are wholly predictable, they are fully deterministic. Start at a known output and the path can be predicted with certainty. These are called “pseudo-random” generators because the numbers only appear unpredictable.

And what does that mean? Appear unpredictable? Well, it means not-random, that we can prove that a set of premises exist which predict the series perfectly. The difference between a “random” generator is that we can prove no such set of premises exist.

## Winner Announced in What Do You Call A Believer In Scientism Contest

At least they’ll subsidize your cane.

Last week we asked what is the best word or term for a die-hard believer in scientism? The response was overwhelming!

More than 50 people entered some 100 words. I grouped these into three-plus-one categories: Honorable Mentions, Runner Ups, and Top 10. But there can be only one! Winner, that is, who will receive a Kindle copy of Iain Murray’s not-to-be-missed Stealing You Blind.

All complaints or suggestions about the entries, rules, or my judging may be entered at this site.

Special Mention

Thinkologist (davebowne).

Love it, but I was saving this for a special use on certified Experts. So it was not in the running for the contest.

Honorable Mentions

In no particular order:

Evidencist (Francsois). Inscienaty, Narrow-Minder, Rationer, Reasoner, Sciencerapist (Pedro Erik). Science Zombie (JohnK; Davebowne). Bright (Thinkling; Mariner). Supercilioust (HowardW). Scimoron (GaryL). Positivist (Philip Neal). Psysciphilliac (BradH). Sheldonists, Optimists, Triverist (Paul Murphy). Knowlatan, Pedantocrat, Patronisiac (Hamish McCallum). Unscientific Hucksterists, Philistine Scientists, Scientific Excrescence, Vulgar Scientists (Jim Fedako). Scatomancer, Weepees (Bruce Foutch). Scisyphist, Scisyphean (Jonathan S). Scienterrific (An Engineer). Scienzealot (Charles Boncelet), Scientomancer (hmi). Anthropodogmatists (Pangloss). Scinsist (Jeffrey).

Runner Ups

In no particular order:

Scientite (Adrian Hilton). Fatuotist (as in “fatuous”; Bob Mrotek). Scientizer, Scientaster, Scifollogist, Sciphist (Jester). Spockists (Toby Young). Scientician (TheRealAaron). Sciencista (Mike Anderson). Scientocracy (K). Sciphist (John Baglien). Technogogue, Techogogue, Scientologue (Mark Webster). Scientificant, Saintificist, Saintologist, Scientificist (Bob). Aoristicist (Don Jackson). Scienticist (Edmund Kartoffel; David). Scientipher, Scientifie (Mike B). Scientient (Andy). Scientismion, Scientismidel, (Aloysius Hogan, who had 42 generated entries, such as Scientismafuego derived from Cacafuego, a ship all fans of Patrick O’Brian will recognize; only the top are shown).

Top 10

In winning order:

1. Scientificalist (Ye Olde Statisician)
2. Obfuscator Scienista (Bruce Foutch)
4. Sciphiliac (Rich)
5. Scientophile (Andy)
6. Scientiscubus (Aloysius Hogan)
7. Sighintists (John M)
8. Scientocrat (K)
It is not only that Science cannot answer these questions now, but that it never can. All these and many more are forever beyond the reach of empiricism. There is no observation in the universe, nor can there be, nor will there ever be, which proves $e^{i\pi} = -1$. It is impossible to peer at the Unmoved Mover, yet He must be there or, quite literally, nothing would happen.