The Fourth Quadrant: A map of the limits of statistics

Edge essay by Nassim Taleb:

http://www.edge.org/3rd_culture/tale...b08_index.html

Quote:

---

Statistical and applied probabilistic knowledge is the core of knowledge; statistics is what tells you if something is true, false, or merely anecdotal; it is the "logic of science"; it is the instrument of risk-taking; it is the applied tools of epistemology; you can't be a modern intellectual and not think probabilistically—but... let's not be suckers. The problem is much more complicated than it seems to the casual, mechanistic user who picked it up in graduate school. Statistics can fool you. In fact it is fooling your government right now. It can even bankrupt the system (let's face it: use of probabilistic methods for the estimation of risks did just blow up the banking system).

---

Intro by John Brockman:

Quote:

---

Introduction

When Nassim Taleb talks about the limits of statistics, he becomes outraged. "My outrage," he says, "is aimed at the scientist-charlatan putting society at risk using statistical methods. This is similar to iatrogenics, the study of the doctor putting the patient at risk." As a researcher in probability, he has some credibility. In 2006, using FNMA and bank risk managers as his prime perpetrators, he wrote the following:

The government-sponsored institution Fannie Mae, when I look at its risks, seems to be sitting on a barrel of dynamite, vulnerable to the slightest hiccup. But not to worry: their large staff of scientists deemed these events "unlikely."

In the following Edge original essay, Taleb continues his examination of Black Swans, the highly improbable and unpredictable events that have massive impact. He claims that those who are putting society at risk are "no true statisticians", merely people using statistics either without understanding them, or in a self-serving manner. "The current subprime crisis did wonders to help me drill my point about the limits of statistically driven claims," he says.

Taleb, looking at the cataclysmic situation facing financial institutions today, points out that "the banking system, betting against Black Swans, has lost over 1 Trillion dollars (so far), more than was ever made in the history of banking".

But, as he points out, there is also good news.

We can identify where the danger zone is located, which I call "the fourth quadrant", and show it on a map with more or less clear boundaries. A map is a useful thing because you know where you are safe and where your knowledge is questionable. So I drew for the Edge readers a tableau showing the boundaries where statistics works well and where it is questionable or unreliable. Now once you identify where the danger zone is, where your knowledge is no longer valid, you can easily make some policy rules: how to conduct yourself in that fourth quadrant; what to avoid.

—John Brockman

NASSIM NICHOLAS TALEB, essayist and former mathematical trader, is Distinguished Professor of Risk Engineering at New York University’s Polytechnic Institute. He is the author of Fooled by Randomness and the international bestseller The Black Swan.

---

It's a long read, but a good one. Should of course be taken with a pinch of salt. He can be a bit of a jerk :o

Any particular reason he didn't get this published in a math or statistics journal? Where actual experts on the topic would get to agree that he's not spouting BS?

It could be that he antagonises experts, specifically experts in finance, politics and history. It could also be that this is an essay written for Edge that highlights fairly informally one aspect of something he's been pursuing more rigorously for a long time.

http://en.wikipedia.org/wiki/Nassim_...l_publications

He's hardly been hiding his ideas from mathematicians and statisticians, even though he's new to me and perhaps to you :)

One example: http://www.fooledbyrandomness.com/TAS.pdf

I stand corrected, though the guy does seem to make too many definitive statements about areas that are less than settled.

Just to be clear, he isn't trying to launch some alternative discipline of statistics. His scorn is directed towards those who use statistics inappropriately and with dire consequences. I think anyone here can agree with the problems that stem from people using tools they don't really understand on material they don't really understand to make decisions they really don't understand.

His approach seems to be that if you're uncertain of something, you shouldn't do it. The general approach in the social sciences is to acknowledge potential pitfalls, deal with them the best you can, but do the analysis anyway. I think he'd be listened to much more if he tried to help social scientists do the latter better instead of just mocking them.

Quote:

---

Originally Posted by Loki

I stand corrected, though the guy does seem to make too many definitive statements about areas that are less than settled.

---

Like I said, should be taken with a pinch of salt :-D I don't how much of his cockiness is an affectation or a pedagogical tool and how much of it represents his real attitudes. He is, ostensibly, a semi-skeptic. In his more personal writings he occasionally pokes fun at his own over-confidence and incompetence so I'm guessing he's not particularly extremist.

The claims are fairly conventional. His most salient point wrt the subject of the essay is that much of our activity occurs in domains we have little hope of adequately understanding using the tools presently available to us, and that we should therefore set ourselves up to be as resilient as possible (within reason).

Technical appendix: http://www.fooledbyrandomness.com/EDGE/index.html

Another excerpt, because it is a longish read compared to the typical article posted here:

Quote:

---

Phronetic Rules: What Is Wise To Do (Or Not Do) In The Fourth Quadrant

1) Avoid Optimization, Learn to Love Redundancy. [...] Only fools (such as Banks) optimize, not realizing that a simple model error can blow through their capital (as it just did). In one day in August 2007, Goldman Sachs experienced 24 x the average daily transaction volume—would 29 times have blown up the system? The only weak point I know of financial markets is their ability to drive people & companies to "efficiency" (to please a stock analyst’s earnings target) against risks of extreme events.

Indeed some systems tend to optimize—therefore become more fragile. Electricity grids for example optimize to the point of not coping with unexpected surges—Albert-Lazlo Barabasi warned us of the possibility of a NYC blackout like the one we had in August 2003. Quite prophetic, the fellow. Yet energy supply kept getting more and more efficient since. Commodity prices can double on a short burst in demand (oil, copper, wheat) —we no longer have any slack. Almost everyone who talks about "flat earth" does not realize that it is overoptimized to the point of maximal vulnerability.

Biological systems—those that survived millions of years—include huge redundancies. Just consider why we like sexual encounters (so redundant to do it so often!). Historically populations tended to produced around 4-12 children to get to the historical average of ~2 survivors to adulthood.

Option-theoretic analysis: redundancy is like long an option. You certainly pay for it, but it may be necessary for survival.

2) Avoid prediction of remote payoffs—though not necessarily ordinary ones. Payoffs from remote parts of the distribution are more difficult to predict than closer parts.

A general principle is that, while in the first three quadrants you can use the best model you can find, this is dangerous in the fourth quadrant: no model should be better than just any model.

3) Beware the "atypicality" of remote events. There is a sucker's method called "scenario analysis" and "stress testing"—usually based on the past (or some "make sense" theory). Yet I show in the appendix how past shortfalls that do not predict subsequent shortfalls. Likewise, "prediction markets" are for fools. They might work for a binary election, but not in the Fourth Quadrant. Recall the very definition of events is complicated: success might mean one million in the bank ...or five billions!

4) Time. It takes much, much longer for a times series in the Fourth Quadrant to reveal its property. At the worst, we don't know how long. Yet compensation for bank executives is done on a short term window, causing a mismatch between observation window and necessary window. They get rich in spite of negative returns. But we can have a pretty clear idea if the "Black Swan" can hit on the left (losses) or on the right (profits).

The point can be used in climatic analysis. Things that have worked for a long time are preferable—they are more likely to have reached their ergodic states.

5) Beware Moral Hazard. Is optimal to make series of bonuses betting on hidden risks in the Fourth Quadrant, then blow up and write a thank you letter. Fannie Mae and Freddie Mac's Chairmen will in all likelihood keep their previous bonuses (as in all previous cases) and even get close to 15 million of severance pay each.

6) Metrics. Conventional metrics based on type 1 randomness don't work. Words like "standard deviation" are not stable and does not measure anything in the Fourth Quadrant. So does "linear regression" (the errors are in the fourth quadrant), "Sharpe ratio", Markowitz optimal portfolio, ANOVA shmnamova, Least square, etc. Literally anything mechanistically pulled out of a statistical textbook.

My problem is that people can both accept the role of rare events, agree with me, and still use these metrics, which is leading me to test if this is a psychological disorder.

The technical appendix shows why these metrics fail: they are based on "variance"/"standard deviation" and terms invented years ago when we had no computers. One way I can prove that anything linked to standard deviation is a facade of knowledge: There is a measure called Kurtosis that indicates departure from "Normality". It is very, very unstable and marred with huge sampling error: 70-90% of the Kurtosis in Oil, SP500, Silver, UK interest rates, Nikkei, US deposit rates, sugar, and the dollar/yet currency rate come from 1 day in the past 40 years, reminiscent of figure 3. This means that no sample will ever deliver the true variance. It also tells us anyone using "variance" or "standard deviation" (or worse making models that make us take decisions based on it) in the fourth quadrant is incompetent.

7) Where is the skewness? Clearly the Fourth Quadrant can present left or right skewness. If we suspect right-skewness, the true mean is more likely to be underestimated by measurement of past realizations, and the total potential is likewise poorly gauged. A biotech company (usually) faces positive uncertainty, a bank faces almost exclusively negative shocks. I call that in my new project "concave" or "convex" to model error.

8) Do not confuse absence of volatility with absence of risks. Recall how conventional metrics of using volatility as an indicator of stability has fooled Bernanke—as well as the banking system.

Figure 7 Random Walk—Characterized by volatility. You only find these in textbooks and in essays on probability by people who have never really taken decisions under uncertainty.



Figure 8 Random Jump process—It is not characterized by its volatility. Its exits the 80-120 range much less often, but its extremes are far more severe. Please tell Bernanke if you have the chance to meet him.

9) Beware presentations of risk numbers. Not only we have mathematical problems, but risk perception is subjected to framing issues that are acute in the Fourth Quadrant. Dan Goldstein and I are running a program of experiments in the psychology of uncertainty and finding that the perception of rare events is subjected to severe framing distortions: people are aggressive with risks that hit them "once every thirty years" but not if they are told that the risk happens with a "3% a year" occurrence. Furthermore it appears that risk representations are not neutral: they cause risk taking even when they are known to be unreliable.

---

Quote:

---

Originally Posted by Loki

His approach seems to be that if you're uncertain of something, you shouldn't do it.

---

At first glance, yes. As he develops his ideas (in the essay and, more thoroughly, in the books) he clarifies that his approach is primarily to improve the outcomes when engaging in complex activities where we're exposed to known and unknown risks. That's an incomplete summary, but it's a fairly reasonable one. Put more informally, he thinks we should try to not "be the turkey" in his butcher vs. turkey analogy.

Quote:

---

The general approach in the social sciences is to acknowledge potential pitfalls, deal with them the best you can, but do the analysis anyway. I think he'd be listened to much more if he tried to help social scientists do the latter better instead of just mocking them.

---

:-D Everyone needs a hobby :downcast:

On the one hand, competent researchers in the social sciences are fairly aware of the limits of their research, and strive to be as rigorous as is possible in their chosen tasks. Taleb himself credits a number of social scientists for conducting impressive and meaningful research. At the same time, he generally doubts--out of principle--their ability to forecast the future. Social scientists may be aware of their difficulties with forecasting, and of the potential biases and fallacies they may fall prey to. But, by the time their knowledge arrives at the general public's doorstep, it's been cleansed of all caveats and lost much of its (useful) nuance and gained a great deal of political coloration. That's been my experience anyway, I may be exaggerating though.





Of course, I hear experts in certain fields often greatly overestimate their ability while also being crappy :mad: Ie. they are not aware of how wrong they are, while also being wrong very often. I'll try to remember to compile a list of references once I'm done reading.

According to the criticisms page at wiki, most of what he says has been common knowledge for decades, more advanced models of the Black Swan have been created, and he never acknowledges previous work on the subject. Even smart people need to look at what came before them instead of trying to build every argument from thin air.

Quote:

---

Originally Posted by Loki

According to the criticisms page at wiki, most of what he says has been common knowledge for decades, more advanced models of the Black Swan have been created, and he never acknowledges previous work on the subject. Even smart people need to look at what came before them instead of trying to build every argument from thin air.

---

The criticisms section is very informative and entertaining :o there's no denying he has an arrogant attitude that can rub most people the wrong way.

Just so we're clear, The Black Swan, the book on which he's built his popular recognition, is a literary work--but one in which he looks at the centuries of philosophy and science that underlie the concepts he's trying to popularise. It's the only reason I could find it in me to overlook his writing style (and the impression I'd gotten from videos and from wiki) for long enough to get hooked. The basic idea is immediately accessible to most people. The examination of its history, its development and of our curious failures (as a civilisation) to act on that idea, that's interesting!

When it comes to acknowledging prior work, he has done so in the portion of the book I've read so far, and he does so in his academic publications. He's less fastidious in informal contexts but my only problem with that is that I feel a little robbed :cry: not that he's stealing all the glory. After all, it has absolutely no bearing on my life, as long as he isn't putting sinister mind-bugs into my head.







Re. what's common knowledge and what's not, I reckon we have a problem (and a story) when something is common and important knowledge and yet fails to influence the behavior of people and systems in good ways (or at least not in spectacularly bad ways).

Quote:

---

Originally Posted by Loki

His approach seems to be that if you're uncertain of something, you shouldn't do it. The general approach in the social sciences is to acknowledge potential pitfalls, deal with them the best you can, but do the analysis anyway. I think he'd be listened to much more if he tried to help social scientists do the latter better instead of just mocking them.

---

Maybe the fact that social scientists are incentivized to go ahead, slap some assumptions on an uncertain analysis and publish it, rather than holding back when they can't actually provide a proper answer is part of the problem?

Quote:

---

Originally Posted by LittleFuzzy

Maybe the fact that social scientists are incentivized to go ahead, slap some assumptions on an uncertain analysis and publish it, rather than holding back when they can't actually provide a proper answer is part of the problem?

---

Surely an uncertain analysis is better than no analysis, as long as the uncertainty is openly acknowledged? I should also point out that the main strain of this guy's attack - our inability to forecast - is a problem mainly in some parts of economics. Most social scientists don't attempt to predict, though it's implicit in their works.

Quote:

---

Originally Posted by Loki

Surely an uncertain analysis is better than no analysis, as long as the uncertainty is openly acknowledged?

---

Actually I think bad information is worse than no information.

Quote:

---

Originally Posted by LittleFuzzy

Actually I think bad information is worse than no information.

---

Then we should just abandon science, seeing that most of the things we know will probably be shown to be false in some way in the future. Science is about progress; it's not about starting from perfection.

I'm not sure, but I think acting on bad information may sometimes be worse (in terms of consequences) than acting on no information. Eg. cases of experts performing worse than chance. That's all right on a small scale, and, hopefully, their negative outcomes will come to light and add to our knowledge. On a large scale, eg. with Communist-style social engineering, who knows :o



Misguided educational policy for example.

Quote:

---

Originally Posted by Loki

Then we should just abandon science, seeing that most of the things we know will probably be shown to be false in some way in the future. Science is about progress; it's not about starting from perfection.

---

I'll keep in mind you said that the next time you and I are criticizing inadequate climate modeling, which I'm sure is still WAY ahead of most political and economic models.

Quote:

---

Originally Posted by LittleFuzzy

I'll keep in mind you said that the next time you and I are criticizing inadequate climate modeling, which I'm sure is still WAY ahead of most political and economic models.

---

I'm fairly sure it's not. To tie this to Minx's point, my problem isn't so much that the climate change models are incomplete, but rather that A) people claim that our knowledge is far more certain that it really is, and B) we're meant to spend trillions of dollars under the assumption that it is fully certain. If I was to "prove" that larger militaries lead to war, do you really think anyone would take that research and attempt to disband the US army?

Quote:

---

Originally Posted by Aimless

I'm not sure, but I think acting on bad information may sometimes be worse (in terms of consequences) than acting on no information. Eg. cases of experts performing worse than chance. That's all right on a small scale, and, hopefully, their negative outcomes will come to light and add to our knowledge. On a large scale, eg. with Communist-style social engineering, who knows http://www.theworldforgotten.com/ima...on_redface.gif

Misguided educational policy for example.

---

It's not up to scientists to determine whether others act on their information. Again, scientific progress is incremental, which means that at any given moment what we know is flawed in some way. Should scientists refuse to publish any research unless they're 100% certain of its accuracy?

Quote:

---

Taleb, looking at the cataclysmic situation facing financial institutions today, points out that "the banking system, betting against Black Swans, has lost over 1 Trillion dollars (so far), more than was ever made in the history of banking".

---

Including newly created derivatives and MBSs, pushed by math quants, Wall Street and Bankers, unregulated and sold around the world.

Banking instruments were created by bankers? Really? Are you going to blame construction workers for construction next?

Quote:

---

Originally Posted by Loki

It's not up to scientists to determine whether others act on their information. Again, scientific progress is incremental, which means that at any given moment what we know is flawed in some way. Should scientists refuse to publish any research unless they're 100% certain of its accuracy?

---

Well, considering how much of published science is apparently kinda rubbish... maybe! :p After all, good journal real estate is a very scarce resource and the findings, good or bad (mostly bad) clearly have real impacts on decision making as well as on future research.





I'm kidding. Scientists should conduct and publish research. The best level to tackle problems would be at the level of policy makers.

'Sides, it would be cruel to ask scientists to include arguments in support of their irrelevance in their every published paper :(

It's in the interest of individual scientists to go after popular papers that are flawed. Major findings rarely go unchallenged. I really don't see what purpose not publishing flawed findings serves, other than to stifle progress.

Furthermore, policy is never based on objective facts, so I fail to see why the science that policy is based on should be held to such a restrictive standard. At least science provides replicable evidence, along with margins of error. I wish I could say that for most other factors that influence policy-makers. Instead, they rely on common wisdom (let alone pure politics), which is itself based on many hidden assumptions that would be easily discredited if they were ever closely examined.

Quote:

---

Originally Posted by Aimless

It's a long read, but a good one. Should of course be taken with a pinch of salt. He can be a bit of a jerk :o

---

Seems to me like a whole lot of "we knew that already," but like he acknowledged, the short-term incentives are to reduce redundancy and slack that insulate against black swan events, and very little in the way of incentivization to prevent them... even when people are acting rationally, which isn't all the time, either.

My first thought was "point being...?" But maybe the level of understanding around these events is much lower than mine, and the read will do someone some good... though I hardly see anyone who doesn't already know what he's talking about stooping to read for knowledge.

Quote:

---

Originally Posted by GGT

Including newly created derivatives and MBSs, pushed by math quants, Wall Street and Bankers, unregulated and sold around the world.

---

j00s! :up:

:haha::haha::haha:

Quote:

---

Originally Posted by Loki

Banking instruments were created by bankers? Really? Are you going to blame construction workers for construction next?

---

If builders manipulate numbers (the elegance of math) to sell houses built on flawed foundations, then buy insurance policies to profit when the houses sink, sure.

Quote:

---

Originally Posted by CitizenCain

j00s! :up:

:haha::haha::haha:

---

Hasn't that "joke" hit its effective statistical limits by now? :donkey:

Lies, damn lies, and statistics. :mad::up:

5

Quote:

---

Originally Posted by GGT

If builders manipulate numbers (the elegance of math) to sell houses built on flawed foundations, then buy insurance policies to profit when the houses sink, sure.

Hasn't that "joke" hit its effective statistical limits by now? :donkey:

---

Why ask me? I'm not one of the j00s that invents/manipulates the numb3r5. :(

Quote:

---

Originally Posted by CitizenCain

Why ask me? I'm not one of the j00s that invents/manipulates the numb3r5. :(

---

Of course you weren't a math quant or physics PhD, recruited by "Wall Street" (aka the financial industry) to create synthetic CDOs, MBSs, CDSs with crazy algorithms (aka WMD) in exchange for million dollar salaries and bonii.

You're just the one always joking about jews in any thread that mentions banking, finance, or investment math. :rolleyes:

Again, this is not a joke (well, it's funny like a joke, but I'm not joking), this is you jumping into every thread and blaming the world's problems on the evil j00s. Even this thread, which offers a very well-supported and strongly reasoned explanation of why things happened the way they did, in a very accessible way, and your response is... ~"nope, j00s tricked everyone in exchange for j00 gold."

And again, good luck with that whole thing you do where you blame the j00s and proudly proclaim your successful evasion of knowledge, but don't expect everyone to agree with you, or me to not find ways to amuse myself by pointing it out.

Quote:

---

Originally Posted by CitizenCain

Again, this is not a joke (well, it's funny like a joke, but I'm not joking), this is you jumping into every thread and blaming the world's problems on the evil j00s. Even this thread, which offers a very well-supported and strongly reasoned explanation of why things happened the way they did, in a very accessible way, and your response is... ~"nope, j00s tricked everyone in exchange for j00 gold."

And again, good luck with that whole thing you do where you blame the j00s and proudly proclaim your successful evasion of knowledge, but don't expect everyone to agree with you, or me to not find ways to amuse myself by pointing it out.

---

Excuse me, but this thread included limits of math/stats and risk/probability used by Bankers and "Wall Street":

Quote:

---

Statistics can fool you. In fact it is fooling your government right now. It can even bankrupt the system (let's face it: use of probabilistic methods for the estimation of risks did just blow up the banking system).

---

Quote:

---

Taleb, looking at the cataclysmic situation facing financial institutions today, points out that "the banking system, betting against Black Swans, has lost over 1 Trillion dollars (so far), more than was ever made in the history of banking".

---

NASSIM NICHOLAS TALEB, essayist and former mathematical trader, is Distinguished Professor of Risk Engineering at New York University’s Polytechnic Institute. He is the author of Fooled by Randomness and the international bestseller The Black Swan.

In Cain's world, any criticism of the financial industry is actually complaining about jews and jew gold. Even when it comes from math traders or professors in risk engineering. :donkey:

After rubbing my eyes and reading more of this, I can appreciate some of the truisms he's spouting out there because I agree with them. But I can't help avoid the sense that he's just making opinions and slapping-on some statistical vocabulary as part of his crusade against how Wall Street works.