The Book of Why: The New Science of Cause and Effect (Penguin Science)

We have all heard the old saying “correlation is not causation”. This is a problem for statistics, since all it can measure is correlation. Pearl here argues that this is because statisticians are restricting themselves too much, and that it is possible to do more. There is no magic; to get this more, you have to add something into the system, but that something is very reasonable: a causal model.He organises his argument using the three-runged “ladder of causation”. On the bottom rung is pure statistics, reasoning about observations: what is the probability of recovery, found from observing these people who have taken a drug. The second rung allows reasoning about interventions: what is the probability of recovery, if I were to give these other people the drug. And the top rung includes reasoning about counterfactuals: what would have happened if that person had not received the drug?Intervention (rung 2) is different from observation alone (rung 1) because the observations may be (almost certainly are) of a biassed group: observing only those who took the drug for whatever reason, maybe because they were already sick in a particular hospital, or because they were rich enough to afford it, or some other confounding variable. The intervention, however, is a different case: people are specifically given the drug. The purely statistical way of moving up to rung 2 is to run a randomised control trial (RCT), to remove the effect of confounding variables, and thereby to make the observed results the same as the results from intervention. The RCT is often known as the “gold standard” for experimental research for this reason.But here’s the thing: what is a confounding variable, and what is not? In order to know what to control for, and what to ignore, the experimenter has to have some kind of implicit causal model in their head. It has to be implicit, because statisticians are not allowed to talk about causality! Yet it must exist to some degree, otherwise how do we even know which variables to measure, let alone control for? Pearl argues to make this causal model explicit, and use it in the experimental design. Then, with respect to this now explicit causal model, it is possible to reason about results more powerfully. (He does not address how to discover this model: that is a different part of the scientific process, of modelling the world. However, observations can be used to test the model to some degree: some models are simply too causally strong to support the observed situation.)Pearl uses this framework to show how and why the RCT works. More importantly, he also shows that it is possible to reason about interventions sometimes from observations alone (hence data mining pure observations becomes more powerful), or sometimes with fewer controlled variables, without the need for a full RCT. This is extremely useful, since there are many cases where RCTs are unethical, impractical, or too expensive. RCTs are not the “gold standard” after all; they are basically a dumb sledgehammer approach. He also shows how to use the causal model to calculate which variables do need to be controlled for, and how controlling for certain variables is precisely the wrong thing to do.Using such causal models also allows us to ascend to the third rung: reasoning about counterfactuals, where experiments are in principle impossible. This gives us power to reason about different worlds: What’s the probability that Fred would have died from lung cancer if he hadn’t smoked? What’s the probability that heat wave would have happened with less CO2 in the atmosphere?[p51] "probabilities encode our beliefs about a static world, causality tells us whether and how probabilities change when the world changes, be it by intervention or by act of imagination."This is a very nicely written book, with many real world examples. The historical detail included shows how and why statisticians neglected causality. It is not always an easy read – the concepts are quite intricate in places – but it is a crucially important read. We should never again bow down to “correlation is not causation”: we now know how to discover when it is.

This is a book about a fascinating and important subject which is almost incredibly difficult to read. Not because the concepts are excessively challenging, not because the organisation of the book is poor - it is logical, well-structured and uses good examples. Not even because the subject matter is without excitement - the wars of the frequentists and the Bayesians are presented in a suitably dynamic manner.But because it lack formal presentation. Pearl introduces technical terms like do-calculus, confounder, counterfactual, backdoor paths and frontdoor adjustment, and then continues to use them never having given a concise and accessible definition.It's fine to introduce a new concept by means of examples, derivations and formulae, but at some point it is absolutely necessary to provide a concise and formal definition - maybe even two or three definitions, one in strict formal terms, one in ordinary language and one in terms of the application and use of it "An x is an X if and only if it is a y and painted blue by a left handed kangaroo" so the test is "if you think something might by an X, check if it's a y and look for kangaroos, blue paint pots and used brushes".This allows the reader to work through the material which leads to the formal statement, repeatedly if necessary, until he feels happy that he has understood the concept and understands the formal statement. He can then progress to later material, knowing that if the earlier term recurs, and he can't quite remember what it means ("Was it a left or right handed kangaroo?") he can go back to the single, concise formal definition, refresh his understanding (or revert to the material which culminates in the definition if it no longer rings a bell) and get back to the new stuff.All in all, an important and fascinating work with a really annoying flaw.I finally resorted to making my own notes about what the concepts meant, but I'd still rather have the words of the author.If a writer doesn't do this, he makes his work far harder to understand.Pearl writes well

I found this book to be informative, well-written and enjoyable to read. As someone who is trained in traditional statistics I really enjoyed such a clear and enthusiastic account of the possibilities of causal inference.

This is not a book on cause and effect in physics. Instead it tells the story og how classical statistics was separated from cause and effect by its development as a mathematical transformation (a so called "reduction") of observed data, independent of how and why these data were measured. It was argued the the statistical results should be objective without any intervention in the observational process. The resulting correlations cannot, however, tell us anything about cause and effect. R. A. Fisher invented (in 1924) the randomized controlled trial in order to avoid a subjective intervention. This is the old science of cause and effect.The definition og causality is so important, because it determines the time direction of the future and the past. We can only remember the past, not the future. Any intelligence (artificial og natural) must involve causality. This book is about how a new science of cause and effect can be joined to statistics, so a robot with real humanlike intelligence can be created (eventually).This implies that Google's DeepLearning and TensorFlow cannot possibly be real intelligence. They are data driven like classical statistics and do not allow causality.

Although is not easy to read, this book is excellent.

Livro ótimo pra entender sobre correlação e causalidade

Detallado en la exposición del tema.

accessible written style with simple and meaningful examples.fresh air in the world of "big data is all we need".core idea : any scientist needs a model of the real world he wants to represent + you can not escape the problem of causes and effects, even with massive data bases and statistics.

This is the best gift I could give to myself having bought the book after I was scouring for good books. Sorry to those who haven’t done Science and Mathematics this piece of gold is not made for you. Lots of mathematical terms make it difficult for non science and non mathematics.

When learning statistics, students are inundated with the fact that correlation does not imply causation. This may be true, but it begs the question, what does imply causation? This is exactly the question that Judea Pearl and Dana Mackenzie adeptly address in The Book of Why. The book covers why causation is crucial, how the very concept of causation became taboo, and the burgeoning causation revolution that is enriching the sciences. It's an exciting journey! To the authors' credit, they were able to create a captivating narrative with engaging prose about a topic that is commonly construed as dry. They skillfully balance thorough treatment with repetitive drudgery. It is a delightful read.Those in the field of data science and other related disciplines will find this book particularly interesting. It challenges the current prevailing conception that everything that can be known is found in data. The reality is that the only thing a "deep-learning program can do [is] fit a function to data." (p.17) In other words, crunching data simply reveals associations. Real intelligence requires the ability to predict the outcome of conceived events and retrospectively determine alternate outcomes given altered data. This type of insight requires a model of how the data was created. Big data is not the final destination rather it is a milestone.Beyond the technical content, The Book of Why provides a glimpse into how personal bias can influence scientific facts. It serves at yet another reminder that human factors cannot be ignored. Scientists are infallible just like everyone else. The causal revolution is a testament to the bravery of many brilliant individuals who challenged the status quo.A final accolade worthy of mention is the book's accessibility. This isn't reading just for statistics geeks. Any person, regardless of their background, will have no issue keeping pace. Geeks and laymen alike will find it informative and gain a suitable understanding of the subject matter upon completion.