(Source)

Are we really living in the most peaceful time in human history? It all depends on how you measure: Reality Project Episode 2

In his sweeping 2011 work The Better Angels of Our Nature: Why Violence has Declined, Steven Pinker makes the audacious claim, “today we may be living in the most peaceable era in our species’ existence.” To someone who pays even a little attention to constant news reports of violence, both local and international, this statement sounds ludicrous. The grandeur of this assertion should immediately raise our suspicions; as the science popularizer Carl Sagan often said, “extraordinary claims require extraordinary evidence.” Does Pinker have that evidence and does he interpret it correctly?

Over the course of 800 pages, Pinker makes a convincing argument with numbers, figures, and references to many books and articles. Nonetheless, it’s critical that we don’t rely on a single argument to form our worldview. We must remember that data is never objective, and therefore we need to look at the data itself as well as multiple expert interpretations. In this article, we’ll examine the global violent conflict data and different stances on the same numbers. (As a note, Pinker’s work deals with all forms of violence from the individual — homicides, abuse — to the worldwide, but here we’ll focus on international violent conflicts — war — and leave the rest for future articles).

(Source)

Exploring the incredible worldwide gains in prosperity: Reality Project Episode 1

Every day for 25 years, newspapers could have correctly run the headline “130,000 People Escape Extreme Poverty Yesterday” (source). Instead, as far as I can tell, this has appeared on the front page of a major newspaper precisely 0 times. Maybe this explains the incredibly terrible results recorded by people on this simple question (from Gapminder):

Question 1. of the Gapminder test

When this question was asked in the United States, 5% of humans got the correct answer, C: almost halved. Animals, on the other hand, answer this question with 33% accuracy (because they randomly guess).

How is it possible for people in a supposedly literate society like the US to be so wrong about reality? I don’t like to place blame, but in this case, it’s likely justified: the media has succeeded in a campaign to misinform us about the world. Instead of reporting facts, they have chosen to activate our natural inclination towards xenophobia (fear of outsiders) by convincing us there is a “developed world” with well-off people, and then there is everyone else, a mass of poor hordes that will never be able to rise from poverty.

Fortunately, armed with the right data, we can correct our wrong beliefs about poverty. On doing so, we discoverable a remarkable fact: over the past few decades, people all over the world have undergone a remarkable increase in prosperity, resulting in measurably better living standards for billions of people.

Sunrise from the International Space Station (Source)

Becoming less wrong about the world with data

Strange and wonderful things happen when you put down the newspaper, turn off the tv, and decide to look at actual statistics about the world instead of relying on what you’ve been told. The ground begins to shift beneath your feet as you realize the worldview which the news has been feeding you — that the world is on a downward slide — is, based on almost every single factual measure, in direct opposition to reality. Once we look at the data on topics such as wealth, health, human rights, and environmental protection, we are left with no option but to conclude the world has been on an upward trajectory for most of recorded history, we live in the best time ever for human civilization, and the improvement shows no signs of stopping.

Over the past five years, as I’ve slowly weaned myself from a debilitating news addiction, I’ve experienced what can only be described as an update to the worldview-generating software in my brain. Through tens of thousands of book pages, millions of data points, and countless charts, my view of the world as a dark, dangerous place and humanity as an evil force has reversed. Based on the data, the story of the world has flipped: the history of humanity is not one of decline, but a gradual ascent continuing to today.

I did not start out with the aim of becoming an optimist, but a long-term positive outlook is the inevitable conclusion from a fact-based examination of the world.

As a new year begins, and now that I’ve assimilated enough independent sources of data to be confident reality is really better than we think, I’m starting The Reality Project, a data-driven endeavor to present the world as it is through a series of weekly articles. The stated goal is to become less wrong about the world through data, disposing of our distorted worldviews. If, along the way, the data happen to give you a more optimistic view of humanity, then consider it as a beneficial side effect.

A Python toolkit for data science with Medium article statistics

Medium is a great place to write: no distracting features, a large — yet civil — readership, and, best of all, no advertisements. However, one aspect where it falls short is in the statistics you can see for your articles. Sure, you can go to the stats page, but all you get to see is some plain numbers and a bar chart in an awful shade of green. There’s no in-depth analysis of any kind and no way to make sense of the data generated by your articles.

It’s nice when you reach the point where it’s more than your mom reading your articles.

It’s as if Medium said: “let’s build a great blogging platform, but make it as difficult as possible for writers to get insights from their stats.” Although I don’t care about using stats to maximize views (if I wanted to get the most views, all my articles would be 3-minute lists), as a data scientist, I can’t bear the thought of data going unexamined.

Instead of just complaining about the poor state of Medium’s stats, I decidedto do something about it and wrote a Python toolkit to allow anyone to quickly retrieve, analyze, interpret, and make beautiful, interactive plots of their Medium statistics. In this article, I’ll show how to use the tools, discuss how they work, and we’ll explore some insights from my Medium story stats.

The full toolkit for you to use is on GitHub. You can see a usage Jupyter Notebook on GitHub here (unfortunately interactive plots don’t work on GitHub’s notebook viewer) or in full interactive glory on NBviewer here. Contributions to this toolkit are welcome!

Example plot from Python toolkit for analyzing Medium articles

(Source)

Statistics, the lifetime equation, and when data science will end

The pursuit of astronomy has been a gradual process of uncovering the insignificance of humanity. We started out in the center of the universe with the cosmos literally revolving around us. Then we were rudely relegated to one of 8 planets orbiting the sun, a sun which subsequently was revealed to be just one of billions of stars (and not even a large one) in our galaxy.

This galaxy, the majestic Milky Way, seemed pretty impressive until Hubble discovered that all those fuzzy objects in the sky are billions of other galaxies, each of which has billions of stars (potentially with their own intelligent life). The demotion has only continued in the 21st century, as mathematicians and physicists have concluded the universe is one of an infinity of universes collectively called the multiverse.

Go here to be blown away