Social Physics - Reinventing Analytics to Better Predict Human Behaviors
Social physics first emerged over 200 years ago as an attempt to understand society and human behavior using laws similar to those of the physical sciences. But, it wasn’t until the past two decades that we finally had enough data, powerful computers and sophisticated mathematical algorithms to develop quantitative theories of human social interactions. We were now able to reliably predict how large groups of people make decisions by analyzing how information and ideas flow from person to person.
“The engine that drives social physics is big data: the newly ubiquitous digital data now available about all aspects of human life,” wrote MIT professor Alex (Sandy) Pentland in his 2014 book Social Physics: How Good Ideas Spread - The Lessons from a New Science. “Social physics functions by analyzing patterns of human experience and idea exchange within the digital bread crumbs we all leave behind as we move through the world, - call records, credit card transactions, and GPS location fixes, among others. These data tell the story of everyday life by recording what each of us has chosen to do… Who we actually are is more accurately determined by where we spend our time and which things we buy, not just by what we say we do.”
The book was the result of over a decade’s worth of research in the Human Dynamics group at MIT’s Media Lab. Pentland, along with his graduate students and research associates, partnered with a variety of companies to obtain and analyze real-world data, properly anonymized to protect the privacy of the individuals whose behavior was reflected in the data. They eventually discovered that all event-data representing human activity contain a special set of social activity patterns regardless of what the data is about. These patterns are common across all human activities and demographics, and can be used to detect emerging behavioral trends before they can be observed by any other technique.
For example, as described in a 2013 Harvard Business Review paper, Pentland and his then postdoctoral associate Yaniv Altshuler and graduate student Wei Pan devised a research project based on data from the social trading platform eToro. eToro allows single trades, - where investors make decisions on their own, and social trades, - where investors can automatically copy the leading traders in the eToro community and earn money by having their trades copied by others.
After analyzing almost 10 million financial transactions conducted by 1.6 million eToro users, they found that traders fell along a continuum. At one end where the explorers, who came up with investment ideas on their own and followed few, if any, other traders. At the other end where the hyperconnected social learners who followed, and were followed by, many others. Many eToro users were somewhere in the middle, engaging in a moderate level of social learning but also making a fair number of individual trades.
Properly used, social learning made a big difference. “The traders who had the right balance and diversity of ideas in their network - meaning that their social learning was neither too sparse nor too dense - had a return on investment that was 30% higher than the returns of both the isolated traders and those in the herd. In this digital trading environment, the sweet spot resides between the two extremes. This intermediate zone is where social learning - that is, copying successful people - yields real rewards.”
These social physics patterns have been tested across a variety of applications involving people, including strategy formulation in business, economic activity in cities, and, - working with an intelligence agency, - the detection of potential terrorist activity based on Twitter data. As long as the data involves human activity, - regardless of the type of data, the demographic of the users or the size of the data sets, - similar behavioral dynamics apply.
What accounts for such universal social physics principles? The answer most likely lies in evolutionary biology and natural selection. Survival is clearly a key evolutionary imperative. And surviving in a changing environment requires a combination of social learning and new ideas. Humans have thus evolved with the drive to learn from each other. But, at the same time, mutations and innovations will vary among different groups, with natural selection favoring those human groups better able to adapt to changing conditions.
To help me better appreciate social physics, I looked at it through the lens of physics, - my long-ago field of study at the University of Chicago. Think, for example, about finding and tracking potentially hazardous, fast moving, near-Earth objects by detecting very small changes in the sky, or looking for Earth-size extrasolar planets by detecting the faint changes in a star’s light caused by a potential planet quickly passing by. No matter how much data we might have access to, it would be near impossible to detect the weak signals associated with either task without the physics principles and models developed over the past few hundred years.
How about our recent advances in machine learning methods? AI is rapidly becoming one of the most important technologies of our era. Machine learning and related algorithms like deep learning, have played a major role in AI’s recent achievements, giving computers the ability to learn by ingesting and analyzing large amounts of data instead of being explicitly programmed. They’ve enabled the construction of AI algorithms that can be trained with lots and lots of sample inputs, which are subsequently applied to complex problems like language translation, natural language processing, and playing championship-level Go.
Machine learning has been most successful when used for complex computational problems like image and voice recognition, where a huge body of data is available and the data is fairly static. There are many tasks for which machine learning methods aren’t applicable given the current state-of-the-art, such as the analysis of data derived from human behavior.
Human behavior and interactions are dynamic, complex and ever-changing, exhibiting a high degree of variance which make them hard to predict and subject to emergence, - where the whole might well be different from the sum of the parts. Predicting human behavior requires the ability to frequently analyze relatively small data sets collected over short periods of time.
In 2014, Pentland and Altshuler co-founded Endor, an Israeli-based startup with Altshuler as CEO. Endor is focused on making fast, accurate predictions using data derived from human behavior. Its prediction algorithms integrate social physics technologies into its data analytic engine, enabling it to efficiently extract the underlying social attributes embedded in the raw data being analyzed. Endor’s clients include Coca-Cola, Mastercard, Walmart, and BCG.
Earlier this year, the startup announced Endor.Coin, an analytics platform designed to reinvent AI predictions by making them easily accessible to companies and individuals with no data science or AI expertise, thus aiming to become a kind of Google for predictive analytics. It can answer questions like: who are our top customers and how do we acquire more of them?; who is likely to try this newly-launched product?; and where should we open our next store?
Using Endor’s self-service platform, users upload whatever data they want analyzed, such as records of mobile phone calls or credit card purchases. They then use Endor’s query builder wizard to phrase their question in the form “here is an example X, find me more of X.” The platform then extracts clusters with similar behavioral patterns from its millions of raw data points, which it can do much more quickly and accurately than machine learning algorithms. The actual predictions are generated from Endor’s large collections of behavioral clusters using data learning algorithms. Individuals and data owners pay for predictions using EDR tokens, which they can buy any time.
Since it’s searching for patterns, not content, the Endor platform can analyze fully encrypted data sets, allowing customers such as financial companies or health care providers to maintain data privacy. But the more public data the platform can use to generate behavioral clusters across a wide variety of domains, the more accurate its predictions. So, to encourage Endor users to contribute public data, they’re rewarded with EDR tokens when insights derived from their data are used for predictions. A detailed explanation of Endor’s platform, including the social physics technologies on which it’s based, can be found in this comprehensive white paper.
Social physics is a new discipline, still requiring considerable effort and real-world experimentation to demonstrate its value. “In the end, I believe that the potential rewards of a data-driven society operating on the principles of social physics are worth the efforts and the risk,” wrote professor Pentland in the concluding paragraph of Social Physics. “Imagine: We could predict and mitigate financial crashes, detect and prevent infectious disease, use our natural resources more wisely, and encourage creativity to flourish. These dreams used to be the stuff of science-fiction stories, but that fantasy could become a reality - our reality, if we navigate the pitfalls carefully. That is the promise of social physics and a data-driven society.”