new scarcity in the information age

From MIT Media Lab's Alex Pentland, Social Physics:

"One disturbing implication of [our] findings is that our hyperconnected world may be moving toward a state of too much idea flow. In a world of echo chambers, fads and panic are the norm, and it is much harder to make good decisions. This suggests that we need to pay more attention to where our ideas are coming from, and we should actively discount common opinions and keep track of contrarian ideas."

From investor Peter Thiel's Zero to One:

"Courage is in far shorter supply than genius."

Disclaimer: this post is longer than my usual stuff, and it's still a work in progress. I may have huge blind spots and indulgences here, and I'd love to hear where they are or if you got totally lost somewhere in my rambling mess of an essay. But I had to get my thoughts down, so alas here we are.

In classical economics, there's an idea called the Scarcity Principle. If you've ever taken econ, this is probably one of the first things you learn.

The scarcity principle, in (very slightly) technical terms, is the market disequilibrium caused by the limited supply of a good and a corresponding high demand for it. The efficient markets hypothesis implies that when this happens, either supply expands to meet the demand, or prices go up to lower the demand until supply = demand.


It's arbitrage in the most fundamental sense -- if you as a supplier discover a disequilibrium, you start selling that thing at higher prices. Or you start making more of that thing and undercut your competitors at a lower price. Human Craftiness 101.

Scarcity Principle: Used Bikes and Tuna Fish

An example that comes to mind is the used bicycle market. In any bike-friendly city, you can probably buy a used vintage Trek road bike for $150 off Craigslist, refurbish it for $50-100, and then resell it for ~$450-600 on eBay.

There's an unmet demand for vintage bikes that are in great physical condition. These types of bikes are scarce. Your Craigslist sneakiness happened to respond to that scarcity. And this scarcity will remain as long as hipsters keep riding fresh bikes and as long as refurbishing a busted beauty takes more time and effort than a normal person would want.

In the excellent documentary Jiro Dreams of Sushi, you also see this principle in action: the wholesale price of tuna has risen tremendously as the tuna population has dwindled and the global demand for fatty tuna has increased. 

This has led two a few parallel trends: i) things that look and taste like tuna being labeled and sold as tuna, ii) large corporations running tuna farms, basically breeding and "manufacturing" tuna, and iii) high-end restaurants driving up their prices (knowing their customers are price insensitive), thus continuing to buy wild tuna. 

Tuna is the scarce resource. It will remain scarce as long as its texture and flavor remain highly-valued and as long as the tuna population remains endangered.

Building "Old Scarcity"

Guess what the big reveal is? Our highly-professionalized, white-collar knowledge market is also susceptible to the Scarcity Principle (as are our new media darlings out in Silicon Valley). And that is where this blog post is headed.

Over the past 150 years, the US economy has become highly financialized and instrumentalized. The concepts of corporations, public markets, commodities, derivatives, and stakeholders have been formulated to help us regulate, predict, reason about, and exploit human economic activity at a large scale.

Those who can understand the markets can figure out where the money is flowing and redirect some of it to themselves. Managing this complex system in part created new, lucrative livelihoods -- corporate law, investment banking, management consulting, hedge funds, etc. This is the scarcity principle in action-- Old Scarcity, as I'll call it. If public corporations need to drive value to stakeholders, how? If banks need to drive returns to investors, how?

Consultants provide analyses and recommendations to executives, bankers provide financial models and projections to potential investors, lawyers provide due diligence and legal paperwork to all parties involved. What is scarce here is the know-how of navigating a complex system safely and opportunistically (sounds a lot like driving a car, huh). Since money is exchanging hands in such large sums to make these transactions/deals happen, this know-how gets compensated handsomely. Who benefits? Smart, highly analytical, and hard-working elites.

Here's the thing, though: Old Scarcity is subject to the same forces as fish and used bikes. Old Scarcity remains as long as the system in which it operates remains somewhat understandable and predictable. And Old Scarcity remains as long as there aren't enough people with the know-how to do all of this financial wrangling and navigating well.

Dismantling "Old Scarcity"

Have you heard the old folktale of John Henry? John Henry was a skilled steel-driver who beat a steam-powered hammer in a race to build a railroad tunnel, but ultimately died as his heart gave out due to the stress of the job. 

The common takeaway with this story is the power of human industriousness, but really, all I see is a proud guy who lost to a machine. Maybe he should've derived his self-worth from something a machine couldn't replicate as easily. Maybe this is a moving target and what it means to be human keeps evolving as we make earlier versions of ourselves obsolete.

This is the crossroads at which the Old Scarcity industries find themselves at. On one hand, the system itself is becoming far less predictable and far more complex. On the other hand, trading, document review, and data analysis by algorithms are here, and they are beginning to do a better job than their lawyer/banker/consultant counterparts. The know-how is no longer scarce (or even useful all the time).

It will take several decades for this reality to really permeate and take full hold over the labor market, but it's here. Why pay a bunch of 20-somethings premium salaries when a monthly subscription to PandoraTech will do all of their jobs for you? The previously highly lucrative jobs as glorified wheel-greasers and bean counters will soon be done much faster and better by an algorithm written by your geeky cousin. And so the Scarcity Principle in this professional economy will be largely optimized into efficiency by our machine friends.

Building "New Scarcity"

The demise of Old Scarcity will illuminate the New Scarcity. Information and knowledge used to be scarce, it used to be something that institutions hoarded and justifiably charged $200,000 for. But now (almost) anything can, with enough Google-fu, be learned. Of course, some things still can't, and maybe those things are worth investigating further.

So what is scarce in a world where information is abundant? Courage and taste

The courage to create and build things that have very little precedence, or to create anything at all, is scarce. Machine learning works really well with large datasets, but what if the dataset is empty? On the flip side, when you have so much access to information (and thus many logical reasons to not do something), why do anything that will inevitably draw criticism and doubt? Those who are willing to do things are hot commodities.

Taste (the ability to be opinionated, principled, and insightful about things) is scarce. What use is all this data when it drowns, overwhelms, confuses, contradicts? What have you really learned when your information consumption is just an infinite stream of watered-down clickbait? Can you trust your gut? Those who know how to are hot commodities.

When large-scale computing can do most of our repeatable tasks infinitely faster and (for the most part) better than us, what do we have left to do? Create, build, express, tell stories, do things differently, think deeply and holistically. Either that, or become a computer whisperer and tune those algorithms so that more jobs get obsoleted and more people can be forced to think creatively.

It's less about acquiring knowledge, capital, and labor. Those are cheap and easy now. Those are things a computer can do a lot of this for you. But the things that aren't repeatable and scalable yet are things that require a personal touch. Art, self-expression, and exploration. Through this lens, the sciences and the humanities are more similar than we think.

We're already seeing this happen. Curated lists of things are popping up everywhere (and doing fantastically well). Artsy niche businesses, services, and boutiques are popping up at the grassroots level. Tech companies are focusing more and more on designing strong customer experiences. Lawyers are offloading their grunt work to software so that they can focus on building relationships, understanding the big picture, and catching any details that don't appear in databases yet. Some consulting firms are moving their work towards actually implementing their operational recommendations; others are being hired to ask the right questions, rather than to give the "right" answers.

As the emerging wisdom claims, many jobs will be lost, but a new breed of them will probably be created-- some that we can't quite imagine as being jobs yet. Just as several centuries ago, the job title of "he who pores over a spreadsheet" was unfathomable.

It's an exciting time to be thinking and doing.