Hierarchy's Limits: It's the Complexity, Stupid

It's not just the oppressive nature of human hierarchies that we must acknowledge, it's also their scaling limits. Our systems have become top-heavy and ill-equipped to deal with complexity

Eugene Tkachenko

Despite eighteenth-century revolutions in France and America, hierarchy is still the dominant form of social organization throughout most of the world. That is to say, much of the world's people are still stratified like medieval Europe or feudal Japan as compared with modern Switzerland.

Even the United States—that once great beacon of freedom—now bears a striking resemblance to Imperial Rome. The American Founders had made improvements by creating institutional checks on power. Yet the checks are weak. The hierarchy has grown. The paradox here is that the hierarchy will continue to grow as checks weaken further.

But as the hierarchy grows, power becomes unstable. Society is vulnerable.

The Limits of Hierarchy

What we have described is a series of transitions. But why did these transitions take place? Remember that, though dominance hierarchies are moderately sophisticated human systems, they originate by simple means: Men being assholes. More tactfully, people use threats to get others to do what they want.

Elsewhere we have discussed the urge to control, which creates a coercion paradigm that ends up growing to protect and perpetuate itself. We bolstered that claim by citing the anthropological evidence of early states. James C. Scott, for example, teaches us that governments started as protection rackets. After that, you had to figure out how to manage the increasing complexity.

Now that you can see how hierarchies appear and evolve, we need to understand how they break down. To lubricate understanding, we must recognize that human systems operate according to physical laws like other parts of nature. From there, following complexity science, we can discover theoretical limits to how we organize ourselves.

Then we can look for evidence that we are now reaching the limits of the world’s great hierarchies.

The easiest way to illustrate that human systems are subject to physical laws is to think of them as information processors seeking to overcome entropy. The individual processes information and energy. The collective also processes information and energy. People process information to do work. The system processes information to exist.

If we imagine that each individual is a kind of node or router, to which other nodes can respond, it's easy to see that there are limits to what any given node can process. We know that every byte of information in the universe taken together would fry the neural circuitry of any single person, no matter how smart or capable. Likewise, as consumers, it's easy to see practical limits to the number of calories one can consume, the number of decibels one can listen to, or the number of watts one uses to power a smartphone.

When we organize ourselves into different kinds of human systems, information gets processed in different ways. But all systems must respect their physical limits.

In the earliest days of the clan kingdom, the relationship between the individual and the collective was simple. Eventually, though, a single king might direct the actions of a significant number of people, such as men conscripted to build a temple. Then, as smaller kingdoms got subsumed under larger organizations, the collective's complexity increased. But that meant a decrease in complexity for any given individual in the system. In other words, the more you added layers of management, the simpler the orders became. For a slave working in the clan kingdom, the task might be: Dig up slabs from the quarry and organize them by size and color. For slaves working in an empire, the job might only be Dig up slabs from the quarry. A separate task, Organize them according to a plan, would go to another slave.

"Large-scale human systems executed relatively simple behaviors," writes complexity scientist, Yaneer Bar-Yam, “and individuals performed relatively simple individual tasks that were repeated by many individuals over time to have a large-scale effect." So you could have many people working on large-scale projects, such as construction or farming.

But in time, the behavior of the individuals became more varied, as did the collective tasks. Diverse individual actions imply the system overall is becoming more complex. Building a temple would be far more straightforward than managing a distant satrapy.

"This required reducing the branching ratio by adding layers of management that served to exercise local control," Bar-Yam writes. He adds,

As viewed by higher levels of management, each layer simplified the behavior to a point where an individual could control it. The hierarchy acts as a mechanism for communication of information to and from management.

We can imagine messages passing up and down chains of command. Caesar expresses a preference for features in the new coliseum. That message would go to the governor of the Gauls. The governor would discuss matters with the builder, and so on down the line. Those at the site might discuss feedback and report it back up to Caesar, through lots of wax, parchment, and jawboning.

Bar-Yam notes that management starts to act as a filter. Information gets reduced on the way up. Today we call this "briefing." Executives at the top have to process data to make decisions. Without briefs, executives become overloaded. But as decisions get made, information flows down the chain of command. Directives can get more complicated as they travel back down. Middle management is delegated with authority to ensure the simpler executive strategies are sufficiently bulked up and broken down into tactics. Such usually takes the form of detailed plans, divided and subdivided into tasks to be performed by subordinates. As information increases and becomes more varied, the layers of hierarchy stack up.

But when people are arranged in this structure, growing collective complexity depends on reducing individual complexity. This can mean making once-rich information thinner so that subordinates can process it. Whether we're talking about soldiers on the front line or workers on an assembly line, task specialization requires simplification. That means you need more people doing different things, but each person doing fewer things. Again, such branching means more collective complexity, but less individual complexity. 

Trouble is, the branching can't go on forever. Whether in ancient Rome or modern Washington, something has to give.

The Breakdown of Hierarchy

"The point at which the collective complexity is the maximum individual complexity, the process breaks down," says Bar-Yam. 

And this is true for anyone in any management layer, all the way up. In hierarchies, the system retains coherence by having a single person make decisions. But that system is only as complex as its most complex decisionmaker. Hierarchies can provide no greater complexity. Beyond that, there must be some transition. Either managers and subordinates lateralize, taking on more independent decision-making authority, or the system collapses. 

As we start to understand these limits, it's silly to talk about how this or that president handles a complex crisis. More importantly, it is foolish to think he ought to handle such crises.

Nobel Prize-winning economist Friedrich A. Hayek similarly cautioned us. When central economic planning and make-work programs were all the rage, Hayek saw a fatal flaw. Knowledge is distributed among as many individuals as there are in society. Each person's knowledge is informed by the "particular circumstances of time and place."

Hayek's insight is more important than ever. In our dynamic world, the circumstances of time and place are not only mind-bogglingly diverse but changing at a dizzying rate. That's why the great economist urged humanity to put more faith in decentralized entrepreneurs than in centralized High Minds. Not only do entrepreneurs have incentives to focus on local knowledge; they must respect the knowledge transmitted in market prices. 

Who cares about prices?

Prices are information wrapped in incentives. That means prices allow individuals to take action in a vast sea of other minds with different perspectives. Social plans are the purview of experts but almost always formulated far away from our peculiar circumstances, neither informed nor incentivized by the pheromone trails of price signals. Instead, plans originate in the technocrat's mind based on 'knowledge' contrived in the vapors of abstraction. Knowledge forged in the fires of experience, frequently informed by modest failures, discipline the entrepreneur. Of course, the planner will almost always cite some 'study' that confirms her priors. But studies rarely provide local knowledge. And that's their flaw.

I hope you'll permit a long quotation, Dear Reader, to drive this point home. It was written by Scott Shane, a Moscow correspondent for the Baltimore Sun, from 1988 to 1991. In his book Dismantling Utopia: How Information Ended the Soviet Union, Shane offers a passage that should be taught in every economics class. 

My informal survey suggested that some of the longest lines in Moscow were for shoes. At first I assumed that the inefficient Soviet economy did not produce enough shoes, and for that reason, even in the capital, people were forced to line up for hours to buy them…. Then I looked up the statistics.

I was wrong. The Soviet Union was the largest producer of shoes in the world. It was turning out 800 million pairs of shoes a year—twice as many as Italy, three times as many as the United States, four times as many as China. Production amounted to more than three pairs of shoes per year for every Soviet man, woman, and child.

The problem with shoes, it turned out, was not an absolute shortage. It was a far more subtle malfunction. The comfort, the fit, the design, and the size mix of Soviet shoes were so out of sync with what people needed and wanted that they were willing to stand in line for hours to buy the occasional pair, usually imported, that they liked

At the root of the dysfunction was the state's control of information. Prices are information—the information producers need in order to know what and how much to produce. In a market for a product as varied in material and design as footwear, shifting prices are like sensors taped to the skin of a patient in a medical experiment; they provide a constant flow of information about consumer needs and preferences. When the state controlled information, it deprived producers of information about demand.

Shane’s observation doesn't just extend to shoes. When it comes to the forces of Eros Feminine, it extends to every desire of the human heart that anyone can serve. 

That's why Hayek tutted the tendency to privilege abstract expertise over real feedback. Prices flow in dynamic relationships to different currents of local knowledge, which animate our human ecosystems. Prices are thus liquid, changing, and decentralized. 

The architects of large hierarchies tend to prize abstraction over local knowledge, that is, unless they can use big data somehow. Big data allows a hierarchy to grow beyond what humans could accomplish if left to their own devices. But the data must still be interpreted by human brains at some point. Until AI is sufficiently advanced, big data can only do so much. That's because technocratic ‘knowledge’ still works only in predictable patterns of cause and effect, as some part pulled out of a fuselage. Emergence and evolution don't compute with a High Mind who needs to know someone is in control, even as he considers the universe.

"There cannot be design without a designer; contrivance without a contriver; order without choice," wrote Christian apologist William Paley in 1800. 

Paley's position is known as the argument from design, and some use it in theological debates. And Paley's intuitions here are not unwarranted. Emergent systems are counterintuitive. But if intelligent-design thinking has damaged our understanding of cosmology and evolution, it has done more damage to economics. Paley would have been just as gobsmacked as the Soviet apparatchiks as they tried to figure out how a million preferences could design a shoe while a million shoes, contrived by a brilliant contriver, could sit in St. Petersburg warehouses collecting dust.

Through the lens of Eros Feminine, it's easy to see how a Soviet factory could fail to produce the right amount of shoes, or screws, or loaves of bread, all of which most of us take for granted. But this lens should also prompt us to be suspicious of what America's planners are up to today. They're not like the Soviets, exactly. Their plans are more insidious because they're less obvious. Instead of big, nationalized industries, we have a complicated array of dependent industries in a dirigisme matrix. They depend on some regulation here, a subsidy or bailout there. In the American system, planners can paper over errors for a long time. But in time, the failures accumulate.

Authorities simply cannot adequately respond to the decision-making challenges. 

"This is true whether we're talking about dictatorships, or communism … or representative democracies today," Bar-Yam warns.

Hierarchies fail because neither self-appointed dictators nor elected officials can identify what is right for a complex society. Never mind that society is composed of diverse individuals. It's all beyond the comprehension of any single person.

Regardless of any planner's intended result, the outcomes will vary. 

In diversity and scale, society has changed drastically since the time of the pharaohs. Yet most people believe that significant decisions can and ought to be made by authorities on everyone's behalf. Maybe it's a plan to build a new nation amid sectarian violence so democracy will soften the hearts of religious extremists. Maybe it's subsidies for Solyndra solar cells or GE Wind turbines to fashion the energy sector in the image of Bill McKibbin. Maybe it's a scheme to ban worker poverty by setting a price floor on labor, the same rate in Mississippi and Manhattan. Maybe it's a plan to lock everyone down in a pandemic and provide bread and circuses until it goes away. Maybe it's a plan to make healthcare more affordable that somehow makes it less affordable. 

These schemes burn like fires in the planner's mind but so often turn into charred detritus. The red ink hides the original blueprints, but they're there. By the time the errors come to light, the planners have moved on to the next scheme. 

As state hierarchy grows, entrepreneurs learn to find a way to survive in the technocratic matrix. Some find it's good for business because it's bad for their competitors. The paradox here is that these predator corporations become enormous. Their gigantism prompts popular outcries, some of which are justified. Others are just scapegoating. It's fashionable mindlessly to blame 'the corporations' and to think authorities are there to save us. Ironically, voters oblige the technocrats to rein in the very behemoths they had a hand in creating. Regulation almost always makes these companies even bigger as upstarts can't afford to comply. With little competition, the big boys grow bigger. New ventures that would diminish their outsized profits and monopoly growth never come into existence at all.

These unborn ventures trace invisible skylines as ghost firms in a parallel universe—a world that never was. 

When a technocrat sets plans in motion from some federal agency's top floor, she almost always claims knowledge she does not have and destroys phenomena she cannot see. It's counterintuitive. But it's just one reason why it's rash to centralize anything important. Whenever you hear someone cry 'Trust the experts' during a crisis, that's precisely the time to watch, wait, and question. 

Order emerges from human action and not from human design. Because the technocrats' knowledge is limited, technocratic hierarchies are limited. The knowledge required to 'run' the federal government effectively would fry anyone's neural circuitry. Likewise, if a given system isn't equipped to process its own complexity, one of two things will happen: change or collapse. 

It's time, therefore, to mount an ardent challenge to technocracy. Make no mistake; we are still living in a technocratic age. But that age is coming to a close. The only question is whether it will go out with a bang or a whimper.