Antinatalism and reducing suffering: A case of suspicious convergence

First published: Feb. 2021. Last update: Feb. 2021.

Two positions are worth distinguishing. One is the view that we should reduce (extreme) suffering as much as we can for all sentient beings. The other is the view that we should advocate for humans not to have children.

It may seem intuitive to think that the former position implies the latter. That is, to think that the best way to reduce suffering for all sentient beings is to advocate for humans not to have children. My aim in this brief essay is to outline some of the reasons to be skeptical of this claim.

Suspicious convergence

Lewis, 2016 warns of “suspicious convergence”, which he introduces with the following toy example:

Oliver: … Thus we see that donating to the opera is the best way of promoting the arts.

Eleanor: Okay, but I’m principally interested in improving human welfare.

Oliver: Oh! Well I think it is also the case that donating to the opera is best for improving human welfare too.

The general point is that, for any set of distinct altruistic aims or endeavors we may consider, we should be a priori suspicious of the claim that they are perfectly convergent — i.e. that directly pursuing one of them also happens to be the very best thing we can do for achieving the other. Justifying such a belief would require good, object-level reasons. And in the case of the respective endeavors of reducing suffering and advocating for humans not to procreate, we in a sense find the opposite, as there are good reasons to be skeptical of a strong degree of convergence, and even to think that such antinatalist advocacy might increase future suffering.

The marginal impact of antinatalist advocacy

A key point when evaluating the impact of altruistic efforts is that we need to think at the margin: how does our particular contribution change the outcome, in expectation? This is true whether our aims are modest or maximally ambitious — our actions and resources still represent but a very small fraction of the total sum of actions and resources, and we can still only exert relatively small pushes toward our goals.

Direct effects

What, then, is the marginal impact of advocating for people not to have children? One way to try to answer this question is to explore the expected effects of preventing a single human birth. Antinatalist analyses of this question are quick to point out the many harms caused by a single human birth, which must indeed be considered. Yet what these analyses tend not to consider are the harms that a human birth would prevent.

For example, in his book Better Never to Have Been, David Benatar writes about “the suffering inflicted on those animals whose habitat is destroyed by encroaching humans” (p. 224) — which, again, should definitely be included in our analysis. Yet he fails to consider the many births and all the suffering that would be prevented by an additional human birth, such as due to its marginal effects on habitat reduction (“fewer people means more animals“). As Brian Tomasik argues, when we consider a wider range of the effects humans have on animal suffering, “it seems plausible that encouraging people to have fewer children actually causes an increase in suffering and involuntary births.” 

This highlights how a one-sided analysis such as Benatar’s is deeply problematic when evaluating potential interventions. We cannot simply look at the harms prevented by our pet interventions without considering how they might lead to more harm. Both things must be considered.

To be clear, the considerations above regarding the marginal effects of human births on animal suffering by no means represent a complete analysis of the effects of additional human births, or of advocating for humans not to have children. But they do represent compelling reasons to doubt that such advocacy is among the best things we can do to reduce suffering for all sentient beings, at least in terms of the direct effects, which leads us to the next point.

Long-term effects

Some seem to hold that the main reason to advocate against human procreation is not the direct effects, but rather its long-term effects on humanity’s future. I agree that the influence our ideas and advocacy efforts have on humanity’s long-term future are plausibly the most important thing about them, and I think many antinatalists are likely to have a positive influence in this regard by highlighting the moral significance of suffering (and the relative insignificance of pleasure).

But the question is why we should think that the best way to steer humanity’s long-term future toward less suffering is to argue for people not to have children. After all, the space of possible interventions we could pursue to reduce future suffering is vast, and it would be quite a remarkable coincidence if relatively simple interventions — such as advocating for antinatalism or veganism — happened to be the very best way to reduce suffering, or even among the best.

In particular, the greatest risk from a long-term perspective is that things somehow go awfully wrong, and that we counterfactually greatly increase future suffering, either by creating additional sources of suffering in the future, or by simply failing to reduce existing forms of suffering when we could. And advocating for people not to have children seems unlikely to be among the best ways to reduce the risk of such failures — again since the space of possible interventions is vast, and interventions that are targeted more directly at reducing these risks, including the risk of leaving wild-animal suffering unaddressed, are probably significantly more effective than is advocating for humans not to procreate.

Better alternatives?

If our aim is to reduce suffering for all sentient beings, a plausible course of action would be to pursue an open-ended research project on how we can best achieve this aim. This is, after all, not a trivial question, and we should hardly expect the most plausible answers to be intuitive, let alone obvious. Exploring this question requires epistemic humility, and forces us to contend with the vast amount of empirical uncertainty facing any endeavor to create a better world.

I have explored this question at length in Vinding, 2020, part II, as have other individuals and organizations elsewhere. One conclusion that seems quite robust is that we should focus mostly on avoiding bad outcomes, whereas comparatively suffering-free future scenarios merit less priority. Another robust conclusion is that we should pursue a pragmatic and cooperative approach when trying to reduce suffering (see also Vinding, 2020, ch. 10) — not least since future conflicts are one of the main ways in which worst-case outcomes might materialize, and hence we should generally strive to reduce the risk of such conflicts.

In more concrete terms, antinatalists may be more effective if they focus on defending antinatalism for wild animals in particular. This case seems easier to make given the overwhelming amount of suffering and early death in nature, it pertains to a larger number of beings, and it may have more beneficial near-term and long-term effects — being less at risk of increasing non-human suffering in the near term, and plausibly being more conducive to reducing worst-case risks, whether these entail spreading non-human life or simply failing to reduce wild-animal suffering.

Broadly speaking, the aim of reducing suffering would seem to recommend efforts to identify the main ways in which humanity might cause — or prevent — vast amounts of suffering in the future, and to find out how we can best navigate accordingly. None of these conclusions seem to support efforts to convince people not to have children as a particularly promising strategy, though they likely do recommend efforts to promote concern for suffering more generally.

Conversation with David Pearce about digital sentience and the binding problem

Whether digital sentience is possible would seem to matter greatly for our priorities, and so gaining even slightly more refined views on this matter could be quite valuable. Many people appear to treat the possibility, if not indeed the imminence, of digital sentience as a foregone conclusion. David Pearce, in contrast, is skeptical.

Pearce has written and spoken elaborately about his views on consciousness. My sense, however, is that these expositions do not always manage to clearly convey the core, and actually very simple reasons underlying Pearce’s skepticism of digital sentience. The aim of this conversation is to probe Pearce so as to shed greater — or perhaps most of all simpler — light on why he is skeptical, and thus to hopefully advance the discussions on this issue among altruists working to reduce future suffering.


MV: You are skeptical about the possibility of digital sentience. Could you explain why, in simple terms?

DP: Sure. Perhaps we can start by asking why so many people believe that our machines will become conscious (cf. https://www.hedweb.com/quora/2015.html#definition). Consciousness is widely recognised to be scientifically unexplained. But the computer metaphor of mind seems to offer us clues (cf. https://www.hedweb.com/quora/2015.html#braincomp). As far as I can tell, many if not most believers in digital sentience tend to reason along the following lines. Any well-defined cognitive task that the human mind can perform could also be performed by a programmable digital computer (cf. https://en.wikipedia.org/wiki/Turing_machine). A classical Turing machine is substrate-neutral. By “substrate-neutral”, we mean that whether a Turing machine is physically constituted of silicon or carbon or gallium oxide (etc) makes no functional difference to the execution of the program it runs. It’s commonly believed that the behaviour of a human brain can, in principle, be emulated on a classical Turing machine. Our conscious minds must be identical with states of the brain. If our minds weren’t identical with brain states, then dualism would be true (cf. https://www.hedweb.com/quora/2015.html#dualidealmat). Therefore, the behaviour of our minds can in principle be emulated by a digital computer. Moreover, the state-space of all possible minds is immense, embracing not just the consciousness of traditional and enhanced biological lifeforms, but also artificial digital minds and maybe digital superintelligence. Accordingly, the belief that non-biological information-processing machines can’t support consciousness is arbitrary. It’s unjustified carbon chauvinism.

I think most believers in digital sentience would recognise that the above considerations are not a rigorous argument for the existence of inorganic machine consciousness. The existence of machine consciousness hasn’t been derived from first principles. The “explanatory gap” is still unbridged. Yet what is the alternative?

Well, as a scientific rationalist, I’m an unbeliever. Digital computers and the software they run are not phenomenally-bound subjects of experience (cf. https://www.binding-problem.com/). Ascribing sentience to digital computers or silicon robots is, I believe, a form of anthropomorphic projection — a projection their designers encourage by giving their creations cutesy names (“Watson”, “Sophia”, “Alexa” etc). 

Before explaining my reasons for believing that digital computers are zombies, I will lay out two background assumptions. Naturally, one or both assumptions can be challenged, though I think they are well-motivated.

The first background assumption might seem scarcely relevant to your question. Perpetual direct realism is false (cf. https://www.hedweb.com/quora/2015.html#distort). Inferential realism about the external world is true. The subjective contents of your consciousness aren’t merely a phenomenally thin and subtle serial stream of logico-linguistic thought-episodes playing out behind your forehead, residual after-images when you close your eyes, inner feelings and emotions and so forth. Consciousness is also your entire phenomenal world-simulation — what naïve realists call the publicly accessible external world. Unless you have the neurological syndromes of simultanagnosia (the inability to experience more than one object at once) or akinetopsia (“motion blindness”), you can simultaneously experience a host of dynamic objects — for example, multiple players on a football pitch, or a pride of hungry lions. These perceptual objects populate your virtual world of experience from the sky above to your body-image below. Consciousness is all you directly know. The external environment is an inference, not a given.

Let’s for now postpone discussion of how our skull-bound minds are capable of such an extraordinary feat of real-time virtual world-making. The point is that if you couldn’t experience multiple feature-bound phenomenal objects — i.e. if you were just an aggregate of 86 billion membrane-bound neuronal “pixels” of experience — then you’d be helpless. Compare dreamless sleep. Like your enteric nervous system (the “brain-in-the-gut”), your mind-brain would still be a fabulously complex information-processing system. But you’d risk starving to death or getting eaten. Waking consciousness is immensely adaptive. (cf. https://www.hedweb.com/quora/2015.html#evolutionary). Phenomenal binding is immensely adaptive (cf. https://www.hedweb.com/quora/2015.html#purposecon).

My second assumption is physicalism (cf. https://www.hedweb.com/quora/2015.html#materialism). I assume the unity of science. All the special sciences (chemistry, molecular biology etc) reduce to physics. In principle, the behaviour of organic macromolecules such as self-replicating DNA can be described entirely in the mathematical language of physics without mentioning “life” at all, though such high-level description is convenient. Complications aside, no “element of reality” is missing from the mathematical formalism of our best theory of the world, quantum mechanics, or more strictly from tomorrow’s unification of quantum field theory and general relativity.

One corollary of physicalism is that only “weak” emergence is permissible. “Strong” emergence is forbidden. Just as the behaviour of programs running on your PC supervenes on the behaviour of its machine code, likewise the behaviour of biological organisms can in principle be exhaustively reduced to quantum chemistry and thus ultimately to quantum field theory. The conceptual framework of physicalism is traditionally associated with materialism. According to materialism as broadly defined, the intrinsic nature of the physical — more poetically, the mysterious “fire” in the equations — is non-experiential. Indeed, the assumption that quantum field theory describes fields of insentience is normally treated as too trivially obvious to be worth stating explicitly. However, this assumption of insentience leads to the Hard Problem of consciousness. Non-materialist physicalism (cf. https://www.hedweb.com/quora/2015.html#galileoserror) drops this plausible metaphysical assumption. If the intrinsic nature argument is sound, there is no Hard Problem of consciousness: it’s the intrinsic nature of the physical (cf. https://www.hedweb.com/quora/2015.html#definephysical ). However, both “materialist” physicalists and non-materialist physicalists agree: everything that happens in the world is constrained by the mathematical straitjacket of modern physics. Any supposedly “emergent” phenomenon must be derived, ultimately, from physics. Irreducible “strong” emergence would be akin to magic.

Anyhow, the reason I don’t believe in digital minds is that classical computers are, on the premises outlined above, incapable of phenomenal binding. If we make the standard assumption that their 1 and 0s and logic gates are non-experiential, then digital computers are zombies. Less obviously, digital computers are zombies if we don’t make this standard assumption! Imagine, fancifully, replacing non-experiential 1s and 0s of computer software with discrete “pixels” of experience. Run the program as before. The upshot will still be a zombie, more technically a micro-experiential zombie. What’s more, neither increasing the complexity of the code nor exponentially increasing the speed of its execution could cause discrete “pixels” somehow to blend into each other in virtue of their functional role, let alone create phenomenally-bound perceptual objects or a unitary self experiencing a unified phenomenal world. The same is true of a connectionist system (cf. https://en.wikipedia.org/wiki/Connectionism), supposedly more closely modelled on the brain — however well-connected and well-trained the network, and regardless whether its nodes are experiential or non-experiential. The synchronous firing of distributed feature-processors in a “trained up” connectionist system doesn’t generate a unified perceptual object — again on pain of “strong” emergence. AI programmers and roboticists can use workarounds for the inability of classical computers to bind, but they are just that: workarounds.

Those who believe in digital sentience can protest that we don’t know that phenomenal minds can’t emerge at some level of computational abstraction in digital computers. And they are right! If abstract objects have the causal power to create conscious experience, then digital computer programs might be subjects of experience. But recall we’re here assuming physicalism. If physicalism is true, then even if consciousness is fundamental to the world, we can know that digital computers are — at most — micro-experiential zombies.

Of course, monistic physicalism may be false. “Strong” emergence may be real. But if so, then reality seems fundamentally lawless. The scientific world-picture would be false.

Yet how do biological minds routinely accomplish binding if phenomenal binding is impossible for any classical digital computer (cf. https://en.wikipedia.org/wiki/Universal_Turing_machine). Even if our neurons support rudimentary “pixels” of experience, why aren’t animals like us in the same boat as classical digital computers or classically parallel connectionist systems?

I can give you my tentative answer. Naïvely, it’s the reductio ad absurdum of quantum mind: “Schrödinger’s neurons”: https://www.hedweb.com/quora/2015.html#quantumbrain.

Surprisingly, it’s experimentally falsifiable via interferometry: https://en.wikipedia.org/wiki/Quantum_mind#David_Pearce 

Yet the conjecture I explore may conceivably be of interest only to someone who already feels the force of the binding problem. Plenty of researchers would say it’s a ridiculous solution to a nonexistent problem. I agree it’s crazy; but it’s worth falsifying. Other researchers just lump phenomenal binding together with the Hard Problem (cf. https://www.hedweb.com/quora/2015.html#categorize) as one big insoluble mystery they suppose can be quarantined from the rest of scientific knowledge.

I think their defeatism and optimism alike are premature. 

MV: Thanks, David. A lot to discuss there, obviously.

Perhaps the most crucial point to really appreciate in order to understand your skepticism is that you are a strict monist about reality. That is, “the experiential” is not something over and above “the physical”, but rather identical with it (which, to be clear, does not imply that all physical things have minds, or complex experiences). And so if “the mental” and “the physical” are essentially the same ontological thing, or phenomenon, under two different descriptions, then there must, roughly speaking, also be a match in terms of their topological properties.

As Mike Johnson explained your view: “consciousness is ‘ontologically unitary’, and so only a physical property that implies ontological unity … could physically instantiate consciousness.” (Principia Qualia, p. 73). (Note that “consciousness” here refers to an ordered, composite mind; not phenomenality more generally.)

Conversely, a system that is physically discrete or disconnected — say, a computer composed of billiard balls that bump into each other, or lighthouses that exchange signals across hundreds of kilometers — could not, on your view, support a unitary mind. In terms of the analogy of thinking about consciousness as waves, your view is roughly that we should think of a unitary mind as a large, composite wave of sorts, akin to a song, whereas disconnected “pixels of experience” are like discrete microscopic proto-waves, akin to tiny disjoint blobs of sound. (And elsewhere you quote Seth Lloyd saying something similar about classical versus quantum computations: “A classical computation is like a solo voice — one line of pure tones succeeding each other. A quantum computation is like a symphony — many lines of tones interfering with one another.”)

This is why you say that “computer software with discrete ‘pixels’ of experience will still be a micro-experiential zombie”, and why you say that “even if consciousness is fundamental to the world, we can know that digital computers are at most micro-experiential zombies” — it’s because of this physical discreteness, or “disconnectedness”.

And this is where it seems to me that the computational view of mind is also starkly at odds with common sense, as well as with monism. For it seems highly counterintuitive to claim that billiard balls bumping into each other, or lighthouses separated by hundreds of kilometers that exchange discrete signals, could, even in principle, mediate a unitary mind. I wonder whether most people who hold a computational view of mind are really willing to bite this bullet. (Such views have also been elaborately criticized by Mike Johnson and Scott Aaronson — critiques that I have seen no compelling replies to.)

It also seems non-monistic in that it appears impossible to give a plausible account of where a unitary mind is supposed to be found in this picture (e.g. in a picture with discrete computations occurring serially over long distances), except perhaps as a separate, dualist phenomenon that we somehow map onto a set of physically discrete computations occurring over time, which seems to me inelegant and unparsimonious. Not to mention that it gives rise to an explosion of minds, as we can then see minds in a vast set of computations that are somehow causally connected across time and space, with the same computations being included in many distinct minds. This picture is at odds with a monist view that implies a one-to-one correspondence between concrete physical state and concrete mental state — or rather, which sees these two sides as distinct descriptions of the exact same reality.

The question is then how phenomenal binding could occur. You explore a quantum mind hypothesis involving quantum coherence. So what are your reasons for thinking that quantum coherence is necessary for phenomenal binding? Why would, say, electromagnetic fields in a synchronous state not be enough?

DP: If the phenomenal unity of mind is an effectively classical phenomenon, then I have no idea how to derive the properties of our phenomenally bound minds from decohered, effectively classical neurons — not even in principle, let alone in practice. 

MV: And why is that? What is it that makes deriving the properties of our phenomenally bound minds seem feasible in the case of coherent states, unlike in the case of decohered ones?

DP: Quantum coherent states are individual states — i.e. fundamental physical features of the world — not mere unbound aggregates of classical mind-dust. On this story, decoherence (cf. https://arxiv.org/pdf/1911.06282.pdf) explains phenomenal unbinding.

MV: So it is because only quantum coherent states could constitute the “ontological unity” of a unitary, “bound” mind. Decoherent states, on your view, are not and could not be ontologically unitary in the required sense?

DP: Yes!

Digital computing depends on effectively classical, decohered individual bits of information, whether as implemented in Turing’s original tape set-up, a modern digital computer, or indeed if the world’s population of skull-bound minds agree to participate in an experiment to see if a global mind can emerge from a supposed global brain.

One can’t create perceptual objects, let alone unified minds, from classical mind-dust even if strictly the motes of decohered “dust” are only effectively classical, i.e. phase information has leaked away into the environment. If the 1s and 0s of a digital computer are treated as discrete micro-experiential pixels, then when running a program, we don’t need to consider the possibility of coherent superpositions of 1s and 0s/ micro-experiences. If the bits weren’t effectively classical and discrete, then the program wouldn’t execute.

MV: In other words, you are essentially saying that binding/unity between decohered states is ultimately no more tenable than binding/unity between, say, two billard balls separated by a hundred miles? Because they are in a sense similarly ontologically separate?

DP: Yes!

MV: So to summarize, your argument is roughly the following: 

  1. observed phenomenal binding, or a unitary mind, combined with 
  2. an empirically well-motivated monistic physicalism, means that
  3. we must look for a unitary physical state as the “mediator”, or rather the physical description, of mind [since the ontological identity from (2) implies that the phenomenal unity from (1) must be paralleled in our physical description], and it seems that
  4. only quantum coherent states could truly fit the bill of such ontological unity in physical terms.

DP: 1 to 4, yes!

MV: Cool. And in step 4 in particular, to spell that out more clearly, the reasoning is roughly that classical states are effectively (spatiotemporally) serial, discrete, disconnected, etc. Quantum coherent states, in contrast, are a connected, unitary, individual whole.

Classical bits in a sense belong to disjoint “ontological sets”, whereas qubits belong to the same “ontological set” (as I’ve tried to illustrate somewhat clumsily below, and in line with Seth Lloyd’s quote above).

Is that a fair way to put it?

DP: Yes!

I sometimes say who will play Mendel to Zurek’s Darwin is unknown. If experience discloses the intrinsic nature of the physical, i.e. if non-materialist physicalism is true, then we must necessarily consider the nature of experience at what are intuitively absurdly short timescales in the CNS. At sufficiently fine-grained temporal resolutions, we can’t just assume the existence of decohered macromolecules, neurotransmitters, receptors, membrane-bound neurons etc. — they are weakly emergent, dynamically stable patterns of “cat states”. These high-level patterns must be derived from quantum bedrock — which of course I haven’t done. All I’ve done is make a “philosophical” conjecture that (1) quantum coherence mediates the phenomenal unity of our minds; and (2) quantum Darwinism (cf. https://www.sciencemag.org/news/2019/09/twist-survival-fittest-could-explain-how-reality-emerges-quantum-haze) offers a ludicrously powerful selection-mechanism for sculpting what would otherwise be mere phenomenally-bound “noise”.

MV: Thanks for that clarification.

I guess it’s also worth stressing that you do not claim this to be any more than a hypothesis, while you at the same time admit that you have a hard time seeing how alternative accounts could explain phenomenal binding.

Moreover, it’s worth stressing that the conjecture resulting from your line of reasoning above is in fact, as you noted, a falsifiable one — a rare distinction for a theory of consciousness.

A more general point to note is that skepticism about digital sentience need not be predicated on the conjecture you presented above, as there are other theories of mind — not necessarily involving quantum coherence — that also imply that digital computers are unable to mediate a conscious mind (including some of the theories hinted at above, and perhaps other, more recent theories). For example, one may accept steps 1-3 in the argument above, and then be more agnostic in step 4, with openness to the possibility that binding could be achieved in other ways, yet while still considering contemporary digital computers unlikely to be able to mediate a unitary mind (e.g. because of the fundamental architectural differences between such computers and biological brains).

Okay, having said all that, let’s now move on to a slightly different issue. Beyond digital sentience in particular, you have also expressed skepticism regarding artificial sentience more generally (i.e. non-digital artificial sentience). Can you explain the reasons for this skepticism?

DP: Well, aeons of posthuman biological minds probably lie ahead. They’ll be artificial — genetically rewritten, AI-augmented, most likely superhumanly blissful, but otherwise inconceivably alien to Darwinian primitives. My scepticism is about the supposed emergence of minds in classical information processors — whether programmable digital computers, classically parallel connectionist systems or anything else.

What about inorganic quantum minds? Well, I say a bit more e.g. here: https://www.hedweb.com/quora/2015.html#nonbiological

A pleasure-pain axis has been so central to our experience that sentience in everything from worms to humans is sometimes (mis)defined in terms of the capacity to feel pleasure and pain. But essentially, I see no reason to believe that such (hypothetical) phenomenally bound consciousness in future inorganic quantum computers will support a pleasure-pain axis any more than, say, the taste of garlic.

In view of our profound ignorance of physical reality, however, I’m cautious: this is just my best guess!

MV: Interesting. You note that you see no reason to believe that such systems would have a pleasure-pain axis. But what about the argument that pain has proven exceptionally adaptive over the course of biological evolution, and might thus plausibly prove adaptive in future forms of evolution as well (assuming things won’t necessarily be run according to civilized values)? 

DP: Currently, I can’t see any reason to suppose hedonic tone (or the taste of garlic) could be instantiated in inorganic quantum computers. If (a big “if”) the quantum-theoretic version of non-materialist physicalism is true, then subjectively it’s like something to be an inorganic quantum computer, just as it’s like something subjectively to be superfluid helium — a nonbiological macro-quale. But out of the zillions of state-spaces of experience, why expect the state-space of phenomenally-bound experience that inorganic quantum computers hypothetically support will include hedonic tone? My guess is that futuristic quantum computers will instantiate qualia for which humans have no name nor conception and with no counterpart in biological minds.

All this is very speculative! It’s an intuition, not a rigorous argument.

MV: Fair enough. What then is your view of hypothetical future computers built from biological neurons?

DP: Artificial organic neuronal networks are perfectly feasible. Unlike silicon-based “neural networks” — a misnomer in my view — certain kinds of artificial organic neuronal networks could indeed suffer. Consider the reckless development of “mini-brains”.

MV: Yeah, it should be uncontroversial that such developments entail serious risks.

Okay, David. What you have said here certainly provides much food for thought. Thanks a lot for patiently exploring these issues with me, and not least for all your work and your dedication to reducing the suffering of all sentient beings.

DP: Thank you, Magnus. You’re very kind. May I just add a recommendation? Anyone who hasn’t yet done so should read your superb Suffering-Focused Ethics (2020).

Two biases relevant to expected AI scenarios

My aim in this essay is to briefly review two plausible biases in relation to our expectations of future AI scenarios. In particular, these are biases that I think risk increasing our estimates of the probability of a local, so-called FOOM takeoff.

An important point to clarify from the outset is that these biases, if indeed real, do not in themselves represent reasons to simply dismiss FOOM scenarios. It would clearly be a mistake to think so. But they do, I submit, constitute reasons to be somewhat more skeptical of them, and to re-examine our beliefs regarding FOOM scenarios. (Stronger, more direct reasons to doubt FOOM have been reviewed elsewhere.)

Egalitarian intuitions looking for upstarts

The first putative bias has its roots in our egalitarian origins. As Christopher Boehm argues in his Hierarchy in the Forrest, we humans evolved in egalitarian tribes in which we created reverse dominance hierarchies to prevent domineering individuals from taking over. Boehm thus suggests that our minds are built to be acutely aware of the potential for any individual to rise and take over, perhaps even to the extent that we have specialized modules whose main task is to be attuned to this risk.

Western “Great Man” intuitions

The second putative bias is much more culturally contingent, and should be expected to be most pronounced in Western (“WEIRD“) minds. As Joe Henrich shows in his book The WEIRDest People in the World, Western minds are uniquely focused on individuals, so much so that their entire way of thinking about the world tends to revolve around individuals and individual properties (as opposed to thinking in terms of collectives and networks, which is more common among East Asian cultures).

The problem is that this Western, individualist mode of thinking, when applied straightforwardly to the dynamics of large-scale societies, is quite wrong. For while it may be mnemonically pragmatic to recount history, including the history of ideas and technology, in terms of individual actions and decisions, the truth is usually far more complex than this individualist narrative lets on. As Henrich argues, innovation is largely the product of large-scale systemic factors (such as the degree of connectedness between people), and these factors are usually far more important than is any individual, suggesting that Westerners tend to strongly overestimate the role that single individuals play in innovation and history more generally. Henrich thus alleges that the Western way of thinking about innovation reflects an “individualism bias” of sorts, and further notes that:

thinking about individuals and focusing on them as having dispositions and kind of always evaluating everybody [in terms of which] attributes they have … leads us to what’s called “the myth of the heroic inventor”, and that’s the idea that the great advances in technology and innovation are the products of individual minds that kind of just burst forth and give us these wonderful inventions. But if you look at the history of innovation, what you’ll find time after time was that there was lucky recombinations, people often invent stuff at the same time, and each individual only makes a small increment to a much larger, longer process.

In other words, innovation is the product of numerous small and piecemeal contributions to a much greater extent than Western “Great Man” storytelling suggests. (Of course, none of this is to say that individuals are unimportant, but merely that Westerners seem likely to vastly overestimate the influence that single individuals have on history and innovation.)

Upshot

If we have mental modules specialized to look for individuals that accumulate power and take control, and if we have expectations that roughly conform to this pattern in the context of future technology, with one individual entity innovating its way to a takeover, it seems that we should at least wonder whether this expectation may derive partly from our forager-age intuitions rather than resting purely on solid epistemics. Especially when this view of the future seems in strong tension with our actual understanding of innovation. This understanding being that innovation — contra Western intuition — is distributed, with increases in abilities generally the product of countless “small” insights and tools rather than a few big ones.

Both of the tendencies listed above lead us (or in the second case, mostly Westerners) to focus on individual agents rather than larger, systemic issues that may be crucial to future outcomes, yet which are less intuitively appealing for us to focus on. And there may well be more general explanations for this lack of appeal than just the two reasons listed above. The fact that there were no large-scale systemic issues of any kind for almost all of our species’ history renders it unsurprising that we are not particularly prone to focus on such issues (except for local signaling purposes).

Perhaps we need to control for this, and try to look more toward systemic issues than we are intuitively inclined to do. After all, the claim that the future will be dominated by AI systems in some form need not imply that the best way to influence that future is to focus on individual AI systems, as opposed to broader, institutional issues.

Suffering-focused ethics and the importance of happiness

It seems intuitive to think that suffering-focused moral views imply that it is unimportant whether people live rich and fulfilling lives. Yet the truth, I will argue, is in many ways the opposite — especially for those who are trying to reduce suffering effectively with their limited resources.

Personal sustainability and productivity

A key reason why we need to live fulfilling lives is that we cannot work to reduce suffering in sustainable ways otherwise. Indeed, not only is a reasonably satisfied mind a precondition for sustainable productivity in the long run, but also for our productivity on a day-to-day basis, which is often aided by a strong passion and excitement about our work projects. Suffering-focused ethics by no means entails that excitement and passion should be muted.

Beyond aiding our productivity in work-related contexts, a strong sense of well-being also helps us be more resilient in the face of life’s challenges — things that break, unexpected expenses, unfriendly antagonists, etc. Cultivating a sense of fulfillment and a sound mental health can help us better handle these things as well.

Signaling value

This reason pertains to the social rather than the individual level. If we are trying to create positive change in the world, it generally does not help if we ourselves seem miserable. People often decide whether they want to associate with (or distance themselves from) a group of people based on perceptions of the overall wellness and mental health of its adherents. And while this may seem unfair, it is also not entirely unreasonable, as these factors arguably do constitute some indication of the practical consequences of associating with the group in question.

This hints at the importance of avoiding this outcome; to show that a life in short supply of true fulfillment is not in fact what suffering-focused views ultimately recommend. After all, if failing to prioritize our own well-being has bad consequences in the bigger picture, such as scaring people away from joining our efforts to create a better future, then this failure is not recommended by consequentialist suffering-focused views.

To be clear, my point here is not that suffering-focused agents should be deceptive and try to display a fake and inflated sense of well-being (such deception itself would have many bad consequences). Rather, the point is that we have good reasons to prioritize getting to a place of genuine health and well-being, both for the sake of our personal productivity and our ability to inspire others.

A needless hurdle to the adoption of suffering-focused views

A closely related point has to do with people’s evaluations of suffering-focused views more directly (as opposed to the evaluations of suffering-focused communities and their practical efforts). People are likely to judge the acceptability of a moral view based in part on the expected psychological consequences of its adoption — will it enable me to pursue the lifestyle I want, to maintain my social relationships, and to seem like a good and likeable person?

Indeed, modern moral and political psychology suggests that these social and psychological factors are strong determinants of our moral and political views, and that we usually underestimate just how much these “non-rationalist” factors influence our views (see e.g. Haidt, 2012, part III; Tuschman, 2013, ch. 22; Simler, 2016; Tooby, 2017).

This is then another good reason to seek to both emphasize and exemplify the compatibility of suffering-focused views and a rich and fulfilling life. Again, if failing in this regard tends to prevent people from prioritizing the reduction of suffering, then a true extrapolation of suffering-focused views will militate against such a failure, and instead recommend a focus on cultivating an invitingly healthful state of mind.

In sum, while it may seem counterintuitive, there is in fact no inherent tension between living a happy and rewarding life and at the same time being committed to reducing the most intense forms of suffering. On the contrary, these pursuits can be quite complementary. As I have argued above, living healthy and flourishing lives ourselves is helpful for the endeavor of reducing suffering in various ways. Conversely, being strongly dedicated to this endeavor, while admittedly challenging at times, can positively enhance the richness of our lives, providing us with a powerful source of meaning and purpose. It is okay to revel in the unspeakable profundity and significance of this purpose.

Underappreciated consequentialist reasons to avoid consuming animal products

While there may be strong deontological or virtue-ethical reasons to avoid consuming animal products (“as far as is possible and practicable”), the consequentialist case for such avoidance is quite weak.

Or at least this appears to be a common view in some consequentialist-leaning circles. My aim in this post is to argue against this view. On a closer look, we find many strong consequentialist reasons to avoid the consumption of animal products.

The direct effects on the individuals we eat

99 percent of animals raised for food in the US, and more than 90 percent globally, live out their lives on factory farms. These are lives of permanent confinement to very small spaces, often involving severe abuse, as countless undercover investigations have revealed. And their slaughter frequently involves extreme suffering as well — for example, about a million chickens and turkeys are boiled alive in the US every year, and fish, the vast majority of farmed vertebrates, are usually slaughtered without any stunning. They are routinely suffocated to death, frozen to death, and cut in ways that leave them to bleed to death (exsanguination). 

Increasing such suffering via one’s marginal consumption is bad on virtually all consequentialist views. And note that, empirically, it turns out that people who aspire to avoid meat from factory farmed animals (“conscientious omnivores”) actually often do not (John & Sebo, 2020, 3.2; Rothgerber, 2015). And an even greater discrepancy between ideals and actuality is found in the behavior of those who believe that the animals they eat are “treated well”, which in the US is around 58 percent of people, despite the fact that over 99 percent of farm animals in the US live on factory farms (Reese, 2017).

Furthermore, even in Brian Tomasik’s analyses that factor in the potential of animal agriculture to reduce wild-animal suffering, the consumption of virtually all animal “products” is recommended against — including eggs and meat from fish (farmed and wild-caught), chickens, pigs, and (especially) insects. Brian argues that the impact of not consuming meat is generally positive, both because of the direct marginal impact (“avoiding eating one chicken or fish roughly translates to one less chicken or fish raised and killed”) and because of the broader social effects (more on the latter below).

The above is an important consequentialist consideration against consuming animal products. Yet unfortunately, consequentialist analyses tend to give far too much weight to this consideration alone, and to treat it as the end-all be-all of consequentialist arguments against consuming animal products when, in fact, it is not necessarily even one of the most weighty arguments.

Institutional effects

Another important consideration has to do with the institutional effects of animal consumption. These effects seem superficially similar to those discussed in the previous point, yet they are in fact quite distinct.

Anti-charity

For one, there is the increased financial support to an industry that not only systematically harms currently existing individuals, but which also, perhaps more significantly, actively works to undermine moral concern for future non-human individuals. It does this through influential lobbying activities and by advertising in ways that effectively serve as propaganda against non-human animals (that is certainly what we would call it in the human case if an industry continually worked to legitimize the exploitation and killing of certain human individuals; in fact, “propaganda” may be overly euphemistic).

Supporting this industry can be seen as anti-charity of sorts, as it pushes us away from betterment for non-human animals at the level of our broader institutions. And this effect could well be more significant than the direct marginal impact on non-human beings consumed, as such institutional factors may be a greater determinant of how many such beings will suffer in the future.

Not only are these institutional effects negative for future farmed animals, but the resulting reinforcement of speciesism and apathy toward non-human animals in general likely also impedes concern for wild animals in particular. And given the numbers, this effect may be even more important than the negative effect on future farmed animals.

Anti-activism

Another institutional effect is that, when we publicly buy or consume animal products, we signal to other people that non-human individuals can legitimately be viewed as food, and that we approve of the de facto horrific institution of animal agriculture. This signaling effect is difficult to avoid even if we do not in fact condone most of the actual practices involved. After all, virtually nobody condones the standard practices, such as the castration of pigs without anesthetics. And yet virtually all of us still condone these practices behaviorally, and indeed effectively support their continuation.

In this way, publicly buying or consuming animal products can, regardless of one’s intentions, end up serving as miniature anti-activism against the cause of reducing animal suffering — it serves to normalize a collectively perpetrated atrocity — while choosing to forego such products can serve as miniature activism in favor of the cause.

One may object that the signaling effects of such individual actions are insignificant. Yet we are generally not inclined to say the same about the signaling effects of, say, starkly racist remarks, even when the individuals whom the remarks are directed against will never know about them (e.g. when starkly anti-black sentiments are shared in forums with white people only). The reason, I think, is that we realize that such remarks do have negative effects down the line, and we realize that these effects are not minor.

It is widely acknowledged that, to human psychology, racism is a ticking bomb that we should make a consistent effort to steer away from, lest we corrode our collective attitudes and in turn end up systematically exploiting and harming certain groups of individuals. We have yet to realize that the same applies to speciesism.

For a broader analysis of the social effects of the institution of animal exploitation, see (John & Sebo, 2020, 3.3). Though note that I disagree with John and Sebo’s classical utilitarian premise, which would allow us to farm individuals, and even kill them in the most horrible ways, provided that their lives were overall “net positive” (the horrible death included). I think this notion of “net positive” needs to be examined at length, especially in the interpersonal context where some beings’ happiness is claimed to outweigh the extreme suffering of others.

Influence on our own perception

The influence on our own attitudes and thinking is another crucial factor. Indeed, for a consequentialist trying to think straight about how to prioritize one’s resources for optimal impact, this may be the most important reason not to consume animal products.

Moral denigration is a well-documented effect

Common sense suggests that we cannot think clearly about the moral status of a given group of individuals as long as we eat them. Our evolutionary history suggests the same: it was plausibly adaptive in our evolutionary past to avoid granting any considerable moral status to individuals categorized as “food animals”.

Psychological studies bear out common sense and evolution-based speculation. In Don’t Mind Meat? The Denial of Mind to Animals Used for Human Consumption, Brock Bastian and colleagues demonstrated that people tend to ascribe diminished mental capacities to “food animals”; that “meat eaters are motivated to deny minds to food animals when they are reminded of the link between meat and animal suffering”; and that such mind denial is increased when people expect to eat meat in the near future.

Another study (Bratanova et al., 2011) found that:

categorization as food — but not killing or human responsibility — was sufficient to reduce the animal’s perceived capacity to suffer, which in turn restricted moral concern.

This finding is in line with the prevalence of so-called consistency effects, our psychological tendency to adapt beliefs that support our past and present behavior (see Salamon & Rayhawk’s Cached Selves and Huemer, 2010, “5.d Coherence bias”). For example, “I eat animals, and hence animals don’t suffer so much and don’t deserve great moral consideration”. 

And yet another study (Loughnan et al., 2010) found that the moral numbing effects of meat eating applied to other non-human animals as well, suggesting that these numbing effects may extend to wild animals:

Eating meat reduced the perceived obligation to show moral concern for animals in general and the perceived moral status of the [animal being eaten].

(See also Jeff Sebo’s talk A utilitarian case for animal rights and John & Sebo, 2020, 3.2.)

These studies confirm a point that a number of philosophers have been trying to convey for a while (see John & Sebo, 2020, 3.2 for a brief review). Here is Peter Singer in Practical Ethics (as quoted in ibid.):

it would be better to reject altogether the killing of animals for food, unless one must do so to survive. Killing animals for food makes us think of them as objects that we can use as we please …

And such objectification, in turn, has horrendous consequences. This is usually quite obvious in the human case: few people are tempted to claim that it would be inconsequential if we began eating a given group of humans, even if we stipulated that these humans had the same mental abilities as, say, pigs. Singer’s point about objectification is obvious to most people in this case, and most consequentialists would probably say that raising, killing, and eating humans could only be recommended by very naive and incomplete consequentialist analyses detached from the real world — not least the realities of human psychology. Yet the same ought to be concluded when the beings in question possess not just the minds but also the bodies of pigs.

Relatedly, in the hypothetical case where systematic exploitation of certain humans is the norm, few consequentialists would be tempted to say that abstention from the consumption of human products (e.g. human body parts or forcefully obtained breast milk) is insignificant, or say that it is not worth sticking with it because other things are more important. For on reflection, when we put on the more sophisticated consequentialist hat, we realize that such abstention probably is an important component of the broader set of actions that constitutes the ethically optimal path forward. The same ought to be concluded, I submit, in the non-human case.

Note, finally, that even if we believed ourselves to be exceptions to all of the psychological tendencies reviewed above — a belief we should be skeptical of given the prevalence of illusory superiority — it would still be hypocritical and a failure of integrity if we ourselves did not follow a norm that we would recommend others to follow. And consequentialists have good reasons to show high integrity.

Self-serving biases

This is more of a meta consideration suggesting that 1) we should be skeptical of convenient conclusions, and 2) we should adhere to stricter principles than a naive consequentialist analysis might imply.

A good reason to adhere to reasonably strict principles is that, if we loosen our principles and leave everything up for case-by-case calculation, we open the door for biases to sneak in.

As Jamie Mayerfeld writes in Suffering and Moral Responsibility (p. 121):

An agent who regarded [sound moral principles] as mere rules of thumb would ignore them whenever she calculated that compliance wasn’t necessary to minimize the cumulative badness of suffering. The problem is that it might also be in her own interest to violate these principles, and self-interest could distort her calculations, even when she calculated sincerely. She could thus acquire a pattern of violating the principles even when compliance with them really was necessary to prevent the worst cumulative suffering. To avoid this, we would want her to feel strongly inhibited from violating the principles. Inhibitions of this kind can insulate agents from the effect of biased calculations.

And there are indeed many reasons to think that our “calculations” are strongly biased against concern for non-human individuals and against the conclusion that we should stop consuming them. For example, there is the fact that people who do not consume animal products face significant stigma — for example, one US study found that people tended to evaluate vegans more negatively than other minority groups, such as atheists and homosexuals; “only drug addicts were evaluated more negatively than vegetarians and vegans”. And a recent study suggested that fear of stigmatization is among the main reasons why people do not want to stop eating animal products. Yet fear of stigmatization is hardly, on reflection, a sound moral reason to eat animal products.

A more elaborate review of relevant biases can be found in (Vinding, 2018, “Bias Alert: We Should Expect to Be Extremely Biased”; Vinding, 2020, 11.5).

Human externalities

Defenses of the consumption of non-human individuals often rest on strongly anthropocentric values (which cannot be justified). But even on such anthropocentric terms, a surprisingly strong case can in fact be made against animal consumption given the negative effects animal agriculture has on human health — effects that individual consumption will also contribute to on the margin.

First, as is quite salient these days, animal agriculture significantly increases the risk of zoonotic diseases. Many of the most lethal diseases of the last century were zoonotic diseases that spread to humans due to animal agriculture and/or animal consumption, including the 1918 flu (50-100 million deaths), AIDS (30-40 million deaths), the Hong Kong flu (1-4 million deaths), and the 1957-1958 flu (1-4 million deaths). The same is true of the largest epidemics so far in this century, such as SARS, Ebola, COVID-19, and various bird and swine flus.

As noted in (Babatunde, 2011):

A remarkable 61 percent of all human pathogens, and 75 percent of new human pathogens, are transmitted by animals, and some of the most lethal bugs affecting humans originate in our domesticated animals.

Antibiotic resistance is another health problem exacerbated by animal agriculture. Each year in the US, more than 35,000 people die from antibiotic-resistant infections, which is more than twice the annual number of US gun homicides. And around 80 percent of all antibiotics used in the US are given to non-human animals — often simply to promote growth rather than to fight infections. In other words, animal agriculture is a key contributor to antibiotic resistance.

The environmental effects of animal agriculture represent another important factor, or rather set of factors. There is pollution — “ammonia pollution linked to U.S. farming may impose human health costs that are greater than the profits earned by agricultural exports”. There are greenhouse gases contributing significantly to climate change. There is nitrate contamination of the groundwater from manure:

The EPA found that nitrates are the most widespread agricultural contaminant in drinking water wells and estimates that 4.5 million people [in the US] are exposed to elevated nitrate levels from drinking water wells. Nitrates, if they find their way into the groundwater, can potentially be fatal to infants.

Beyond the environmental effects, there are also significant health risks associated with the direct consumption of animal products, including red meat, chicken meat, fish meat, eggs and dairy. Conversely, significant health benefits are associated with alternative sources of protein, such as beans, nuts, and seeds. This is relevant both collectively, for the sake of not supporting industries that actively promote poor human nutrition in general, as well as individually, to maximize one’s own health so one can be more effectively altruistic.

A more thorough review of the human costs of animal agriculture are found in (Vinding, 2014, ch. 2).

In sum, one could argue that we also have a strong obligation to our fellow humans to avoid contributing to the various human health problems and risks caused by animal agriculture.

Both/And

What I have said above may seem in tension with the common consequentialist critique that says that animal advocates focus too much on individual consumer behavior. Yet in reality, there is no tension. It is both true, I submit, that avoiding the consumption of animal products is important (in purely consequentialist terms) and that most animal advocates focus far too much on individual consumer change compared to institutional change and wild-animal suffering. The latter point does not negate the former (the same view is expressed in John & Sebo, 2020, 3.3).

When Machines Improve Machines

The following is an excerpt from my book Reflections on Intelligence (2016/2020).

 

The term “Artificial General Intelligence” (AGI) refers to a machine that can perform any task at least as well as any human. This is often considered the holy grail of artificial intelligence research, and also the thing that many consider likely to give rise to an “intelligence explosion”, the reason being that machines then will be able to take over the design of smarter machines, and hence their further development will no longer be held back by the slowness of humans. Luke Muehlhauser and Anna Salamon express the idea in the following way:

Once human programmers build an AI with a better-than-human capacity for AI design, the instrumental goal for self-improvement may motivate a positive feedback loop of self-enhancement. Now when the machine intelligence improves itself, it improves the intelligence that does the improving.

(Muehlhauser & Salamon, 2012, p. 13)

This seems like a radical shift, yet is it really? As author and software engineer Ramez Naam has pointed out (Naam, 2010), not quite, since we already use our latest technology to improve on itself and build the next generation of technology. As I argued in the previous chapter, the way new tools are built and improved is by means of an enormous conglomerate of tools, and newly developed tools merely become an addition to this existing set of tools. In Naam’s words:

[A] common assertion is that the advent of greater-than-human intelligence will herald The Singularity. These super intelligences will be able to advance science and technology faster than unaugmented humans can. They’ll be able to understand things that baseline humans can’t. And perhaps most importantly, they’ll be able to use their superior intellectual powers to improve on themselves, leading to an upward spiral of self improvement with faster and faster cycles each time.

In reality, we already have greater-than-human intelligences. They’re all around us. And indeed, they drive forward the frontiers of science and technology in ways that unaugmented individual humans can’t.

These superhuman intelligences are the distributed intelligences formed of humans, collaborating with one another, often via electronic means, and almost invariably with support from software systems and vast online repositories of knowledge.

(Naam, 2010)

The design and construction of new machines is not the product of human ingenuity alone, but of a large system of advanced tools in which human ingenuity is just one component, albeit a component that plays many roles. And these roles, it must be emphasized, go way beyond mere software engineering – they include everything from finding ways to drill and transport oil more effectively, to coordinating sales and business agreements across countless industries.

Moreover, as Naam hints, superhuman intellectual abilities already play a crucial role in this design process. For example, computer programs make illustrations and calculations that no human could possibly make, and these have become indispensable components in the design of new tools in virtually all technological domains. In this way, superhuman intellectual abilities are already a significant part of the process of building superhuman intellectual abilities. This has led to continued growth, yet hardly an intelligence explosion.

Naam gives a specific example of an existing self-improving “superintelligence” (a “super” goal achiever, that is), namely Intel:

Intel employs giant teams of humans and computers to design the next generation of its microprocessors. Faster chips mean that the computers it uses in the design become more powerful. More powerful computers mean that Intel can do more sophisticated simulations, that its CAD (computer aided design) software can take more of the burden off of the many hundreds of humans working on each chip design, and so on. There’s a direct feedback loop between Intel’s output and its own capabilities. …

Self-improving superintelligences have changed our lives tremendously, of course. But they don’t seem to have spiraled into a hard takeoff towards “singularity”. On a percentage basis, Google’s growth in revenue, in employees, and in servers have all slowed over time. It’s still a rapidly growing company, but that growth rate is slowly decelerating, not accelerating. The same is true of Intel and of the bulk of tech companies that have achieved a reasonable size. Larger typically means slower growing.

My point here is that neither superintelligence nor the ability to improve or augment oneself always lead to runaway growth. Positive feedback loops are a tremendously powerful force, but in nature (and here I’m liberally including corporate structures and the worldwide market economy in general as part of ‘nature’) negative feedback loops come into play as well, and tend to put brakes on growth.

(Naam, 2010)

I quote Naam at length here because he makes this important point well, and because he is an expert with experience in the pursuit of using technology to make better technology. In addition to Naam’s point about Intel and other companies that improve themselves, I would add that although these are enormous competent collectives, they still only constitute a tiny part of the larger collective system that is the world economy that they contribute modestly to, and which they are entirely dependent upon.

“The” AI?

The discussion above hints at a deeper problem in the scenario Muelhauser and Salomon lay out, namely the idea that we will build an AI that will be a game-changer. This idea seems widespread in modern discussions about both risks and opportunities of AI. Yet why should this be the case? Why should the most powerful software competences we develop in the future be concentrated into anything remotely like a unitary system?

The human mind is unitary and trapped inside a single skull for evolutionary reasons. The only way additional cognitive competences could be added was by lumping them onto the existing core in gradual steps. But why should the extended “mind” of software that we build to expand our capabilities be bound in such a manner? In terms of the current and past trends of the development of this “mind”, it only seems to be developing in the opposite direction: toward diversity, not unity. The pattern of distributed specialization mentioned in the previous chapter is repeating itself in this area as well. What we see is many diverse systems used by many diverse systems in a complex interplay to create ever more, increasingly diverse systems. We do not appear to be headed toward any singular super-powerful system, but instead toward an increasingly powerful society of systems (Kelly, 2010).

Greater Than Individual or Collective Human Abilities?

This also hints at another way in which our speaking of “intelligent machines” is somewhat deceptive and arbitrary. For why talk about the point at which these machines become as capable as human individuals rather than, say, an entire human society? After all, it is not at the level of individuals that accomplishments such as machine building occurs, but rather at the level of the entire economy. If we talked about the latter, it would be clear to us, I think, that the capabilities that are relevant for the accomplishment of any real-world goal are many and incredibly diverse, and that they are much more than just intellectual: they also require mechanical abilities and a vast array of materials.

If we talked about “the moment” when machines can do everything a society can, we would hardly be tempted to think of these machines as being singular in kind. Instead, we would probably think of them as a society of sorts, one that must evolve and adapt gradually. And I see no reason why we should not think about the emergence of “intelligent machines” with abilities that surpass human intellectual abilities in the same way.

After all, this is exactly what we see today: we gradually build new machines – both software and hardware – that can do things better than human individuals, but these are different machines that do different things better than humans. Again, there is no trend toward the building of disproportionally powerful, unitary machines. Yes, we do see some algorithms that are impressively general in nature, but their generality and capabilities still pale in comparison to the generality and the capabilities of our larger collective of ever more diverse tools (as is also true of individual humans).

Relatedly, the idea of a “moment” or “event” at which machines surpass human abilities is deeply problematic in the first place. It ignores the many-faceted nature of the capabilities to be surpassed, both in the case of human individuals and human societies, and, by extension, the gradual nature of the surpassing of these abilities. Machines have been better than humans at many tasks for centuries, yet we continue to speak as though there will be something like a “from-nothing-to-everything” moment – e.g. “once human programmers build an AI with a better-than-human capacity for AI design”. Again, this is not congruous with the way in which we actually develop software: we already have software that is superhuman in many regards, and this software already plays a large role in the collective system that builds smarter machines.

A Familiar Dynamic

It has always been the latest, most advanced tools that, in combination with the already existing set of tools, have collaborated to build the latest, most advanced tools. The expected “machines building machines” revolution is therefore not as revolutionary as it seems at first sight. The “once machines can program AI better than humans” argument seems to assume that human software engineers are the sole bottleneck of progress in the building of more competent machines, yet this is not the case. But even if it were, and if we suddenly had a thousand times as many people working to create better software, other bottlenecks would quickly emerge – materials, hardware production, energy, etc. All of these things, indeed the whole host of tasks that maintain and grow our economy, are crucial for the building of more capable machines. Essentially, we are returned to the task of advancing our entire economy, something that pretty much all humans and machines are participating in already, knowingly or not, willingly or not.

By themselves, the latest, most advanced tools do not do much. A CAD program alone is not going to build much, and the same holds true of the entire software industry. In spite of all its impressive feats, it is still just another cog in a much grander machinery.

Indeed, to say that software alone can lead to an “intelligence explosion” – i.e. a capability explosion – is akin to saying that a neuron can hold a conversation. Such statements express a fundamental misunderstanding of the level at which these accomplishments are made. The software industry, like any software program in particular, relies on the larger economy in order to produce progress of any kind, and the only way it can do so is by becoming part of – i.e. working with and contributing to – this grander system that is the entire economy. Again, individual goal-achieving ability is a function of the abilities of the collective. And it is here, in the entire economy, that the greatest goal-achieving ability is found, or rather distributed.

The question concerning whether “intelligence” can explode is therefore essentially: can the economy explode? To which we can answer that rapid increases in the growth rate of the world economy certainly have occurred in the past, and some argue that this is likely to happen again in the future (Hanson 1998/2000, 2016). However, there are reasons to be skeptical of such a future growth explosion (Murphy, 2011; Modis, 2012; Gordon, 2016; Caplan, 2016; Vinding, 2017b; Cowen & Southwood, 2019).

“Intelligence Though!” – A Bad Argument

A type of argument often made in discussions about the future of AI is that we can just never know what a “superintelligent machine” could do. “It” might be able to do virtually anything we can think of, and much more than that, given “its” vastly greater “intelligence”.

The problem with this argument is that it again rests on a vague notion of “intelligence” that this machine “has a lot of”. For what exactly is this “stuff” it has a lot of? Goal-achieving ability? If so, then, as we saw in the previous chapter, “intelligence” requires an enormous array of tools and tricks that entails much more than mere software. It cannot be condensed into anything we can identify as a single machine.

Claims of the sort that a “superintelligent machine” could just do this or that complex task are extremely vague, since the nature of this “superintelligent machine” is not accounted for, and neither are the plausible means by which “it” will accomplish the extraordinarily difficult – perhaps even impossible – task in question. Yet such claims are generally taken quite seriously nonetheless, the reason being that the vague notion of “intelligence” that they rest upon is taken seriously in the first place. This, I have tried to argue, is the cardinal mistake.

We cannot let a term like “superintelligence” provide a carte blanche to make extraordinary claims or assumptions without a bare minimum of justification. I think Bostrom’s book Superintelligence is an example of this. Bostrom worries about a rapid “intelligence explosion” initiated by “an AI” throughout the book, yet offers very little in terms of arguments for why we should believe that such a rapid explosion is plausible (Hanson, 2014), not to mention what exactly it is that is supposed to explode (Hanson, 2010; 2011a).

No Singular Thing, No Grand Control Problem

The problem is that we talk about “intelligence” as though it were a singular thing; or, in the words of brain and AI researcher Jeff Hawkins, as though it were “some sort of magic sauce” (Hawkins, 2015). This is also what gives rise to the idea that “intelligence” can explode, because one of the things that this “intelligence” can do, if you have enough of it, is to produce more “intelligence”, which can in turn produce even more “intelligence”.

This stands in stark contrast to the view that “intelligence” – whether we talk about cognitive abilities in particular or goal-achieving abilities in general – is anything but singular in nature, but rather the product of countless clever tricks and hacks built by a long process of testing and learning. On this latter view, there is no single master problem to crack for increasing “intelligence”, but rather just many new tricks and hacks we can discover. And finding these is essentially what we have always been doing in science and engineering.

Robin Hanson makes a similar point in relation to his skepticism of a “blank-slate AI mind-design” intelligence explosion:

Sure if there were a super mind theory that allowed vast mental efficiency gains all at once, but there isn’t. Minds are vast complex structures full of parts that depend intricately on each other, much like the citizens of a city. Minds, like cities, best improve gradually, because you just never know enough to manage a vast redesign of something with such complex inter-dependent adaptations.

(Hanson, 2010)

Rather than a concentrated center of capability that faces a grand control problem, what we see is a development of tools and abilities that are distributed throughout the larger economy. And we “control” – i.e. specify the function of – these tools, including software programs, gradually as we make them and put them to use in practice. The design of the larger system is thus the result of our solutions to many, comparatively small “control problems”. I see no compelling reason to believe that the design of the future will be any different.


See also Chimps, Humans, and AI: A Deceptive Analogy.

Consciousness – Orthogonal or Crucial?

The following is an excerpt from my book Reflections on Intelligence (2016/2020).

 

A question often considered open, sometimes even irrelevant, when it comes to “AGIs” and “superintelligences” is whether such entities would be conscious. Here is Nick Bostrom expressing such a sentiment:

By a “superintelligence” we mean an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills. This definition leaves open how the superintelligence is implemented: it could be a digital computer, an ensemble of networked computers, cultured cortical tissue or what have you. It also leaves open whether the superintelligence is conscious and has subjective experiences.

(Bostrom, 2012, “Definition of ‘superintelligence’”)

This is false, however. On no meaningful definition of “more capable than the best human brains in practically every field, including scientific creativity, general wisdom, and social skills” can the question of consciousness be considered irrelevant. This is like defining a “superintelligence” as an entity “smarter” than any human, and to then claim that this definition leaves open whether such an entity can read natural language or perform mathematical calculations. Consciousness is integral to virtually everything we do and excel at, and thus if an entity is not conscious, it cannot possibly outperform the best humans “in practically every field”. Especially not in “scientific creativity, general wisdom, and social skills”. Let us look at these three in turn.

Social Skills

Good social skills depend on an ability to understand others. And in order to understand other people, we have to simulate what it is like to be them. Fortunately, this comes quite naturally to most of us. We know what it is like to consciously experience emotions such as sadness, fear, and joy directly, and this enables us to understand where people are coming from when they report and act on these emotions.

Consider the following example: without knowing anything about a stranger you observe on the street, you can roughly know how that person would feel and react if they suddenly, by the snap of a finger, had no clothes on right there on the street. Embarrassment, distress, wanting to cover up and get away from the situation are almost certain to be the reaction of any randomly selected person. We know this, not because we have read about it, but because of our immediate simulations of the minds of others – one of the main things our big brains evolved to do. This is what enables us to understand the minds of other people, and hence without running this conscious simulation of the minds of others, one will have no chance of gaining good social skills and interpersonal understanding.

But couldn’t a computer just simulate people’s brains and then understand them without being conscious? Is the consciousness bit really relevant here?

Yes, consciousness is relevant. At the very least, it is relevant for us. Consider, for instance, the job of a therapist, or indeed the “job” of any person who attempts to listen to another person in a deep conversation. When we tell someone about our own state or situation, it matters deeply to us that the listener actually understands what we are saying. A listener who merely pretends to feel and understand would be no good. Indeed, this would be worse than no good, as such a “listener” would then essentially be lying and deceiving in a most insensitive way, in every sense of the word.

Frustrated Human: “Do you actually know the feeling I’m talking about here? Do you even know the difference between joy and hopeless despair?”

Unconscious liar: “Yes.”

Whether someone is actually feeling us when we tell them something matters to us, especially when it comes to our willingness to share our perspectives, and hence it matters for “social skills”. An unconscious entity cannot have better social skills than “the best human brains” because it would lack the very essence of social skills: truly feeling and understanding others. Without a conscious mind there is no way to understand what it is like to have such a mind.

General Wisdom

Given how relevant social skills are for general wisdom, and given the relevance of consciousness for social skills, the claim that consciousness is irrelevant to general wisdom should already stand in serious doubt at this point.

Yet rather than restricting our focus to “general wisdom”, let us consider ethics in its entirety, which, broadly construed at least, includes any relevant sense of “general wisdom”. For in order to reason about ethics, one must be able to consider and evaluate questions like the following:

Can certain forms of suffering be outweighed by a certain amount of happiness?

Does the nature of the experience of suffering in some sense demand that reducing suffering is given greater moral priority than increasing happiness (for the already happy)?

Can realist normative claims be made on the basis of the properties of such experiences?

One has to be conscious to answer such questions. That is, one must know what such experiences are like in order to understand their experiential properties and significance. Knowing what terms like “suffering” and “happiness” refer to – i.e. knowing what the actual experiences of suffering and happiness are like – is as crucial to ethics as numbers are to mathematics.

The same point holds true about other areas of philosophy that bear on wisdom, such as the philosophy of mind: without knowing what it is like to have a conscious mind, one cannot contribute to the discussion about what it is like to have one and what the nature of consciousness is. Indeed, an unconscious entity has no idea about what the issue is even about in the first place.

So both in ethics and in the philosophy of mind, an unconscious entity would be less than clueless about the deep questions at hand. If an entity not only fails to surpass humans in this area, but fails to even have the slightest clue about what we are talking about, it hardly surpasses the best human brains in practically every field. After all, these questions are also relevant to many other fields, ranging from questions in psychology to questions concerning the core foundations of knowledge.

Experiencing and reasoning about consciousness is a most essential part of “human abilities”, and hence an entity that cannot do this cannot be claimed to surpass humans in the most important, much less all, human abilities.

Scientific Creativity

The third and final ability mentioned above that an unconscious entity can supposedly surpass humans in is scientific creativity. Yet scientific creativity must relate to all fields of knowledge, including the science of the conscious mind itself. This is also a part of the natural world, and a most relevant one at that.

Experiencing and accurately reporting what a given state of consciousness is like is essential for the science of mind, yet an unconscious entity obviously cannot do such a thing, as there is no experience it can report from. It cannot display any scientific creativity, or even produce mere observations, in this most important science. Again, the most it can do is produce lies – the very anti-matter of science.

 

Compassionate Free Speech

Two loose currents appear to be in opposition in today’s culture. One is animated by a strong insistence on empathy and compassion as core values, the other by a strong insistence on free speech as a core value. These two currents are often portrayed as though they must necessarily be in conflict. I think this is a mistake.

To be sure, the two values described above can be in tension, and none of them strictly imply the other. But it is possible to reconcile them in a refined and elegant synthesis. That, I submit, is what we should be aiming for. A synthesis of two vital and mutually reinforcing values.

Definitions and outline

It is crucial to distinguish 1) social and ethical norms, and 2) state-enforced laws. The argument I make here pertains to the first level. That is, I am arguing that we should aim to observe and promote ethical norms of compassion and open conversation respectively.

What do I mean by these terms? Compassion is commonly defined as “sympathetic consciousness of others’ distress together with a desire to alleviate it”. I here use the term in a broader sense that also covers related virtues such as understanding, charity, and kindness.

By norms of open conversation, or free expression, I mean norms that enable people to express their honest views openly, even when these views are controversial and uncomfortable. These norms do not entail that speech should be wholly unrestricted; after all, virtually everyone agrees that defamation and incitements to commit severe crimes should be illegal, as they commonly are.

My view is that we should roughly think of these two broad values as prima facie duties: we should generally strive to observe norms of compassion and open conversation, except in (rare) cases where other duties or virtues override these norms.

Below is a short defense of these two respective values, highlighting their importance in their own right. This is followed by a case that these values are not only compatible, but indeed strongly complementary. Finally, I explore what I see as some of the causes of our current state of polarization, and suggest five heuristics that might be useful going forward.

Brief defenses

Free speech

There are many strong arguments in favor of free speech. A famous collection of such arguments is On Liberty (1859) by John Stuart Mill, whose case for free speech is primarily based on the harm principle: the only reason power can legitimately be exercised over any individual against their will is to prevent harm to others.

This principle is intuitively compelling, although it leaves it quite unspecified what exactly counts as a harm to others. That is perhaps the main crux in discussions about free speech, and this alone provides an argument in favor of free and open expression. For how can we clarify what should count as sufficient harm to others to justify the exercise of power if not through open discussion?

A necessary corrective to biased, fallible minds

Another important argument Mill makes in favor of free speech is based not merely on the rights of the speaker, but in equal part on the rights of the would-be listeners, who are also robbed by the suppression of free expression:

[T]he peculiar evil of silencing the expression of an opinion is, that it is robbing the human race; posterity as well as the existing generation; those who dissent from the opinion, still more than those who hold it. If the opinion is right, they are deprived of the opportunity of exchanging error for truth: if wrong, they lose, what is almost as great a benefit, the clearer perception and livelier impression of truth, produced by its collision with error.

In essence, Mill argues that, contrary to the annoyance we may instinctively feel, we should in fact be grateful for having our cherished views challenged, not least because it can help clarify and update our views.

Today, Mill’s argument can be further bolstered by a host of well-documented psychological biases. We now know that we are all vulnerable to confirmation bias, the bandwagon effect, groupthink, etc. These biases make it all too easy for us to deceive ourselves into thinking that we already possess the whole truth, although we most certainly do not. Consequently, if we want to hold reasonable beliefs, we should welcome and appreciate those who challenge the pitfalls of our groupish minds — pitfalls that we may otherwise be content to embrace in what I would argue is ultimately indulgent self-betrayal.

After all, how can we know that our attempts to protect ourselves from hearing views we dislike are not essentially unconscious attempts to protect our own confirmation bias? Free and open conversation is our best debiasing tool. Our manifest fallibility renders free speech essential.

Strategic reasons

An altogether different argument in favor of honoring principles of free speech is that a failure to do so is strategically unwise. Indeed, as free-speech defender Noam Chomsky argues, there are several reasons to consider the suppression of free speech a tactical error if we are trying to create a good society.

First, reinforcing a norm of suppressing speech can have the unintended consequence of leading all sides, and perhaps eventually governments, to consider it increasingly legitimate to suppress certain forms of speech. “If they can suppress speech, why shouldn’t we?” The effects of such a regression would be worst for those who lack power.

Second, seeking to suppress speech is likely to backfire and to strengthen the other side, by making that side look more appealing than it in fact is — the suppressed becomes alluring — and by making the side that seeks to suppress speech look unreasonable, as though they are unable to muster a defense of their views.

When people try to make us do something, we tend to react negatively and to distance ourselves, even if we agreed with them from the outset (cf. psychological reactance). This is another strong reason against suppressing free expression, and against giving people the impression that they are not allowed to discuss or think certain things. It is human nature to react by asserting one’s freedom in defiance, even if it means voting for a president that one would otherwise have voted against.

(Weak norms of free expression are thus a democratic problem in more than one way: it can keep citizens from voting in accordance with their ideal preferences both by making them ill-informed and by provoking votes of defiance.)

Steven Pinker has made a related point: if we place certain issues beyond the bounds of acceptable discourse, many people are likely to seek out discussion of these issues from unsavory sources, which can in turn put people on a path toward extreme and uncompassionate views. This parallels one of the main arguments made against the prevailing drug laws of today: such restrictions merely push the whole business into an underground market where people get dangerously polluted goods.

As Ayishat Akanbi eloquently put it (paraphrased slightly): if we suppress ideas, they will “operate with insidious undertones”, and we in effect “push people into the arms of extremism.”

Compassion

I will allow myself to let my defense of compassion be even briefer still, as I have already made an elaborate defense of it in my book Suffering-Focused Ethics: Defense and Implications.

The short case is this: Suffering, especially the most intense suffering, truly matters. It is truly bad and truly worth preventing. Consequently, a desire to alleviate intense suffering is simply the most sensible response. Only a failure to connect with the reality of suffering can leave us apathetic. That is the simplest and foremost reason why compassion is of paramount importance.

(This was also John Stuart Mill’s ultimate value, and the core motivation animating his defense of free speech: a concern for the well-being of sentient beings.)

Another reason to be compassionate, including in the broader sense of being kind and understanding, is that such an attitude has great instrumental benefits at the level of our communication and relations: it fosters trust and cooperation, which in turn enables win-win interactions.

However, to say that we should be compassionate is not to say that we should be game-theoretically naive in the sense of kindly allowing others to walk all over us. Compassion is wholly compatible with, and indeed mandates, tit for tat and assertiveness in the face of transgressions.

Lastly, it is worth emphasizing that compassion and empathy are not partisan values. Empathy is a human universal, and compassion has been considered a virtue in all major world traditions, as well as in most political movements, including political conservatism. Indeed, people of all political orientations score high on the harm/care dimension in Jonathan Haidt’s moral foundations framework. It really is a value on which people show uniquely wide agreement, at least on reflection. When they are not on Twitter.

Compassion and free speech: complementary values

As noted above, the two values I defend here do not strictly imply each other, at least in some purely theoretical sense. But they are strongly complementary in many regards.

By analogy, consider two classical virtues: honesty and courage. Strictly speaking, one can be honest without being courageous (an honest person may lack the courage to save a drowning person) and one can be courageous without being honest (a courageous person may fail to see the value of honesty). But it is also clear that these virtues often do enhance each other. Greater courageousness generally allows one to be more honest, and conversely, being more honest can foster greater courageousness, such as by giving one less to hide and be timid about.

The same applies to compassion and free expression.

How free speech aids compassion

Compassion and the compassionate project can be aided by free speech in various ways. For example, to alleviate and prevent suffering effectively with our limited resources, we need to be able to discuss controversial ideas. We need to be able to discuss and measure different values and priorities against each other, including values that many people consider sacred and hence offensive to discuss.

As a case in point, in my latest book, I defend the moral primacy of reducing extreme suffering, even above other values that many people may consider sacred, and I further discuss the difficult question of which causes we should prioritize so as to best reduce extreme suffering. My arguments will no doubt be deeply offensive and infuriating to many, and I believe a substantial number of people would like to see my ideas suppressed if they could. This is not, of course, unique to my views: all treatises and positions on ethics are bound to be deemed too offensive and too dangerous by some.

This highlights the importance of free speech for ethics in general, and for the project of reducing suffering in particular. To conduct this most difficult conversation about what matters and what our priorities should be, we need a culture that allows, indeed cultivates, this conversation — not a culture that stifles it. People who want to reduce suffering should thus have a strong interest in preserving and advancing free speech norms.

If I am missing important considerations about how we can best reduce suffering, as I most surely am, yet few people dare to publicly lambaste my flaws and defend alternative, perhaps even more controversial priorities, then I, and my project of advancing compassion, will indeed be robbed.

Another way in which free speech aids compassion is that, put simply, encouraging the free expression of and listening to each others’ underlying grievances can help us build mutual understanding, and in turn enable us to address our problems in cooperative ways. As Noam Chomsky notes in the context of hateful ideologies:

If you have a festering sore, the cure is not to irritate it, but to find out what its roots are and where it comes from, and to deal with those. Racist and other such speech is a festering sore. By silencing it, you simply amplify its appeal, and even lend it a veneer of respectability, as in fact we’ve seen very clearly in the last couple of years. And what has to be done, plainly, is to confront it, and to ask where it comes from, and to try to deal with the roots of such ideas. That’s the way to extirpate the ugliness and evil that lies behind such phenomena.

Andrew Yang makes similar points about the problem of white supremacy in the United States: a root source of this problem is often a sense of fear and lack of opportunity, not inherent evil or apathy, and hence listening to and addressing this underlying problem may be among the best ways to abate white supremacy.

Yang is inspired by the work of Deeyah Khan, who maintains that the best solution to extremist ideologues is to engage in conversation and to seek to understand, not to shut down the conversation. (I recommend watching Khan’s documentary White Right: Meeting the Enemy.)

So while compassion per se does not directly imply free speech at some purely theoretical level, I would argue that a sophisticated and fully extrapolated version of compassion and the compassionate project does necessitate strong norms of free and open expression at the practical level.

How compassion aids free speech

One of the ways in which compassion can aid open conversation is exemplified in Deeyah Khan’s documentary mentioned above: she sits down and listens to white nationalists, seeking to understand them with compassion, which allows them to identify and express their own underlying issues, such as feelings of fear, vulnerability, and unworthiness. Such things can be difficult to share in apathetic and antagonistic environments, be they the macho ingroup or the angry outgroup. “Fuck you, racist” does not quite invite a response of “I’m afraid and hurting” as much as does, “How are you feeling, and what really motivates you?” On the contrary, it probably just serves to reinforce the facade of the pain.

We may not usually think of conditions that further the sharing of our underlying worries and vulnerabilities as a matter of free speech, perhaps because we all help perpetuate norms that suppress honesty about these things. But if free speech norms are essentially about enabling us to dare express the truth, then our de facto suppression of our inmost worries and vulnerabilities is indeed a free speech issue — and a rather consequential one at that (as I think Khan’s White Right makes clear). Compassion may well be the best remedy we have to our truth-subduing culture of suppressing our core worries and vulnerabilities.

A related way in which compassion, specifically the virtue of being charitable, is important for free speech is, quite simply, that we suffocate free speech in its absence. If people hold back from expressing a nuanced view because they know they will be strawmanned and vilely attacked based on bad-faith misinterpretations, then the state of free expression, and of our public conversation in general, will be poor indeed.

In contrast, free speech will thrive when we do the opposite: when everyone engages with the strongest version of their opponents’ view — i.e. steel mans it — so that people feel positively motivated to present nuanced views and arguments in the expectation of being critiqued in good faith.

That, needless to say, is far from the state we are currently in.

Why we fail so spectacularly today

We are currently witnessing a primitive tribal dynamic exacerbated by the fact that we inhabit a treacherous environment to which we are not yet adapted, neither biologically nor culturally. I am speaking, of course, of the environment of screen-to-screen interaction.

Yet we should be clear that values and politics were never easy spheres to navigate in the first place. They have always been minefields. Politics is a notorious mind-killer for deep evolutionary reasons, and our political behavior is often more about signaling our group affiliations than it is about creating good policies. This is true not just of the “other side”; it is true of us all, though we remain largely unaware of and self-deceived about it.

Thus, our predicament is that we care deeply about loyalty signaling, and such signaling has now become dangerously inflated. Moreover, we often use beliefs, ostensibly all about tracking reality, as ornaments that signal our group loyalty.

A hostage crisis instilling false assumptions

The two loose social currents I mentioned in the introduction should, I submit, be understood in this light. Specifically, values centered on empathy and compassion have become an ornament of sorts that signals loyalty to one side, while values centered on free speech have become a loyalty signal to another side. To be clear, I am not saying these values are merely ornaments; they clearly are not. A value can be an ornament displayed with pride and be sincerely held at the same time. Yet our natural inclination to signal group loyalty can lead us to only express our support for one of these values, and to underemphasize the “opposing” value, even if we in fact do favor it.

In this way, the values of compassion and free speech have to some extent become hostages in a primitive tribal game, which in turn gives the false impression that there must be some deep conflict between these values, and that people must choose one or the other, as opposed to these values being, as I have argued, strongly complementary (with occasional and comparatively minor tensions).

And because many of us are loosely affiliated with one of these groups, and because we have expressed support for what appears to be its core value in the past, we may unconsciously buy the tacit premise that this is now the kind of person we are and should keep on being. Supporters of free speech may thus feel nudged to display insensitivity in order to signal their loyalty and consistency, while supporters of anti-discrimination may feel nudged to oppose free speech.

Uncharitable claims beyond belief

A sad feature of this dynamic, and something that helps fuel it further, is how incredibly uncharitable the outer flanks of these two tribal currents are to the other side.

“The PC-policing SJWs don’t care about the hard facts and just want to suppress them.”

“The free speech bros don’t care about minorities and just want to oppress them.”

To say that people are failing to steel man here would be quite the understatement. Indeed, this barely even qualifies as a straw man. It is more like the scream-man version of the other side: the worst, most scary version of the other side’s position one could come up with. And this scream man is repeatedly rehearsed in the partitioned echo halls of Twitter to the extent that people start believing these preposterously uncharitable narratives about the Scary Other.

It is a tragedy of the commons phenomenon: people are gleefully rewarded in their ingroup each time they promulgate the scream man of the other side, and so it feels right to do so for individuals in these respective groups. But in the bigger picture, it just leaves everyone much worse off. It is a Red Queen’s race in which one must run ever faster just to stand still: people on both sides increasingly need to identify the other side as something akin to evil monsters in order to maintain their status in the ingroup.

Distributions and common knowledge

To be sure, there are serious problems with significant numbers of people who conform too closely to the cartoon descriptions above. But a crucial point is that we must think in terms of statistical distributions. Specifically, the most loud-mouthed and scary two percent of the “other side” — a minority that tends to get a disproportionate amount of attention — should not be taken to represent everyone on that “side”, let alone its most reasonable representatives.

Indeed, making it common knowledge that the worst elements on these respective sides do not speak for everyone on that side, and that a majority of people on both teams actually disagree with the excesses of the extremists on both sides, may well be among the best ways to weaken these extremist elements and the polarization we are currently witnessing.

Which leads us to a significant problem with the tribal mess in which we find ourselves: people in these notional groups tend to be remarkably bad at criticizing the harmful tendencies of their own “team”. There does indeed appear to be a tendency among certain defenders of free speech to fail to criticize and condemn those who discriminate against minorities. Likewise, there really does seem to be a tendency among certain progressives to fail to criticize and condemn those who suppress discussions of contentious issues.

This failure to speak out against the worst elements of one’s “own side”, side A, with sufficient force creates the impression, on side B, that most people on side A actually agree with these worst elements. That is how damning it is that we fail to criticize the transparent excesses of our ingroup in clear terms.

We may speculate whether these failures of moral clarity are best explained by group-signaling biases, temperamental differences (which may be partially innate), or some combination of the two. Yet whatever their origin, these failures to criticize the ingroup do merit serious critique and self-reflection.

At cross-purposes

A problem with our failure to be charitable and to think in terms of distributions is that people end up talking past each other: both sides tend to criticize a straw-man version of the other side based on the rabid tail-end elements of that side, which most people on the other side really do disagree with (although they may, as mentioned above, fail to express this disagreement with sufficient clarity).

This frequently results in debates with two sides that are in large part talking at cross-purposes: one side mostly defends free speech, the other mostly defends anti-discrimination, as though these were necessarily in great conflict (in the case of the debate linked to above, the two sides are arguing about two very different conceptions of “political correctness”). The failure to explore the compatibility and mutual complementarity of these values is striking.

The perils of screen-to-screen interaction

As noted above, our current mode of interaction only aggravates our political imbecility. When engaged in face-to-face interaction, we naturally relate to and empathize with the person before us, and we have a strong interest in keeping our interaction cordial so as to prevent it from escalating into conflict.

In screen-to-screen interactions, by contrast, our circuits for interpersonal interaction are all but inert, as we find ourselves shielded off from salient feedback and danger. Social media is road rage writ large. A road rage that renders it extra difficult to be charitable, and which renders it far more tempting to paint the outgroup in a bad light than it could ever be in a face-to-face environment, where preposterous straw men would be called out and challenged in real time.

As a study on political polarization on Twitter put it:

Many messages contain sentiments more extreme than you would expect to encounter in face-to-face interactions, and the content is frequently disparaging of the identities and views associated with users across the partisan divide.

The result is a steady rise in polarization: people in group A publicize a preposterous straw man of group B — a clear signal that they are not part of that group — which increases the incentive for members of group B to signal distance from group A. The members of group B then send such a signal by broadcasting a preposterous straw man of group A, which in turn encourages those in group A to present an even more preposterous straw man of group B, and so on.

How can we reverse this vicious spiral? The age of social media calls for new norms.

Better norms for screen communication

Human culture has adapted to technological changes before, and it seems that we have no choice but to do the same today, in the face of our current state of cultural maladaptation. The following are five heuristics, or norms, that I think are likely to be useful in this regard.

1. The face-to-face heuristic

In light of the above, it seems sensible to adopt the precept of communicating online in roughly the same way we would communicate face-to-face. Our skills in face-to-face interaction have deep biological and cultural bases, and hence this heuristic is a cheap way to tap into a well-honed toolbox for functional human communication.

One effect of employing this heuristic will likely be a reduction of sarcastic and taunting comments. Such comments are rarely useful for taking our conversations to the next level, as we tend to realize face-to-face.

2. The nuance heuristic

As I argue in my defense of nuance, much of the tension and miscommunication we see today could likely be lessened greatly if we adopted more nuanced perspectives. Not seeing everything through the lens of black-or-white thinking, acknowledging grains of truth in different perspectives, and representing beliefs in terms of graded credences rather than posturing with overconfident all-or-nothing credences.

These are the remedies for dissolving the cartoon narratives that currently appear to divide us in fundamental ways, and which make mutual understanding and cooperation seem impossible. This appearance must go.

3. The steel-man heuristic

I have already mentioned this, but it really cannot be said enough: we must strive to be charitable and to steel man the views of our opponents, especially since our road-rage-behind-the-screen predicament makes it easier than ever to do the opposite.

Whenever we summarize and criticize the view of the other side, we should stop and ask ourselves: is this really the most honest statement of their view I can muster, let alone the strongest one? If I think their view is painfully stupid, do I really fully understand it? Do I really know what it entails and the best arguments that support it?

4. Compassion for the outgroup

As noted above, compassion really is a consensus value, if ever there were one. The disagreement mostly arises when it comes to which individuals we should extend our compassion to. Both of the notional “sides”, or social currents, described here suffer from selective compassion: they generally fail to show sufficient compassion and respect for the other side, which renders productive conversation difficult.

This point needs to be stressed with unique fervor today, as screens are an all too powerful catalyst for outgroup apathy.

5. Criticizing the ingroup

Condemning the excesses of one’s (vaguely associated) ingroup is also uniquely important today. Why? Because we now see large numbers of people behaving badly on social media, and our intuitions are statistically illiterate: we do not intuitively understand how a faction endorsing a certain view or behavior can simultaneously be large in number and constitute but a small minority of a given group. The world is big, and we mostly do not understand that.

Only if we counter these excesses by clearly proving that most people in the “ingroup” actually do disagree with the extremists will it become clear to the other side — and perhaps also to one’s own side — that the extremists truly are a disapproved minority, one that people so far have failed to criticize mostly because of intellectual cowardice and bystander apathy.

Such ingroup criticism is how we stop the vicious spiral of increasing polarization described above.

 

We have created a polarized society in which too many feel pushed toward a needlessly narrow set of values — compassion or free speech, choose one! We are pushed in this way, not by totalitarian laws, but by modes of communication to which we are not yet adapted, and which we are navigating with patently defunct norms.

Norms are often more important than laws. Most of us can think of judgments from our peers that would be worse than a minor prison sentence. Hence, totalitarian laws are not required for free expression to be stifled into a de facto draconian state. The notion that harshly punitive norms do not restrict speech in costly ways is naive.

Sure, we should be free to judge others based on the things they say. But just how harshly should we judge people for discussing controversial views? And do we understand the risks and the strategic costs associated with such judgments, and with attempts to suppress certain views? If we place ourselves in opposition to free speech, and then give people the ultimatum of siding either with “us” or with “them”, a lot of people are going to choose the other side, even if that side has features they find genuinely worrying.

The choice between free speech or compassion is a false one, peddled by those who follow social trends as opposed to philosophical principles. Nothing, save primitive tribal and signaling dynamics, really prevents us from charting the balanced middle path of a free and compassionate society.

Ten Biases Against Prioritizing Wild-Animal Suffering

I recommend reading the short and related post Why Most People Don’t Care About Wild-Animal Suffering by Ben Davidow.

The aim of this essay is to list some of the reasons why animal advocates and aspiring effective altruists may be biased against prioritizing wild-animal suffering. These biasing factors are, I believe, likely to significantly distort the views and priorities of most people who hold impartial moral views concerned about the suffering of all non-human animals.

1. Historical momentum and the status quo

The animal rights movement has, historically, been almost exclusively concerned with the protection of non-human animals exploited by humans. Very little attention has been devoted to suffering in nature for natural reasons. And to the extent the issue has been mentioned by philosophers in the past, it has rarely been framed as something that we ought to do something about.

Only in recent decades has the view that wild-animal suffering deserves serious attention in our practical deliberations been defended more explicitly. And the people who have defended this view have, of course, still been a tiny minority among activists concerned about animal suffering, and they have so far had little impact on the focus and activism of the animal movement at large.

This historical background matters greatly, since we humans very much have a social epistemology: we tend to pick up the views of our peers. For example, most people adopt the religion that is most popular in their geographical region, even if it is not the most rational belief system on reflection. And a similar pattern applies to our views in general. It is truly rare for people to think critically and independently.

Thus, if most people concerned about non-human animals — including our own mentors and personal heroes — have focused almost exclusively on the plight of non-human animals exploited by humans, then we are likely to be strongly inclined to do the same, even if this is not the most rational focus on reflection (in terms of how we can have the best impact on the margin).

2. Emotionally salient footage

Closely related to the point above is the fact that footage of suffering “farm animals” constitutes almost all of the disturbing footage we see of animal suffering. Whether on social media or in documentary movies about animal rights, the vast majority of the content encountered by the average animal activist shows cows, pigs, and chickens who are suffering at human hands.

Note how unrepresentative this picture is: a great majority of the animal suffering we observe occurs at human hands, although the vast majority of all suffering beings on the planet are found in nature. It is difficult to see how this can give us anything but a skewed sense of what is actually happening on our planet.

Yet not only will most of us have been exposed to far more suffering occurring at human hands, but we probably also tend to see the victims of such suffering with very different eyes compared to how we see the victims of natural processes. When we, as animal activists, see pigs and chickens suffer at human hands, we look at these beings with sympathy. We feel moral outrage. But when we see a being suffer in nature for natural reasons — for example, a baby elephant getting eaten alive — we are probably more hesitant about activating this same sympathy. Sure, we may lament the suffering and feel bad for the victim. But we do not truly see ourselves in the victim’s place. We do not look at the situation with moral eyes that cry “this is unacceptable”.

It is difficult to overstate the significance of this point. For while we may like to think of our activism and moral priorities as being animated chiefly by reasoned arguments, the truth is that salient experiences tend to matter just as much, if not more, for our moral motivation. It is one thing to think that wild-animal suffering is important, but it is quite another to feel it. The latter renders action less optional.

If we had only seen more footage of wild-animal suffering, and — most crucially — dared to behold such footage with truly sympathetic eyes, we would probably feel its moral gravity much more clearly, and in turn feel more motivated to address the problem. It seems unlikely that the priorities of the animal movement would be largely the same if more than 99 percent of the horrible footage encountered by animal activists had displayed the suffering of wild animals.

3. Perpetrator bias

Another relevant bias to control for is what I have called the perpetrator bias: we seem to care more about suffering when it is caused by a moral agent who has brought it about by intentional action (Vinding, 2020, 7.7). By extension, we tend to neglect suffering when it is not caused by intentional actions, such as when it occurs in nature for natural reasons. This bias, and its relevance to our appraisals of wild-animal suffering, has been explored in (Tomasik, 2013; Davidow, 2013).

As both Tomasik and Davidow argue, this bias could well be among the main reasons why most people, and indeed most animal advocates, tend to neglect the problem of wild-animal suffering. Our moral psychology is very much set up to track the transgressions of perpetrators, which can leave us relatively unmoved by suffering that involves no perpetrators, even if our reflected view is that all suffering should matter equally. After all, the core programming of our moral cognition does not change instantly just because a few of the modules in our minds have come to endorse a more advanced, impartial view.

4. Omission bias

Some version of the omission bias — our tendency to judge harmful acts of omission more leniently than harmful acts of commission, even when the consequences are the same — may be another reason why people with impartial views give less priority to wild-animal suffering than they ideally should. Our moral psychology is plausibly often motivated to focus on wrongs that we can be perceived to be responsible for, and for which we may be blamed.

Suffering caused by humans is in some sense done by “us”, and hence we may instinctively feel that we are more blameworthy for allowing such suffering to occur compared to allowing the suffering of wild animals. This might in turn incline us toward focusing on the former rather than the latter. Yet from an impartial perspective, this is not a sound reason for prioritizing human-caused suffering over “natural” suffering.

5. Scope neglect

Numbers are commonly invoked as one of the main reasons for focusing on “farm animals”. For example, there are about a hundred times as many non-human animals used and killed for food as there are companion animals, and hence we should generally spend our limited resources on helping the former rather than the latter. What is less commonly acknowledged, however, is that a similar thing can be said about wild animals, who, even if we only count vertebrates, outnumber vertebrates used and killed for food at least a thousand times (and perhaps more than 100,000 times).

Such numbers are notoriously difficult for us to internalize in our moral outlook. Our minds were simply not built to feel the significance of several orders of magnitude. Consequently, we have to make an arduous effort to really appreciate the force of this consideration.

6. Invertebrate neglect

Related to, and amplifying, the scope-neglect consideration is our neglect of invertebrate suffering. Not only are domesticated vertebrates outnumbered by wild vertebrates by at least a thousand times, but wild vertebrates are, in turn, outnumbered by wild invertebrates by at least ten thousand times (and perhaps by more than ten million times).
.


Put differently, more than 99.99 percent of all animals are invertebrates, and virtually all of them live in the wild. Taking the suffering of invertebrates into account thus gives us another strong — and widely ignored — reason in favor of prioritizing wild-animal suffering. A
nd in line with the point about the significance of emotionally salient footage, it may be that we need to watch footage of harmed invertebrates in order for us to fully appreciate the weight of this consideration.

7. Thinking we can have no impact

A common objection against focusing on wild-animal suffering is that the problem is intractable — if we could do anything about it, then we should prioritize it, but there just isn’t anything we can do at this point.

This is false in two principal ways. First, we humans already make countless decisions that influence animals in the wild (and we will surely make even more significant such decisions in the future). For example, the environmental policies adopted by our societies already influence large numbers of non-human animals in significant ways, and it would be false to claim that such policies are impossible to influence. After all, environmental groups have already been able to influence such policies to a considerable extent. Sadly, such groups have routinely pushed for policies that are profoundly speciesist and harmful for non-human animals — often with support from animal advocates, which shows how important it is that animal activists do not blindly endorse environmentalist policies, and how important it is that we reflect on the relationship between environmentalist ethics and animal ethics. And, of course, beyond influencing large-scale policy decisions, there are also many interventions we can make on a smaller scale that still help non-human animals in significant ways.

Second, we can help wild animals in indirect ways: by arguing against speciesism and for the importance of taking wild-animal suffering into consideration, as well as by establishing a research field focused on how we can best help wild animals on a large scale. Such indirect work, i.e. work that does not lead to direct interventions in the near term, may be the most important thing we can do at this point, even as our current wildlife policies and direct interventions are already hugely consequential.

So the truth is that there is much we can do at this point to work for a future with fewer harms to wild animals.

8. Underestimating public receptivity

There are reasons to think that animal advocates strongly underestimate public receptivity to the idea that wild-animal suffering matters and is worth reducing (see also what I have written elsewhere concerning the broader public’s receptivity to antispeciesist advocacy).

One reason could be that animal advocates themselves tend to find the idea controversial, and they realize that veganism is already quite controversial to most people. Hence, they reason, if they, as animal advocates, find the idea so controversial, and if most people find mere veganism to be highly controversial, then surely the broader public must find concern for wild-animal suffering extremely controversial.

Yet such an expectation is heavily distorted by the idiosyncratic position in which vegans find themselves. The truth is that most people may well view things the opposite way: veganism is controversial to them because they are currently heavily invested — socially and habit-wise — in non-veganism. By contrast, most people are not heavily invested in non-intervention with respect to wild animals, and thus have little incentive to oppose it.

The following is a relevant quote from Oscar Horta that summarizes his experience of giving talks about the issues of speciesism and wild-animal suffering at various high schools (my own software-assisted translation):

Intervention to help animals is easily accepted
There are many antispeciesist activists who are afraid to defend the idea of helping animals in the wild. Even if these activists totally agree with the idea, they believe that most people will reject it completely, and even consider the idea absurd. However, among the people attending the talks there was a very wide acceptance of the idea. Radical cases of intervention were not raised in the talks, but all the examples presented were well accepted. These included cases of injured, sick or trapped animals being rescued; orphan animal shelters; medical assistance to sick or injured animals; vaccination of wild animals; and provision of food for animals at risk of starvation. In sum, there does not seem to be any reason to be afraid of conveying this idea in talks of this type.

Of course, the claim here is not that everybody, or even most people, will readily agree with the idea of helping wild animals — many will surely resist it strongly. But the same holds true of all advocacy on behalf of non-human animals, and the point is that, contrary to our intuitive expectations, public receptivity to helping non-human animals in nature may in many ways be greater than their receptivity to helping “farm animals” (although receptivity toward the latter also appears reasonably high when the issue is framed in terms of institutional change rather than individual consumer change).

9. Overlooking likely future trajectories

As I have noted elsewhere:

Veganism is rising, and there are considerable incentives entirely separate from concern for nonhuman animals to move away from the production of animal “products”. In economic terms, it is inefficient to sustain an animal in order to use her flesh and skin rather than to grow meat and other animal-derived products directly, or replace them with plant-based alternatives. Similarly strong incentives exist in the realm of public health, which animal agriculture threatens by increasing the risks of zoonotic diseases, antibiotic resistant bacteria like MRSA, and cardiovascular disease. These incentives, none of which have anything to do with concern for nonhuman animals per se, could well be pushing humanity toward veganism more powerfully than anything else.

So despite the bleakness of the current situation, there are many incentives that appear to push humanity toward the abolition of animal exploitation, and we may even be moving in that direction faster than most of us expect (this is not, of course, a reason to be complacent about the unspeakable moral atrocity of “animal farming”, but it is something to take into account in our approach to helping future beings as much as we can).

In contrast, there are no corresponding incentives that lead us to help non-human animals in nature, and thus no strong reasons to think that humanity (including environmentalists, sadly) will take the interests of wild animals sufficiently into account if we do not advocate on their behalf.

Advocacy focused on wild animals is already vastly neglected in the animal movement today, and when we consider what the future is likely to look like, the level of priority animal advocates currently devote to the problem of wild-animal suffering seems even more disproportionate still.

10. Long-term nebulousness bias

This last bias is a bit more exotic and applies mostly to so-called longtermist effective altruists. People who focus on improving the long-term future can risk ending up with a rather nebulous sense of how to act and what to prioritize: there are so many hypothetical cause areas to consider, and it is often difficult to find tractable ways to further a given cause. Moreover, since there tends to be little real-world data that can help us make progress on these issues, longtermists are often forced to rely mostly on speculation — which in turn opens the floodgates for overconfidence in such speculations. In other words, focusing on the long-term future can easily lead us to rely far too strongly on untested abstractions, and to pay insufficient attention to real-world data and existing problems.

In this way, a (naive) longtermist focus may lead us to neglect concrete problems that evidently do have long-term relevance, and which we can take clear steps toward addressing today. We neglect such problems not only because most of our attention is devoted to more speculative things, but also because these concrete problems do not seem to resemble the “ultimate thing” that clearly improves the long-term future far better than other, merely decent focus areas. Unfortunately, such an “ultimate thing” is, I would argue, unlikely to ever be found. (And if one thinks one has found it, there are reasons to be skeptical.)

In effect, a naive longtermist focus can lead us to overlook just how promising work to reduce wild-animal suffering in fact is, and how long a list of compelling reasons one can give in its favor: in terms of scale, it vastly dominates all other sources of currently existing suffering; it is, as argued above, a tractable problem where there are fairly concrete and robust ways to make progress; and the problem is likely to exist and be dominant in scale for a long time — centuries, at least.

More than that, work to reduce wild-animal suffering is also likely to have many good flow-through effects. For example, such work is probably among the most promising actions we can take to prevent the spread of animal suffering to space, which is one of the least speculative s-risks (i.e. risks of astronomical future suffering) — there are already people who actively advocate that humanity should spread nature to space, and concrete proposals for how it could be accomplished already exist.

The risk of spreading wild-animal suffering to space appears greater than the risk of spreading factory farming to space, not least in light of the point made in the previous section concerning the incentives and future technologies that are likely to render factory farming obsolete. One may, of course, object that the risks of astronomical future suffering we reduce by addressing factory farming today do not involve factory farming itself but rather future analogs of it. This is a fair point, and such risks of future analogs to factory farming should indeed be taken seriously. However, by the same token, one can argue that we also address future analogs to wild-animal suffering by working on that problem today, and indeed further argue that this would be a superior focus.

After all, work to address wild-animal suffering appears more wide-ranging and inclusive than does work to address factory farming — for example, it is difficult to imagine a future where we address wild-animal suffering (and analog problems) yet fail to address factory farming (and analog problems). Future scenarios where we address the latter yet fail to address the former seem more plausible, since addressing wild-animal suffering takes a greater level of moral sophistication: it not only requires that we avoid directly harming other beings, but also that we actively help them.

Which brings us to another positive secondary effect of focusing on wild-animal suffering: such a focus embodies and reinforces the virtue of factoring in numbers in our moral deliberations, as well as the virtue of extending our circle of moral concern — and responsibility — to even include beings who suffer for reasons we ourselves had no hand in. It is a focus that reflects a truly universal view of our moral obligations, and it does this to a significantly greater extent than a mere opposition to factory farming or (anthropogenic) animal exploitation in general.

To be clear, I am not claiming that wild-animal suffering is necessarily the best thing to focus on for people trying to reduce suffering in the long-term future (I myself happen to think suffering-focused research of a more general nature is somewhat better). But I do claim that it is a decent candidate, and a better candidate than one is likely to realize when caught up in speculative far-mode sequence thinking.

Either/Or: a false choice

To say that most of us likely have strong biases against prioritizing wild-animal suffering, and that we should give it much greater priority, is not to say that we cannot still support efforts to abolish animal exploitation, and indeed do effective work toward this end.

As I have argued elsewhere, one of the many advantages of antispeciesist advocacy is that it encompasses all non-human animals and all the suffering they endure — anthropogenic as well as naturogenic.


Addendum: An important bias I left out above is the “proportion bias” (Vinding, 2020, 7.6), also known as “proportion dominance“:  our tendency to care more about helping 10 out of 10 individuals than we care about helping 10 out of 100, even though the impact is the same. This bias is especially relevant in the context of wild-animal suffering given the enormous scale at which it continually occurs as a backdrop to any altruistic effort we may pursue.

In terms of biases in the other direction, Jacy Reese has suggested some biases that may favor a focus on wild-animal suffering (though note that he largely agrees with me: “there are no ‘similarly strong’ biases [in the other direction] in the sense that, among self-identified animal advocates, the biases away from wild animal suffering are much stronger than biases toward”). I have shared my views on Jacy’s points on Twitter.

Chimps, Humans, and AI: A Deceptive Analogy

The prospect of smarter-than-human artificial intelligence (AI) is often presented and thought of in terms of a simple analogy: AI will stand in relation to us the way we stand in relation to chimps. In other words, AI will be qualitatively more competent and powerful than us, and its actions will be as inscrutable to humans as current human endeavors (e.g. science and politics) are to chimps.

My aim in this essay is to show that this is in many ways a false analogy. The difference in understanding and technological competence found between modern humans and chimps is, in an important sense, a zero-to-one difference that cannot be repeated.

How are humans different from chimps?

A common answer to this question is that humans are smarter. Specifically, at the level of our individual cognitive abilities, humans, with our roughly three times larger brains, are just far more capable.

This claim no doubt contains a large grain of truth, as humans surely do beat chimps in a wide range of cognitive tasks. Yet it is also false in some respects. For example, chimps have superior working memory compared to humans, and apparently also beat humans in certain video games, including games involving navigation in complex mazes.

But researchers who study human uniqueness actually provide some rather different, more specific answers to this question. If we focus on individual mental differences in particular, researchers have found that, crudely speaking, humans are different from chimps in three principal ways: 1) we can learn language, 2) we have a strong orientation toward social learning, and 3) we are highly cooperative (among our ingroup, compared to chimps).

These differences have in turn resulted in two qualitative differences in the abilities of humans and chimps in today’s world.

I. Symbolic language

The first is that we humans have acquired an ability to think and communicate in terms of symbolic language that represents elaborate concepts. We can learn about the deep history of life and the universe, as well as the likely future of the universe, including the fundamental limits to future space travel and future computations. Any educated human can learn a good deal about these things whereas no chimp can.

Note how this is truly a zero-to-one difference: no symbolic language versus an elaborate symbolic language through which knowledge can be represented and continually developed (see chapter 1 in Deacon, 1997). It is the difference between having no science of physics versus having an elaborate such science with which we can predict future events and put hard limits on future possibilities.

This zero-to-one difference cannot really be repeated. Given that we already have physical models that predict, say, the future motion of planets and the solar system to a fairly high degree of accuracy, the best one can do in this respect is to (slightly) improve the accuracy of these predictions. Such further improvements cannot be compared to going from zero physics to current physics.

The same point applies to our scientific understanding more generally: we currently have theories that work decently well at explaining most of the phenomena around us. And though one can significantly improve the accuracy and sophistication of many of these theories, any such further improvement would be much less significant than the qualitative leap from absolutely no conceptual models to the entire collection of models and theories we currently have.

For example, going from no understanding of evolution by natural selection to the elaborate understanding of biology we have today cannot be matched, in terms of qualitative and revolutionary leaps, by further refinements in biology. We have already mapped out the core basics of biology (in fact a great deal more than that), and this can only be done once.

This is not an original point. Robin Hanson has made essentially the same point in response to the notion that future machines will be “as incomprehensible to us as we are to goldfish”:

This seems to me to ignore our rich multi-dimensional understanding of intelligence elaborated in our sciences of mind (computer science, AI, cognitive science, neuroscience, animal behavior, etc.).

… the ability of one mind to understand the general nature of another mind would seem mainly to depend on whether that first mind can understand abstractly at all, and on the depth and richness of its knowledge about minds in general. Goldfish do not understand us mainly because they seem incapable of any abstract comprehension. …

It seems to me that human cognition is general enough, and our sciences of mind mature enough, that we can understand much about quite a diverse zoo of possible minds, many of them much more capable than ourselves on many dimensions.

Ramez Naam has argued similarly in relation to the idea that there will be some future time or intelligence that we are fundamentally unable to understand. He argues that our understanding of the future is growing rather than shrinking as time progresses, and that AI and other future technologies will not be beyond comprehension:

All of those [future technologies] are still governed by the laws of physics. We can describe and model them through the tools of economics, game theory, evolutionary theory, and information theory. It may be that at some point humans or our descendants will have transformed the entire solar system into a living information processing entity — a Matrioshka Brain. We may have even done the same with the other hundred billion stars in our galaxy, or perhaps even spread to other galaxies.

Surely that is a scale beyond our ability to understand? Not particularly. I can use math to describe to you the limits on such an object, how much computing it would be able to do for the lifetime of the star it surrounded. I can describe the limit on the computing done by networks of multiple Matrioshka Brains by coming back to physics, and pointing out that there is a guaranteed latency in communication between stars, determined by the speed of light. I can turn to game theory and evolutionary theory to tell you that there will most likely be competition between different information patterns within such a computing entity, as its resources (however vast) are finite, and I can describe to you some of the dynamics of that competition and the existence of evolution, co-evolution, parasites, symbiotes, and other patterns we know exist.

Chimps cannot understand human politics and science to a similar extent. Thus, the truth is that there is a strong disanalogy between the understanding chimps have of humans versus the understanding that we humans — thanks to our conceptual tools — can have of any possible future intelligence (in physical and computational terms, say).

Note that the qualitative leap reviewed above was not one that happened shortly after human ancestors diverged from chimp ancestors. Instead, it was a much more recent leap that has been unfolding gradually since the first humans appeared, and which has continued to accelerate in recent centuries, as we have developed ever more advanced science and mathematics. In other words, this qualitative step has been a product of cultural evolution just as much as biological evolution. Early humans presumably had a roughly similar potential to learn modern language, science, mathematics, etc. But such conceptual tools could not be acquired in the absence of a surrounding culture able to teach these innovations.

Ramez Naam has made a similar point:

If there was ever a singularity in human history, it occurred when humans evolved complex symbolic reasoning, which enabled language and eventually mathematics and science. Homo sapiens before this point would have been totally incapable of understanding our lives today. We have a far greater ability to understand what might happen at some point 10 million years in the future than they would to understand what would happen a few tens of thousands of years in the future.

II. Cumulative technological innovation

The second zero-to-one difference between humans and chimps is that we humans build things. Not just that we build things, but that we refine our technology over time. After all, many non-human animals use tools in the form of sticks and stones, and some even shape primitive tools of their own. But only humans improve and build upon the technological inventions of their ancestors.

Consequently, humans are unique in expanding their abilities by systematically exploiting their environment, molding the things around them into ever more useful self-extensions. We have turned wildlands into crop fields; we have created technologies that can harvest energy — from oil, gas, wind, and sun — and we have built external memories far more reliable than our own, such as books and hard disks.

This is another qualitative leap that cannot be repeated: the step from having absolutely no cumulative technology to exploiting and optimizing our external environment toward our own ends. The step from having no external memory to having the current repository of stored human knowledge at our fingertips, and from harvesting absolutely no energy (other than through individual digestion) to collectively harvesting and using hundreds of quintillions of Joules every year.

To be sure, it is possible to improve on and expand these innovations. We can harvest greater amounts of energy, for example, and create even larger external memories. Yet these are merely quantitative differences, and humanity indeed continually makes such improvements each year. They are not zero-to-one differences that only a new species could bring about. And what is more, we know that the potential for making further technological improvements is, at least in many respects, quite limited.

Take energy efficiency as an example. Many of our machines and energy harvesting technologies have already reached a significant fraction of the maximally possible efficiency. For example, electric motors and pumps tend to have around 90 percent energy efficiency, and the best solar panels have an efficiency greater than 40 percent. So as a matter of hard physical limits, many of our technologies cannot be made orders of magnitude more efficient; in fact, a large number of them can at most be marginally improved.

In sum, we are unique in being the first species that systematically sculpted our surrounding environment and turned it into ever-improving tools, many of which have near-maximal efficiency. This step cannot be repeated, only expanded further.


Just like the qualitative leap in our symbolic reasoning skills, the qualitative leap in our ability to create technology and shape our environment emerged, not between chimps and early humans, but between early humans and today’s humans, as the result of a cultural process occurring over thousands of years. In fact, the two leaps have been closely related: our ability to reason and communicate symbolically has enabled us to create cumulative technological innovation. Conversely, our technologies have allowed us to refine our knowledge and conceptual tools, by enabling us to explore and experiment, which in turn made us able to build even better technologies with which we could advance our knowledge even further, and so on.

This, in a nutshell, is the story of the growth of human knowledge and technology, a story of recursive self-improvement (see Simler, 2019, “On scientific networks”). It is not really a story about the individual human brain per se. After all, the human brain does not accomplish much in isolation (nor is it the brain with the largest number of neurons; several species have more neurons in the forebrain). It is more a story about what happened between and around brains: in the exchange of information in networks of brains and in the external creations designed by them. A story made possible by the fact that the human brain is unique in being by far the most cultural brain of all, with its singular capacity to learn from and cooperate with others.

The range of human abilities is surprisingly wide

Another way in which an analogy to chimps is frequently drawn is by imagining an intelligence scale along which different species are ranked, such that, for example, we have “rats at 30, chimps at 60, the village idiot at 90, the average human at 98, and Einstein at 100”, and where future AI may in turn be ranked many hundreds of points higher than Einstein. According to this picture, it is not just that humans will stand in relation to AI the way chimps stand in relation to humans, but that AI will be far superior still. The human-chimp analogy is, on this view, a severe understatement of the difference between humans and future AI.

Such an intelligence scale may seem intuitively compelling, but how does it correspond to reality? One way to probe this question is to examine the range of human abilities in chess. The standard way to rank chess skills is with the Elo rating system, which is a good predictor of the outcomes of chess games between different players, whether human, digital, or otherwise.

An early human beginner will have a rating around 300, a novice around 800, and a rating in the range 2000-2199 is ranked as “Expert”. The highest rating ever achieved is 2882 by Magnus Carlsen.

How large is this range of chess skills in an absolute sense? Remarkably large, it turns out. For example, it took more than four decades from when computers were first able to beat a human chess novice (the 1950s), until a computer was able to beat the best human player (1997, officially). In other words, the span from novice to Kasparov corresponded to more than four decades of progress in hardware — i.e. a million times more computing power — and software. This alone suggests that the human range of chess skills is rather wide.

Yet the range seems even broader when we consider the upper bounds of chess performance. After all, the fact that it took computers decades to go from human novice to world champion does not mean that the best human is not still ridiculously far from the best a computer could be in theory. Surprisingly, however, this latter distance does in fact seem quite small. Estimates suggest that the best possible chess machine would have an Elo rating around 3600, which means that the relative distance between the best possible computer and the best human is only around 700 Elo points (the Elo rating is essentially a measure of relative distance; 700 Elo points corresponds to a winning percentage of around 1.5 percent for the losing player).

This implies that the distance between the best human (Carlsen) and a chess “Expert” (someone belonging to the top 5 percent of chess players) is similar to the distance between the best human and the best possible chess brain, while the distance between a human beginner and the best human is far greater (2500 Elo points). This stands in stark contrast to the intelligence scale outlined above, which would predict the complete opposite: the distance from a human novice to the best human should be comparatively small whereas the distance from the best human to the optimal brain should be the larger one by far.


It may be objected that chess is a bad example, and that it does not really reflect what is meant by the intelligence scale above. But the question is then what would be a better measure. After all, a similar story seems to apply to other games, such as shogi and go: the human range of abilities is surprisingly wide and the best players are significantly closer to optimal than they are to novice players.

In fact, one can argue that the objection should go in the opposite direction, as human brains are not built for chess, and hence we should expect even the best humans to be far from optimal at it. We should expect to be much closer to “optimal” at solving problems that are more important for our survival, such as social cognition and natural language processing — skills that most people are wired to master at super-Carlsen levels.

Regardless, the truth is that humans are mastering ever more “games”, literal as well as figurative ones, at optimal or near-optimal levels. Not because evolution “just so happened to stumble upon the most efficient way to assemble matter into an intelligent system”, but rather because it created a species able to make cultural and technological progress toward ever greater levels of competence.

The cultural basis of the human capability expansion

The intelligence scale outlined above misses two key points. First, human abilities are not a constant. Whether we speak of individual abilities (e.g. the abilities of elite chess players) or humanity’s collective abilities (e.g. building laptops and sending people to the moon), it is clear that our abilities have increased dramatically as our culture and technology have expanded.

Second, because human abilities are not a constant, the range of human abilities is far wider, in an absolute sense, than the intelligence scale outlined above suggests, as it has grown and still continues to grow over time.

Chess is a good example of this. Untrained humans and chimps have the same (non-)skill level at chess. Yet thanks to culture, some people can learn to master the game. A wealthy society can allow people to specialize in chess, and makes it possible for knowledge to accumulate in books and experts. Eventually, it enables learning from super-human chess engines, whose innovations we can adopt just as we do those of other humans.

And yet we humans expand our abilities to a much greater extent than the example of increased human chess abilities suggests, as we not only expand our abilities by stimulating our brains with progressively better forms of practice and information, but also by extending ourselves directly with technology. For example, we can all use a chess engine to find great chess moves for us. Our latest technologies enable us to accomplish ever-more tasks that no human could ever accomplish unaided.

Worth noting in this regard is that this self-extension process seems to have slowed down in recent decades, likely because we have reaped most low-hanging fruits already, and in some respects because it is impossible to improve things much further (we already mentioned energy efficiency as an example where we are getting close to the upper limits in many respects).

This suggests that not only is there not a qualitative leap similar to that between chimps and modern humans ahead of us, but that even a quantitative growth explosion, with relative growth rates significantly higher than what we have seen in the past, should not be our default expectation either (for some support for this claim, see “Peak growth might lie in the past” in Vinding, 2017).

Why this is relevant

The errors of the human-chimp analogy are worth highlighting for a few reasons. First, the analogy can lead us to overestimate how much everything will change with AI. It leads us to expect qualitative leaps of sorts that cannot be repeated.

Second, the human-chimp analogy makes us underestimate how much we currently know and are able to understand. To think that intelligent systems of the future will be as incomprehensible to us today as human affairs are to chimps is to underestimate how extensive and universal our current knowledge of the world in fact is — not just when it comes to physical and computational limits, but also in relation to general economic and game-theoretic principles. We know a good deal about economic growth, for example, and this knowledge has a lot to say about how we should expect future intelligent systems to grow. In particular, it suggests that local AI-FOOM growth is unlikely.

The analogy can thus have an insidious influence by making us feel like current data and trends cannot be trusted much, because look how different humans are from chimps, and look how puny the human brain is compared to ultimate limits. I think this is exactly the wrong way to think about the future. We should base our expectations on a deep study of past trends, including the actual evolution of human competences — not simple analogies.

Relatedly, the human-chimp analogy is also relevant in that it can lead us to grossly overestimate the probability of an AI-FOOM scenario. That is, if we get the story about the evolution of human competences so wrong that we think the differences we observe today between chimps and modern humans reduce mostly to a story about changes in individual brains, then we are likely to have similarly inaccurate expectations about what comparable innovations in some individual machine are able to effect on their own.

If the human-chimp analogy leads us to (marginally) overestimate the probability of a FOOM scenario, it may nudge us toward focusing too much on some single, concentrated future thing that we expect to be all-important: the AI that suddenly becomes qualitatively more competent than humans. In effect, the human-chimp analogy can lead us to neglect broader factors, such as cultural and institutional developments.

Note that the above is by no means a case for complacency about risks from AI. It is important that we get a clear picture of such risks, and that we allocate our resources accordingly. But this requires us to rely on accurate models of the world. If we overemphasize one set of risks, we are by necessity underemphasizing others.

Blog at WordPress.com.

Up ↑