Why Altruists Should Perhaps Not Prioritize Artificial Intelligence: A Lengthy Critique

The following is a point-by-point critique of Lukas Gloor’s essay Altruists Should Prioritize Artificial Intelligence. My hope is that this critique will serve to make it clear — to Lukas, myself, and others — where and why I disagree with this line of argument, and thereby hopefully also bring some relevant considerations to the table with respect to what we should be working on to best reduce suffering. I should like to note, before I begin, that I have the deepest respect for Lukas, and that I consider his work very important and inspiring.

Below, I quote every paragraph from the body of Lukas’ article, which begins with the following abstract:

The large-scale adoption of today’s cutting-edge AI technologies across different industries would already prove transformative for human society. And AI research rapidly progresses further towards the goal of general intelligence. Once created, we can expect smarter-than-human artificial intelligence (AI) to not only be transformative for the world, but also (plausibly) to be better than humans at self-preservation and goal preservation. This makes it particularly attractive, from the perspective of those who care about improving the quality of the future, to focus on affecting the development goals of such AI systems, as well as to install potential safety precautions against likely failure modes. Some experts emphasize that steering the development of smarter-than-human AI into beneficial directions is important because it could make the difference between human extinction and a utopian future. But because we cannot confidently rule out the possibility that some AI scenarios will go badly and also result in large amounts of suffering, thinking about the impacts of AI is paramount for both suffering-focused altruists as well as those focused on actualizing the upsides of the very best futures.

An abstract of my thoughts on this argument:

My response to this argument is twofold: 1) I do not consider the main argument presented by Lukas, as I understand it, to be plausible, and 2) I think we should think hard about whether we have considered the opportunity cost carefully enough. We should not be particularly confident, I would argue, that any of us have found the best thing to focus on to reduce the most suffering.

I do not think the claim that “altruists can expect to have the largest positive impact by focusing on artificial intelligence” is warranted. In part, my divergence from Lukas rests on empirical disagreements, and in larger part it stems from what may be called “conceptual disagreements” — I think most talk about “superintelligence” is conceptually confused. For example, intelligence as “cognitive abilities” is liberally conflated with intelligence as “the ability to achieve goals in general”, and this confusion does a lot of deceptive work.

I would advocate for more foundational research into the question of what we ought to prioritize. Artificial intelligence undoubtedly poses many serious risks, yet it is important that we maintain a sense of proportion with respect to these risks relative to other serious risks, many of which we have not even contemplated yet.

I will now turn to the full argument presented by Lukas.

I. Introduction and definitions

Terms like “AI” or “intelligence” can have many different (and often vague) meanings. “Intelligence” as used here refers to the ability to achieve goals in a wide range of environments. This definition captures the essence of many common perspectives on intelligence (Legg & Hutter, 2005), and conveys the meaning that is most relevant to us, namely that agents with the highest comparative goal-achieving ability (all things considered) are the most likely to shape the future.

A crucial thing to flag is that “intelligence” here refers to the ability to achieve goals — not to scoring high on an IQ test, or “intelligence” as “advanced cognitive abilities”. And these are not the same, and should not be conflated (indeed, this is one of the central points of my book Reflections on Intelligence, which dispenses with the muddled term “intelligence” at an early point, and instead examines the nature of this better defined “ability to achieve goals” in greater depth).

While it is true that the concept of goal achieving is related to the concept of IQ, the latter is much narrower, as it relates to a specific class of goals. Boosting the IQ of everyone would not immediately boost our ability to achieve goals in every respect — at least not immediately, and not to the same extent across all domains. For even if we all woke up with an IQ of 200 tomorrow, all the external technology with which we run and grow our economy would still be the same. Our cars would drive just as fast, the energy available to us would be the same, and so would the energy efficiency of our machines. And while a higher IQ might now enable us to grow this external technology faster, there are quite restricting limits to how much it can grow. Most of our machines and energy harvesting technology cannot be made many times more efficient, as their efficiency is already a significant fraction — 15 to 40 percent — of the maximum physical limit. In other words, their efficiency cannot be doubled more than a couple of times, if even that.

One could then, of course, build more machines and power plants, yet such an effort would itself be constrained strongly by the state of our external technology, including the energy available to us; not just by the cognitive abilities available. This is one of the reasons I am skeptical of the idea of AI-powered runaway growth. Yes, greater cognitive abilities is a highly significant factor, yet there is just so much more to growing the economy and our ability to achieve a wide range of goals than that, as evidenced by the fact that we have seen a massive increase in computer-powered cognitive abilities — indeed, exponential growth for many decades by many measures — and yet we have continued to see fairly stable, in fact modestly declining, economic growth.

If one considers the concept of “increase in cognitive powers” to be the same as “increase in the ability to achieve goals, period” then this criticism will be missed. “I defined intelligence to be the ability to achieve goals, so when I say intelligence is increased, then all abilities are increased.” One can easily come to entertain a kind of motte and bailey argument in this way, by moving back and forth between this broad notion of intelligence as “the ability to achieve goals” and the more narrow sense of intelligence as “cognitive abilities”. To be sure, a statement like the one above need not be problematic as such, as long as one is clear that this concept of intelligence lies very far from “intelligence as measured by IQ/raw cognitive power”. Such clarity is often absent, however, and thus the statement is quite problematic in practice, with respect to the goals of communicating clearly and not confusing ourselves.

Again, my main point here is that increasing cognitive powers should not be conflated with increasing the ability to achieve goals in general — in every respect. I think much confusion springs from a lack of clarity on this matter.

While everyday use of the term “intelligence” often refers merely to something like “brainpower” or “thinking speed,” our usage also presupposes rationality, or goal-optimization in an agent’s thinking and acting. In this usage, if someone is e.g. displaying overconfidence or confirmation bias, they may not qualify as very intelligent overall, even if they score high on an IQ test. The same applies to someone who lacks willpower or self control.

This is an important step toward highlighting the distinction between “goal achieving ability” and “IQ”, yet it is still quite a small step, as it does not really go much beyond distinguishing “high IQ” from “optimal cognitive abilities for goal achievement”. We are still talking about things going on in a single human head (or computer), while leaving out the all-important aspect that is (external) culture and technology. We are still not talking about the ability to achieve goals in general.

Artificial intelligence refers to machines designed with the ability to pursue tasks or goals. The AI designs currently in use – ranging from trading algorithms in finance, to chess programs, to self-driving cars – are intelligent in a domain-specific sense only. Chess programs beat the best human players in chess, but they would fail terribly at operating a car. Similarly, car-driving software in many contexts already performs better than human drivers, but no amount of learning (at least not with present algorithms) would make [this] software work safely on an airplane.

My only comment here would be that it is not quite clear what counts as artificial intelligence. For example, would a human, edited as well as unedited, count as “a machine designed with the ability to pursue tasks or goals”? And could not all software be considered “designed with the ability to pursue tasks or goals”, and hence all software would be artificial intelligence by this definition? If so, we should then just be clear that this definition is quite broad, including both all humans and all software, and more.

The most ambitious AI researchers are working to build systems that exhibit (artificial) general intelligence (AGI) – the type of intelligence we defined above, which enables the expert pursuit of virtually any task or objective.

This is where the distinction we drew above becomes relevant. While the claim quoted above may be true in one sense, we should be clear that the most ambitious AI researchers are not working to increase “all our abilities”, including our ability to get more energy out of our steam engines and solar panels. Our economy arguably works on that broader endeavor. AI researchers, in contrast, work only on bettering what may be called “artificial cognitive abilities”, which, granted, may in turn help spur growth in many other areas (although the degree to which it would do so is quite unclear, and likely surprisingly limited in the big picture, since “growth may be constrained not by what we are good at but rather by what is essential and yet hard to improve”).

In the past few years, we have witnessed impressive progress in algorithms becoming more and more versatile. Google’s DeepMind team for example built an algorithm that learned to play 2-D Atari games on its own, achieving superhuman skill at several of them (Mnih et al., 2015). DeepMind then developed a program that beat the world champion in the game of Go (Silver et al., 2016), and – tackling more practical real-world applications – managed to cut down data center electricity costs by rearranging the cooling systems.

I think it is important not to overstate recent progress compared to progress in the past. We also saw computers becoming better than humans at many things several decades ago, including many kinds of mathematical calculations (and people also thought that computers would soon beat humans at everything back then). So superhuman skill at many tasks is not what is new and unique about recent progress, but rather that these superhuman skills have been attained via self-training, and, as Lukas notes, that the skills achieved by this training seem of a broader, more general nature than the skills of a single algorithm in the past.

And yet the breadth of these skills should not be overstated either, as the skills cited are all acquired in a rather expensive trial-and-error fashion with readily accessible feedback. This mode of learning surely holds a lot of promise in many areas, yet there are reasons to be skeptical that such learning can bring us significantly closer to achieving all the cognitive and motor abilities humans have (see also David Pearce’s “Human’s and Intelligent Machines“; one need not agree with Pearce on everything to agree with some of his reasons to be skeptical).

That DeepMind’s AI technology makes quick progress in many domains, without requiring researchers to build new architecture from scratch each time, indicates that their machine learning algorithms have already reached an impressive level of general applicability. (Edit: I wrote the previous sentence in 2016. In the meantime [January 2018] DeepMind went on to refine its Go-playing AI, culminating in a version called AlphaGo Zero. While the initial version of DeepMind’s Go-playing AI started out with access to a large database of games played by human experts, AlphaGo Zero only learns through self-play. Nevertheless, it managed to become superhuman after a mere 4 days of practice. After 40 days of practice, it was able to beat its already superhuman predecessor 100–0. Moreover, Deepmind then created the version AlphaZero, which is not a “Go-specific” algorithm anymore. Fed with nothing but the rules for either Go, chess, or shogi, it managed to become superhuman at each of these games in less than 24 hours of practice.)

This is no doubt impressive. Yet it is also important not to overstate how much progress that was achieved in 24 hours of practice. This is not, we should be clear, a story about innovation going from zero to superhuman in 24 hours, but rather the story of immense amounts of hardware developed over decades which has then been fed with an algorithm that has also been developed over many years by many people. And then, this highly refined algorithm running on specialized, cutting-edge hardware is unleashed to reach its dormant potential.

And this potential was, it should be noted, not vastly superior to the abilities of previous systems. In chess, for instance, AlphaZero beat the chess program Stockfish (although Stockfish author Tord Romstad notes that it was a version that was a year old and not running on optimal hardware) 25 times as white, 3 as black, and drew the remaining 72 times. Thus, it was significantly better, yet it still did not win in most of the games. Similarly, in Go, AlphaZero won 60 games and lost 40, while in Shogi it won 90 times, lost 8, and drew twice.

Thus, AlphaZero undoubtedly comprised clear progress with respect to these games, yet not an enormous leap that rendered it unbeatable, and certainly not a leap made in a single day.

The road may still be long, but if this trend continues, developments in AI research will eventually lead to superhuman performance across all domains. As there is no reason to assume that humans have attained the maximal degree of intelligence (Section III), AI may soon after reaching our own level of intelligence surpass it.

Again, I would start by noting that human “intelligence” as our “ability to achieve goals” is strongly dependent on the state of our technology and culture at large, not merely our raw cognitive powers. And the claim made above that there is no reason to believe that humans have attained “the maximal degree of intelligence” seems, in this context, to mostly refer to our cognitive abilities rather than our ability to achieve goals in general. For with respect to our ability to achieve goals in general, it is clear that our abilities are not maximal, but indeed continually growing, largely as the result of better software and better machines. Thus, there is not a dichotomous relationship between “human abilities to achieve goals” and “our machines’ abilities to achieve goals”. And given that our ability to achieve goals is in many ways mostly limited by what our best technology can do — how fast our airplanes can fly, how fast our hardware is, how efficient our power plants are, etc. — it is not clear why some other agent or set of agents coming to control this technology (which is extremely difficult to imagine in the first place given the collaborative nature of the grosser infrastructure of this technology) should be vastly more capable of achieving goals than humans powered by/powering this technology.

As for AI surpassing “our own level of intelligence”, one can say that, at the level of cognitive tasks, machines have already been vastly superhuman in many respects for many years — in virtually all mathematical calculations, for instance. And now also in many games, ranging from Atari to Go. Yet, as noted above, I would argue that, so far, such progress has comprised a clear increase in human “intelligence” in the general sense: it has increased our ability to achieve goals.

Nick Bostrom (2014) popularized the term superintelligence to refer to (AGI-)systems that are vastly smarter than human experts in virtually all respects. This includes not only skills that computers traditionally excel at, such as calculus or chess, but also tasks like writing novels or talking people into doing things they otherwise would not. Whether AI systems would quickly develop superhuman skills across all possible domains, or whether we will already see major transformations with [superhuman skills in] just a [few] such domains while others lag behind, is an open question.

I would argue that our machines already have superhuman skills in countless domains, and that this has indeed already given rise to major transformations, in one sense of this term at least.

Note that the definitions of “AGI” and “superintelligence” leave open the question of whether these systems would exhibit something like consciousness.

I have argued to the contrary in the chapter “Consciousness — Orthogonal or Crucial?” in Reflections on Intelligence.

This article focuses on the prospect of creating smarter-than-human artificial intelligence. For simplicity, we will use the term “AI” in a non-standard way here, to refer specifically to artificial general intelligence (AGI).

Again, I would flag that the meaning of the term general intelligence, or AGI, in this context is not clear. It was defined above as the ability that “enables the expert pursuit of virtually any task or objective”. Yet the ability of humans to achieve goals in general is, I would still argue, in large part the product of their technology and culture at large, and AGI, as Lukas uses it here, does not seem to refer to anything remotely like this, i.e. “the sum of the capabilities of our technology and culture”. Instead, it seems to refer to something much more narrow and singular — something akin to “a system that possesses (virtually) all the cognitive abilities that a human does, and which possesses them at a similar or greater level”. I think this is worth highlighting.

The use of “AI” in this article will also leave open how such a system is implemented: While it seems plausible that the first artificial system exhibiting smarter-than-human intelligence will be run on some kind of “supercomputer,” our definition allows for alternative possibilities.

Again, what does “smarter-than-human intelligence” mean here? Machines can already do things that no unaided human can. It seems to refer to what I defined above: “a system that possesses (virtually) all the cognitive abilities that a human does, and which possesses them at a similar or greater level” — not the ability to achieve goals in general. And as for when a computer might have “(virtually) all the cognitive abilities that a human does”, it seems highly doubtful that any system will ever suddenly emerge with them all, given the modular, many-faceted nature of our minds. Instead, it seems much more likely that the gradual process of machines becoming better than humans at particular tasks will continue in its usual, gradual way. Or so I have argued.

The claim that altruists should focus on affecting AI outcomes is therefore intended to mean that we should focus on scenarios where the dominant force shaping the future is no longer (biological) human minds, but rather some outgrowth of information technology – perhaps acting in concert with biotechnology or other technologies. This would also e.g. allow for AI to be distributed over several interacting systems.

I think this can again come close to resembling a motte and bailey argument: it seems very plausible that the future will not be controlled mostly by what we would readily recognize as biological humans today. Yet to say that we should aim to impact such a future by no means implies that we should aim to impact, say, a small set of AI systems which might determine the entire future based on their goal functions (note: I am not saying Lukas has made this claim above, but this is often what people seem to consider the upshot of arguments of this kind, and also what it seems to me that Lukas is arguing below, in the rest of his essay). Indeed, the claim above is hardly much different from saying that we should aim to impact the long-term future. But Lukas seems to be moving back and forth between this general claim and the much narrower claim that we should focus on scenarios involving rapid growth acceleration driven mostly by software, which is the kind of scenario his essay seems almost exclusively focused on.

II. It is plausible that we create human-level AI this century

Even if we expect smarter-than-human artificial intelligence to be a century or more away, its development could already merit serious concern. As Sam Harris emphasized in his TED talk on risks and benefits of AI, we do not know how long it will take to figure out how to program ethical goals into an AI, solve other technical challenges in the space of AI safety, or establish an environment with reduced dangers of arms races. When the stakes are high enough, it pays to start preparing as soon as possible. The sooner we prepare, the better our chances of safely managing the upcoming transition.

I agree that it is worth preparing for high-stakes outcomes. But I think it is crucial that we get a clear sense of what these might look like, as well as how likely they are. “Altruists Should Prioritize Exploring Long-Term Future Outcomes, and Work out How to Best Influence Them”. To say that we should focus on “artificial intelligence”, which has a rather narrow meaning in most contexts (something akin to a software program), when we really mean that we should focus on the future of goal achieving systems in general is, I think, somewhat misleading.

The need for preparation is all the more urgent given that considerably shorter timelines are not out of the question, especially in light of recent developments. While timeline predictions by different AI experts span a wide range, many of those experts think it likely that human-level AI will be created this century (conditional on civilization facing no major disruptions in the meantime). Some even think it may emerge in the first half of this century: In a survey where the hundred most-cited AI researchers were asked in what year they think human-level AI is 10% likely to have arrived by, the median reply was 2024 and the mean was 2034. In response to the same question for a 50% probability of arrival, the median reply was 2050 with a mean of 2072 (Müller & Bostrom, 2016).1

Again, it is important to be careful about definitions. For what is meant by “human-level AI” in this context? The authors of the cited source are careful to define what they mean: “Define a ‘high–level machine intelligence’ (HLMI) as one that can carry out most human professions at least as well as a typical human.”

And yet even this definition is quite vague, since “most human professions” is not a constant. A couple of hundred years ago, the profession of virtually all humans was farming, whereas only a couple percent of people in developed nations are employed in farming today. And this is not an idle point, because as machines become able to do jobs hitherto performed by humans, market forces will push humans to take new jobs that machines cannot do. And these new jobs may be those that require abilities that it will take many centuries for machines to acquire, if non-biological machines will indeed ever acquire them (this is not necessarily that implausible, as these abilities may include “looking like a real, empathetic biological human who ignites our brain circuits in the right ways”).

Thus, the questionnaire above seems poorly defined. And if it asks about most current human professions, its relevance appears quite limited; also because the nature of different professions change over time as well. A doctor today does not do all the same things a doctor did a hundred years ago, and the same will likely apply to doctors of the future. In other words, also within existing professions can we expect to see humans move toward doing the things that machines cannot do/we do not prefer them to do, even as machines become ever more capable.

While it could be argued that these AI experts are biased towards short timelines, their estimates should make us realize that human-level AI this century is a real possibility.

Yet we should keep in mind what they were asked about, and how relevant this is. Even if most (current?) human professions might be done by machines within this century, this does not imply that we will see “a system that possesses (virtually) all the cognitive abilities that a human does, and which possesses them at a similar or greater level” within this century. These are quite different claims.

The next section will argue that the subsequent transition from human-level AI to superintelligence could happen very rapidly after human-level AI actualizes. We are dealing with the decent possibility – e.g. above 15% likelihood even under highly conservative assumptions – that human intelligence will be surpassed by machine intelligence later this century, perhaps even in the next couple of decades. As such a transition will bring about huge opportunities as well as huge risks, it would be irresponsible not to prepare for it.

I want to flag, again, that it is not clear what “human-level AI” means. Lukas seemed to first define intelligence as something like “the ability to achieve goals in general”, which I have argued is not really what he means here (indeed, it is a rather different beast which I seek to examine in Reflections on Intelligence). And the two senses of the term “human-level intelligence” mentioned in the previous paragraph — “the ability to do most human professions” versus “possessing virtually all human cognitive abilities” — should not be conflated either. So it is in fact not clear what is being referred to here, although I believe it is the latter: “possessing virtually all human cognitive abilities at a similar or greater level”.

It should be noted that a potentially short timeline does not imply that the road to superintelligence is necessarily one of smooth progress: Metrics like Moore’s law are not guaranteed to continue indefinitely, and the rate of breakthrough publications in AI research may not increase (or even stay constant) either. The recent progress in machine learning is impressive and suggests that fairly short timelines of a decade or two are not to be ruled out. However, this progress could also be mostly due to some important but limited insights that enable companies like DeepMind to reap the low-hanging fruit before progress would slow down again. There are large gaps still to be filled before AIs reach human-level intelligence, and it is difficult to estimate how long it will take researchers to bridge these gaps. Current hype about AI may lead to disappointment in the medium term, which could bring about an “AI safety winter” with people mistakenly concluding that the safety concerns were exaggerated and smarter-than-human AI is not something we should worry about yet.

This seems true, yet it should also be conceded that a consistent lack of progress in AI would count as at least weak evidence against the claim that we should mainly prioritize what is usually referred to as “AI safety“. And more generally, we should be careful not to make the hypothesis “AI safety is the most important thing we could be working on” into an unfalsifiable one.

As for Moore’s law, not only is it “not guaranteed to continue indefinitely”, but we know, for theoretical reasons, that it must come to an end within a decade, at least in its original formulation concerning silicon transistors, and progress has indeed already been below the prediction of “the law” for some time now. And the same can be said about other aspects in hardware progress: it shows signs of waning off.

If AI progress were to slow down for a long time and then unexpectedly speed up again, a transition to superintelligence could happen with little warning (Shulman & Sandberg, 2010). This scenario is plausible because gains in software efficiency make a larger comparative difference to an AI’s overall capabilities when the hardware available is more powerful. And once an AI develops the intelligence of its human creators, it could start taking part in its own self-improvement (see section IV).

I am not sure I understand the claims being made here. With respect to the first argument about gains in efficiency, the question is how likely we should expect such gains to be if progress has been slow for long. Other things being equal, this would seem less likely in a time where growth is slow than in a time when it is fast, and especially if there is not much growth in hardware either, since hardware growth may in large part be driving growth in software.

I am not sure I follow the claim about AI developing the intelligence of its human creators, and then taking part in its own improvement, but I would just note, as Ramez Naam has argued, that AI, and our machines in general, are already playing a significant role in their own improvement in many ways. In other words, we already actively use our best, most capable technology to build the next generation of such technology.

Indeed, on a more general, yet also less directly relevant note, I would also add that we humans have in some sense been using our most advanced cognitive tools to build the next generation of such tools for hundreds of thousands of years. For over the course of evolution, individual humans have been using the best of their cognitive abilities to select the mates who had the best total package (they could get), of which cognitive abilities were a significant part. In this sense, the idea that “dumb and blind” evolution created intelligent humans is actually quite wrong. The real story is rather one of cognitive abilities actively selecting cognitive abilities (along with other things). A gradual design process over the course of which ever greater cognitive powers were “creating” and in turn created.

For AI progress to stagnate for a long period of time before reaching human-level intelligence, biological brains would have to have surprisingly efficient architectures that AI cannot achieve despite further hardware progress and years of humans conducting more AI research.

Looking over the past decades of AI research and progress, we can say that it indeed has been a fairly long period of time since computers first surpassed humans in the ability to do mathematical calculations, and yet there are still many things humans can do which computers cannot, such as having meaningful conversations with other humans, learning fast from a few examples, and experiencing and expressing feelings. And yet these examples still mostly pertain to cognitive abilities, and hence still overlook other abilities that are also relevant with respect to machines taking over human jobs (if we focus on that definition of “human-level AI”), such as having the physical appearance of a real, biological human, which does seem in strong demand in many professions, especially in the service industry.

However, as long as hardware progress does not come to a complete halt, AGI research will eventually not have to surpass the human brain’s architecture or efficiency anymore. Instead, it could become possible to just copy it: The “foolproof” way to build human-level intelligence would be to develop whole brain emulation (WBE) (Sandberg & Bostrom, 2008), the exact copying of the brain’s pattern of computation (input-output behavior as well as isomorphic internal states at any point in the computation) onto a computer and a suitable virtual environment. In addition to sufficiently powerful hardware, WBE would require scanning technology with fine enough resolution to capture all the relevant cognitive function, as well as a sophisticated understanding of neuroscience to correctly draw the right abstractions. Even though our available estimates are crude, it is possible that all these conditions will be fulfilled well before the end of this century (Sandberg, 2014).

Yet it should be noted that there are many who doubt that this is a foolproof way to build “human-level intelligence” (a term that in this context again seems to mean “a system with roughly the same cognitive abilities as the human brain”). Many doubt that it is even a possibility, and they do so for many different reasons (e.g. that a single, high-resolution scanning of the brain is not enough to capture and enable an emulation of its dynamic workings; that a digital computer cannot adequately simulate the physical complexity of the brain, and that such a computer cannot solve the so-called binding problem.)

Thus, it seems to stand as an open question whether mind uploading is indeed possible, let alone feasible (and it also seems that many people in the broader transhumanist community, who tend to be the people who write and talk the most about mind uploading, could well be biased toward believing it possible, as many of them seem to hope that it can save them from death).

The perhaps most intriguing aspect of WBE technology is that once the first emulation exists and can complete tasks on a computer like a human researcher can, it would then be very easy to make more such emulations by copying the original. Moreover, with powerful enough hardware, it would also become possible to run emulations at higher speeds, or to reset them back to a well-rested state after they performed exhausting work (Hanson, 2016).

Assuming, of course, that WBE will indeed be feasible in the first place. Also, it is worth noting that Robin Hanson himself is critical of the idea that WBEs would be able to create software that is superior to themselves very quickly; i.e. he expects a WBE economy to undergo “many doublings” before it happens.

Sped-up WBE workers could be given the task of improving computer hardware (or AI technology itself), which would trigger a wave of steeply exponential progress in the development of superintelligence.

This is an exceptionally strong claim that would seem in need of justification, and not least some specification, given that it is not clear what “steeply exponential progress in the development of superintelligence” refers to in this context. It hardly means “steeply exponential progress in the development of a super ability to achieve goals in general”, including in energy efficiency and energy harvesting. Such exponential progress is not, I submit, likely to follow from progress in computer hardware or AI technology alone. Indeed, as we saw above, such progress cannot happen with respect to the energy efficiency of most of our machines, as physical limits mean that it cannot double more than a couple of times.

But even if we understand it to be a claim about the abilities of certain particular machines and their cognitive abilities more narrowly, the claim is still a dubious one. It seems to assume that progress in computer hardware and AI technology is constrained chiefly by the amount of hours put into it by those who work on it directly, as opposed to also being significantly constrained by countless other factors, such as developments in other areas, e.g. in physics, production, and transportation, many of which imply limits on development imposed by factors such as hardware and money, not just the amount of human-like genius available.

For example, how much faster should we expect the hardware that AlphaZero was running on to have been developed and completed if a team of super-WBEs had been working on it? Would the materials used for the hardware have been dug up and transported significantly faster? Would they have been assembled significantly faster? Perhaps somewhat, yet hardly anywhere close to twice as fast. The growth story underlying many worries about explosive AI growth is quite detached from how we actually improve our machines, including AI (software and hardware) as well as the harvesting of the energy that powers it (Vaclav Smil: “Energy transitions are inherently gradual processes and this reality should be kept in mind when judging the recent spate of claims about the coming rapid innovative take-overs […]”). Such growth is the result of countless processes distributed across our entire economy. Just as nobody knows how to make a pencil, nobody, including the very best programmers, knows (more than a tiny part of) how to make better machines.

To get a sense of the potential of this technology, imagine WBEs of the smartest and most productive AI scientists, copied a hundred times to tackle AI research itself as a well-coordinated research team, sped up so they can do years of research in mere weeks or even days, and reset periodically to skip sleep (or other distracting activities) in cases where memory-formation is not needed. The scenario just described requires no further technologies beyond WBE and sufficiently powerful hardware. If the gap from current AI algorithms to smarter-than-human AI is too hard to bridge directly, it may eventually be bridged (potentially very quickly) after WBE technology drastically accelerates further AI research.

As far as I understand, much of the progress in machine learning in modern times was essentially due to modern hardware and computing power that made it possible to implement old ideas invented decades ago (of course then implemented with all the many adjustments and tinkering whose necessity and exact nature one cannot foresee from the drawing board). In other words, software progress was hardly the most limiting factor. Arguably, the limiting factor was rather that the economy just had not caught up to be able to make hardware advanced enough to implement these theoretical ideas successfully. And it also seems to me quite naive to think that better hardware design, and genius ideas about how to make hardware more generally, was and is a main limiting factor in our growth of computer hardware. Such progress tends to rest critically on other progress in other kinds of hardware and globally distributed production processes. Processes that no doubt can be sped up, yet hardly that significantly by advanced software alone, in large part because such progress is limited by the fact that many of the crucial processes involved in this progress, such as digging up, refining, and transporting materials, are physical processes that can only go so fast.

Beyond that, there is also an opportunity cost consideration that is ignored by the story of fast growth above. For the hardware and energy required for this team of WBEs could otherwise have been used to run other kinds of computations that could help further innovation, including those we already run on full steam to further progress — CAD programs, simulations, equation solving. And it is not clear that using all this hardware for WBEs would be a better use of hardware than would running these other programs, whose work may be considered a limiting factor to AI progress at a similar level as more “purely” human or human-like work is. Indeed, we should not expect engineers and companies to do these kinds of things with their computing resources if they were not among the most efficient things they could do with them. And even if WBEs are a better use of hardware for fast progress, it is far from clear that it would be that much better.

The potential for WBE to come before de novo AI means that – even if the gap between current AI designs and the human brain is larger than we thought – we should not significantly discount the probability of human-level AI being created eventually. And perhaps paradoxically, we should expect such a late transition to happen abruptly. Barring no upcoming societal collapse, believing that superintelligence is highly unlikely to ever happen requires not only confidence that software or “architectural” improvements to AI are insufficient to ever bridge the gap, but also that – in spite of continued hardware progress – WBE could not get off the ground either. We do not seem to have sufficient reason for great confidence in either of these propositions, let alone both.

Again, what does the term “superintelligence” refer to here? Above, it was defined as “(AGI-)systems that are vastly smarter than human experts in virtually all respects”. And given that AGI is defined as a general ability to pursue goals, and that “smart” here presumably means “better able to achieve goals”, one can say that the definition of superintelligence given here translates to “a system that pursues goals better than human experts in virtually all areas”. Yet we are already building systems that satisfy this definition of superintelligence. Our entire economy is already able to do tasks that no single human expert could ever accomplish. But superintelligence likely refers to something else here, something along the lines of: “a system that is vastly more cognitively capable than any human expert in virtually all respects”. And yet, even by this definition, we already have computer systems that can do countless cognitive tasks much better than any human, and the super system that is the union of all these systems can therefore, in many respects at least, be considered to have vastly superior cognitive abilities relative to humans. And systems composed of humans and technology are clearly vastly more capable than any human expert alone in virtually all respects.

In this sense, we clearly do have “superintelligence” already, and we are continually expanding its capabilities. And, with respect to worries above a FOOM takeover, it seems highly unlikely that a single, powerful machine could ever overtake and become more powerful than the entire collective that is the human-machine civilization, which is not to say that low-risk events should be dismissed. But they should be measured against other risks we could be focusing on.

III. Humans are not at peak intelligence

Again, it is important to be clear about what we mean by “intelligence”. Most cognitively advanced? Or best able to achieve goals in general? Humans extended by technology can clearly increase their intelligence, i.e. ability to achieve goals, significantly. We have done so consistently over the last few centuries, and we continue to do so today. And in a world where humans build this growing body of technology to serve their own ends, and in some cases build it to be provably secure, it is far from clear that some non-human system with much greater cognitive powers than humans (which, again, already exists in many domains) will also become more capable of achieving goals in general than humanity, given that it is surrounded by a capable super-system of technology designed for and by humans, controlled by humans, to serve their ends. Again, this is not to say that one should not worry about seemingly improbable risks — we definitely should — but merely that we should doubt the assumption that our making machines more cognitively capable will necessarily imply that they will be better able to achieve goals in general. Again, despite being related, these two senses of “intelligence” must not be confused.

It is difficult to intuitively comprehend the idea that machines – or any physical system for that matter – could become substantially more intelligent than the most intelligent humans. Because the intelligence gap between humans and other animals appears very large to us, we may be tempted to think of intelligence as an “on-or-off concept,” one that humans have and other animals do not. People may believe that computers can be better than humans at certain tasks, but only at tasks that do not require “real” intelligence. This view would suggest that if machines ever became “intelligent” across the board, their capabilities would have to be no greater than those of an intelligent human relying on the aid of (computer-)tools.

Again, we should be clear that the word “intelligence” here seems to mean “most cognitively capable” rather than “best able to achieve goals in general”. And the gap between the “intelligence”, as in the ability to achieve goals, of humans and other animals does arguably not appear very large when we compare individuals. Most other animals can do things that no single human can do, and to the extent we humans can learn to do things other animals naturally beat us at, e.g. lift heavier objects or traverse distances faster than speedy animals, we do so by virtue of technology, in essence the product of collective, cultural evolution.

And even with respect to cognitive abilities, one can argue that humans are not superior to other animals in a general sense. We do not have superior cognitive abilities with respect to echo location, for example, much less long-distance navigation. Nor are humans superior when it comes to all aspects of short-term/working memory

Measuring goal achieving ability in general, as well as abilities to solve cognitive tasks in particular, along a single axis may be useful in some contexts, yet it can easily become meaningless when the systems being compared are not sufficiently similar. 

But this view is mistaken. There is no threshold for “absolute intelligence.” Nonhuman animals such as primates or rodents differ in cognitive abilities a great deal, not just because of domain-specific adaptations, but also due to a correlational “g factor” responsible for a large part of the variation across several cognitive domains (Burkart et al., 2016). In this context, the distinction between domain-specific and general intelligence is fuzzy: In many ways, human cognition is still fairly domain-specific. Our cognitive modules were optimized specifically for reproductive success in the simpler, more predictable environment of our ancestors. We may be great at interpreting which politician has the more confident or authoritative body language, but deficient in evaluating whose policy positions will lead to better developments according to metrics we care about. Our intelligence is good enough or “general enough” that we manage to accomplish impressive feats even in an environment quite unlike the one our ancestors evolved in, but there are many areas where our cognition is slower or more prone to bias than it could be.

I agree with this. I would just note that “intelligence” here again seems to be referring to cognitive abilities, not the ability to achieve goals in general, and that we humans have expanded both over time via culture: our cognitive abilities, as measured by IQ, have increased significantly over the last century, while our ability to achieve goals in general has expanded much more still as we have developed ever more advanced technology.

Intelligence is best thought of in terms of a gradient. Imagine a hypothetical “intelligence scale” (inspired by part 2.1 of this FAQ) with rats at 100, chimpanzees at, say, 350, the village idiot at 400, average humans at 500 and Einstein at 750.2 Of course, this scale is open at the top and could go much higher.

Again, intelligence here seems to refer to cognitive abilities, not the ability to achieve goals in general. Einstein was likely not better at shooting hoops than the average human, or indeed more athletic in general (by all appearances), although he was much more cognitively capable, at least in some respects, than virtually all other humans.

To quote Bostrom (2014, p. 44): “Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization – a niche we filled because we got there first, not because we are in any sense optimally adapted to it.”

Again, the words “smart” and “stupid” here seem to pertain to cognitive abilities, not the ability to achieve goals in general. And this phrasing is misleading, as it seems to presume that cognitive ability is all it takes to build an advanced civilization, which is not the case. In fact, humans are not the species with the biggest brain on the planet, or even the species with the biggest cerebral cortex; indeed, long-finned pilot whales have more than twice as many neocortical neurons.

What we are, however, is a species with a lot of unique tools — fine motor hands, upright walk, vocal cords, a large brain with a large prefrontal cortex, etc. — which together enabled humans to (gradually build a lot of tools with which they could) take over the world. Remove just one of these unique tools from all of humanity, and we would be almost completely incapable. And this story of a multiplicity of components that are all necessary yet insufficient for the maintenance and growth of human civilization is even more true today, where we have countless external tools — trucks, the internet, computers, screwdrivers, etc. — without which we could not maintain our civilization. And the necessity of all these many different components seems overlooked by the story that views advanced cognitive abilities as the sole driver, or near enough, of growth and progress in the ability to achieve goals in general. This, I would argue, is a mistake.

Thinking about intelligence as a gradient rather than an “on-or-off” concept prompts a Copernican shift of perspective. Suddenly it becomes obvious that humans cannot be at the peak of possible intelligence. On the contrary, we should expect AI to be able to surpass us in intelligence just like we surpass chimpanzees.

Depending on what we mean by the word “intelligence”, one can argue that computers have already surpassed humans. If we define “intelligence” to be “that which is measured by an IQ test”, for example, then computers have already been better than humans in at least some of these tests for a few years now.

In terms of our general ability to achieve goals, however, it is not clear that computers will so readily surpass humans, in large part because we do not aim to build them to be better than humans in many respects. Take self-repair, for example, which is something human bodies, just like virtually all animal bodies, are in a sense designed to do — indeed, most of our self-repair mechanisms are much older than we are as a species. Evolution has built humans to be competent and robust autonomous systems who do not for the most part depend on a global infrastructure to repair their internal parts. Our computers, in contrast, are generally not built to be self-repairing, at least not at the level of hardware. Their notional thrombocytes are entirely external to themselves, in the form of a thousand and one specialized tools and humans distributed across the entire economy. And there is little reason to think that this will change, as there is little incentive to create self-repairing computers. We are not aiming to build generally able, human-independent computers in this sense.

Biological evolution supports the view that AI could reach levels of intelligence vastly beyond ours. Evolutionary history arguably exhibits a weak trend of lineages becoming more intelligent over time, but evolution did not optimize for intelligence (only for goal-directed behavior in specific niches or environment types). Intelligence is metabolically costly, and without strong selection pressures for cognitive abilities specifically, natural selection will favor other traits. The development of new traits always entails tradeoffs or physical limitations: If our ancestors had evolved to have larger heads at birth, maternal childbirth mortality would likely have become too high to outweigh the gains of increased intelligence (Wittman & Wall, 2007). Because evolutionary change happens step-by-step as random mutations change the pre-existing architecture, the changes are path dependent and can only result in local optima, not global ones.

Here we see how the distinction between “intelligence as cognitive abilities” and “intelligence as the ability to achieve goals” is crucial. Indeed, the example provided above clearly proves the point that advanced cognitive abilities are often not the most relevant thing for achieving goals, since the goal of surviving and reproducing was often not best achieved, as Lukas hints, with the best cognitive abilities. Often it was better achieved with longer teeth or stronger muscles. Or a prettier face.

So the question is: why do we think that advanced cognitive abilities are, to a first approximation, identical with the ability to achieve goals? And, more importantly, why do we imagine that this lesson about the sub-optimality of spending one’s limited resources on better cognitive abilities does not still hold today? Why should cognitive abilities be the sole optimal thing, or near enough, to spend all one’s resources on in order to best achieve a broad range of goals? I would argue that it is not. It was not optimal in the past (with respect to the goal of survival), and it does not seem to be optimal today either.

It would be a remarkable coincidence if evolution had just so happened to stumble upon the most efficient way to assemble matter into an intelligent system.

But it would be less remarkable if it had happened to assemble matter into a system that is broadly capable of achieving a broad range of goals, and which another system, especially one that is not built over a billion year process to be robust and highly autonomous, cannot readily outdo in terms of autonomous function. It would also not be that remarkable if biological humans, functioning within a system built by and for biological humans, happened to be among the most capable systems within such a system, not least given all the legal, social and political aspects this system entails.

Beyond that, one can dispute the meaning of “intelligent system” in the quote above, but if we look at the intelligent system that is our civilization at large, one can say that the optimization going on at this level is not coincidental but indeed deliberate, often aiming toward peak efficiency. Thus, in this regard as well, we should not be too surprised if our current system is quite efficient and competent relative to the many constraints we are facing.

But let us imagine that we could go back to the “drawing board” and optimize for a system’s intelligence without any developmental limitations. This process would provide the following benefits for AI over the human brain (Bostrom, 2014, p. 60-61):

Free choice of substrate: Signal transmission with computer hardware is millions of times faster than in biological brains. AI is not restricted to organic brains, and can be built on the substrate that is overall best suited for the design of intelligent systems.

Supersizing:” Machines have (almost) no size-restrictions. While humans with elephant-sized brains would run into developmental impossibilities, (super)computers already reach the size of warehouses and could in theory be built even bigger.

No cognitive biases: We should be able to construct AI in a way that uses more flexible heuristics, and always the best heuristics for a given context, to prevent the encoding or emergence of substantial biases. Imagine the benefits if humans did not suffer from confirmation biasoverconfidencestatus quo biasetc.!

Modular superpowers: Humans are particularly good at tasks for which we have specialized modules. For instance, we excel at recognizing human faces because our brains have hard-wired structures that facilitate that facial recognition in particular. An artificial intelligence could have many more such specialized modules, including extremely useful ones like a module for programming.

Editability and copying: Software on a computer can be copied and edited, which facilitates trying out different variations to see what works best (and then copying it hundreds of times). By contrast, the brain is a lot messier, which makes it harder to study or improve. We also lack correct introspective access to the way we make most of our decisions, which is an important advantage that (some) AI designs could have.

Superior architecture: Starting anew, we should expect it to be possible to come up with radically more powerful designs than the patchwork architecture that natural selection used to construct the human brain. This difference could be enormously significant.

It should be noted that computers already 1) can be built with a wide variety of substrates, 2) can be supersized, 3) do not tend to display cognitive biases, 4) have modular superpowers, 5) can be edited and copied (or at least software readily can), 6) can be made with any architecture we can come up with. All of these advantages exist and are being exploited already, just not as much as they can be. And it is not clear why we should expect future change to be more radical than the change we have seen in past decades in which we have continually built ever more competent computers which can do things that no human can by exploiting these advantages.

With regard to the last point, imagine we tried to optimize for something like speed or sight rather than intelligence. Even if humans had never built anything faster than the fastest animal, we should assume that technological progress – unless it is halted – would eventually surpass nature in these respects. After all, natural selection does not optimize directly for speed or sight (but rather for gene copying success), making it a slower optimization process than those driven by humans for this specific purpose. Modern rockets already fly at speeds of up to 36,373 mph, which beats the peregrine falcon’s 240 mph by a huge margin. Similarly, eagle vision may be powerful, but it cannot compete with the Hubble space telescope. (General) intelligence is harder to replicate technologically, but natural selection did not optimize for intelligence either, and there do not seem to be strong reasons to believe that intelligence as a trait should differ categorically from examples like speed or sight, i.e., there are as far as we know no hard physical limits that would put human intelligence at the peak of what is possible.3

Again, what is being referred to by the word “intelligence” here seems to be cognitive abilities, not the ability to achieve goals in general. And with respect to cognitive abilities in particular, it is clear that computers already beat humans by a long shot in countless respects. So the point Lukas is making here is clearly true.

Another way to develop an intuition for the idea that there is significant room for improvement above human intelligence is to study variation in humans. An often-discussed example in this context is the intellect of John von Neumann. Von Neumann was not some kind of an alien, nor did he have a brain twice as large as the human average. And yet, von Neumann’s accomplishments almost seem “superhuman.” The section in his Wikipedia entry that talks about him having “founded the field of Game theory as a mathematical discipline” – an accomplishment so substantial that for most other intellectual figures it would make up most of their Wikipedia page – is just one out of many of von Neumann’s major achievements.

There are already individual humans (with normal-sized brains) whose intelligence vastly exceeds that of the typical human. So just how much room there is above their intelligence? To visualize this, consider for instance what could be done with an AI architecture more powerful than the human brain running on a warehouse-sized supercomputer.

A counterpoint to this line of reasoning can be found by contemplating chess ratings. Ratings of the skills of chess players are usually done via the so-called Elo rating system, which measures the relative skills of different players against each other. A beginner will usually have a rating around 800, whereas a rating in the range 2000-2199 ranks one as a chess “Expert”, and a ranking of 2400 and above renders one a “Senior Master”. The highest rating ever achieved was 2882 by Magnus Carlsen. Surely, this amount of variation must be puny given that all the humans who have ever played chess have roughly the same brain sizes and structures. And yet it turns out that human variation in chess ability is in fact quite enormous in an absolute sense.

For example, it took more than four decades from computers were able to beat a chess beginner (the 1950s), until they were able to beat the very best human player (1997 officially). Thus, the span from ordinary human beginner to the best human expert was more than four decades of progress in hardware — i.e. a million times more computing power — and software. That seems quite a wide range.

And yet the range seems even broader if we consider the ultimate limits of optimal chess play. For one may argue that the fact that it took computers a fairly long time to go from the average human level to the level of the best human does not mean that the best human is not still ridiculously far from the best a computer could be in theory. Surprisingly, however, this latter distance does in fact seem quite small, at least in one sense. For estimates suggest that the best possible chess machine would have an Elo rating around 3600, which means that the relative distance between the best possible computer and the best human is only around 700 Elo points, implying that the distance between the best human and a chess “Expert” is similar to the distance between the best human and the best possible chess brain, while the distance between an ordinary human beginner and the best human is far greater.

It seems plausible that a similar pattern obtains with respect to many other complex cognitive tasks. Indeed, it seems plausible that many of our abilities, especially those we evolved to do well, such as our ability to interact with other humans, have an “Elo rating” quite close to the notional maximum level for most humans.

IV. The transition from human to superhuman intelligence could be rapid

Perhaps the people who think it is unlikely that superintelligent AI will ever be created are not objecting to it being possible in principle. Maybe they think it is simply too difficult to bridge the gap from human-level intelligence to something much greater. After all, evolution took a long time to produce a species as intelligent as humans, and for all we know, there could be planets with biological life where intelligent civilizations never evolved.4 But considering that there could come a point where AI algorithms start taking part in their own self-improvement, we should be more optimistic.

We should again be clear that the term “superintelligent AI” seems to refer to a system with greater cognitive abilities, across a wide range of tasks, than humans. As for “a point where AI algorithms start taking part in their own self-improvement”, it should be noted, again, that we already use our best software and hardware in the process of developing better software and hardware. True, they are only a part of a process that involves far more elements, yet this is true of most everything that we produce and improve in our economy: many contributions drawn from and distributed across our economy at large are required. And we have good reason to believe that this will continue to be true of the construction of more capable machines in the future.

AIs contributing to AI research will make it easier to bridge the gap, and could perhaps even lead to an acceleration of AI progress to the point that AI not only ends up smarter than us, but vastly smarter after only a short amount of time.

Again, we already use our best software and hardware to contribute to AI research, and yet we do not appear to see acceleration in the growth of our best supercomputers. In fact, in terms of their computing power, we see a modest decline.

Several points in the list of AI advantages above – in particular the advantages derived from the editability of computer software or the possibility for modular superpowers to have crucial skills such as programming – suggest that AI architectures might both be easier to further improve than human brains, and that AIs themselves might at some point become better at actively developing their own improvements.

Again, computers are already “easier to further improve than human brains” in these ways, and our hardware and software are already among the most active parts in their own improvement. So why should we expect to see a different pattern in the future from the pattern we see today of gradual, slightly declining growth?

If we ever build a machine with human-level intelligence, it should then be comparatively easy to speed it up or make tweaks to its algorithm and internal organization to make it more powerful. The updated version, which would at this point be slightly above human-level intelligence, could be given the task of further self-improvement, and so on until the process runs into physical limits or other bottlenecks.

Or better yet than “human-level intelligence” would be if we built software that was critical for the further development of more powerful computers. And we in fact already have such software, many different kinds of it, and yet it is not that easy to simply “speed it up or make tweaks to its algorithm and internal organization to make it more powerful”. More generally, as noted above, we already use our latest, updated technology to improve our latest, updated technology, and the result is not rapid, runaway growth.

Perhaps self-improvement does not have to require human-level general intelligence at all. There may be comparatively simple AI designs that are specialized for AI science and (initially) lack proficiency in other domains. The theoretical foundations for an AI design that can bootstrap itself to higher and higher intelligence already exist (Schmidhuber, 2006), and it remains an empirical question where exactly the threshold is after which AI designs would become capable of improving themselves further, and whether the slope of such an improvement process is steep enough to go on for multiple iterations.

Again, I would just reiterate that computers are already an essential component in the process of improving computers. And the fact that humans who need to sleep and have lunch breaks are also part of this improvement process does not seem a main constraint on it compared to other factors, such as physical limitations implied by transportation and the assemblage of materials. Oftentimes in modern research, computers run simulations at their maximum capacity while the humans do their sleeping and lunching, in which case these resting activities (through which humans often get their best ideas) do not limit progress much at all, whereas the available computing power does.

For the above reasons, it cannot be ruled out that breakthroughs in AI could at some point lead to an intelligence explosion (Good, 1965; Chalmers, 2010), where recursive self-improvement leads to a rapid acceleration of AI progress. In such a scenario, AI could go from subhuman intelligence to vastly superhuman intelligence in a very short timespan, e.g. in (significantly) less than a year.

“It cannot be ruled out” can be said of virtually everything; the relevant question is how likely we should expect these possibilities to be. Beyond that, it is also not clear what would count as a “rapid acceleration of AI progress”, and thus what exactly it is that cannot be ruled out. AI going from subhuman performance to vastly greater than human performance in a short amount of time has already been seen in many different domains, including Go most recently.

But if one were to claim, to take a specific claim, that it cannot be ruled out that an AI system will improve itself so much that it can overpower human civilization and control the future, then I would argue that the reasoning above does not support considering this a likely possibility, i.e. something that is more likely to happen than, say, one in a thousand.

While the idea of AI advancing from human-level to vastly superhuman intelligence in less than a year may sound implausible, as it violates long-standing trends in the speed of human-driven development, it would not be the first time where changes to the underlying dynamics of an optimization process cause an unprecedented speed-up. Technology has been accelerating ever since innovations (such as agriculture or the printing press) began to feed into the rate at which further innovations could be generated.5

In the endnote “5” referred to above, Lukas writes:

[…] Finally, over the past decades, many tasks, including many areas of research and development, have already been improved through outsourcing them to machines – a process that it is still ongoing and accelerating.

That this process of outsourcing of tasks is accelerating seems in need of justification. We have been outsourcing tasks to machines in various ways and at a rapid pace for at least two centuries now, and so it is not a trivial claim that this process is accelerating.

Compared to the rate of change we see in biological evolution, cultural evolution broke the sound barrier: It took biological evolution a few million years to improve on the intelligence of our ape-like ancestors to the point where they became early hominids. By contrast, technology needed little more than ten thousand years to progress from agriculture to space shuttles.

And I would argue that the reason technology could grow so fast is because an ever larger system of technology consisting of an ever greater variety of tools was contributing to it through recursive self-improvement — human genius was but one important component. And I think we have good reason to think the same about the future.

Just as inventions like the printing press fed into – and significantly sped up – the process of technological evolution, rendering it qualitatively different from biological evolution, AIs improving their own algorithms could cause a tremendous speed-up in AI progress, rendering AI development through self-improvement qualitatively different from “normal” technological progress.

I think there is very little reason to believe this story. Again, we already use our best machines to build the next generation of machines. “Normal” technological progress of the kind we see today already depends on computers running programs created to optimize future technology as efficiently as they can, and it is far from clear that running a more human kind of program would be a more efficient use of resources toward this end.

It should be noted, however, that while the arguments in favor of a possible intelligence explosion are intriguing, they nevertheless remain speculative. There are also some good reasons why some experts consider a slower takeoff of AI capabilities more likely. In a slower takeoff, it would take several years or even decades for AI to progress from human to superhuman intelligence.

Again, the word “intelligence” here seems to refer to cognitive abilities, not the ability to achieve goals in general. And it is again not clear what it means to say that it might “take several years or even decades for AI to progress from human to superhuman intelligence”, since computers have already been more capable than humans at a wide variety of cognitive tasks for many decades. So I would argue that this statement suffers from a lack of conceptual clarity.

Unless we find decisive arguments for one scenario over the other, we should expect both rapid and comparably slow takeoff scenarios to remain plausible. It is worth noting that because “slow” in this context also includes transitions on the order of ten or twenty years, it would still be very fast practically speaking, when we consider how much time nations, global leaders or the general public would need to adequately prepare for these changes.

To reiterate the statement I just made, it is not clear what a fast takeoff means in this context given that computers are already vastly superior to humans in many domains, and probably will continue to beat humans at ever more tasks before they come close to being able to do virtually all cognitive tasks humans can do. So what it is we are supposed to consider plausible is not entirely clear. As for whether it is plausible for rapid progress to occur over a wide range of cognitive tasks such that an AI system becomes able to take over the world, I would argue that we have not seen arguments to support this claim.

V. By default, superintelligent AI would be indifferent to our well-being

The typical mind fallacy refers to the belief that other minds operate the same way our own does. If an extrovert asks an introvert, “How can you possibly not enjoy this party; I talked to half a dozen people the past thirty minutes and they were all really interesting!” they are committing the typical mind fallacy.

When envisioning the goals of smarter-than-human artificial intelligence, we are in danger of committing this fallacy and projecting our own experience onto the way an AI would reason about its goals. We may be tempted to think that an AI, especially a superintelligent one, will reason its way through moral arguments6 and come to the conclusion that it should, for instance, refrain from harming sentient beings. This idea is misguided, because according to the intelligence definition we provided above – which helps us identify the processes likely to shape the future – making a system more intelligent does not change its goals/objectives; it only adds more optimization power for pursuing those objectives.

Again, we need to be clear about what “smarter-than-human artificial intelligence” means here. In this case, we seem to be talking about a fairly singular and coherent system, a “mind” of sorts — as opposed to a thousand and one different software programs that do their own thing well — and hence in this regard it seems that the term “smarter-than-human artificial intelligence” here refers to something that is quite similar to a human mind. We are seemingly also talking about a system that “would reason about its goals”.

It seems worth noting that this is quite different from how we think about contemporary software programs, even including the most advanced ones such as AlphaZero and IBM’s Watson, which we are generally not tempted to consider “minds”. Expecting competent software programs of the future to be like minds may itself be to commit a typical mind fallacy of sorts, or perhaps just a mind fallacy. It is conceivable that software will continue to outdo humans at many tasks without acquiring anything resembling what we usually conceive of as a mind.

Another thing worth clarifying is what we mean by the term “by default” here. Does it refer to what AI systems will be built to do by our economy in the absence of altruistic intervention? If “by default” means that which our economy will naturally tend to produce, it seems likely that future AI indeed will be programmed to not be indifferent, at least in a behavioral sense, to human well-being “by default”. Indeed, it seems a much greater risk that future software systems will be constructed to act in a way that exclusively benefits, and is indifferent toward anything else than, human beings. In other words, that it will share our speciesist bias, with catastrophic consequences ensuing.

My point here is merely that, just as it is almost meaningless to claim that biological minds will not care about our well-being by default, as it lacks any specification of what “by default” means — given what evolutionary history? — so is it highly unclear what “by default” means when we are talking about machines created by humans. It seems to assume that we are going to suddenly have a lot of “undirected competence” delivered to us which does not itself come with countless sub-goals and adaptations built into it to attain ends desired by human programmers, and, perhaps to a greater extent, markets.

To give a silly example, imagine that an arms race between spam producers and companies selling spam filters leads to increasingly more sophisticated strategies on both sides, until the side selling spam filters has had it and engineers a superintelligent AI with the sole objective to minimize the number of spam emails in their inboxes.

Again, I would flag that it is not clear what “superintelligent AI” means here. Does it refer to a system that is better able to achieve goals across the board than humans? Or merely a system with greater cognitive abilities than any human expert in virtually all domains? Even if it is merely the latter, it is unlikely that a system developed by a single team of software developers will have much greater cognitive competences across the board than the systems developed by other competing teams, let alone those developed by the rest of the economy combined.

With its level of sophistication, the spam-blocking AI would have more strategies at its disposal than normal spam filters.

Yet how many more? What could account for this large jump in capabilities from previous versions of spam filters? What is hinted here seems akin to the sudden emergence of a Bugatti in the Stone Age. It does not seem credible.

For instance, it could try to appeal to human reason by voicing sophisticated, game-theoretic arguments against the negative-sum nature of sending out spam. But it would be smart enough to realize the futility of such a plan, as this naive strategy would backfire because some humans are trolls (among other reasons). So the spam-minimizing AI would quickly conclude that the safest way to reduce spam is not by being kind, but by gaining control over the whole planet and killing everything that could possibly try to trick its spam filter.

First of all, it is by no means clear that this would be “the safest way” to minimize spam. Indeed, I would argue that trying to gain control in this way would be a very bad action in expectation with respect to the goal of minimizing spam.

But even more fundamentally, the scenario above seems to assume that it would be much easier to build a system with the abilities to take over the world than it would to properly instantiate the goals we want it to achieve. For instance, in the case of earlier versions of AlphaZero, these were all equally aligned with the goal of winning Go. The hard problem was to make it more capable at doing it. The assumption that the situation would be inverted with respect to future goal implementation seems to me unwarranted. Not because the goals are necessarily easy to instantiate, but because the competences in question appear extremely difficult to create. The scenario described above seems to ignore this consideration, and instead assumes that the default scenario is that we will suddenly get advanced machines with a lot of competence, but where we do not know how to direct this competence toward doing what we want it to, as opposed to gradually directing and integrating these competences as they are (gradually) acquired. Beyond that, on a more general note, I think many aspiring effective altruists who worry about AI safety tend to underestimate the extent to which computer programmers are already focused on making software do what they intend it to.

Moreover, the scenario considered here also seems to assume that it would be relatively easy to make a competent machine optimize a particular goal insistently, and I would also question that this is anything less than extremely difficult. In other words, not only do I think it is extremely difficult to create the competences in question, as noted above, but I also think it is extremely difficult to orient all these competences, not just a few subroutines, toward insistently accomplishing some perverse goal. For this reason too, I think one should be highly skeptical of scenarios of this kind.

The AI in this example may fully understand that humans would object to these actions on moral grounds, but human “moral grounds” are based on what humans care about – which is not the minimization of spam! And the AI – whose whole decision architecture only selects for actions that promote the terminal goal of minimizing spam – would therefore not be motivated to think through, let alone follow our arguments, even if it could “understand” them in the same way introverts understand why some people enjoy large parties.

I think this is inaccurate. Any goal-oriented agent would be motivated to think through these things for the same reason that we humans are motivated to think through what those who disagree with us morally would say and do: because it impacts how we ourselves can act effectively toward our goals (this, we should be honest, is also often why humans think about the views and arguments made by others; not because of a deep yearning for truth and moral goodness but for purely pragmatic and selfish reasons). Thus, it makes sense to be mindful of those things, especially given that one has imperfect information and an imperfect ability to predict the future, no matter how “smart” one is.

The typical mind fallacy tempts us to conclude that because moral arguments appeal to us,7 they would appeal to any generally intelligent system. This claim is after all already falsified empirically by the existence of high-functioning psychopaths. While it may be difficult for most people to imagine how it would feel to not be moved by the plight of anyone but oneself, this is nothing compared to the difficulties of imagining all the different ways that minds in general could be built. Eliezer Yudkowsky coined the term mind space to refer to the set of all possible minds – including animals (of existing species as well as extinct ones), aliens, and artificial intelligences, as well as completely hypothetical “mind-like” designs that no one would ever deliberately put together. The variance in all human individuals, throughout all of history, only represents a tiny blob in mind space.

Yes, but this does not mean that the competences of human minds only span a tiny range of the notional “competence range” of various abilities. As we saw in the example of chess above, humans span a surprisingly large range, and the best humans are surprisingly close to the best mind possible. And with respect to the competences required for navigating within a world built by and for humans, it is not that unreasonable to believe that, on a continuum that measures competence across these many domains with a single measure, we are probably quite high and quite difficult to beat. This is not arrogance. It is merely to acknowledge the contingent structure of our civilization, and the fact that it is adapted to many contingent features of the human organism in general, including the human mind in particular.

Some of the minds outside this blob would “think” in ways that are completely alien to us; most would lack empathy and other (human) emotions for that matter; and many of these minds may not even relevantly qualify as “conscious.”

Most of these minds would not be moved by moral arguments, because the decision to focus on moral arguments has to come from somewhere, and many of these minds would simply lack the parts that make moral appeals work in humans. Unless AIs are deliberately designed8 to share our values, their objectives will in all likelihood be orthogonal to ours (Armstrong, 2013).

Again, an agent trying to achieve goals in our world need not be moved by moral arguments in an emotional sense in order to pay attention to them and the preferences of humans more generally, and to choose to avoid causing chaos. Second, the question is why we should expect future software designed by humans to not be “deliberately designed to share our values”? And what marginal difference should we expect altruists to be able to make on them? And how would this influence best be achieved?

VI. AIs will instrumentally value self-preservation and goal preservation

Even though AI designs may differ radically in terms of their top-level goals, we should expect most AI designs to converge on some of the same subgoals. These convergent subgoals (Omohundro, 2008; Bostrom, 2012) include intelligence amplification, self-preservation, goal preservation and the accumulation of resources. All of these are instrumentally very useful to the pursuit of almost any goal. If an AI is able to access the resources it needs to pursue these subgoals, and does not explicitly have concern for human preferences as (part of) its top-level goal, its pursuit of these subgoals is likely to lead to human extinction (and eventually space colonization; see below).

Again, what does “AI design” refer to in this context? Presumably a machine that possesses most of the cognitive abilities a human does to a similar or greater extent, and, on top of that, this machine is in some sense highly integrated into something akin to a coherent unified mind subordinate to a few supreme “top-level goals”. Thus, when Lukas writes “most AI designs” above, he is in fact referring to most systems that meet a very particular definition of “AI”, and one which I strongly doubt will be anywhere close to the most prevalent source of “machine competence” in the future (note that this is not to say that software, as well as our machines in general, will not become ever more competent in the future, but merely that such greater competences may not be subordinate to one goal to rule them all, or a few for that matter).

Beyond that, the claim that such a capable machine of the future seeking to achieve these subgoals is likely to lead to human extinction is a very strong claim that is not supported here, nor in the papers cited. More on this below.

AI safety work refers to interdisciplinary efforts to ensure that the creation of smarter-than-human artificial intelligence will result in excellent outcomes rather than disastrous ones. Note that the worry is not that AI would turn evil, but that indifference to suffering and human preferences will be the default unless we put in a lot of work to ensure that AI is developed with the right values.

Again, I would take issue with this “default” claim, as I would argue that “a lot of work” is exactly what we should expect that there will be made to ensure that future software will do what humans want it to. And the question is, again, how much of a difference altruists should expect to make here, as well as how to best make it.

VI.I Intelligence amplification

Increasing an agent’s intelligence improves its ability to efficiently pursue its goals. All else equal, any agent has a strong incentive to amplify its intelligence. A real-life example of this convergent drive is the value of education: Learning important skills and (thinking-)habits early in life correlates with good outcomes. In the AI context, intelligence amplification as a convergent drive implies that AIs with the ability to improve their own intelligence will do so (all else equal). To self-improve, AIs would try to gain access to more hardware, make copies of themselves to increase their overall productivity, or devise improvements to their own cognitive algorithms.

Again, what does the word “intelligence” mean in this context? Above, it was defined as “the ability to achieve goals in a wide range of environments”, which means that what is being said here reduces to the tautological claim that increasing an agent’s ability to achieve goals improves its ability to achieve goals. If one defines “intelligence” to refer to cognitive abilities, however, the claim becomes less empty. Yet it also becomes much less obvious, especially if one thinks in terms of investments of marginal resources, as it is questionable whether investing in greater cognitive abilities (as opposed to a prettier face or stronger muscles) is the best investment one can make with respect to the goal of achieving goals “in general”.

On a more general note, I would argue that “intelligence amplification”, as in “increasing our ability to achieve goals”, is already what we collectively do in our economy to a great extent, although this increase is, of course, much broader than one merely oriented toward optimizing cognitive abilities. We seek to optimize materials, supply chains, transportation networks, energy efficiency, etc. And it is not clear why this growth process should speed up significantly due to greater machine capabilities in the future than it has in the past, where more capable machines also helped grow the economy in general, as well as to increase the capability of machines in particular.

More broadly, intelligence amplification also implies that an AI would try to develop all technologies that may be of use to its pursuits.

Yet should we expect such “an AI” to be better able to develop “all technologies that may be of use to its pursuits” better than entire industries currently dedicated to it, let alone our entire economy? Indeed, should we even expect it to contribute significantly, i.e. double current growth rates across the board? I would argue that this is most dubious.

I.J. Good, a mathematician and cryptologist who worked alongside Alan Turing, asserted that “the first ultraintelligent machine is the last invention that man need ever make,” because once we build it, such a machine would be capable of developing all further technologies on its own.

To say that a single machine would be able to develop all further technologies on its own is, I submit, unsound. For what does “on its own” mean here? “On its own” independently of the existing infrastructure of machines run by humans? Or “on its own” as in taking over this entire infrastructure? And how exactly could such a take-over scenario occur without destroying the productivity of this system? None of these scenarios seem plausible.

VI.II Goal preservation

AIs would in all likelihood also have an interest in preserving their own goals. This is because they optimize actions in terms of their current goals, not in terms of goals they might end up having in the future.

This again seems to assume that we will create highly competent systems which will be subordinate to a single or a few explicit goals that it will insistently optimize all its actions for. Why should we believe this?

Another critical note of mine on this idea quoted from elsewhere:

Stephen Omohundro (Omohundro, 2008) argues that a chess-playing robot with the supreme goal of playing good chess would attempt to acquire resources to increase its own power and work to preserve its own goal of playing good chess. Yet in order to achieve such complex subgoals, and to even realize they might be helpful with respect to achieving the ultimate goal, this robot will need access to, and be built to exercise advanced control over, an enormous host of intellectual tools and faculties. Building such tools is extremely hard and requires many resources, and harder still, if at all possible, is it to build them so that they are subordinate to a single supreme goal. And even if all this is possible, it is far from clear that access to these many tools would not enable – perhaps even force – this now larger system to eventually “reconsider” the goals that it evolved from. For instance, if the larger system has a sufficient amount of subsystems with sub-goals that involve preservation of the larger system of tools, and if the “play excellent chess” goal threatens, or at least is not optimal with respect to, this goal, could one not imagine that, in some evolutionary competition, these sub-goals could overthrow the supreme goal?

Footnote: After all, humans are such a system of competing drives, and it has been argued (e.g. in Ainslie, 2001 [Breakdown of Will]) that this competition is what gives us our unique cognitive strengths (as well as weaknesses). Our ultimate goals, to the extent we have any, are just those that win this competition most of the time.

And Paul Christiano has also described agents that would not be subject to this “basic drive” of self-preservation described by Omohundro.

Lukas continues:

From the current goal’s perspective, a change in the AI’s goal function is potentially disastrous, as the current goal would not persevere. Therefore, AIs will try to prevent researchers from changing their goals.

Granted that such a highly competent system is built so as to be subordinate to a single goal in this way, which I do not think there is good reason to consider likely to be the case in future AI systems “by default”.

Consequently, there is pressure for AI researchers to get things right on the first try: If we develop a superintelligent AI with a goal that is not quite what we were after – because someone made a mistake, or was not precise enough, or did not think about particular ways the specified goal could backfire – the AI would pursue the goal that it was equipped with, not the goal that was intended. This applies even if it could understand perfectly well what the intentioned goal was. This feature of going with the actual goal instead of the intended one could lead to cases of perverse instantiation, such as the AI “paralyz[ing] human facial musculatures into constant beaming smiles” to pursue an objective of “make us smile” (Bostrom, 2014, p. 120).

This again seems to assume that this “first superintelligent AI” would be so much more powerful than everything else in the world, yet why should we expect a single system to be so much more powerful than everything else across the board? Beyond that, it also seems to assume that the design of this system would happen in something akin to a single step — that there would be a “first try”. Yet what could a first try consist in? How could a super capable system emerge in the absence of a lot of test models that are slightly less competent? I think this “first try” idea betrays an underlying belief in a sudden growth explosion powered by a single, highly competent machine, which, again, I would argue is highly unlikely in light of what we know about the nature of the growth of the capabilities of machines.

VI.III Self-preservation

Some people have downplayed worries about AI risks with the argument that when things begin to look dangerous, humans can literally “pull the plug” in order to shut down AIs that are behaving suspiciously. This argument is naive because it is based on the assumption that AIs would be too stupid to take precautions against this.

There is a difference between being “stupid” and being ill-informed. And there is no reason to think that an extremely cognitively capable agent will be informed about everything relevant to its own self-preservation. To think otherwise is to conflate great cognitive abilities with near-omniscience.

Because the scenario we are discussing concerns smarter-than-human intelligence, an AI would understand the implications of losing its connection to electricity, and would therefore try to proactively prevent being shut down any means necessary – especially when shutdown might be permanent.

Even if all implications were understood by such a notional agent, this by no means implies that an attempt to stop its termination would be successful, nor particularly likely, or indeed even possible.

This is not to say that AIs would necessarily be directly concerned about their own “death” – after all, whether an AI’s goal includes its own survival or not depends on the specifics of its goal function. However, for most goals, staying around pursuing one’s goal will lead to better expected goal achievement. AIs would therefore have strong incentives to prevent permanent shutdown even if their goal was not about their own “survival” at all. (AIs might, however, be content to outsource their goal achievement by making copies of themselves, in which case shutdown of the original AI would not be so terrible as long as one or several copies with the same goal remain active.)

I would question the tacit notion that the self-preservation of such a machine could be done with a significantly greater level of skill than could the “counter self-preservation” work of the existing human-machine civilization. After all, why should a single system be so much more capable than the rest of the world at any given task? Why should humans not develop specialized software systems and other machines that enable them to counteract and overpower rogue machines, for example by virtue of having more information and training? What seems described here as an almost sure to happen default outcome strikes me as highly unlikely. This is not to say that one should not worry about small risks of terrible outcomes, yet we need to get a clear view of the probabilities if we are to make a qualified assessment of the expected value of working on these risks.

The convergent drive for self-preservation has the unfortunate implication that superintelligent AI would almost inevitably see humans as a potential threat to its goal achievement. Even if its creators do not plan to shut the AI down for the time being, the superintelligence could reasonably conclude that the creators might decide to do so at some point. Similarly, a newly-created AI would have to expect some probability of interference from external actors such as the government, foreign governments or activist groups. It would even be concerned that humans in the long term are too stupid to keep their own civilization intact, which would also affect the infrastructure required to run the AI. For these reasons, any AI intelligent enough to grasp the strategic implications of its predicament would likely be on the lookout for ways to gain dominance over humanity. It would do this not out of malevolence, but simply as the best strategy for self-preservation.

Again, to think that a single agent could gain dominance over the rest of the human-machine civilization in which it would find itself appears extremely unlikely. What growth story could plausibly lead to this outcome?

This does not mean that AIs would at all times try to overpower their creators: If an AI realizes that attempts at trickery are likely to be discovered and punished with shutdown, it may fake being cooperative, and may fake having the goals that the researchers intended, while privately plotting some form of takeover. Bostrom has referred to this scenario as a “treacherous turn” (Bostrom, 2014, p. 116).

We may be tempted to think that AIs implemented on some kind of normal computer substrate, without arms or legs for mobility in the non-virtual world, may be comparatively harmless and easy to overpower in case of misbehavior. This would likely be a misconception, however. We should not underestimate what a superintelligence with access to the internet could accomplish. And it could attain such access in many ways and for many reasons, e.g. because the researchers were careless or underestimated its capacities, or because it successfully pretended to be less capable than it actually was. Or maybe it could try to convince the “weak links” in its [team] of supervisors to give it access in secret – promising bribes. Such a strategy could work even if most people in the developing team thought it would be best to deny their AI internet access until they have more certainty about the AI’s alignment status and its true capabilities. Importantly, if the first superintelligence ever built was prevented from accessing the internet (or other efficient channels of communication), its impact on the world would remain limited, making it possible for other (potentially less careful) teams to catch up. The closer the competition, the more the teams are incentivized to give their AIs riskier access over resources in a gamble for the potential benefits in case of proper alignment.

Again, this all seems to assume a very rapid take-off in capabilities with one system being vastly more capable than all others. What reasons do we have to consider such a scenario plausible? Barely any, I have argued.

The following list contains some examples of strategies a superintelligent AI could use to gain power over more and more resources, with the goal of eventually reaching a position where humans cannot harm or obstruct it. Note that these strategies were thought of by humans, and are therefore bound to be less creative and less effective than the strategies an actual superintelligence would be able to devise.

  • Backup plans: Superintelligent AI could program malware of unprecedented sophistication that inserted partial copies of itself into computers distributed around the globe (adapted from part 3.1.2 of this FAQ). This would give it further options to act even if its current copy was destroyed or if its internet connection was cut. Alternatively, it could send out copies of its source code, alongside detailed engineering instructions, to foreign governments, ideally ones who have little to lose and a lot to gain, with the promise of helping them attain world domination if they build a second version of the AI and handed it access to all their strategic resources.
  • Making money: Superintelligent AI could easily make fortunes with online poker, stock markets, scamming people, hacking bank accounts, etc.9
  • Influencing opinions: Superintelligent AI could fake convincing email exchanges with influential politicians or societal elites, pushing an agenda that serves its objectives of gaining power and influence. Similarly, it could orchestrate large numbers of elaborate sockpuppet accounts on social media or other fora to influence public opinion in favorable directions.
  • Hacking and extortion: Superintelligent AI could hack into sensitive documents, nuclear launch codes or other compromising assets in order to blackmail world leaders into giving it access over more resources. Or it could take over resources directly if hacking allows for it.
  • (Bio-)engineering projects: Superintelligent AI could pose as the head researcher of a biology lab and send lab assistants instructions to produce viral particles with specific RNA sequences, which then, unbeknownst to the people working on the project, turned out to release a deadly virus that incapacitated most of humanity.10

Through some means or another – and let’s not forget that the AI could well attempt many strategies at once to safeguard against possible failure in some of its pursuits – the AI may eventually gain a decisive strategic advantage over all competition (Bostrom, 2014, p. 78-90). Once this is the case, it would carefully build up further infrastructure on its own. This stage will presumably be easier to reach as the world economy becomes more and more automated.

These various strategies could also be pursued by other agents, and indeed by vast systems of agents and programs. Why should one such agent be much more competent than others at doing any of these things?

Once humans are no longer a threat, the AI would focus its attention on natural threats to its existence. It would for instance notice that the sun will expand in about seven billion years to the point where existence on earth will become impossible. For the reason of self preservation alone, a superintelligent AI would thus eventually be incentivized to expand its influence beyond Earth.

Following the arguments I have made above (as well as here), I would argue that such a take-over of the world subordinate to a single or a few goals originally instilled in a single machine is extremely unlikely.

VI.IV Resource accumulation

For the fulfillment of most goals, accumulating as many resources as possible is an important early step. Resource accumulation is also intertwined with the other subgoals in that it tends to facilitate them.

The resources available on Earth are only a tiny fraction of the total resources that an AI could access in the entire universe. Resource accumulation as a convergent subgoal implies that most AIs would eventually colonize space (provided that it is not prohibitively costly), in order to gain access to the maximum amount of resources. These resources would then be put to use for the pursuit of its other subgoals and, ultimately, for optimizing its top-level goal.

Superintelligent AI might colonize space in order to build (more of) the following:

  • Supercomputers: As part of its intelligence enhancement, an AI could build planet-sized supercomputers (Sandberg, 1999) to figure out the mysteries of the cosmos. Almost no matter the precise goal, having an accurate and complete understanding of the universe is crucial for optimal goal achievement.
  • Infrastructure: In order to accomplish anything, an AI needs infrastructure (factories, control centers, etc.) and “helper robots” of some sort. This would be similar (but much larger in scale) to how the Manhattan Project had its own “project sites” and employed tens of thousands of people. While some people worry that an AI would enslave humans, these helpers would more plausibly be other AIs specifically designed for the tasks at hand.
  • Defenses: An AI could build shields to protect itself or other sensitive structures from cosmic rays. Perhaps it would build weapon systems to deal with potential threats.
  • Goal optimization: Eventually, an AI would convert most of its resources into machinery that directly achieves its objectives. If the goal is to produce paperclips, the AI will eventually tile the accessible universe with paperclips. If the goal is to compute pi to as many decimal places as possible, the AI will eventually tile the accessible universe with computers to compute pi. Even if an AI’s goal appears to be limited to something “local” or “confined,” such as e.g. “protect the White House,” the AI would want to make success as likely as possible and thus continue to accumulate resources to better achieve that goal.

To elaborate on the point of goal optimization: Humans tend to be satisficers with respect to most things in life. We have minimum requirements for the quality of the food we want to eat, the relationships we want to have, or the job we want to work in. Once these demands are met and we find options that are “pretty good,” we often end up satisfied and settle down on the routine. Few of us spend decades of our lives pushing ourselves to invest as many waking hours as sustainably possible into systematically finding the optimal food in existence, the optimal romantic partner, or anything really.

AI systems on the other hand, in virtue of how they are usually built, are more likely to act as maximizers. A chess computer is not trying to look for “pretty good moves” – it is trying to look for the best move it can find with the limited time and computing power it has at its disposal. The pressure to build ever more powerful AIs is a pressure to build ever more powerful maximizers. Unless we deliberately program AIs in a way that reduces their impact, the AIs we build will be maximizers that never “settle” or consider their goals “achieved.” If their goal appears to be achieved, a maximizer AI will spend its remaining time double- and triple-checking whether it made a mistake. When it is only 99.99% certain that the goal is achieved, it will restlessly try to increase the probability further – even if this means using the computing power of a whole galaxy to drive the probability it assigns to its goal being achieved from 99.99% to 99.991%.

Because of the nature of maximizing as a decision-strategy, a superintelligent AI is likely to colonize space in pursuit of its goals unless we program it in a way to deliberately reduce its impact. This is the case even if its goals appear as “unambitious” as e.g. “minimize spam in inboxes.”

Why should we expect a single machine to be better able to accumulate resources than other actors in the economy, much less whole teams of actors powered by specialized software programs optimized toward that very purpose? Again, what seems to be considered the default outcome here is one that I would argue is extremely unlikely. This is still not to say that we then have reason to dismiss such a scenario. Yet it is important that we make an honest assessment of its probability if we are to make qualified assessments of the value of prioritizing it.

VII. Artificial sentience and risks of astronomical suffering

Space colonization by artificial superintelligence would increase goal-directed activity and computations in the world by an astronomically large factor.11

So would space colonization driven by humans. And it is not clear why we should expect a human-driven colonization to increase goal-directed computations any less. Beyond that, such human-driven colonization also seems much more likely to happen than does rogue AI colonization. 

If the superintelligence holds objectives that are aligned with our values, then the outcome could be a utopia. However, if the AI has randomly, mistakenly, or sufficiently suboptimally implemented values, the best we could hope for is if all the machinery it used to colonize space was inanimate, i.e. not sentient. Such an outcome – even though all humans would die – would still be much better than other plausible outcomes, because it would at least not contain any suffering. Unfortunately, we cannot rule out that the space colonization machinery orchestrated by a superintelligent AI would also contain sentient minds, including minds that suffer. The same way factory farming led to a massive increase in farmed animal populations, multiplying the direct suffering humans cause to animals by a large factor, an AI colonizing space could cause a massive increase in the total number of sentient entities, potentially creating vast amounts of suffering.

The same applies to a human-driven colonization, which I would still argue seems a much more likely outcome. So why should we focus more on colonization driven by rogue AI?

The following are some ways AI outcomes could result in astronomical amounts of suffering:

Suffering in AI workers: Sentience appears to be linked to intelligence and learning (Daswani & Leike, 2015), both of which would be needed (e.g. in robot workers) for the coordination and execution of space colonization. An AI could therefore create and use sentient entities to help it pursue its goals. And if the AI’s creators did not take adequate safety measures or program in compassionate values, it may not care about those entities’ suffering in their assistance.

Optimization for sentience: Some people want to colonize space in order for there to be more life or (happy) sentient minds. If the AI in question has values that reflect this goal, either because human researchers managed to get value loading right (or “half-right”), or because the AI itself is sentient and values creating copies of itself, the result could be astronomical numbers of sentient minds. If the AI does not accurately assess how happy or unhappy these beings are, or if it only cares about their existence but not their experiences, or simply if something goes wrong in even a small portion of these minds, the total suffering that results could be very high.

Ancestor simulations: Turning history and (evolutionary) biology into an empirical science, AIs could run many “experiments” with simulations of evolution on planets with different starting conditions. This would e.g. give the AIs a better sense of the likelihood of intelligent aliens existing, as well as a better grasp on the likely distribution of their values and whether they would end up building AIs of their own. Unfortunately, such ancestor simulations could recreate millions of years of human or wild-animal suffering many times in parallel.

Warfare: Perhaps space-faring civilizations would eventually clash, with at least one of the two civilizations containing many sentient minds. Such a conflict would have vast frontiers of contact and could result in a lot of suffering.

All of these scenarios could also occur in a human-driven colonization, which I would argue is significantly more likely to happen. So again: why should we focus more on colonization driven by rogue AI?

More ways AI scenarios could contain astronomical amounts of suffering are described here and here. Sources of future suffering are likely to follow a power law distribution, where most of the expected suffering comes from a few rare scenarios where things go very wrong – analogous to how most casualties are the result of very few, very large wars; how most of the casualty-risks from terrorist attacks fall into tail scenarios where terrorists would get their hands on weapons of mass destruction; or how most victims of epidemics succumbed to the few very worst outbreaks (Newman, 2005). It is therefore crucial to not only to factor in which scenarios are most likely to occur, but also how bad scenarios would be should they occur.

Again, most of the very worst scenarios could well be due to human-driven colonization, such as US versus China growth races taken beyond Earth. So, again, why focus mostly on colonization scenarios driven by rogue AI? Beyond that, the expected value of influencing a broad class of medium-value outcomes could easily be much higher than the expected value of influencing much fewer, much higher-stakes outcomes, provided that the outcomes that fall into this medium value class are sufficiently probable and amenable to impact. In other words, it is by no means far-fetched to imagine that we can take actions that are robust over a wide range of medium-value outcomes, and that such actions are in fact best in expectation.

Critics may object because the above scenarios are largely based on the possibility of artificial sentience, particularly sentience implemented on a computer substrate. If this turns out to be impossible, there may not be much suffering in futures with AI after all. However, computer-based minds also being able to suffer in the morally relevant sense is a common implication in philosophy of mind. Functionalism and type A physicalism (“eliminativism”) both imply that there can be morally relevant minds on digital substrates. Even if one were skeptical of these two positions and instead favored the views of philosophers like David Chalmers or Galen Strawson (e.g. Strawson, 2006), who believe consciousness is an irreducible phenomenon, there are at least some circumstances under which these views would also allow for computer-based minds to be sentient.12 Crude “carbon chauvinism,” or a belief that consciousness is only linked to carbon atoms, is an extreme minority position in philosophy of mind.

The case for artificial sentience is not just abstract but can also be made on the intuitive level: Imagine we had whole brain emulation with a perfect mapping from inputs to outputs, behaving exactly like a person’s actual brain. Suppose we also give this brain emulation a robot body, with a face and facial expressions created with particular attention to detail. The robot will, by the stipulations of this thought experiment, behave exactly like a human person would behave in the same situation. So the robot-person would very convincingly plead that it has consciousness and moral relevance. How certain would we be that this was all just an elaborate facade? Why should it be?

Because we are unfamiliar with artificial minds and have a hard time experiencing empathy for things that do not appear or behave in animal-like ways, we may be tempted to dismiss the possibility of artificial sentience or deny artificial minds moral relevance – the same way animal sentience was dismissed for thousands of years. However, the theoretical reasons to anticipate artificial sentience are strong, and it would be discriminatory to deny moral consideration to a mind simply because it is implemented on a substrate different from ours. As long as we are not very confident indeed that minds on a computer substrate would be incapable of suffering in the morally relevant sense, we should believe that most of the future’s expected suffering is located in futures where superintelligent AI colonizes space.

I fail to see how this final conclusion is supported by the argument made above. Again, human-driven colonization seems to pose at least as big a risk of outcomes of this sort.

One could argue that “superintelligent AI” could travel much faster and convert matter and energy into ordered computations much faster than a human-driven colonization could, yet I see little reason to expect a rogue AI-driven colonization to be significantly more effective in this regard than a human civilization powered by advanced tools built to be as efficient as possible. For instance, why should “superintelligent AI” be able to build significantly faster spaceships? I would expect both tail-end scenarios — i.e. both maximally sentient rogue AI-driven colonization and maximally sentient human-driven colonization —  to converge toward an optimal expansion solution in a relatively short time, at least on cosmic timescales.

VIII. Impact analysis

The world currently contains a great deal of suffering. Large sources of suffering include for instance poverty in developing countries, mental health issues all over the world, and non-human animal suffering in factory farms and in the wild. We already have a good overview – with better understanding in some areas than others – of where altruists can cost-effectively reduce substantial suffering. Charitable interventions are commonly chosen according to whether they produce measurable impact in the years or decades to come. Unfortunately, altruistic interventions are rarely chosen with the whole future in mind, i.e. with a focus on reducing as much suffering as possible for the rest of time, until the heat death of the universe.13 This is potentially problematic, because we should expect the far future to contain vastly more suffering than the next decades, not only because there might be sentient beings around for millions or billions of years to come, but also because it is possible for Earth-originating life to eventually colonize space, which could multiply the total amount of sentient beings many times over. While it is important to reduce the suffering of sentient beings now, it seems unlikely that the most consequential intervention for the future of all sentience will also be the intervention that is best for reducing short-term suffering.

I think this is true, but also because the word “best” here refers to two very narrow peaks that have to coincide in a very large landscape. In contrast, I do not think it seems unlikely that the best, most robust interventions we can make to influence the long-term future are also highly robust and positive with respect to the short-term future, such as promoting concern for suffering as well as greater moral consideration of neglected beings.

And given that the probability of extinction (evaluated from now) increases over time, and hence that one should discount the value of influencing the long-term future of civilization by a certain factor, it in fact seems reasonable to choose actions that seem positive both in the short and long term.

Instead, as judged from the distant future, the most consequential development of our decade would more likely have something to do with novel technologies or the ways they will be used.

And when it comes to how technologies will be used, it is clear that influencing ideas matters a great deal. By analogy, we have also seen important technologies developed in the past, and yet ideas seem to have been no less significant, such as specific religions (e.g. Islam and Christianity) as well as political ideologies (e.g. communism and liberalism). One may, of course, argue that it is very difficult to influence ideas on a large scale, yet the same can be said about influencing technology. Indeed, influencing ideas, whether broadly or narrowly, might just be the best way to influence technology.

And yet, politics, science, economics and especially the media are biased towards short timescales. Politicians worry about elections, scientists worry about grant money, and private corporations need to work on things that produce a profit in the foreseeable future. We should therefore expect interventions targeted at the far future to be much more neglected than interventions targeted at short-term sources of suffering.

Admittedly, the far future is difficult to predict. If our models fail to account for all the right factors, our predictions may turn out very wrong. However, rather than trying to simulate in detail through everything that might happen all the way into the distant future – which would be a futile endeavor, needless to say – we should focus our altruistic efforts on influencing levers that remain agile and reactive to future developments. An example of such a lever is institutions that persist for decades or centuries. The US Constitution for instance still carries significant relevance in today’s world, even though it was formulated hundreds of years ago. Similarly, the people who founded the League of Nations after World War I did not succeed in preventing the next war, but they contributed to the founding and the charter of its successor organization, the United Nations, which still exerts geopolitical influence today. The actors who initially influenced the formation of these institutions as well as their values and principles, had a long-lasting impact.

In order to positively influence the future for hundreds of years, we fortunately do not need to predict the next hundreds of years in detail. Instead, all we need to predict is what type of institutions – or, more generally, stable and powerful decision-making agencies – are most likely to react to future developments maximally well.14

AI is the ultimate lever through which to influence the future. The goals of an artificial superintelligence would plausibly be much more stable than the values of human leaders or those enshrined in any constitution or charter. And a superintelligent AI would, with at least considerable likelihood, remain in control of the future not only for centuries, but for millions or even billions of years to come. In non-AI scenarios on the other hand, all the good things we achieve in the coming decade(s) will “dilute” over time, as current societies, with all their norms and institutions, change or collapse.

In a future where smarter-than-human artificial intelligence won’t be created, our altruistic impact – even if we manage to achieve a lot in greatly influencing this non-AI future – would be comparatively “capped” and insignificant when contrasted with the scenarios where our actions do affect the development of superintelligent AI (or how AI would act).15

I think this is another claim that is widely overstated, and which I have not seen a convincing case for. Again, this notion that “an artificial superintelligence”, a single machine with much greater cognitive powers than everything else, will emerge and be programmed to be subordinate to a single goal that it would be likely to preserve does not seem credible to me. Sure, we can easily imagine it as an abstract notion, but why should we think such a system will ever emerge? The creation of such a system is, I would argue, far from being a necessary, or even particularly likely, outcome of our creating ever more competent machines.

And even if such a system did exist, it is not even clear, as Robin Hanson has argued, that it would be significantly more likely to preserve its values than would a human civilization — not so much because one should expect humans to be highly successful at it, but rather because there are also reasons to think that it would be unlikely for such a “superintelligent AI” to do it (such as those mentioned in my note on Omohundro’s argument above, as well as those provided by Hanson, e.g. that “the values of AIs with protected values should still drift due to influence drift and competition”).

We should expect AI scenarios to not only contain the most stable lever we can imagine – the AI’s goal function which the AI will want to preserve carefully – but also the highest stakes.

Again, I do not think a convincing case has been made for either of these claims. Why would the stakes be higher than in a human-driven colonization, which we may expect, for evolutionary reasons, to be performed primarily by those who want to expand and colonize as much and as effectively as possible?

In comparison with non-AI scenarios, space colonization by superintelligent AI would turn the largest amount of matter and energy into complex computations.

It depends on what we mean by non-AI scenarios. Scenarios where humans use advanced tools, such as near-maximally fast spaceships and near-optimal specialized software, to fill up space with sentient beings at a near maximal rate is, I would argue, not only at least as conceivable but also at least as likely as similar scenarios brought about by the kind of AI Lukas seems to have in mind here.

In a best-case scenario, all these resources could be turned into a vast utopia full of happiness, which provides as strong incentive for us to get AI creation perfectly right. However, if the AI is equipped with insufficiently good values, or if it optimizes for random goals not intended by its creators, the outcome could also include astronomical amounts of suffering. In combination, these two reasons of highest influence/goal-stability and highest stakes build a strong case in favor of focusing our attention on AI scenarios.

Again, things could also go very wrong or very well with human-driven colonization, so there does not seem a big difference in this regard either.

While critics may object that all this emphasis on the astronomical stakes in AI scenarios appears unfairly Pascalian, it should be noted that AI is not a frivolous thought experiment where we invoke new kinds of physics to raise the stakes.

Right, but the kind of AI system envisioned here does, I would argue, rest on various, highly questionable conceptions of how a single system could grow, as well as what the design of future machines are likely to be like. And I would argue, again, that such a system is highly unlikely to emerge.

Smarter-than-human artificial intelligence and space colonization are both realistically possible and plausible developments that fit squarely into the laws of nature as we currently understand them.

A Bugatti appearing in the Stone Age also in some sense fits squarely into the laws of nature as we currently understand them. Yet that does not mean that such a car was likely to emerge in that time, once we consider the history and evolution of technology. Similarly, I would argue that the scenario Lukas seems to have hinted at throughout his piece is a lot less credible than what this appeal to compatibility with the laws of nature would seem to suggest.

If either of them turn out to be impossible, that would be a big surprise, and would suggest that we are fundamentally misunderstanding something about the way physical reality works. While the implications of smarter-than-human artificial intelligence are hard to grasp intuitively, the underlying reasons for singling out AI as a scenario to worry about are sound.

Well, I have tried to argue to the contrary here. Much more plausible would it be, I think, to argue that the scenario Lukas envisions is one scenario among others that warrants some priority.

As illustrated by Leó Szilárd’s lobbying for precautions around nuclear bombs well before the first such bombs were built, it is far from hopeless to prepare for disruptive new technologies in advance, before they are completed.

This text argued that altruists concerned about the quality of the future should [be] focusing their attention on futures where AI plays an important role.

I would say that the argument that has been made is much more narrow than that, since “AI” here is used in a relatively narrow sense in the first place, and because it is a very particular scenario involving such narrowly defined AI that Lukas has been focusing on the most here — as far as I can tell, it is a scenario where a single system takes over the world and determines the future based on a single, arduously preserved goal. There are many other scenarios we can envision in which AI, both in the ordinary sense as well as in the more narrow sense invoked here by Lukas, plays “an important role”, including scenarios involving human-driven space colonization.

This can mean many things. It does not mean that everyone should think about AI scenarios or technical work in AI alignment directly. Rather, it just means we should pick interventions to support according to their long-term consequences, and particularly according to the ways in which our efforts could make a difference to futures ruled by superintelligent AI. Whether it is best to try to affect AI outcomes in a narrow and targeted way, or whether we should go for a broader strategy, depends on several factors and requires further study.

FRI has looked systematically into paths to impact for affecting AI outcomes with particular emphasis on preventing suffering, and we have come up with a few promising candidates. The following list presents some tentative proposals:

It is important to note that human values may not affect the goals of an AI at all if researchers fail to solve the value-loading problem. Raising awareness of certain values may therefore be particularly impactful if it concerns groups likely to be in control of the goals of smarter-than-human artificial intelligence.

Further research is needed to flesh out these paths to impact in more detail, and to discover even more promising ways to affect AI outcomes.

Lukas writes about the implications of his argument that it means that “we should pick interventions to support according to their long-term consequences”. I agree with this completely. He then continues to write, “and particularly according to the ways in which our efforts could make a difference to futures ruled by superintelligent AI”. And this claim, as I understand it, is what I would argue has not been justified. Again, to argue that one should grant it some priority, even significant priority, along with many other scenarios, is a plausible claim, but not, I would argue, that it should be granted greater priority than all other things.

And as for how we can best reduce suffering in the future, I would agree with pretty much all the proposals Lukas suggests, although I would argue that things like promoting concern for suffering and widening our moral circles (and we should do both) become even more important when we take other scenarios into consideration, such as human-driven colonization. In other words, these things seem even more robust and more positive when we also consider these other high-stakes scenarios.

Beyond that, I would also note that we likely have moral intuitions that make a notional rogue AI-takeover seem worse in expectation than what a more detached analysis relative to a more impartial moral ideal such as “reduce suffering” would suggest. Furthermore, it should be noted that many of those who focus most prominently on AI safety (for example, people at MIRI and FHI) seem to have values according to which it is important that humans maintain control or remain in existence, which may render their view that AI safety is the most important thing to focus on less relevant for other value systems than one might intuitively suppose.

To zoom out a bit, one way to think about my disagreement with Lukas, as well as the overall argument I have tried to make here, is that one can view Lukas’ line of argument as consisting of a certain number of steps where, in each of them, he describes a default scenario he believes to be highly probable, whereas I generally find these respective “default” scenarios quite improbable. And when one then combines our respective probabilities into a single measure of the probability that the grosser scenario Lukas envisions will occur, one gets a very different overall probability for Lukas and myself respectively. It may look something like this, assuming Lukas’ argument consists of eight steps, each assigned a certain probability which then gets multiplied by the rest (i.e. P(A) * P(B|A) * P(C|B) * . . . ):

L: 0.98 * 0.96 * 0.93 * 0.99 * 0.95 * 0.99 * 0.97 * 0.98 ≈ 0.77

M: 0.1 * 0.3 * 0.01 * 0.1 * 0.2 * 0.08 * 0.2 * 0.4 ≈ 0.00000004

(These particular numbers are just more or less random ones I have picked for illustrative purposes, except that their approximate range do illustrate where I think the respective credences of Lukas and myself roughly lie with regard to most of the arguments discussed throughout this essay.)

And an important point to note here is that even if one disagrees both with Lukas and me on these respective probabilities, and instead picks credences roughly in-between those of Lukas and me, or indeed significantly closer to those of Lukas, the overall argument I have made here still stands, namely that it is far from clear that scenarios of the kind Lukas outlines are the most important ones to focus on to best reduce suffering. For then the probability of Lukas’ argument being correct/the probability that the scenario Lukas envisions will occur (one can think of it in both ways, I think, even if these formulations are not strictly equivalent) becomes something like the following:

In-between credence: 0.5^8 ≈ 0.004

Credences significantly closer to Lukas’: 0.75^8 ≈ 0.1

Which would not seem to support the conclusion that a focus on the AI-scenarios Lukas has outlined should dominate other scenarios we can envision (e.g. human-driven colonization).

Lukas ends his post on the following note:

As there is always the possibility that we have overlooked something or are misguided or misinformed, we should remain open-minded and periodically rethink the assumptions our current prioritization is based on.

With that, I could not agree more. In fact, this is in some sense the core point I have been trying to make here.

Moral Circle Expansion Might Increase Future Suffering

Expanding humanity’s moral circle so that it includes all sentient beings seems among the most urgent and important missions before us. And yet there is a significant risk that such greater moral inclusion might in fact end up increasing future suffering. As Brian Tomasik notes:

One might ask, “Why not just promote broader circles of compassion, without a focus on suffering?” The answer is that more compassion by itself could increase suffering. For example, most people who care about wild animals in a general sense conclude that wildlife habitats should be preserved, in part because these people aren’t focused enough on the suffering that wild animals endure. Likewise, generically caring about future digital sentience might encourage people to create as many happy digital minds as possible, even if this means also increasing the risk of digital suffering due to colonizing space. Placing special emphasis on reducing suffering is crucial for taking the right stance on many of these issues.

Indeed, many classical utilitarians do include non-human animals in their moral circle, yet they still consider it permissible, indeed in some sense morally required of us, that we bring individuals into existence so that they can live “net positive lives” and we can eat them (I have argued that this view is mistaken, almost regardless of what kind of utilitarian view one assumes). And some even seem to think that most lives on factory farms might plausibly be such “net positive lives”. A wide circle of moral consideration clearly does not guarantee an unwillingness to allow large amounts of suffering to be brought into the world.

More generally, there is a considerable number of widely subscribed ethical positions that favor bringing about larger rather than smaller populations of the beings who belong to our moral circle, at least provided that certain conditions are met in the lives of these beings. And many of these ethical positions have quite loose such conditions, which implies that they can easily permit, and even demand, the creation of a lot of suffering for the sake of some (supposedly) greater good.

Indeed, the truth is that even if we require an enormous amount of happiness (or an enormous amount of other intrinsically good things) to outweigh a given amount of suffering, this can still easily permit the creation of large amounts of suffering, as illustrated by the following consideration (quoted from the penultimate chapter of my book on effective altruism):

[…] consider the practical implications of the following two moral principles: 1) we will not allow the creation of a single instance of the worst forms of suffering […] for any amount of happiness, and 2) we will allow one day of such suffering for ten years of the most sublime happiness. What kind of future would we accept with these respective principles? Imagine a future in which we colonize space and maximize the number of sentient beings that the accessible universe can sustain over the entire course of the future, which is probably more than 10^30. Given this number of beings, and assuming these beings each live a hundred years, principle 2) above would appear to permit a space colonization that all in all creates more than 10^28 years of [extreme suffering], provided that the other states of experience are sublimely happy. This is how extreme the difference can be between principles like 1) and 2); between whether we consider suffering irredeemable or not. And notice that even if we altered the exchange rate by orders of magnitude — say, by requiring 10^15 times more sublime happiness per unit of extreme suffering than we did in principle 2) above — we would still allow an enormous amount of extreme suffering to be created; in the concrete case of requiring 10^15 times more happiness, we would allow more than 10,000 billion years of [the worst forms of suffering].

This highlights the importance of thinking deeply about which trade-offs, if any, we find acceptable with respect to the creation of suffering, including extreme suffering.

The considerations above concerning popular ethical positions that support larger future populations imply that there is a risk — a seemingly low yet still significant risk — that a more narrow moral circle may in fact lead to less future suffering for the morally excluded beings (e.g. by making efforts to bring these beings into existence, on Earth and beyond, less likely).

Implications

In spite of this risk, I still consider generic moral circle expansion quite positive in expectation. Yet it seems less positive, and arguably significantly less robust (with respect to the goal of reducing extreme suffering) than does the promotion of suffering-focused valuesAnd it seems less robust and less positive still than the twin-track strategy of focusing on both expanding our moral circle and deepening our concern for suffering. Both seem necessary yet insufficient on their own. If we deepen concern for suffering without broadening the moral circle, our deepened concern risks failing to pertain to the vast majority of sentient beings. On the other hand, if we broaden our moral circle without deepening our concern for suffering, we may end up allowing the beings within our moral circle to endure enormous amounts of suffering, including extreme suffering.

Those who seek to minimize extreme suffering should seek to avoid both these pitfalls by pursuing the twin-track approach.

The Principle of Sympathy for Intense Suffering

This essay was first published as a chapter in my book Effective Altruism: How Can We Best Help Others? which is available for free download here. The chapter that precedes it makes a general case for suffering-focused ethics, whereas this chapter argues for a particular suffering-focused view.


The ethical view I would advocate most strongly is a suffering-focused view that centers on a core principle of Sympathy for Intense Suffering, or SIS for short, which roughly holds that we should prioritize the interests of those who are, or will be, in a state of extreme suffering. In particular: that we should prioritize their interest in avoiding such suffering higher than anything else.[1]

One can say that this view takes its point of departure in classical utilitarianism, the theory that we should maximize the net sum of happiness minus suffering. Yet it questions a tacit assumption, a particular existence claim, often held in conjunction with the classical utilitarian framework, namely that for every instance of suffering, there exists some amount of happiness that can outweigh it.

This is a deeply problematic assumption, in my view. More than that, it is peculiar that classical utilitarianism seems widely believed to entail this assumption, given that (to my knowledge) none of the seminal classical utilitarians — Jeremy Bentham, John Stuart Mill, and Henry Sidgwick — ever argued for this existence claim, or even discussed it.[2] Thus, it seems that the acceptance of this assumption is no more entailed by classical utilitarianism, defined as the ethical view, or views, expressed by these utilitarian philosophers, than is its rejection.

The question of whether this assumption is reasonable ties into a deeper discussion about how to measure and weigh happiness and suffering against each other, and I think this is much less well-defined than is commonly supposed (even though the trickiness of the task is often acknowledged).[3] The problem is that we have a common sense view that goes something like the following: if a conscious subject deems some state of suffering worth experiencing in order to attain some given pleasure, then this pleasure is worth the suffering. And this common sense view may work for most of us most of the time.[4] Yet it runs into problems in cases where the subject deems their suffering so unbearable that no amount of happiness could ever outweigh it.

For what would the common sense view say in such a situation? That the suffering indeed cannot be outweighed by any pleasure? That would seem an intuitive suggestion, yet the problem is that we can also imagine the case of an experience of some pleasure that the subject, in that experience-moment, deems so great that it can outweigh even the worst forms of suffering, which leaves us with mutually incompatible value claims (although it is worth noting that one can reasonably doubt the existence of such positive states, whereas, as we shall see below, the existence of correspondingly negative experiences is a certainty).[5] How are we to evaluate these claims?

The aforementioned common sense method of evaluation has clearly broken down at this point, and is entirely silent on the matter. We are forced to appeal to another principle of evaluation. And the principle I would argue we should employ is, as hinted above, to choose to sympathize with those who are worst off — those who are experiencing intense suffering. Hence the principle of sympathy for intense suffering: we should sympathize with, and prioritize, the evaluations of those subjects who deem their suffering unoutweighable, even if only for a brief experience-moment, and thus give total priority to helping these subjects. More precisely, we should minimize the amount of such experience-moments of extreme suffering.[6] That, on this account of value, is the greatest help we can do for others.

This principle actually seems to have a lot of support from common sense and “common wisdom”. For example, imagine two children are offered to ride a roller coaster, one of whom would find the ride very pleasant, while the other child would find it very unpleasant, and imagine, furthermore, that the only two options available are that they either both ride or neither of them ride (and if neither of them ride, they are both perfectly fine).[7] Whose interests should we sympathize with and favor? Common sense would appear to favor the child who would not want to take the ride. The mere pleasure of the “ride-positive” child does not justify a violation of the interest of the other child not to suffer a very unpleasant experience. The interest in not enduring such suffering seems far more fundamental, and hence to have ethical primacy, compared to the relatively trivial and frivolous interest of having a very pleasant experience.[8]

Arguably, common sense even suggests the same in the case where there are many more children who would find the ride very pleasant, while still only one child who would find it very unpleasant (provided, again, that the children will all be perfectly fine if they do not ride). Indeed, I believe a significant fraction of people would say the same no matter how many such “ride-positive” children we put on the scale: it would still be wrong to give them the ride at the cost of forcing the “ride-negative” child to undergo the very unpleasant experience.[9]

And yet the suffering in this example — a very unpleasant experience on a roller coaster — can hardly be said to count as remotely extreme, much less an instance of the worst forms of suffering; the forms of suffering that constitute the strongest, and in my view overwhelming, case for the principle of sympathy for intense suffering. Such intense suffering, even if balanced against the most intense forms of pleasure imaginable, only demands even stronger relative sympathy and priority. However bad we may consider the imposition of a very unpleasant experience for the sake of a very pleasant one, the imposition of extreme suffering for the sake of extreme pleasure must be deemed far worse.

The Horrendous Support for SIS

The worst forms of suffering are so terrible that merely thinking about them for a brief moment can leave the average sympathetic person in a state of horror and darkness for a good while, and therefore, quite naturally, we strongly prefer not to contemplate these things. Yet if we are to make sure that we have our priorities right, and that our views about what matters most in this world are as well-considered as possible, then we cannot shy away from the task of contemplating and trying to appreciate the disvalue of these worst of horrors. This is no easy task, and not just because we are reluctant to think about the issue in the first place, but also because it is difficult to gain anything close to a true appreciation of the reality in question. As David Pearce put it:

It’s easy to convince oneself that things can’t really be that bad, that the horror invoked is being overblown, that what is going on elsewhere in space-time is somehow less real than this here-and-now, or that the good in the world somehow offsets the bad. Yet however vividly one thinks one can imagine what agony, torture or suicidal despair must be like, the reality is inconceivably worse. Hazy images of Orwell’s ‘Room 101’ barely hint at what I’m talking about. The force of ‘inconceivably’ is itself largely inconceivable here.[10]

Nonetheless, we can still gain at least some, admittedly rather limited, appreciation by considering some real-world examples of extreme suffering (what follows are examples of an extremely unpleasant character that may be triggering and traumatizing).

One such example is the tragic fate of the Japanese girl Junko Furuta who was kidnapped in 1988, at the age of 16, by four teenage boys. According to their own trial statements, the boys raped her hundreds of times; “inserted foreign objects, such as iron bars, scissors and skewers into her vagina and anus, rendering her unable to defecate and urinate properly”; “beat her several times with golf clubs, bamboo sticks and iron rods”; “used her as a punching bag by hanging her body from the ceiling”; “dropped barbells onto her stomach several times”; “set fireworks into her anus, vagina, mouth and ear”; “burnt her vagina and clitoris with cigarettes and lighters”; “tore off her left nipple with pliers”; and more. Eventually, she was no longer able to move from the ground, and she repeatedly begged the boys to kill her, which they eventually did, after 44 days.[11]

An example of extreme suffering that is much more common, indeed something that happens countless times every single day, is being eaten alive, a process that can sometimes last several hours with the victim still fully conscious of being devoured, muscle by muscle, organ by organ. A harrowing example of such a death that was caught on camera (see the following note) involved a baboon tearing apart the hind legs of a baby gazelle and eating this poor individual who remained conscious for longer than one would have thought and hoped possible.[12] A few minutes of a much more protracted such painful and horrifying death can be seen via the link in the following note (lions eating a baby elephant alive).[13] And a similar, yet quicker death of a man can be seen via the link in the following note.[14] Tragically, the man’s wife and two children were sitting in a car next to him while it happened, yet they were unable to help him, and knowing this probably made the man’s experience even more horrible, which ties into a point made by Simon Knutsson:

Sometimes when the badness or moral importance of torture is discussed, it is described in terms of different stimuli that cause tissue damage, such as burning, cutting or stretching. But one should also remember different ways to make someone feel bad, and different kinds of bad feelings, which can be combined to make one’s overall experience even more terrible. It is arguably the overall unpleasantness of one’s experience that matters most in this context.[15]

After giving a real-world example with several layers of extreme cruelty and suffering combined, Knutsson goes on to write:

Although this example is terrible, one can imagine how it could be worse if more types of violence and bad feelings were added to the mix. To take another example: [Brian] Tomasik often talks about the Brazen bull as a particularly bad form of torture. The victim is locked inside a metal bull, a fire is lit underneath the bull and the victim is fried to death. It is easy to imagine how this can be made worse. For example, inject the victim with chemicals that amplify pain and block the body’s natural pain inhibitors, and put her loved ones in the bull so that when she is being fried, she also sees her loved ones being fried. One can imagine further combinations that make it even worse. Talking only of stimuli such as burning almost trivializes how bad experiences can be.[16]

Another example of extreme suffering is what happened to Dax Cowart. In 1973, at the age of 25, Dax went on a trip with his father to visit land that he considered buying. Unfortunately, due to a pipeline leak, the air over the land was filled with propane gas, which is highly flammable when combined with oxygen. As they started their car, the propane ignited, and the two men found themselves in a burning inferno. Dax’s father died, and Dax himself had much of his hands, eyes, and ears burned away; two thirds of his skin was severely burned.[17]

The case of Dax has since become quite famous, not only, or even mainly, because of the extreme horror he experienced during this explosion, but because of the ethical issues raised by his treatment, which turned out to be about as torturous as the explosion itself. For Dax himself repeatedly said, immediately after the explosion as well as for months later, that he wanted to die more than anything else, and that he did not want to be subjected to any treatment that would keep him alive. Nonetheless, he was forcibly treated for a period of ten months, during which he tried to take his life several times.
Since then, Dax has managed to recover and live what he considers a happy life — he successfully sued the oil company responsible for the pipeline leak, which left him financially secure; he earned a law degree; and got married. Yet even so, he still wishes that he had been killed rather than treated. In Dax’s own view, no happiness could ever compensate for what he went through.[18]

This kind of evaluation is exactly what the ethical principle advocated here centers on, and what the principle amounts to is simply a refusal to claim that Dax’s evaluation, or any other like it, is wrong. It maintains that we should not allow the occurrence of such extreme horrors for the sake of any intrinsic good, and hence that we should prioritize alleviating and preventing them over anything else.[19]

One may object that the examples above do not all comprise clear cases where the suffering subject deems their suffering so bad that nothing could ever outweigh it. And more generally, one may object that there can exist intense suffering that is not necessarily deemed so bad that nothing could outweigh it, either because the subject is not able to make such an evaluation, or because the subject just chooses not to evaluate it that way. What would the principle of sympathy for intense suffering say about such cases? It would say the following: in cases where the suffering is intense, yet the sufferers choose not to deem it so bad that nothing could outweigh it (we may call this “red suffering”), we should prioritize reducing suffering of the kind that would be deemed unoutweighable (what we may call “black suffering”). And in cases where the sufferers cannot make such evaluations, we may say that suffering at a level of intensity comparable to the suffering deemed unoutweighable by subjects who can make such evaluations should also be considered unoutweighable, and its prevention should be prioritized over all less intense forms of suffering.

Yet this is, of course, all rather theoretical. In practice, even when subjects do have the ability to evaluate their experience, we will, as outside observers, usually not be able to know what their evaluation is — for instance, how someone who is burning alive might evaluate their experience. In practice, all we can do is make informed assessments of what counts as suffering so intense that such an evaluation of unoutweighability would likely be made by the sufferer, assuming an idealized situation where the sufferer is able to evaluate the disvalue of the experience.[20]

 

I shall spare the reader from further examples of extreme suffering here in the text, and instead refer to sources, found in the following note, that contain additional cases that are worth considering in order to gain a greater appreciation of extreme suffering and its disvalue.[21] And the crucial question we must ask ourselves in relation to these examples — which, as hinted by the quote above by Knutsson, are probably far from the worst possible manifestations of suffering — is whether the creation of happiness or any other intrinsic good could ever justify the creation, or the failure to prevent, suffering this bad and worse. If not, this implies that our priority should not be to create happiness or other intrinsic goods, but instead to prevent extreme suffering of this kind above anything else, regardless of where in time and space it may risk emerging.

Objections to SIS

Among the objections against this view I can think of, the strongest, at least at first sight, is the sentiment: but what about that which is most precious in your life? What about the person who is most dear to you? If anything stands a chance of outweighing the disvalue of extreme suffering, surely this is it. In more specific terms: does it not seem plausible to claim that, say, saving the most precious person in one’s life could be worth an instance of the very worst form of suffering?

Yet one has to be careful about how this question is construed. If what we mean by “saving” is that we save them from extreme suffering, then we are measuring extreme suffering against extreme suffering, and hence we have not pointed to a rival candidate for outweighing the superlative disvalue of extreme suffering. Therefore, if we are to point to such a candidate, “saving” must here mean something that does not itself involve extreme suffering, and, if we wish to claim that there is something wholly different from the reduction of suffering that can be put on the scale, it should preferably involve no suffering at all. So the choice we should consider is rather one between 1) the mixed bargain of an instance of the very worst form of suffering, i.e. black suffering, and the continued existence of the most precious person one knows, or 2) the painless discontinuation of the existence of this person, yet without any ensuing suffering for others or oneself.

Now, when phrased in this way, choosing 1) may not sound all that bad to us, especially if we do not know the one who will suffer. Yet this would be cheating — nothing but an appeal to our faulty and all too partial moral intuitions. It clearly betrays the principle of impartiality,[22] according to which it should not matter whom the suffering in question is imposed upon; it should be considered equally disvaluable regardless.[23] Thus, we may equivalently phrase the choice above as being between 1) the continued existence of the most precious person one knows of, yet at the price that this being has to experience a state of extreme suffering, a state this person deems so bad that, according to them, it could never be outweighed by any intrinsic good, or 2) the discontinuation of the existence of this being without any ensuing suffering. When phrased in this way, it actually seems clearer to me than ever that 2) is the superior choice, and that we should adopt the principle of sympathy for intense suffering as our highest ethical principle. For how could one possibly justify imposing such extreme, and in the mind of the subject unoutweighable, suffering upon the most precious person one knows, suffering that this person would, at least in that moment, rather die than continue to experience? In this way, for me at least, it is no overstatement to say that this objection against the principle of sympathy for intense suffering, when considered more carefully, actually ends up being one of the strongest cases for it.

Another seemingly compelling objection would be to question whether an arbitrarily long duration of intense, yet, according to the subject, not unoutweighable suffering, i.e. red suffering, is really less bad than even just a split second of suffering that is deemed unoutweighable, i.e. black suffering. Counter-intuitively, my response, at least in this theoretical case, would be to bite the bullet and say “yes”. After all, if we take the subject’s own reports as the highest arbiter of the (dis)value of experiential states, then the black suffering cannot be outweighed by anything, whereas the red suffering can. Also, it should be noted that this thought experiment likely conflicts with quite a few sensible, real-world intuitions we have. For instance, in the real world, it seems highly likely that a subject who experiences extreme suffering for a long time will eventually find it unbearable, and say that nothing can outweigh it, contrary to the hypothetical case we are considering. Another such confounding real-world intuition might be one that reminds us that most things in the real world tend to fluctuate in some way, and hence, intuitively, it seems like there is a significant risk that a person who endures red suffering for a long time will also experience black suffering (again contrary to the actual conditions of the thought experiment), and perhaps even experience a lot of it, in which case this indeed is worse than a mere split second of black suffering on any account.

Partly for this latter reason, my response would also be different in practice. For again, in the real world, we are never able to determine the full consequences of our actions, and nor are we usually able to determine from the outside whether someone is experiencing red or black suffering, which implies that we have to take uncertainty and risks into account. Also because, even if we did know that a subject deemed some state of suffering as “merely” red at one point, this would not imply that their suffering at other moments where they appear to be in a similar state will also be deemed red as opposed to black. For in the real world it is indeed to be expected that significant fluctuations will occur, as well as that “the same suffering”, in one sense at least, will be felt as worse over time. Indeed, if the suffering is extreme, it all but surely will be deemed unbearable eventually.

Thus, in the real world, any large amount of extreme suffering is likely to include black suffering too, and therefore, regardless of whether we think some black suffering is worse than any amount of red suffering, the only reasonable thing to do in practice is to avoid getting near the abyss altogether.

Bias Alert: We Prefer to Not Think About Extreme Suffering

As noted above, merely thinking about extreme suffering can evoke unpleasant feelings that we naturally prefer to avoid. And this is significant for at least two reasons. First, it suggests that thinking deeply about extreme suffering might put our mental health at risk, and hence that we have good reason, and a strong personal incentive, to avoid engaging in such deeper thinking. Second, in part for this first reason, it suggests that we are biased against thinking deeply about extreme suffering, and hence biased against properly appreciating the true horror and disvalue of such suffering. Somewhat paradoxically, (the mere thought of) the horror of extreme suffering keeps us from fully appreciating the true scope of this horror. And this latter consideration is significant in the context of trying to fairly evaluate the plausibility of views that say we should give special priority to such suffering, including the view presented above.

Indeed, one can readily tell a rather plausible story about how many of the well-documented biases we reviewed previously might conspire to produce such a bias against appreciating the horror of suffering.[24] For one, we have wishful thinking, our tendency to believe as true what we wish were true, which in this case likely pulls us toward the belief that it can’t be that bad, and that, surely, there must be something of greater value, some grander quest worth pursuing in this world than the mere negative, rather anti-climatic “journey” of alleviating and preventing extreme suffering. Like most inhabitants of Omelas, we wishfully avoid giving much thought to the bad parts, and instead focus on all the good — although our sin is, of course, much greater than theirs, as the bad parts in the real world are indescribably worse on every metric, including total amount, relative proportions, and intensity.

To defend this wishfully established view, we then have our confirmation bias. We comfortably believe that it cannot really be that bad, and so in perfect confirmation bias textbook-style, we shy away from and ignore data that might suggest otherwise. We choose not to look at the horrible real-world examples that might change our minds, and to not think too deeply about the arguments that challenge our merry conceptions of value and ethics. All of this for extremely good reasons, of course. Or at least so we tell ourselves.[25]

Next, we have groupthink and, more generally, our tendency to conform to our peers. Others do not seem to believe that extreme suffering is that horrible, or that reducing it should be our supreme goal, and thus our bias to conform smoothly points us in the same direction as our wishful thinking and confirmation bias. That direction being: “Come on, lighten up! Extreme suffering is probably not that bad, and it probably can be outweighed somehow. This is what I want to believe, it is what my own established and comfortable belief says, and it is what virtually all my peers seem to believe. Why in the world, then, would I believe anything else?”

Telling such a story of bias might be considered an unfair move, a crude exercise in pointing fingers at others and exclaiming “You’re just biased!”, and admittedly it is to some extent. Nonetheless, I think two things are worth noting in response to such a sentiment. First, rather than having its origin in finger pointing at others, the source of this story is really autobiographical: it is a fair characterization of how my own mind managed to repudiate the immense horror and primacy of extreme suffering for a long time. And merely combining this with the belief that I am not a special case then tentatively suggests that a similar story might well apply to the minds of others too.

Second, it should be noted that a similar story cannot readily be told in the opposite direction — about the values defended here. In terms of wishful thinking, it is not particularly wishful or feel-good to say that extreme suffering is immensely bad, and that there is nothing of greater value in the world than to prevent it. That is not a pretty or satisfying story for anyone. The view also seems difficult to explain via an appeal to confirmation bias, since many of those who hold this view of extreme suffering, including myself, did not hold it from the outset, but instead changed their minds toward it upon considering arguments and real-world examples that support it. The same holds true of our tendency to conform to our peers. For although virtually nobody appears to seriously doubt that suffering has disvalue, the view that nothing could be more important than preventing extreme suffering does not seem widely held, much less widely expressed. It lies far from the narrative about the ultimate mission and future purpose of humanity that prevails in most circles, which runs more along the lines of “Surely it must all be worth it somehow, right?”

This last consideration about how we stand in relation to our peers is perhaps especially significant. For the truth is that we are a signalling species: we like to appear cool and impressive.[26] And to express the view that nothing matters more than the prevention of extreme suffering seems a most unpromising way of doing so. It has a strong air of darkness and depression about it, and, worst of all, it is not a signal of strength and success, which is perhaps what we are driven the most to signal to others, prospective friends and mates alike. Such success signalling is not best done with darkness, but with light: by exuding happiness, joy, and positivity. This is the image of ourselves, including our worldview, that we are naturally inclined to project, which then ties into the remark made above — that this view does not seem widely held, “much less widely expressed”. For even if we are inclined to hold this view, we appear motivated to not express it, lest we appear like a sad loser.

 

In sum, by my lights, effective altruism proper is equivalent to effectively reducing extreme suffering. This, I would argue, is the highest meaning of “improving the world” and “benefiting others”, and hence what should be considered the ultimate goal of effective altruism. The principle of sympathy for intense suffering argued for here stems neither from depression, nor resentment, nor hatred. Rather, it simply stems, as the name implies, from a deep sympathy for intense suffering.[27] It stems from a firm choice to side with the evaluations of those who are superlatively worst off, and from this choice follows a principled unwillingness to allow the creation of such suffering for the sake of any amount of happiness or any other intrinsic good. And while it is true that this principle has the implication that it would have been better if the world had never existed, I think the fault here is to be found in the world, not the principle.

Most tragically, some pockets of the universe are in a state of insufferable darkness — a state of black suffering. In my view, such suffering is like a black hole that sucks all light out of the world. Or rather: the intrinsic value of all the light of the world pales in comparison to the disvalue of this darkness. Yet, by extension, this also implies that there is a form of light whose value does compare to this darkness, and that is the kind of light we should aspire to become, namely the light that brightens and prevents this darkness.[28] We shall delve into how this can best be done shortly, but first we shall delve into another issue: our indefensibly anthropocentric take on altruism and “philanthropy”.


 

(For the full bibliography, see the end of my book.)

[1] This view is similar to what Brian Tomasik calls consent-based negative utilitarianism: http://reducing-suffering.org/happiness-suffering-symmetric/#Consent-based_negative_utilitarianism
And the Organisation for the Prevention of Intense Suffering (OPIS) appears founded upon a virtually identical principle: http://www.preventsuffering.org/
I do not claim that this view is original; merely that it is important.

[2] And I have read them all, though admittedly not their complete works. Bentham can seem to come close in chapter 4 of his Principles of Morals and Legislation, where he outlines a method for measuring pain and pleasure. One of the steps of this method consists in summing up the values of “[…] all the pleasures on one side and of all the pains on the other.” And later he writes of this process that it is “[…] applicable to pleasure and pain in whatever form they appear […]”. Yet he does not write that the sum will necessarily be finite, nor, more specifically, whether every instance of suffering necessarily can be outweighed by some pleasure. I suspect Bentham, as well as Mill and Sidgwick, never contemplated this question in the first place.

[3] A recommendable essay on the issue is Simon Knutsson’s “Measuring Happiness and Suffering”: https://foundational-research.org/measuring-happiness-and-suffering/

[4] However, a defender of tranquilism would, of course, question whether we are indeed talking about a pleasure outweighing some suffering rather than it, upon closer examination, really being a case of a reduction of some form of suffering outweighing some other form of suffering

[5] And therefore, if one assumes a framework of so-called moral uncertainty, it seems that one should assign much greater plausibility to negative value lexicality than to positive value lexicality (cf. https://foundational-research.org/value-lexicality/), also in light of the point made in the previous chapter that many have doubted the positive value of happiness (as being due to anything but its absence of suffering), whereas virtually nobody has seriously doubted the disvalue of suffering.

[6] But what if there are several levels of extreme suffering, where an experience on each level is deemed so bad that no amount of experiences on a lower level could outweigh it? This is a tricky issue, yet to the extent that these levels of badness are ordered such that, say, no amount of level I suffering can outweigh a single instance of level II suffering (according to a subject who has experienced both), then I would argue that we should give priority to reducing level II suffering. Yet what if level I suffering is found to be worse than level II suffering in the moment of experiencing it, while level II suffering is found to be worse than level I suffering when it is experienced? One may then say that the evaluation should be up to some third experience-moment with memory of both states, and that we should trust such an evaluation, or, if this is not possible, we may view both forms of suffering as equally bad. Whether such dilemmas arise in the real world, and how to best resolve them in case they do, stands to me as an open question.
Thus, cf. the point about the lack of clarity and specification of values we saw two chapters ago, the framework I present here is not only not perfectly specific, as it surely cannot be, but it is admittedly quite far from it indeed. Nonetheless, it still comprises a significant step in the direction of carving out a clearer set of values, much clearer than the core value of, say, “reducing suffering”.

[7] A similar example is often used by the suffering-focused advocate Inmendham.

[8] This is, of course, essentially the same claim we saw a case for in the previous chapter: that creating happiness at the cost of suffering is wrong. The principle advocated here may be considered a special case of this claim, namely the special case where the suffering in question is deemed irredeemably bad by the subject.

[9] Cf. the gut feeling many people seem to have that the scenario described in The Ones Who Walk Away from Omelas should not be brought into the world regardless of how big the city of Omelas would be. Weak support for this claim is also found in the following survey, in which a plurality of people said that they think future civilization should strive to minimize suffering (over, for instance, maximizing positive experiences): https://futureoflife.org/superintelligence-survey/

[10] https://www.hedweb.com/negutil.htm
A personal anecdote of mine in support of Pearce’s quote is that I tend to write and talk a lot about reducing suffering, and yet I am always unpleasantly surprised by how bad it is when I experience even just borderline intense suffering. I then always get the sense that I have absolutely no idea what I am talking about when I am talking about suffering in my usual happy state, although the words I use in that state are quite accurate: that it is really bad. In those bad states I realize that it is far worse than we tend to think, even when we think it is really, really bad. It truly is inconceivable, as Pearce writes, since we simply cannot simulate that badness in a remotely faithful way when we are feeling good, quite analogously to the phenomenon of binocular rivalry, where we can only perceive one of two visual images at a time.

[11] https://ripeace.wordpress.com/2012/09/14/the-murder-of-junko-furuta-44-days-of-hell/
https://en.wikipedia.org/wiki/Murder_of_Junko_Furuta

[12] https://www.youtube.com/watch?v=PcnH_TOqi3I

[13] https://www.youtube.com/watch?v=Lc63Rp-UN10

[14] https://www.abolitionist.com/reprogramming/maneaters.html

[15] http://www.simonknutsson.com/the-seriousness-of-suffering-supplement

[16] http://www.simonknutsson.com/the-seriousness-of-suffering-supplement

[17] Dax describes the accident himself in the following video:
https://www.youtube.com/watch?v=M3ZnFJGmoq8

[18] Brülde, 2010, p. 576; Benatar, 2006, p. 63.

[19] And if one thinks such extreme suffering can be outweighed, an important question to ask oneself is: what exactly does it mean to say that it can be outweighed? More specifically, according to whom, and measured by what criteria, can such suffering be outweighed? The only promising option open, it seems, is to choose to prioritize the assessments of beings who say that their happiness, or other good things about their lives, can outweigh the existence of such extreme suffering — i.e. to actively prioritize the evaluations of such notional beings over the evaluations of those enduring, by their own accounts, unoutweighable suffering. What I would consider a profoundly unsympathetic choice.

[20] This once again hints at the point made earlier that we in practice are unable to specify in precise terms 1) what we value in the world, and 2) how to act in accordance with any set of plausible values. Rough, qualified approximations are all we can hope for.

[21] http://reducing-suffering.org/the-horror-of-suffering/
http://reducing-suffering.org/on-the-seriousness-of-suffering/
http://www.simonknutsson.com/the-seriousness-of-suffering-supplement
https://www.youtube.com/watch?v=RyA_eF7W02s&

[22] Or one could equivalently say that it betrays the core virtue of being consistent, as it amounts to treating/valuing similar beings differently.

[23] I make a more elaborate case for this conclusion in my book You Are Them.

[24] One might object that it makes little sense to call a failure to appreciate the value of something a bias, as this is a moral rather than an empirical disagreement, to which I would respond: 1) the two are not as easy to separate as is commonly supposed (cf. Putnam, 2002), 2) one clearly can be biased against fairly considering an argument for a moral position — for instance, we can imagine an example where someone encounters a moral position and then, due to being brought up in a culture that dislikes that moral position, fails to properly engage with and understand this position, although this person would in fact agree with it upon reflection; such a failure can fairly be said to be due to bias — and 3) at any rate, the question concerning what it is like to experience certain states of consciousness is a factual matter, including how horrible they are deemed from the inside, and this is something we can be factually wrong about as outside observers.

[25] Not that sparing our own mental health is not a good reason for not doing something potentially traumatizing, but the question is just whether it is really worth letting our view of our personal and collective purpose in life be handicapped and biased, at the very least less well-informed than it otherwise could be, for that reason. Whether such self-imposed ignorance can really be justified, both to ourselves and the world at large.

[26] Again, Robin Hanson and Kevin Simler’s book The Elephant in the Brain makes an excellent case for this claim.

[27] And hence being animated by this principle is perfectly compatible with living a happy, joyous, and meaningful life. Indeed, I would argue that it provides the deepest meaning one could possibly find.

[28] I suspect both the content and phrasing of the last couple of sentences are inspired by the following quote I saw written on Facebook by Robert Daoust: “What is at the center of the universe of ethics, I suggest, is not the sun of the good and its play of bad shadows, but the black hole of suffering.”

Suffering-Focused Ethics

This essay was first published as a chapter in my book Effective Altruism: How Can We Best Help Others? which is available for free download here.


The view of values I would favor falls within a broader class of ethical views one may call suffering-focused ethics, which encompasses all views that give special priority to the alleviation and prevention of suffering. I will review some general arguments and considerations in favor of such views in this chapter, arguments that individually and collectively can support granting moral priority to suffering.[1] This general case will then be followed by a more specific case for a particular suffering-focused view — what I consider to be the strongest and most convincing one — in the next chapter.

It should be noted, however, that not all effective altruists agree with this view of values. Many appear to view the creation of happiness — for example, via the creation of new happy beings, or by raising the level of happiness of the already happy — as having the same importance as the reduction of “equal” suffering. I used to hold this view as well. Yet I have changed my mind in light of considerations of the kind presented below.[2]

The Asymmetries

We have already briefly visited one asymmetry that seems to exist, at least in the eyes of many people, between suffering and happiness, namely the so-called Asymmetry in population ethics, which roughly says that we have an obligation to avoid bringing miserable lives into the world, but no obligation to bring about happy lives. To the extent we agree with this view, it appears that we agree that we should assign greater moral value and priority to the alleviation and prevention of suffering over the creation of happiness, at least in the context of the creation of new lives.

A similar view has been expressed by philosopher Jan Narveson, who has argued that there is value in making people happy, but not in making happy people.[3] Another philosopher who holds a similar view is Christoph Fehige, who defends a position he calls antifrustrationism, according to which we have obligations to make preferrers satisfied, but no obligations to make satisfied preferrers.[4] Peter Singer, too, has expressed a similar view in the past:

The creation of preferences which we then satisfy gains us nothing. We can think of the creation of the unsatisfied preferences as putting a debit in the moral ledger which satisfying them merely cancels out. […] Preference Utilitarians have grounds for seeking to satisfy their wishes, but they cannot say that the universe would have been a worse place if we had never come into existence at all.[5]

In terms of how we choose to prioritize our resources, there does indeed, to many of us at least, seem something highly unpalatable, not to say immoral and frivolous, about focusing on creating happiness de novo rather than on alleviating and preventing suffering first and foremost. As philosopher Adriano Mannino has expressed it:

What’s beyond my comprehension is why turning rocks into happiness elsewhere should matter at all. That strikes me as okay, but still utterly useless and therefore immoral if it comes at the opportunity cost of not preventing suffering. The non-creation of happiness is not problematic, for it never results in a problem for anyone (i.e. any consciousness-moment), and so there’s never a problem you can point to in the world; the non-prevention of suffering, on the other hand, results in a problem.[6]

And in the case of extreme suffering, one can argue that the word “problem” is a strong contender for most understated euphemism in history. Mannino’s view can be said to derive from what is arguably an intuitive and common-sense “understanding of ethics as being about solving the world’s problems: We confront spacetime, see wherever there is or will be a problem, i.e. a struggling being, and we solve it.”[7]

Simon Knutsson has expressed a similar sentiment to the opportunity cost consideration expressed by Mannino above, and highlighted the crucial juxtaposition we must consider:

When spending resources on increasing the number of beings instead of preventing extreme suffering, one is essentially saying to the victims: “I could have helped you, but I didn’t, because I think it’s more important that individuals are brought into existence. Sorry.”[8]

Philosopher David Benatar defends an asymmetry much stronger than the aforementioned Asymmetry in population ethics, as he argues that we not only should avoid bringing (overtly) miserable lives into existence, but that we ideally should avoid bringing any lives into existence at all, since coming into existence is always a harm on Benatar’s account. Explained simply, Benatar’s main argument rests on the premise that the absence of suffering is good, while the absence of happiness is not bad, and hence the state of non-existence is good (“good” + “not bad” = “good”), whereas the presence of suffering and happiness is bad and good respectively, and hence not a pure good, which renders it worse than the state of non-existence according to Benatar.[9]

Beyond this asymmetry, Benatar further argues that there is an asymmetry in how much suffering and happiness our lives contain — e.g. that the worst forms of suffering are far worse than the best pleasures are good; that we almost always experience some subtle unpleasantness, dissatisfaction, and preference frustration; and that there are such negative things as chronic pain, impairment, and trauma, yet no corresponding positive things, like chronic pleasure.[10] And the reason that we fail to acknowledge this, Benatar argues, is that we have various, well-documented psychological biases which cause us to evaluate our lives in overly optimistic terms.[11]

It seems worth expanding a bit on this more quantitative asymmetry between the respective badness and goodness of suffering and happiness. For even if one rejects the notion that there is a qualitative difference between the moral status of creating happiness and preventing suffering — e.g. that a failure to prevent suffering is problematic, while a failure to create happiness is not — it seems difficult to deny Benatar’s claim that the worst forms of suffering are far worse than the best of pleasures are good. Imagine, for example, that we were offered ten years of the greatest happiness possible on the condition that we must undergo some amount of hellish torture in order to get it. How much torture would we be willing to endure in order to get this prize? Many of us would reject the offer completely and prefer a non-existent, entirely non-problematic state over any mixture of hellish torture and heavenly happiness.

Others, however, will be willing to accept the offer and make a sacrifice. And the question is then how big a sacrifice one could reasonably be willing to make? Seconds of hellish torture? A full hour? Perhaps even an entire day? Some might go as far as saying an entire day, yet it seems that no matter how much one values happiness, no one could reasonably push the scale to anywhere near 50/50. That is, no one could reasonably choose to endure ten years of hellish torture in order to attain ten years of sublime happiness.

Those who would be willing to endure a full day of torture in order to enjoy ten years of paradise are, I think, among those who are willing to push it the furthest in order to attain such happiness, and yet notice how far they are from 50/50. We are not talking 80/20, 90/10, or even 99/1 here. No, one day of hell for 3650 days of paradise roughly corresponds to a “days of happiness to days of suffering” ratio of 99.97 to 0.03. And that is for those who are willing to push it.[12]

So not only is there no symmetry here; the moral weight of the worst of suffering appears to be orders of magnitude greater than that of the greatest happiness, which implies that the prevention of suffering appears the main name of the ethical game on any plausible moral calculus. Even on a view according to which we are willing to really push it and endure what is, arguably by most accounts, an unreasonable amount of suffering in order to gain happiness, the vast majority of moral weight is still found in preventing suffering, at least when speaking in terms of durations of the best and worst potential states. And one can reasonably argue that this is also true of the actual state of the world, as Arthur Schopenhauer did when comparing “the feelings of an animal engaged in eating another with those of the animal being eaten.”[13]

A more general and qualitative asymmetry between the moral status of happiness and suffering has been defended by philosopher Karl Popper:

I believe that there is, from the ethical point of view, no symmetry between suffering and happiness, or between pain and pleasure. […] In my opinion human suffering makes a direct moral appeal, namely, the appeal for help, while there is no similar call to increase the happiness of a man who is doing well anyway. A further criticism of the Utilitarian formula “Maximize pleasure” is that it assumes a continuous pleasure-pain scale which allows us to treat degrees of pain as negative degrees of pleasure. But, from the moral point of view, pain cannot be outweighed by pleasure, and especially not one man’s pain by another man’s pleasure. Instead of the greatest happiness for the greatest number, one should demand, more modestly, the least amount of avoidable suffering for all; […][14]

David Pearce, who identifies as a negative utilitarian, describes his view in a similar way:

Ethical negative-utilitarianism is a value-system which challenges the moral symmetry of pleasure and pain. It doesn’t question the value of enhancing the happiness of the already happy. Yet it attaches value in a distinctively moral sense of the term only to actions which tend to minimise or eliminate suffering. This is what matters above all else.[15]

Neither Popper nor Pearce appear to deny that there is value in happiness. Instead, what they deny is that the value there may be in creating happiness is comparable to the value of reducing suffering. In Pearce’s words, increasing the happiness of the already happy does not carry value in the distinctively moral sense that reducing suffering does; in Popper’s words, suffering makes a direct moral appeal for help, while the state of those who are doing well does not.

Expressed in other words, one may say that the difference is that suffering, by its very nature, carries urgency, whereas the creation of happiness does not, at least not in a similar way. (Popper put it similarly: “[…] the promotion of happiness is in any case much less urgent than the rendering of help to those who suffer […]”[16]) We would rightly rush to send an ambulance to help someone who is enduring extreme suffering, yet not to boost the happiness of someone who is already happy, no matter how much we may be able to boost it. Similarly, if we had pills that could raise the happiness of those who are already doing well to the greatest heights possible, there would be no urgency in distributing these pills (to those already doing well), whereas if a single person fell to the ground in unbearable agony right before us, there would indeed be an urgency to help. Increasing the happiness of the already happy is, unlike the alleviation of extreme suffering, not an emergency.

A similar consideration about David Pearce’s abolitionist project described in the previous chapter — the abolition of suffering throughout the living world via biotechnology — appears to lend credence to this asymmetrical view of the moral status of the creation of happiness versus the prevention of suffering. For imagine we had completed the abolitionist project and made suffering non-existent for good. The question is then whether it can reasonably be maintained that our moral obligations would be exactly the same after this completion. Would we have an equally strong duty or obligation to move sentience to new heights after we had abolished suffering? Or would we instead have discharged our prime moral obligation, and thus have reason to lower our shoulders and breathe a deep and justified sigh of moral relief? I think the latter.

Another reason in favor of an asymmetrical view is that, echoing Benatar somewhat, it seems that the absence of extreme happiness cannot be considered bad in remotely the same way that the absence of extreme suffering can be considered good. For example, if a person is in a state of dreamless sleep rather than having the experience of a lifetime, this cannot reasonably be characterized as a disaster or a catastrophe; the difference between these two states does not seem to carry great moral weight. Yet when it comes to the difference between sleeping and being tortured, we are indeed talking about a difference that does carry immense moral weight, and the realization of the worse rather than the better outcome would indeed amount to a catastrophe.

The final asymmetry I shall review in this section is one that is found more on a meta-level, namely in the distribution of views concerning the moral value of the creation of happiness and the prevention of suffering. For in our broader human conversation about what has value, very few seem to have seriously disputed the disvalue of suffering and the importance of preventing it. Indeed, to the extent that we can find a value that almost everyone agrees on, it is this: suffering matters. In contrast, there are many who have disputed the value and importance of creating more happiness, including many of the philosophers mentioned in this section; many thinkers in Eastern philosophy for whom moksha, liberation from suffering, is the highest good; as well as many thinkers in Western philosophy, with roots all the way back to Epicurus, for whom ataraxia, an untroubled state free from distress, was the highest aim. Further elaboration on a version of this view of happiness follows in the next section.

This asymmetry in consensus about the value and moral status of creating happiness versus preventing suffering also counts as a weak reason for giving greater priority to the latter.

Tranquilism: Happiness as the Absence of Suffering

Author Lukas Gloor defends a view he calls tranquilism, which — following Epicurus and his notion of ataraxia, as well as the goal of moksha proposed as the highest good by many Eastern philosophers[17] — holds that the value of happiness lies in its absence of suffering.[18] Thus, according to tranquilism, states of euphoric bliss are not of greater value than, say, states of peaceful contentment free of any negative components. Or, for that matter, than a similarly undisturbed state of dreamless sleep or insentience. In other words, states of happiness are of equal value to nothing, provided that they are shorn of suffering.

In this way, tranquilism is well in line with the asymmetry in moral status between happiness and suffering defended by Karl Popper and David Pearce: that increasing the happiness of the already happy does not have the moral value that reducing suffering does. And one may even argue that it explains this asymmetry: if the value of happiness lies in its absence of suffering, then it follows that creating happiness (for those not suffering) cannot take precedence over reducing suffering. Moving someone from zero to (another kind of) zero can never constitute a greater move on the value scale than moving someone from a negative state to a (however marginally) less negative one.[19]

To many of us, this is a highly counter-intuitive view, at least at first sight. After all, do we not seek pleasure almost all the time, often at the seemingly justified cost of suffering? Yet one can frame this seeking in another way that is consistent with tranquilism, by viewing our search for pleasure as really being an attempt to escape suffering and dissatisfaction. On this framing, what appears to be going from neutral to positive is really going from a state of negativity, however subtle, to a state that is relieved, at least to some extent, from this negativity. So, on this view, when we visit a friend we have desired to see for some time, we do not go from a neutral to a positive state, but instead just remove our craving for their company and the dissatisfaction caused by their absence. So too with the pleasure of physical exercise: it is liberating in that it gives us temporary freedom from the bad feelings and moods that follow from not exercising. Or even the pleasure of falling in love, which provides refreshing relief from the boredom and desire we are otherwise plagued by.

Psychologist William James seemed to agree with this view of happiness:

Happiness, I have lately discovered, is no positive feeling, but a negative condition of freedom from a number of restrictive sensations of which our organism usually seems the seat. When they are wiped out, the clearness and cleanness of the contrast is happiness. This is why anaesthetics make us so happy.[20]

As did Arthur Schopenhauer:

[…] evil is precisely that which is positive,[21] that which makes itself palpable, and good, on the other hand, i.e. all happiness and gratification, is that which is negative, the mere abolition of a desire and extinction of a pain.[22]

And here is how Lukas Gloor explains it:

In the context of everyday life, there are almost always things that ever so slightly bother us. Uncomfortable pressure in the shoes, thirst, hunger, headaches, boredom, itches, non-effortless work, worries, longing for better times. When our brain is flooded with pleasure, we temporarily become unaware of all the negative ingredients of our stream of consciousness, and they thus cease to exist. Pleasure is the typical way in which our minds experience temporary freedom from suffering, which may contribute to the view that happiness is the symmetrical counterpart to suffering, and that pleasure, at the expense of all other possible states, is intrinsically important and worth bringing about.[23]

One may object that the implication that mere contentment has the same value as the greatest euphoric bliss seems implausible, and thus counts against tranquilism. Yet whether this is indeed implausible depends on the eyes that look. For consider it this way: if someone who experiences “mere contentment” without any negative cravings[24] whatsoever, and thus does not find the experience insufficient in any way, who are we to say that they are wrong about their state, and that they actually should want something better? Tranquilism denies that such a “merely content” person is wrong to claim that their state is perfect. Indeed, tranquilism is here in perfect agreement with this person, and hence this implication of tranquilism is at least not implausible from this person’s perspective, which one may argue is the most relevant perspective to consider in this context of discussing whether said person is in a suboptimal state. The perspective from which this implication appears implausible, a proponent of tranquilism may argue, is only from the perspective of someone who is not in perfect contentment — one who desires euphoric bliss, for oneself and others, and in some sense feels lacking, i.e. a negative craving, about its absence.

Another apparent, and perhaps intuitive, reason to reject tranquilism is that it appears to imply that happiness is not really that wonderful — that the best experience one has ever had was not really that great. Yet it is important to make clear that tranquilism implies no such thing. On the contrary, according to tranquilism, experiences of happiness without any suffering are indeed (together with other experiential states that are absent of suffering) experiences of the most wonderful kind, and they are by no means less wonderful than they are felt. What tranquilism does say, however, is that the value of such states is due to their absence of suffering, and that the creation of such happy states cannot justify the creation of suffering.

Yet even so, even while allowing us to maintain the view that happiness is wonderful, tranquilism is still, at least for many of us, really not a nice way to think about the world, and about the nature of value in particular, as we would probably all like to think that there exists something of truly positive value in the realm of conscious experience beyond merely the absence of negative experiences or cravings. Yet this want of ours — this negative craving, one could say — should only make us that much more skeptical of any reluctance we may have to give tranquilism a fair hearing. And even if, upon doing so, one does not find tranquilism an entirely convincing or exhaustive account of the respective (dis)value of happiness and suffering, it seems difficult to deny that there is a significant grain of truth to it.

The implications of tranquilism are clear: creating more happiness (for the currently non-existent or otherwise not suffering) has neutral value, while there is value in the alleviation and prevention of suffering, a value that, as noted above, nobody seriously questions.

Creating Happiness at the Cost of Suffering Is Wrong

In this section I shall not argue for a novel, separate point, but instead invoke some concrete examples that help make the case for a particular claim that follows directly from many of the views we have seen above, the claim being that it is wrong to create happiness at the cost of suffering.

One obvious example of such gratuitous suffering would be that of torturing a single person for the enjoyment of a large crowd.[25] If we think happiness can always outweigh suffering, we seem forced to say that, yes, provided that the resulting enjoyment of the crowd is great enough, and if other things are equal, then such happiness can indeed outweigh and justify torturing a single person. Yet that seems misguided. A similar example to consider is that of a gang rape: if we think happiness can always outweigh suffering, then such a rape can in principle be justified, provided that the pleasure of the rapists is sufficiently great. Yet most people would find this proposition utterly wrong.

One may object that these thought experiments bring other issues into play than merely that of happiness versus suffering, which is a fair point. Yet we can in a sense control for these by reversing the purpose of these acts so that they are about reducing suffering rather than increasing happiness for a given group of individuals. So rather than the torture of a single person being done for the enjoyment of a crowd, it is now done in order to prevent a crowd from being tortured; rather than the rape being done for the pleasure of, say, five people, it is done to prevent five people from being raped. While we may still find it most unpalatable to give the go signal for such preventive actions, it nonetheless seems clear that torturing a single person in order to prevent the torture of many people would be the right thing to do, and that having less rape occur is better than having more.

A similar example, which however does not involve any extreme suffering, is the situation described in Ursula K. Le Guin’s short story The Ones Who Walk Away from Omelas. The story is about an almost utopian city, Omelas, in which everyone lives an extraordinarily happy and meaningful life, except for a single child who is locked in a basement room, fated to live a life of squalor:

The child used to scream for help at night, and cry a good deal, but now it only makes a kind of whining, “eh-haa, eh-haa,” and it speaks less and less often. It is so thin there are no calves to its legs; its belly protrudes; it lives on a half-bowl of corn meal and grease a day. It is naked. Its buttocks and thighs are a mass of festered sores, as it sits in its own excrement continually.[26]

The story ends by describing some people in the city who appear to find the situation unacceptable and who choose not to take part in it any more — the ones who walk away from Omelas.

The relevant question for us to consider here is whether we would walk away from Omelas, or perhaps rather whether we would choose to bring a condition like Omelas into existence in the first place. Can the happy and meaningful lives of the other people in Omelas justify the existence of this single, miserable child? Different people have different intuitions about it; some will say that it depends on how many people live in Omelas. Yet to many of us, the answer is “no” — the creation of happiness is comparatively frivolous and unnecessary, and it cannot justify the creation of such a victim, of such misery and suffering.[27] A sentiment to the same effect was expressed in the novel The Plague, by Albert Camus: “For who would dare to assert that eternal happiness can compensate for a single moment’s human suffering?”[28]

A “no” to the creation of Omelas would also be supported by the Asymmetry in population ethics, according to which it has neutral value to add a happy life to Omelas, while adding this one miserable child has negative value, and hence the net value of the creation of Omelas is negative.

The examples visited above all argue for the claim that it is wrong to impose certain forms of suffering on someone for the sake of creating happiness, where the forms of suffering have gradually been decreasing in severity. And one may argue that the plausibility of the claims these respective examples have been used to support has been decreasing gradually too, and for this very reason: the less extreme the suffering, the less clear it is that happiness could never outweigh it. And yet even in the case of the imposition of the mildest of suffering — a pinprick, say — for the sake of the creation of happiness, it is far from clear, upon closer examination, that this should be deemed permissible, much less an ethical obligation. Echoing the passage by Camus above, would it really be right to impose a pinprick on someone in order to create pleasure for ourselves or others, or indeed for the very person we do it on, provided that whomever would gain the happiness is doing perfectly fine already, and thus that the resulting happiness would not in fact amount to a reduction of suffering? Looking only at, or rather from, the perspective of that moment’s suffering itself, the act would indeed be bad, and the question is then what could justify such badness, given that the alternative was an entirely trouble-free state. If one holds that being ethical means to promote happiness over suffering, not to create happiness at the cost of suffering, the answer is “nothing”.

Two Objections

Finally, it is worth briefly addressing two common objections against suffering-focused ethics, the first one being that not many people have held such a view, which makes it appear implausible. The first thing to say in response to this claim is that, even if it were true, the fact that a position is not widely held is not a strong reason to consider it implausible, especially if one thinks one has strong, object-level reasons to consider it plausible, and, furthermore, if one believes there are human biases[29] that can readily explain its (purportedly) widespread rejection. The second thing to say is that the claim is simply not true, as there are many thinkers, historical as well as contemporary ones, who have defended views similar to those outlined here (see the following note for examples).[30]

Another objection is that suffering-focused views have unappealing consequences, including that, according to such views, it would be right to kill everyone (or “destroy the world”). One reply to this claim is that at least some suffering-focused views do not have this implication. For example, in his book The Battle for Compassion: Ethics in an Apathetic Universe, Jonathan Leighton argues for a pragmatic position he calls “negative utilitarianism plus”, according to which we should aim to do our best to reduce preventable suffering, yet where we can still “categorically refuse to intentionally destroy the planet and eliminate ourselves and everything we care about in the process […]”.[31]

Another reply is that, as Simon Knutsson has argued at greater length,[32] other ethical views that have a consequentialist component seem about as vulnerable to similar objections. For instance, if maximizing the sum of happiness minus suffering were our core objective, it could be said that we ought to kill people in order to replace them with happier beings. One may then object, quite reasonably, that this is unlikely to be optimal in practice, yet one can argue — as reasonably, I believe — that the same holds true of trying to destroy the world in order to reduce suffering: it does not seem the best we can do in practice. I shall say a bit more about this last point in the penultimate chapter on future directions.

 

Having visited this general case for suffering-focused ethics, we shall now turn to what is arguably the strongest case for such a view — the appeal to sympathy for intense suffering.


 

(For the full bibliography, see the end of my book.)

[1] This chapter is inspired by other resources that also advocate for suffering-focused ethics, such as the following:
https://foundational-research.org/the-case-for-suffering-focused-ethics/
https://www.utilitarianism.com/nu/nufaq.html
https://www.youtube.com/watch?v=4OWl5nTctYI
https://www.hedweb.com/negutil.htm
Pearce, 2017, part II
A more elaborate case for focusing on suffering can be found in Jamie Mayerfeld’s Suffering and Moral Responsibility.

[2] Not least have I changed my mind about whether a term like “equal suffering” is at all meaningful in general.

[3] Narveson, 1973.

[4] Fehige, 1998.

[5] Singer, 1980b. However, Singer goes on to say about this view of coming into existence that it “perhaps, is a reason to combine [preference and hedonistic utilitarianism]”. Furthermore, Singer seems to have moved much closer toward, and to now defend, hedonistic utilitarianism, whereas he was arguably primarily a preference utilitarian when he made the quoted statement.

[6] Quoted from a Facebook conversation.

[7] https://foundational-research.org/the-case-for-suffering-focused-ethics/

[8] http://www.simonknutsson.com/the-one-paragraph-case-for-suffering-focused-ethics

[9] Benatar, 2006, chapter 2.

[10] Benatar, 2006, chapter 3.

[11] Benatar, 2006, chapter 3.

[12] One may object that our choosing such a skewed trade-off is merely a reflection of our contingent biology, and that it may be possible to create happiness so great that most people would consider a single day of it worth ten years of the worst kinds of suffering our biology can support. To this I would respond that such a possibility remains hypothetical, indeed speculative, and that we should base our views mainly on the actualities we know rather than such hypothetical (and wishful) possibilities. After all, it may also be, indeed it seems about equally likely, that suffering can be far worse than the worst suffering our contingent biology can support, and, furthermore, it may be that the pattern familiar from our contingent biology only repeats itself in this realm of theoretical maxima; i.e. that such maximal suffering can only be deemed far more disvaluable than the greatest bliss possible can be deemed valuable.

[13] Schopenhauer, 1851/1970, p. 42.

[14] Popper, 1945/2011, note 2 to chapter 9.

[15] https://www.hedweb.com/negutil.htm

[16] Popper, 1945/2011, note 6 to chapter 5.

[17] Some version of the concept of moksha is central to most of the well-known Eastern traditions, such as Buddhism (nirvana), Hinduism, Jainism, and Sikhism (mukti).

[18] https://foundational-research.org/tranquilism/
Thus, the view is not that happiness is literally the absence of suffering, which is, of course, patently false — insentient rocks are obviously not happy — but rather that the value of happiness lies in its absence of suffering.

[19] It should be noted, however, that one need not hold this tranquilist view of value in order to agree with Popper’s and Pearce’s position. For example, one can also view happiness as being strictly more valuable than nothing, while still maintaining that the value of raising the happiness of the already happy is always less than the value of reducing suffering. An intuitive way of formalizing this view would be by representing the value of states of suffering with negative real numbers, while representing the value of states of pure happiness with hyperreal numbers greater than 0, yet smaller than any positive real number, allowing us to assign some states of pure happiness greater value than others. On tranquilism, by contrast, all states of (pure) happiness would be assigned exactly the value 0.

[20] James, 1901.

[21] The terms “positive” and “negative” here respectively refer to the presence and absence of something.

[22] Schopenhauer, 1851/1970, p. 41.

[23] https://foundational-research.org/tranquilism/

[24] I happen to disagree with Gloor’s particular formulation of tranquilism when he writes: “According to tranquilism, a state of consciousness is negative or disvaluable if and only if it contains a craving for change.” For it seems to me that even intense cravings for change (for a different sex position, say) can feel perfectly fine and non-negative; that euphoric desire, say, is not an oxymoron. The term “negative cravings” avoids this complication.

[25] There are various versions of this example. A common one is whether it can be right to make gladiators fight for the enjoyment of a full colosseum, which is often raised as a problematic question for (certain versions of) utilitarianism.

[26] Guin, 1973/1992.

[27] And even though many will probably insist that the child’s suffering is a worthy sacrifice, the fact that it only takes a single life of misery to bring the value of a whole paradisiacal city into serious question, as it seems to do for most people, is yet another strong hint that there is an asymmetry between the (dis)value of happiness and suffering.

[28] Camus, 1947/1991, p. 224.

[29] Cf. Benatar, 2006, chapter 3.

[30] See section 2.2.14 here https://www.utilitarianism.com/nu/nufaq.html as well as http://www.simonknutsson.com/thoughts-on-ords-why-im-not-a-negative-utilitarian

[31] Leighton, 2011, p. 96.

[32] http://www.simonknutsson.com/the-world-destruction-argument/

In Defense of Nuance

The world is complex. Yet most of our popular stories and ideologies tend not to reflect this complexity. Which is to say that our stories and ideologies, and by extension we, tend to have insufficiently nuanced perspectives on the world.

Indeed, falling into a simple narrative through which we can easily categorize and make sense of the world — e.g. “it’s all God’s will”; “it’s all class struggle”; “it’s all the muslims’ fault”; “it’s all a matter of interwoven forms of oppression” — is a natural and extremely powerful human temptation. And something social constructivists get very right is that this narrative, the lens through which we see the world, influences our experience of the world to an extent that is difficult to appreciate.

So much more important, then, that we suspend our urge to embrace simplistic narratives to (mis)understand the world through. In order to navigate wisely in the world, we need to have views that reflect its true complexity; not views that merely satisfy our need for simplicity (and social signaling; more on this below). For although simplicity can be efficient, and to some extent is necessary, it can also, when too much too relevant detail is left out, be terribly costly. And relative to the needs of our time, I think most of us naturally err on the side of being expensively unnuanced, painting a picture of the world with far too few colors.

Thus, the straightforward remedy I shall propose and argue for here is that we need to control for this. We need to make a conscious effort to gain more nuanced perspectives. This is necessary as a general matter, I believe, if we are to be balanced and well-considered individuals who steer clear of self-imposed delusions and instead act wisely toward the betterment of the world. Yet it is also necessary for our time in particular. More specifically, it is essential in addressing the crisis that human conversation seems to be facing in the Western world at this point in time. A crisis that largely seems the result of an insufficient amount of nuance in our perspectives.

Some Remarks on Human Nature

There are certain facts about the human condition that we need to put on the table and contend with. These are facts about our limits and fallibility which should give us all pause about what we think we know — both about the world in general as well as ourselves in particular.

For one, we have a whole host of well-documented cognitive biases. There are far too many for me to list them all here, yet some of the most important ones are: confirmation bias (the tendency of our minds to search for, interpret, and recall information that confirms our pre-existing beliefs); wishful thinking (our tendency to believe what we wish were true); and overconfidence bias (our tendency to have excessive confidence in our own beliefs; in one study, people who reported to be 100 percent certain about their answer to a question were correct less than 85 percent of the time). And while we can probably all recognize these pitfalls in other people, it is much more difficult to realize and admit that they afflict ourselves as well. In fact, our reluctance to realize this is itself a well-documented bias, known as the bias blindspot.

Beyond realizing that we have fallible minds, we also need to realize the underlying context that has given rise to much of this fallibility, and which continues to fuel it, namely: our social context — both the social context of our evolutionary history as well as of our present condition. We humans are deeply social creatures, and it shows at every level of our design, including the level of our belief formation. And we need to be acutely aware of this if we are to form reasonable beliefs with minimal amounts of self-deception.

Yet not only are we social creatures, we are also, by nature, deeply tribal creatures. As psychologist Henri Tajfel showed, one need only assign one group of randomly selected humans the letter “A” and another randomly selected group the letter “B” in order for a surprisingly strong in-group favoritism to emerge. This method for studying human behavior is known as the minimal group paradigm, and it shows something about us that history should already have taught us a long time ago: that human tribalism is like gasoline just waiting for a little spark to be ignited.

This social and tribal nature of ours has implications for how we act and what we believe. It is, for instance, largely what explains the phenomenon of groupthink, which is when our natural tendency toward (in-)group conformity leads to a lack of dissenting viewpoints among individuals in a given group, which then, in turn, leads to poor decisions by these individuals.

Indeed, our beliefs about the world are far more socially influenced than we realize. Not just in the obvious way that we get our views from others around us — often without much external validation or testing — but also in that we often believe things in order to signal to others that we possess certain desirable traits, or that we are loyal to them. This latter way of thinking about our beliefs is quite at odds with how we prefer to think about ourselves, yet the evidence for this unflattering view is difficult to deny at this point.

As authors Robin Hanson and Kevin Simler argue in their recent book The Elephant in the Brain, we humans are strategically self-deceived about our own motives, including when it comes to what motivates our beliefs. Beliefs, they argue, serve more functions than just the function of keeping track of what is true of the world. For while beliefs surely do have this practical function, they also often serve a very different, very social function, which is to show others what kind of person we are and what kind of groups we identify with. This makes beliefs much like clothes, which have the practical function of keeping us warm while, for most of us, also serving the function of signaling our taste and group affiliations. And one of Hanson’s and Simler’s essential points is that we are not aware of the fact that we do this, and that there is an evolutionary reason for this: if we realized (clearly) that we believe certain things for social reasons, and if we realized that we display our beliefs with overconfidence, we would be much less convincing to those we are trying to convince and impress.

Practical Implications of Our Nature

This brief survey of the natural pitfalls and fallibilities of our minds is far from exhaustive, of course. But it shall suffice for our purposes. The bottom line is that we are creatures who naturally want our pre-existing beliefs confirmed, and who tend to display too high levels of confidence about these beliefs. We do this in a social context, and many of the beliefs we hold serve non-epistemic functions within this context, which include the tribal function of showing others how loyal we are to certain groups, as well as how worthy we are as friends and mates. In other words, we have a natural pull to impress our peers, not just with our behavior but also with our beliefs. And, for socially strategic reasons, we are quite blind to the fact that we do this.

So what, then, is the upshot of all of this? It is clear, I submit, that these facts about ourselves do have significant implications for how we should comport ourselves. In short, they imply that we have a lot to control for if we aspire to have reasonable beliefs — and our own lazy mind, with all its blindspots and craving for simple comfort, is not our friend in this endeavor. The fact that we are naturally biased and tendentious implies that we should doubt our own beliefs and motives. And it implies that we need to actively seek out the counter-perspectives and nuance that our confirmation bias, this vile bane of reason, so persistently struggles to keep us from accessing.

Needless to say, these are not the norms that govern our discourse at this point in time. Sadly, what plays out right now is mostly the unedited script of tribal, confirmation biasing human nature, unfazed by the prefrontal interventions that seem just about the only hope for our rewriting this script into something better.

The Virtues of the Good Conversationist

Let us elaborate a bit on the implications of our fallibility, and the precepts we should follow if we want to control for these unflattering tendencies and pitfalls of human nature. Recall the study cited above: people who reported to be 100 percent certain about their answer to a question were correct less than 85 percent of the time. The fact that we can be so wrong — more than 15 percent of the time when we claim perfect certainty(!) — implies, among other things, that when someone tells us we are wrong, we seem to have a prima facie reason to listen and try our best to understand what they are saying, as they may just be right. Of course, the tendency toward overconfidence will all but surely be shared by this other person as well, who could also be wrong. And our task then lies in finding out which it is. This is the importance of conversation. It is nothing less than the best tool we have, collectively, against being misguided. And that is why we have to become good conversationists.

What does it take to become that? At the very least, it requires an awareness of our biases, and a deliberate effort to counteract them.

Countering Confirmation Bias

To counteract our confirmation bias, we need to loosen our attachment to pre-existing beliefs, and to seek out viewpoints and arguments that may contradict them. The imperative of doing this derives from nothing less than the basic epistemic necessity of taking all relevant data into consideration rather than a small cherry-picked selection. For the truth is that we all cherry-pick data a little bit here and there in favor of our own position, and so by hearing from people with opposing views, and by examining their cherry-picked data and their particular emphasis and interpretation, we will, in the aggregate, tend to get a more balanced picture of the issue at hand.

And, importantly, we should strive to engage with these other views in a charitable way: by assuming good faith on behalf of the proponents of any position; by trying to understand their view as well as possible; and by then engaging with the strongest possible version of that position (i.e. the steel man rather than the straw man version of it). Indeed, it is difficult to overstate just how much the state of human conversation would improve if we all just followed this simple precept: be charitable.

Countering Wishful Thinking

Our propensity for wishful thinking should make us skeptical of beliefs that are convenient and which match up with what we want to be true. If we want there to be a God, and we believe there is one, then this should make us at least a little skeptical of this convenient belief. By extension, our attraction toward the wishful also implies that we should pay more attention to information and arguments that suggest conclusions which are inconvenient or otherwise contrary to what we wish were true. Do we believe the adoption of a vegan lifestyle would be highly inconvenient for us personally? Then we should probably expect to be more than a little biased against any argument in its favor, and indeed, if we suspect the argument has merit, be inclined to ignore it altogether rather than giving it a fair hearing.

Countering Overconfidence Bias

When it comes to correcting for our overconfidence bias, the key virtue to embrace is intellectual humility (or at least so it seems to me). That is, to admit and speak as though we have a limited and fallible perspective on things. In this respect, it also helps to be aware of the social factors that might be driving our overconfidence much of the time. As noted above, we often express certainty in order to signal to third parties, as well as to instill strong doubts in those we engage with. And we do this without being aware of it. This social function of confidence should lead us to update away from bravado and toward being more measured. Again: to be intellectually humble.

Countering In-Group Conformity

Another way in which social forces make us less than reasonable is by compelling us to conform to our peers. As hinted above, our beliefs are subject to in-group favoritism, which highlights the importance of being (especially) skeptical of the beliefs we share with groups that we affiliate closely with, and to practice playing the devil’s advocate against these beliefs. And, by extension, to try to be extra charitable toward the beliefs held by the notional out-group, whether it be “the Left” or “the Right”, “the religious” or “the atheists”.

Beyond that, we should also be aware that our minds likely often paint the out-group in an unfairly unfavorable light, viewing them as much less sincere and well-intentioned — one may even say more evil — than they actually are, however misguided (we may think) their particular views are. And it seems a natural temptation for us to try to score points by publicly broadcasting such a negative view of the out-group as a way of showing our in-group just how unlikely we are to change affiliation.

Thinking in Degrees of Certainty

It seems that we have a tendency to express our views in a very binary, 0-or-1 fashion. We tend to be either clearly for something or clearly against it, be it abortion, efforts to prevent climate change, the death penalty, or universal health care. And it seems to me that what we express outwardly is generally much more absolutist, i.e. more purely 0 or 1, than what happens inwardly, under the hood — perhaps even underneath our conscious awareness — where there is probably more conflicting data than what we are aware of and allow ourselves to admit.

I have observed this pattern in conversations: people will argue strongly for a given position which they continue to insist on, until, quite suddenly it seems, they say that they accept the opposite conclusion. In terms of their outward behavior, they went from 0 to 1 quite rapidly, although it seems likely that the process that took place underneath the hood was much more continuous — a more gradual move from 0 to 1, where the signal “express 1 now” was then passed at some threshold.

An extreme example of similar behavior found in recent events is that of Omarosa Manigault Newman, who was the so-called Director of African-American Outreach for Donald Trump’s presidential campaign in 2016. She went from describing Trump in adulating terms and calling him “a trailblazer on women’s issues”, to being strongly against him and calling him a racist and a misogynist. It seems unlikely that this shift was based purely on evidence she encountered after she made her adulating statements. There probably was a lot of information in her brain that contradicted the claim of Trump’s status as such a trailblazer, but which she ignored and suppressed. And the reason why is quite obvious: she had a political aim. She needed to broadcast the message that Trump was a good person to further a campaign and to further her own career tied to this campaign. It was about signaling first, not truth-tracking (which is not to say that she did not sincerely believe what she said, but her sincere belief was probably just conveniently biased).

The important thing to realize, of course, is that this applies to all of us. We are all inclined to be more like a politician than a scientist in many situations. In particular, we are all inclined to believe and express either a pure 0 or a pure 1 for social reasons. And the nature of these social reasons may vary. It may be about signaling opposition to someone who believes the opposite, or about signaling loyalty to a given group (few groups rally around low-credence claims). It may also be about signaling that we have a mind that is of a strong conviction. After all, doubt is generally not sexy. Just consider the words we usually associate with it, such as uncertainty, confusion, and indecision. Certainty, on the other hand, signals strength, and is commonly associated with more positive words such as decisiveness, confidence, resoluteness, and firmness. And so, for this reason as well, it only seems natural that we would generally be inclined to signal certainty rather than doubt, even when we do not possess anything close to justified certainty.

Fortunately, there exists a corrective for our tendency toward 0-or-1 thinking, which is to think in terms of credences along a continuum, ranging from 0 to 1. For one, this would constitute a more honest form of communication, in that it would force us to carefully weigh all the information that our brain keeps hidden from us, as well as to express its underlying credence in detail — as opposed to merely expressing whether this credence has crossed some given threshold. Yet perhaps even more significantly, thinking in terms of such a continuum would also help subvert the tribal aspect of our either-or thinking by placing us all in the same boat: the boat of degrees of certainty, in which the only thing that differs between us is our level of certainty in any given claim. For example, think how strange it would be for a religious believer to present their religious beliefs by saying that their credence in the existence of a God lies around 93 percent. This is a much weaker statement, in terms of its social signaling function, than a statement such as “I am a Christian”.

Such an honest, more detailed description of one’s beliefs is not good for keeping groups divided by different beliefs. Indeed, it is good for the exact opposite: it helps us move toward a more open and sincere conversation about what we in fact believe and why, regardless of our group affiliations.

Different Perspectives Can Be Equally True

There are two common definitions of the term “perspective” that are quite different, yet closely related at the same time. One is “a mental outlook/point of view”, while the other is “the art of representing three-dimensional objects on a two-dimensional surface”. And they are related in that the latter can be viewed as a metaphor for the former: our particular perspective, the representation of the world we call our point of view, is in a sense a limited two-dimensional representation of a more complex, multi-dimensional reality. A representation that is bound to leave out a lot of information about this reality. The best we can do, then, is to try to paint the two-dimensional canvas that is our mind so as to make it as rich and informative as possible about the complex and many-faceted world we inhabit.

And an important point for us to realize in our quest for more balanced and nuanced views, as well as for the betterment of human conversation, is to realize that seemingly conflicting reports of different perspectives on the same underlying reality can in fact all be true, as hinted by the following illustrations:

 

Image result for perspective

Image result for perspective

Image result for perspective perceptual

 

The same object can have very different reflections when viewed from different angles. Similarly, the same events can be viewed very differently by different people who each have their own unique dispositions and prior experiences. And these different views can all be true; John really does see X when he looks at this event, while Jane really does see Y. And, like the square- and circle-shaped reflections above, X and Y need not be incompatible. (A similar sentiment is reflected in the Jain doctrine of Anekantavada.)

And even when someone does get something wrong, they may nonetheless still be reporting the appearance of the world as it is revealed to them as honestly and accurately as they can. For example, to many of us, it really does seem as though the lines in the following picture are not parallel, although they in fact are:

 

Image result for visual illusions

 

Which is merely to state the obvious point that it is possible, indeed quite common, to be perfectly honest and wrong at the same time, which is worth keeping in mind when we engage with people whom we think are obviously wrong; they usually think they are right, and that we are obviously wrong — and perhaps even dishonest.

Another important point the visual illusion above hints at is that we should be careful not to confuse external reality with our representation of it. Our conscious experience of the external world is not, obviously, the external world itself. And yet we tend to speak as though it were; as though our experience of the external world were not a sophisticated representation, but instead the external world as it is in itself.

This is an evolutionarily adaptive illusion no doubt, but it is an illusion nonetheless. All we ever inhabit is, in the words of David Pearce, our own world simulation, a world of conscious experience residing in our head. And given that we all find ourselves stuck in — or indeed as — such separate, albeit mutually communicating bubbles, it is not so strange that we can have so many disagreements about what we think reality is like. All we have to go on is our own private, phenomenal cartoon model of each other and the world at large; a cartoon model that may get many things right, but which is also sure to miss a lot of important things.

Framing Shapes Our Perspective

From the vantage point of our respective world simulations, we each interpret information from the external world with our own unique framing. And this framing in part determines how we will experience it, as demonstrated by the following illustration, where one can change one’s framing so as to either see a duck or a rabbit:

 

Image result for duck rabbit

 

As well as the following illustration where one’s framing determines whether one sees a cube from above or below — or indeed just a two dimensional pattern without depth:

 

Image result for dual perspective picture

Sometimes, as in the examples above, our framing is readily alterable. In other cases, however, it can be more difficult to just switch our framing, as when it comes to how different people with different life experiences will naturally interpret the same scenario in very different ways. For instance, a physicist might enter a room and see a lot of interesting physical phenomena there. Air consisting of molecules which bounce around in accord with the laws of thermodynamics; sound waves that travel adiabatically across the room; long lamps dangling in their natural frequency while emitting photons. An artistic person, in contrast, may enter the same room and instead see a lot of people. And this person may view these people as a sea of flowing creative potential in the process of being unleashed, inspired by deeply emotional music and a warm glowing light that fits perfectly with the atmosphere of the music.

Although these two perspectives on the events of this same room are very different, none of them are necessarily wrong. Indeed, they seem perfectly compatible, despite their representing what seems to be two very different cognitive styles — two different paradigms of thinking and perceiving, one may say. And what is important to realize is that a similar story applies to all of us. We all experience the world in different ways, due to our differing biological dispositions, life experiences, and vantage points. And while these different experiences are not necessarily incompatible, it can nonetheless be difficult to achieve mutual understanding between such differing perspectives.

Acknowledging Many Perspectives Is Not a Denial of Truth

It should be noted, however, that none of the above makes a case for the relativistic claim that there are no truths. On the contrary, what the above implies is indeed that it is a truth — as hard and strong as could be — that different individuals can have different perspectives and experiences in reaction to the same external reality, and that it is possible for such differing perspectives to all have merit, even if they seem in tension with each other. And to acknowledge this fact by no means amounts to the illogical statement that no given perspective can ever be wrong and make false claims about reality — that, sadly, is clearly all too common. This middle-position of rejecting both the claim that there is only one valid perspective and the claim that there are no truths is, I submit, the only reasonable one on offer.

And the fact that there can be merit in a plurality of perspectives implies that, beyond conceiving of our credences along a continuum ranging from 0 to 1, we also need to think in terms of a diversity of continua in a more general sense if we are to gain a fuller, more nuanced understanding that does justice to reality, including the people around us with whom we interact. More than just thinking in terms of shades of grey found in-between the two endpoints of black and white, we need to think in terms of many different shades of many different colors.

At the same time, it is also important to acknowledge the limits of our understanding of other minds and experiences we have not had. This does not amount to some obscure claim about how we each have our own, wholly incommensurable experiences, and hence that mutual understanding between individuals with different backgrounds is impossible. Rather, it is simply to acknowledge that psychological diversity is real, which implies that we should be careful to avoid the so-called typical mind fallacy, as well as to acknowledge that at least some experiences just cannot be conveyed faithfully with words alone to those who have not had them. And this does, at the very least, pose a challenge to the endeavor of communicating with and understanding each other. For example, most of us have never tried experiencing extreme forms of suffering, such as the experience of being burned alive. And beyond describing this class of experiences with thin yet accurate labels such as “horrible” and “bad”, most of us are surely very ignorant — luckily for us.

However, this realization that we do not know what certain experiences are like is in fact itself an important insight that does help expand and advance our outlook. For it at least helps us realize that our own understanding, as well as the range and variety of experiences we are familiar with, are far from exhaustive. With this realization in mind, we can look upon a state of absolute horror and admit that we have virtually no understanding of just how bad it is, which, I submit, comprises a significantly greater understanding than does beholding it with both the same absence of comprehension, and the absence of the admission of this absent comprehension. The realization that we are ignorant itself constitutes knowledge of sorts. The kind of knowledge that makes us rightfully humble.

Grains of Truth in Different Perspectives

Even when two different perspectives indeed are in conflict with each other, this does not imply that they are necessarily both entirely wrong, as there can still be significant grains of truth in both of them. Most of today’s widely endorsed perspectives and narratives make a wide range of claims and arguments, and even if not all of these stand up to scrutiny, many of them often do, at least when modified slightly. And part of being charitable is to seek out such grains of truth in a position one does not agree with. This can also help us realize which truths and plausible claims that might motivate people to support (what we consider) misguided views, and thus help further mutual understanding among us. Therefore, this seems a reasonable precept to follow as well: sincerely ask what might be the grains of truth in the views you disagree with. One can almost always find something, and often a good deal more than one would naively have thought.

As mentioned earlier, it is also possible for different perspectives to support what seems to be very different positions on the same subject without necessarily being wrong in any way; if they have different lenses, looking in different directions. Indeed, different perspectives on the same issue are often merely the result of different emphases which each focus on certain framings and sets of data rather than others. And thus seemingly incompatible perspectives may in fact all be right about the particular aspects of a given subject that they emphasize, which is why it is important to seek out different treatments of the same subject from multiple angles. Oftentimes, it is not that novel perspectives show our current perspective wrong, but merely that it is insufficiently unnuanced — i.e. that we have failed to take certain things into account, such as alternative framings, particular kinds of data, and critical counter-considerations.

This is, I believe, a common pattern in human conversation, and another sense in which we should be mindful of the possible existence of different grains of truth, namely: when different views on the same subject are all completely true, yet where each of them merely comprise a small grain in the larger mosaic that is the complete truth. And hence we should remind ourselves, as stated in the illustration above, that just because we are right does not mean that the person who says something else on the same subject is wrong.

Having made a general case for nuance, let us now turn our eyes toward our time in particular, and why it is especially important to actively seek to be nuanced and charitable today.

Our Time Is Different

Every period in history likely sees itself as uniquely unique. Yet in terms of how humanity communicates, it is clear that our time indeed is a highly unique one. For never before in history has human communication been so screen-based as it is today. Or, expressed equivalently: never before has so much of our communication been without face-to-face interaction. And this has significant implications for how and what we communicate.

It is clear that our brains process communication through a screen in a very different way. Writing a message in a Facebook group consisting of a thousand people does not, for most of us, feel remotely the same as delivering the same message in front of a thousand people crowd. And a similar discrepancy between the two forms of communication is found when we interact with just a single person, which is no wonder. Communication through a screen consists of a string of black and white symbols. Face-to-face interaction, in contrast, is composed of multiple streams of information. We read off important cues from a person’s face and posture, as well as from the tone and pace of their voice.

All this information provides a much more comprehensive, one might indeed say more nuanced, picture of the state of mind of the person we are interacting with. We get the verbal content of the conversation (as we would through a screen), plus a ton of information about the emotional state of the other. And beyond being informative, this information also serves the purpose of making the other person relatable. It makes the reality of their individuality and emotions almost impossible to deny, which is much less true when we communicate through a screen.

Indeed, it is as though these two forms of communication activate entirely different sets of brain circuits. Not only in that we communicate via a much broader bandwidth and likely see each other as more relatable when we communicate face-to-face, but also in that face-to-face communication naturally motivates us to be civil and agreeable. When we are in the direct physical presence of someone else, we have a strong interest in keeping things civil enough to allow our co-existence in the same physical space. When we interact through a screen, however, this is no longer a necessity. The notional brain circuitry underlying peaceful co-existence with antagonists can more safely be put on stand-by mode.

The reality of these differences between the two forms of communication has, I would argue, some serious implications. First of all, it highlights the importance of being aware that these two forms of communication indeed are very different, and that we are, in various ways, quite handicapped communicators when we communicate through a screen, often entering a state of mind that perhaps only a sociopath would be able to maintain in a face-to-face interaction. A handicap that further implies that we should be even more aware of the tendencies reviewed above when interacting through a screen, as these tendencies then become much easier and more tempting to engage in. It is (even) more difficult to relate to those who disagree with us, and we have (even) less of an incentive to understand them properly and be civil. Which is to say that it is (even) more difficult to be charitable. Written communication through a screen makes it easier than ever before to paint the out-group antagonists we interact with in an unreasonably unfavorable light.

And our modern means of communication arguably also make it easier than ever before to not interact with the out-group at all, as the internet has made it possible for us to diverge into our own respective in-group echo chambers to an extent not possible in the past. It is therefore now easy to end up in communities in which we continuously echo data that supports our own narrative, which ultimately gives us a one-sided and distorted picture of reality. And while it may be easier than ever to find counter-perspectives if we were to look for them, this is of little use if we mostly find ourselves collectively indulging in our own in-group confirmation bias. As we often do. For instance, feminists may find themselves mostly informing each other about how women are being discriminated against, while men’s rights activists may disproportionally share and discuss ways in which men are discriminated against. And so by joining only one of these communities, one is likely to end up with a skewed, insufficiently nuanced view of reality.

This mode of interaction has serious sociological implications. Indeed, the change in our style of interaction brought about by the internet is probably in large part why, in spite of the promise technology seemed to hold to connect us with each other, we now appear increasingly balkanized, divided along various lines in ways that feed into our tribal nature all too well. Democrats and republicans, for example, increasingly see each other as a “threat to the nation’s well-being” — significantly more so than they did even just ten years ago. This is a real problem that does not seem to be going away on its own. And one of the greatest hopes we have for improving this situation is, I submit, to become aware of and actively try to control for our own pitfalls. Especially when we interact through screens.

With all the information we have reviewed thus far in mind, let us now turn to some concrete examples of heated issues that divide people today, and where more nuanced perspectives and a greater commitment to being charitable are desperately needed. (I should note, however, that given the brevity of the following remarks, what I write here on these issues is, needless to say, itself bound to fail to express a highly nuanced perspective, as that would require a longer treatment. Nonetheless, the following brief remarks will at least gesture at some ways in which we can generally be more nuanced about these topics.)

Sex Discrimination

As hinted above, there are two groups that seem to tell very different stories about the state of sex discrimination in our world today. On the one hand there are the feminists, who seem to argue that women generally face much more discrimination than men, and on the other, there are the so-called men’s rights activists, who seem to argue that men are, at least in some parts of the world, generally the more discriminated sex. And these two claims surely cannot both be right, can they?

If one were to define sex discrimination in terms of some single general measure, a “General Discrimination Factor”, then no, they could not both be right. Yet if one instead talks about concrete forms of discrimination, then it is entirely possible, and indeed clearly the case, that women are discriminated against more than men in some respects, while men face more discrimination in other respects. And it is arguably also much more fruitful to talk about such concrete cases than it is to talk about discrimination “in general”. (In response to those who insist that it is obvious that women face more discrimination everywhere, almost regardless of how one constructs such a general measure, I would recommend watching the documentary The Red Pill, and, for a more academic treatment, reading David Benatar’s The Second Sexism.)

For example, it is a well-known fact that women have, historically, been granted the right to vote much later than men have, which undeniably constitutes a severe form of discrimination against women. Similarly, women have also historically been denied the right to take a formal education, and they still are in many parts of the world. In general, women have been denied many of the opportunities that men have had, including access to professions in which they were clearly more than competent to contribute. These are all undeniable facts about undeniably severe forms of discrimination.

However, tempting as it may be to infer, none of this implies that men have not also faced severe discrimination in the past, nor that they evade such discrimination today. For example, it is generally only men who have been subject to conscription — i.e. forced duty to enlist for state service, such as in the military. Historically, as well as today, men have been forced by law to join the military and go to war, often without returning — whether they wanted to or not (sure, some men wanted to join the military, yet the fact that some men wanted to do this does not imply that making it compulsive for virtually all men and only men is not discriminatory; as a side note, it should be noted that many feminists have criticized conscription).

Thus, at a global level, it is true to say that, historically as well as today, women have generally faced more discrimination in terms of their rights to vote and pursue an education, as well as in their professional opportunities in general, while men have faced more discrimination in terms of state-enforced duties.

Different forms of discrimination against men and women are also present at various other levels. For example, in one study where the same job application was sent to different scientists, and where half of the applications had a female name on them, the other half a male name, the “female applicants” were generally rated as less competent, and the scientists were willing to pay the “male applicants” more than 14 percent more.

The same general pattern seems reported by those who have conducted a controlled experiment in being a man and a women from “the inside”, namely from transgender men (those who have transitioned from being a woman to being a man). Many of these men report being viewed as more competent after their transition, as well as being listened to more and interrupted less. This also fits with the finding that both men and women seem to interrupt women more than they interrupt men.

At the same time, many of these transgender men also generally report that people seem to care less about them now that they are men. As one transgender man wrote about the change in his experience:

What continues to strike me is the significant reduction in friendliness and kindness now extended to me in public spaces. It now feels as though I am on my own: No one, outside of family and close friends, is paying any attention to my well-being.

Such anecdotal reports seem well in line with the finding that both men and women show more aggression toward men than women, as well as with recent research (see page 137) conducted by social psychologist Tania Reynolds, which among other things found that:

[…] female harm or disadvantage evoked more sympathy and outrage and was perceived as more unfair than equivalent male harm or disadvantage. Participants more strongly blamed men for their own disadvantages, were more supportive of policies that favored women, and donated more to a female-only (vs male-only) homeless shelter. Female participants showed a stronger in-group bias, perceiving women’s harm as more problematic and more strongly endorsed policies that favored women.

As these examples show, it seems that men and women are generally discriminated against in different ways. And it is worth noting that these different forms of discrimination are probably in large part the natural products of our evolutionary history rather than some deliberate, premeditated conspiracy (which is obviously not to say that they are ethically justified).

Yet deliberation and premeditation is exactly what is required if we are to step beyond such discrimination. More generally, what seems required is that we get a clearer view of the ways in which women and men face discrimination, and that we then take active steps toward remedying these problems. Something that is only possible if we allow ourselves enough of a nuanced perspective to admit that both women and men are subject to serious discrimination and injustice.

Intersectionality

It seems that many progressives are inspired by the theoretical framework called intersectionality, according to which we should seek to understand many aspects of the modern human condition in terms of interlocking forms of power, oppression, and privilege. One problem with relying on this framework is that it can easily become like only seeing nails when all one has is a hammer. If one insists on understanding the world predominantly in terms of oppression and social privilege, one risks seeing it in many places where it is not, as well as overemphasizing its relevance in many cases — and, by extension, to underemphasize the importance of other factors.

As with most popular ideas, there is no doubt a significant grain of truth in some of what intersectional theory talks about, such as the fact that discrimination is a very real phenomenon, that privilege is too, and that both of these phenomena can compound. Yet the narrow focus on only social explanations and versions of these phenomena means that intersectional theory misses a lot about the nature of discrimination and privilege. For example, some people are privileged to be born with genes that predispose them to be very happy, while others have genes that dispose them to have chronic depression. Such two people may be of the same race, gender, and sexuality, and they may be equally able-bodied. Yet such two people will most likely have very different opportunities and quality of life. A similar thing can be said about genetic differences that predispose individuals to have a higher or lower IQ, as well as about genetic differences that make people more or less physically attractive.

Intersectional theory seems to have very little to say about such cases, even as these genetic factors seem able to impact opportunities and quality of life to a similar degree as discrimination and social exclusion. Indeed, it seems that intersectional theory actively ignores, or at the very least underplays, the relevance of such factors — what may be called biological privileges in general — perhaps because they go against the tacit assumption that inequity and other bad things must be attributable to an oppressive agent or social system in some way, as opposed to just being the default outcome one should expect to find in an apathetic universe.

In general, it seems that intersectional theory significantly underestimates the importance of biology, which is, of course, by no means a mistake that is unique to intersectionality in particular. And it is indeed understandable how such an underestimation can emerge. For the truth is that many of the most relevant human traits, including those of personality and intelligence, are strongly influenced by both genetic and environmental factors. Indeed, around 40-60 percent of the variance of such traits tends to be explained by genetics, and, consequently, the amount of variance explained by the environment lies roughly in this range as well. This means that, with respect to these traits, it is both true to say that cultural factors are extremely significant, and to say that biological factors are extremely significant. And the mistake that many seem to make, including many proponents of intersectionality, is to believe that one of these truths rules out the other.

Another critique one can direct toward intersectional theory is that it often makes asymmetrical claims about how one group, “the privileged”, are unable to understand the experiences of another group of individuals, “the unprivileged”, whatever form the privilege and lack thereof may take. Yet it is rarely conceded that this argument can also, with roughly as much plausibility, be made the other way around: that the (allegedly) unprivileged might not fully understand the experience of the (allegedly) privileged, and that they may, in effect, overstate the differences in their experience, and overstate how easy the (allegedly) privileged in fact have it. A commitment to intellectual openness and honesty would at least require us to not dismiss this possibility out of hand.

A similar critique that intersectional theorists ought to contend with is that some of the people whom intersectional theory maintains are discriminated against and oppressed themselves argue that they are not, and some indeed further argue that many of the solutions and practical steps supported by intersectional theorists are often harmful rather than beneficial. Such voices must, at least, be counted as weak anomalies relative to the theory, and considered worthy of serious engagement.

More generally, a case can be made that intersectional theory greatly overemphasizes group membership and identities in its analyses of and attempts to address societal problems. As Brian Tomasik notes:

[…] I suspect it’s tempting for our tribalistic primate brains to overemphasize identity membership and us-vs.-them thinking when examining social ills, rather than just focusing on helping people in general with whatever problems they have. For example, I suspect that one of the best ways to help racial minorities in the USA is to reduce poverty (such as through, say, universal health insurance), rather than exploring ever more intricate nuances of social-justice theory.

A regrettable complication that likely bolsters the focus of intersectionalists is that many people seem to flatly deny that there are any grains of truth to any of the claims intersectional theory makes. Some claim, for instance, that there is no such thing as being transgendered, and that there barely is such a thing as racial or sex discrimination in the Western world today. Rather than serving as a meaningful critique of the overreaches of intersectionality, such unnuanced and ill-informed statements seem likely to only help convince intersectionalists that they are uniquely right while others are dangerously wrong, as well as to suggest to them that more radical tactics may be needed, since current tactics clearly do not work to make other people see basic reality for what it is.

This speaks to the more general point that if we make measured views a rarity, and convince ourselves that all one can do is join either team A or team B — e.g. “camp discrimination exists” or “camp discrimination does not exist” — then we only push people toward division. We risk finding ourselves in a run-away spiral where people try to eagerly signal that they do not belong to the other team, which may in turn push us toward ever more extreme views. The alternative option to this tribal game is to simply aspire toward, and express, measured and nuanced views. That might just be the best remedy against such polarization and toward reasonable consensus. Whether our tribal brains indeed want such a consensus is, of course, a separate question.

A final critique I would direct at mainstream intersectional theory is that, despite its strong focus on unjustified discrimination, it nonetheless generally fails to acknowledge and examine what is, I have argued, the greatest, most pervasive, and most harmful form of discrimination that exists today, namely: speciesism, the unjustified discrimination against individuals based on their species membership. The so-called argument from species overlap is rarely examined, nor are the implications that follow, including when it comes to what equality in fact entails. This renders mainstream versions of intersectionality, as a theory of discrimination against vulnerable individuals, a complete failure.

Political Correctness

Another controversial issue closely related to intersectionality is that of political correctness. What do we mean by political correctness? The answer is actually not straightforward, since the term has a rather complex history throughout which it has had many different meanings. Yet one sense of the term that was at least prominent at one point refers simply to conduct and speech that embodies fairness and common decency toward others, especially in a way that avoids offending particular groups of people. In this sense of the term, political correctness is about not referring to people with ethnic slurs, such as “nigger” and “paki”, or homophobic slurs, such as “faggot” and “dyke”. A more recent sense of the term, in contrast, refers to instances where such a commitment to not offend people has been taken too far (in the eyes of those who use the term), which is arguably the sense in which it is most commonly used today.

This then leads us to what seems the quintessential point of contention when it comes to political correctness, namely: what is too far? What does the optimum level of decency entail? And the only reasonable answer, I believe, will have to be a nuanced one found between the two extremes of “nothing is too offensive” and “everything is too offensive”.

Some seem to approach this subject with the rather unnuanced attitude that feelings of being offended do not matter in any way whatsoever. Yet this view seems difficult to maintain, however, at least if one is called a pejorative name in an unjoking manner oneself. For most people, such name-calling is likely to hurt — indeed, it can easily hurt quite a lot. And significant amounts of hurt and unpleasantness do, I submit, matter. A universe with fewer, less intense feelings of offense is, other things being equal, better than a universe with more, more intense feelings of offense.

Yet the words “other things being equal” should not be missed here. For the truth is that there can be, indeed there clearly is, a tension between 1) risking to offend people and 2) talking freely and honestly about the realities of life. And it is not clear what the optimal balance is.

Yet what is quite clear, I would argue, is that if we cannot talk in an unrestricted way about what matters most in life, then we have gone too far. In particular, if we cannot draw distinctions between different kinds of discrimination and forms of suffering, and if we are not allowed to weigh these ills against each other to assess which are most urgent, then we have gone too far. For if we deny ourselves a clear sense of proportion with respect to the problems of the world, we end up undermining our ability to sensibly prioritize our limited resources in a world that urgently demands reasonable prioritization. And this is, I submit, much too high a price to pay to avoid the risk of offending people.

Relationship Styles and Promiscuity

Another subject that a lot of people seem to express quite strong and unnuanced positions on is that of sexual promiscuity and relationship styles. For example, some claim that strict monogamy is the only healthy and viable choice for everybody, while others seem to make more or less the same claim about polyamory: that most people would be happier if they were in loving, sexual relationships with more than one person, and that only our modern culture prevents us from realizing this. Similar opinions can be found on the subject of casual sex. Some say it is not a big deal, while others say it is — for everyone.

An essential thing to acknowledge on this subject, it seems, is the reality of individual differences. Most of these strong opinions seem to arise from the fallacious assumption that other people are significantly much like ourselves — i.e. the typical mind fallacy. The truth is that some may well thrive best in monogamous relationships, while others may thrive best in polyamorous relationships; some may well thrive having casual sex, some may not. And in the absence of systematic studies, it is difficult to say which distribution people fall along in these respects — in terms of what circumstance people thrive best in — as well as how much this distribution can be influenced by culture.

None of this is to say that there is no such thing as human nature when it comes to sexuality, but merely that it should be considered an open question just what this nature is exactly, and how much plasticity and individual variation it entails. And we should all admit this much.

Politics and Making the World a Better Place

The subjects of politics and “how to make the world a better place” more generally are both subjects on which people tend to have strong convictions, limited nuance, and powerful incentives to signal group loyalty. Indeed, they are about as good examples as any of subjects where it is important to be charitable and actively seek out nuance, as well as to acknowledge one’s own biased nature.

A significant step we can take toward thinking more clearly about these matters is to adopt the aforementioned virtue of thinking in terms of continuous credences. Just as the expression of a “merely” high credence in the existence of the Christian God is more conducive to open-minded conversation, so is having a “merely” high credence in any given political ideology, principle, or policy likely more conducive to honest and constructive conversations and greater mutual updating.

If nothing else, the fact that the world is so complex implies that we will at least have considerable uncertainty about what the consequences of our actions will be. In many cases, we simply cannot know with great certainty which policy or candidate that is going to be best (relative to any set of plausible values) all things considered. This suggests that our strong convictions about how a given political candidate or policy is all bad, and about how immeasurably greater the alternatives would be, are likely often overstated. More generally, it implies that our estimates of which actions that are best to take, in the realm of politics in particular as well as with respect to improving the world in general, should probably be more measured and humble than they tend to be.

For example: what is your credence that Donald Trump was a better choice (with respect to your core values) than Hillary Clinton for the US presidency in 2016? I suspect most people’s credence on this question is either much too low or much too high relative to what can be justified. For even if one thinks his influence is clearly positive or clearly negative in the short term, this still leaves open the question of what the long-term effects will be. If the short-term effects are negative, for instance, it does not seem entirely implausible that there will be a counter-reaction in the future whose effects will end up being better in the long term, or vice versa. This consideration alone should dampen one’s credence somewhat — away from the extremes and closer toward the middle. A similar argument could be made about grave atrocities and instances of extreme suffering occurring today and in the near future: although it seems unlikely, we cannot exclude that these may in fact lead to a future with fewer atrocities and less suffering in the long term. (Note, however, that none of this implies that one should not fight hard for what one believes to be the best thing; even if one has only, say, a 60 percent credence in some action being better than another, it can still make perfect sense to push very hard for this seemingly better option.)

Or, to take another concrete example: would granting everyone a universal basic income be better (relative to your values) than not doing so? Again, being absolutely certain in either a positive or a negative answer to this question is hardly defensible. More reasonable, it seems, would it be to maintain a credence that lies somewhere in-between. (And in relation to what one’s underlying values are, I would argue that this is one of the very first things we need to reflect upon if we are to make a reasonable effort toward making the world a better place.)

A similar point can be made about existing laws and institutions. When we are young and radical, we have a tendency to find existing laws and social structures to be obviously stupid compared to the brilliant alternatives we ourselves envision. Yet, in reality, our knowledge of the roles played by these existing systems, as well as the consequences of our proposed alternatives, will tend to be quite limited in most cases. And it seems wise to admit this much, and to adjust our credences and plans of action accordingly.

A related pitfall worth avoiding is that of believing a single political candidate or policy to have purely good or purely bad effects; such an outcome seems extraordinarily unlikely. In the words of economist Thomas Sowell, there are no perfect solutions in the real world, only trade-offs. Similarly, it is also worth steering clear of the tendency to look to a single intellectual for the answers to all important questions. For the truth is that we all have blindspots and false beliefs, and virtually everyone is going to be ignorant of things that others would consider common knowledge. Indeed, no single person can read and reflect widely and deeply enough to be an expert on everything of importance. Expertise requires specialization, which means that we must look to different experts if we are to find expert views on a wide range of topics. In other words, the quest for a more complete and nuanced outlook requires us to engage with many different thinkers from very different disciplines.

The preceding notes about ways in which we could be more nuanced on various concrete topics are, of course, merely scratching the surface. Yet they hopefully do serve to establish the core point that nuance is essential if we are to gain a balanced understanding of virtually any complicated issue.

Can We Have Too Much Nuance?

In a piece that argues for the virtues of being nuanced, it seems worth asking whether I am being too one-sided. Might I not be overstating the case in its favor, and should I not be a bit more nuanced about the utility of nuance itself? Indeed, might we not be be able to have too much nuance in some cases?

I would be the first to admit that we probably can have too much nuance in many cases. I will grant that in situations that call for quick action, and where there is not much time to build a nuanced perspective, it may well often be better to act on one’s limited understanding rather than a more nuanced, yet harder-won picture. There are many situations like this, no doubt. Yet at the level of our public conversations, this is not often the case. We usually do have time to build a more nuanced picture, and we are rarely required to act promptly. Indeed, we are rarely required to act at all. And, unthinkable as it may seem, it could just be that expressions of agnosticism, and perhaps no public expressions at all on a given hot topic, would tend to serve everyone better than expressions of poorly considered views.

One could perhaps attempt to make a case against nuance with reference to examples where near-equal weight is granted to all considerations and perspectives — reasonable and less reasonable ones alike. This, one may argue, is a bad thing, and surely demonstrates that there is such a thing as too much nuance. Yet while I would agree that weighing arguments so blindly and undiscerningly is unreasonable, I would not consider this an example of too much nuance as such. For being nuanced does not mean giving equal weight to all arguments a posteriori, after all the relevant arguments have been presented. Instead, what it requires is that we at least consider these relevant arguments, and that we strive to be minimally prejudiced toward them a priori. In other words, the quest for appropriately nuanced perspectives demands us to grant equality of opportunity to all arguments; not equality of outcome.

Another objection one may be tempted to raise against being nuanced and charitable is that it implies that we should be submissive and over-accommodating. This does not follow, however. For to say that we should be charitable is not to say that we cannot be firm in our convictions when such firmness is justified, much less that we should ever tolerate disrespect or unfair treatment; we should not. We have no obligation to tolerate bullies and intimidators, and if someone repeatedly fails to act in a respectful, good-faith manner, we have every right, and arguably even good reason, to remove ourselves from them. After all, the maxim “assume the other person is acting in good faith” does not entail that we should not update this assumption as soon as we encounter evidence that contradicts it. And to assert one’s boundaries and self-respect in light of such updating is perfectly consistent with a commitment to being charitable.

A more plausible critique against being nuanced is that it might in some cases be strategically unwise, and that one should instead advocate for one’s views in an unnuanced, polemic manner in order to better achieve one’s objectives, at least in some cases. I think this is a decent point. Yet I think there are also good reasons to think that this will rarely be the optimal strategy when engaging in public conversations. First of all, we should acknowledge that, even if we were to grant this style of communication superior in a given situation, it still seems advantageous to possess a nuanced understanding of the counter-arguments. For, if nothing else, such an understanding would seem to make one better able to rebut these arguments, regardless of whether one then does so in a nuanced way or not.

And beyond this reason to acquire a nuanced understanding, there are also very good reasons to express such an understanding, as well as to treat the counter-arguments in as fair and measured a way as one can. One reason is the possibility that we might ourselves be wrong, which means that, if we want an honest conversation through which we can make our beliefs converge toward what is most reasonable, then we ourselves also have an interest in seeing the best and most unbiased arguments for and against different views. And hence we ourselves have an interest in moderating our own bravado and confirmation bias which actively keep us from evaluating our pre-existing beliefs as impartially as we should, as well as an interest in trying to express our own views in a measured and nuanced fashion.

Beyond that, there are also reasons to believe that people will be more receptive to one’s arguments if one communicates them in a way that demonstrates a sophisticated understanding of relevant counter-perspectives, and which lays out opposing views as strongly as possible. This will likely lead people to conclude that one’s perspective is at least built in the context of a sophisticated understanding, and it might thus plausibly be read as an honest signal that this perspective may be worth listening to.

Finally, one may object that some subjects just do not call for any nuance whatsoever. For example, should we be nuanced about the Holocaust? This is a reasonable point. Yet even here, I would argue that nuance is still important, in various ways. For one, if we do not have a sufficiently nuanced understanding of the Holocaust, we risk failing to learn from it. For example, to simply believe that the Germans were evil would appear the dangerous thing, as opposed to realizing that what happened was the result of primitive tendencies that we all share, as well as the result of a set of ideas which had a strong appeal to the German people for various reasons — reasons that we should seek to understand.

This is all descriptive, however, and so none of it implies taking a particularly nuanced stance on the ethical status of the Holocaust. Yet even in this respect, a fearless search for nuance and perspective can still be of great importance. In terms of the moral status of historical events, for instance, we should have enough perspective to realize that the Holocaust, although it was the greatest mass killing of humans in history, was by no means the only one; and hence that its ethical status is arguably not qualitatively unique compared to other similar events of the past. Beyond that, we should also admit that the Holocaust is not, sadly, the greatest atrocity imaginable, neither in terms of the number of victims it had, nor in terms of the horrors imposed on its victims. Greater atrocities than the Holocaust are imaginable. And we ought to both seriously contemplate whether such atrocities might indeed be actual, as well as to realize that there is a risk that atrocities that are much greater still may emerge in the future.

Conclusion

Almost everywhere one finds people discussing contentious issues, nuance and self-scrutiny seem to be in short supply. And yet the most essential point of this essay is not really one about looking outward and pointing fingers at others. Rather, the point is, first and foremost, that we all need to look into the mirror and ask ourselves some uncomfortable questions. Self-scrutiny can, after all, only be performed by ourselves.

“How might I be obstructing my own quest for truth?”

“How might my own impulse to signal group loyalty bias my views?”

“What beliefs of mine might mostly serve social rather than epistemic functions?”

Indeed, we all need to take a hard look in the mirror and let ourselves know that we are sure to be biased and wrong in many ways. And more than just realizing that we are wrong and biased, we also need to realize that we are limited creatures. Creatures who view the world from a limited vantage point from which we cannot fully integrate and comprehend all perspectives and modes of consciousness — least of all those we have never been close to experiencing.

We need to remind ourselves, continually and insistently, that we should be charitable and measured, and that we should seek out the grains of truth that may exist in different views so as to gain a more nuanced understanding that better reflects the true complexity of the world. Not least ought we remind ourselves that our brains evolved to express overconfident and unnuanced views for social reasons — especially in ways that favor our in-group and oppose our out-group. And we need to do a great deal of work to control for this. We should seek to scrutinize our in-group narrative, and be especially charitable to the out-group narrative.

None of us will ever be perfect in these regards, of course. Yet we can at least all strive to do better.

Why I Used to Consider the Absence of Sentience Tragic

Whether one considers the absence of sentience bad or neutral — or indeed as good as can be — can matter a lot for one’s ethical and altruistic priorities. Specifically, it can have significant implications for whether one should push for smaller or larger future populations.

I used to be a classical utilitarian. Which is to say, I used to agree with the statement “we ought to maximize the net amount of happiness minus suffering in the world”. And given this view, I found it a direct, yet counter-intuitive implication that the absence of sentience is tragic, and something we ought to minimize by bringing about a maximally large, maximally happy population. My aim in this essay is to briefly present what I consider the main reason why I used to believe this, and also to explain why I no longer hold this view. I am not claiming the reasons I had for believing this are shared by other classical utilitarians, yet I suspect they could be, at least by some.

The Reason: Striving for Consistency

My view that the absence of sentience is tragic and something we ought to prevent mostly derived, I believe, from a wish to be consistent. Given the ostensibly reasonable assumption that death is bad, it would seem to follow, I reasoned, that since death merely amounts to a discontinuation of life — or, seen in a larger perspective, a reduction of the net amount of sentience — the reduction of sentience caused by not giving birth to a new (happy) life should be considered just as bad as the end of a (happy) life. This was counter-intuitive, of course, yet I did not, and still do not, consider immediate intuitions to be the highest arbiters of moral wisdom, and so it did not seem that weird to accept this conclusion. The alternative, if I were to be consistent, would be to bring my view of death in line with my intuition that the absence of sentience is not bad. Yet this was too implausible, since death surely is bad.

This, I believe, was the reasoning behind my considering it a moral obligation to produce a large, happy population. To not do it would, in some ways, be the moral equivalent of committing genocide. My view is quite different now, however.

My Current View of My Past View

I now view this past reasoning of mine as akin to a deceptive trick, like a math riddle where one has to find where the error was made in a series of seemingly valid deductions. You accept that death is tragic. Death means less sentient life than continued life, other things being equal. But a failure to bring a new individual into the world also means less sentient life, other things being equal. So why would you not consider a failure to bring an individual into the world tragic as well?

My current response to this line of reasoning is that death indeed is bad, yet that it is not intrinsically so. What is bad about death, I would argue, is the suffering it causes; not the discontinuation of sentience per se (after all, a discontinuation of sentience occurs every night we go to sleep, which we rarely consider bad, much less tragic). This view is perfectly consistent with the view that it is not tragic to fail to create a new individual.

As I have argued elsewhere, it is somewhat to be expected that we humans consider the death of a close relative or group member to be tragic and highly worth avoiding, given that such a death would tend, evolutionarily speaking, to have been costly to our own biological success in the past. In other words, our view that death is tragic may in large part stem from a penalizing mechanism instilled in us by evolution to prevent us from losing fellow assets who served our hidden biological imperative — assets who had invested a lot into us and whom we had invested a lot into in return. And I believe that my considering the absence of sentience tragic was, crudely speaking, a matter of extending this penalizing mechanism so that it pertained to all insentient parts of the universe. An extension I now consider misguided. I now see nothing tragic whatsoever about the fact that there is no sentient life on Mars.

Other Reasons

There may, of course, be other reasons why a classical utilitarian, including my past self, would consider the absence of sentience tragic. For instance, it seems reasonable to suspect us, or at least many of us, to have an inbuilt drive to maximize the number of our own descendants, or to maximize the future success of our own tribe (the latter goal would probably have aligned pretty well with the former throughout our evolutionary history). It is not clear what would count as “our own tribe” in modern times, yet it seems that many people, including many classical utilitarians, now view humanity as their notional tribe.

A way to control for such a hidden drive, then, would be to ask whether we would accept if the universe were filled up with happy beings who do not belong to our own tribe. For example, would we accept if our future light cone were filled up by happy aliens who, in their quest to maximize net happiness, replaced human civilization with happier beings? (i.e. a utilitronium shockwave of sorts.) An impartial classical utilitarian would happily accept this. The question is whether a human classical utilitarian would too?

Explaining Existence

“Not how the world is, is the mystical, but that it is.”

(“Nicht wie die Welt ist, ist das Mystische, sondern dass sie ist.”)

Ludwig Wittgenstein

 

Why is there something rather than nothing? How can we explain the fact of existence?

This most fundamental question may be worth pondering for various reasons. Such pondering may help sharpen our thinking about the nature of the world, our place within it, and the scope of our understanding. And it may also just lead us to some significant answers to the question itself.

Is Non-Existence Coherent?

I would argue that the key to (dis)solving this mystery lies in questioning the coherence of the idea that there could be nothing in the first place — the notion that non-existence could exist. For existing is, after all, exactly what non-existence, by definition, does not. Non-being, by definition, cannot be. Yet, in asking why there is not nothing, we are indeed, somehow, imagining that it could. Essentially, what we are asking is: why is there not “non-isness“? Why could non-being not have been? The answer, I submit, is that the being of non-being is a contradiction in terms.

If existence were not the case, this would imply non-existence being the case, which is an incoherent notion. More specifically, to say that non-being could be is to contradict the principle of non-contradiction, as one then asks for something, or rather “nothing”, both to be and not be at the same time.

As David Pearce put it:

“One can apparently state the epistemic possibility of nothing having existed rather than something. But it’s unclear how it could make cognitive sense to talk of the epistemic possibility of nothing-or-other having even been the case. For the notion of something-or-other being the case is about as conceptually primitive as one can get. For just what is the (supposedly non-self-refuting) alternative with which one would be contrasting the generic notion of existence – in the sense of something-or-other being the case – that we have at present? The notion doesn’t seem to make any sense. It’s self-stultifying.”

Why Does Anything Exist“, section nine.

Furthermore, even if we were to assume that non-existence could be the case, we would still end up with the conclusion that it actually cannot. For if non-existence were the case, then its being the case would, quite obviously, be a truth, which implies that this truth would at least (also) exist. And yet this truth is not nothing. In other words, it implies the existence of (more of) something. And such a supposedly empty state would in fact imply other properties as well, such as the property of being one (not two or more, as it contains no separation, nor zero, since it does exist by assumption), as well as the property of being free from contradictions (genuine contradictions could not possibly exist in any possible state of existence, much less one that is purportedly empty). Thus, even the notion of a state with no properties other than its mere being is incoherent.

Another way to realize that there could not possibly be nothing, even if we were to pretend that the notion is coherent, is to think in terms of necessary and contingent facts (following the reasoning of Timothy O’Connor found here). For the suggestion that there might have been nothing amounts to the claim that existence might merely be a contingent, not a necessary fact. Yet the fact that we are here proves that existence was, at the very least, a possibility. In other words, the reality of (at least) the possibility of existence is undeniable. And yet the reality of the possibility of existence is not nothing. It is, in fact, something. Thus, even if we assume that the fact of existence is merely contingent, we still end up with the conclusion that it is in fact necessary. The existence of the mere possibility of existence necessarily implies, indeed amounts to, existence in full, and hence the suggestion that existence may merely be contingent, and that there could instead have been absolutely nothing, is revealed to be impossible and indeed incoherent in this way as well.

This may be considered an answer to why there is something rather than nothing: the alternative is simply incoherent, and hence logically impossible. Only “something” could conceivably be the case. And thus, contra Wittgenstein, the real mystery to explain is indeed how the world is, not that it is; to explain which properties the world has, not that it has any. And part of this mystery is to explain why we ever considered the existence of non-existence — as opposed to a very different state of existence — a coherent possibility in the first place, and, by extension, why we ever considered the non-existence of non-existence any more mysterious than the non-existence of square circles.

(And if, and that arguably is a huge if, existence is identical with what we call “physical existence”, then the argument above shows that a physical world must exist, and that its absence is incoherent. Again, this is provided that we assume existence to be identical with “the physical”, which is just an assumption, although I believe one can make a decent case that we have no strong reasons to believe in such a thing as non-physical existence, and hence no strong reasons to doubt this assumption. And if one then further believes that “the physical” is identical with “the mental” — in other words, if one holds a monist ontology that considers both physical and mental descriptions of the world equally valid, which I also consider a highly defensible position — then the argument above shows the necessity of the existence of this monist reality. And all that would then be left to explain, if this assumption happened to be true, is “just” what particular properties and relations that exist within this monist reality.)

No Purpose or Reason Behind Existence, Only Within

The all-inclusive nature of existence implies that, just as there cannot be a mechanism or principle that lies behind or beyond existence, there could not be a reason or purpose behind it either, since behind and beyond existence lies only that which does not exist. And hence there could not possibly be an ultimate purpose, in this sense at least, behind our being here.

Yet this by no means implies, contrary to what may be naturally supposed, that reasons and purposes, of the most real and significant kinds, do not exist within existence. Indeed, it is obvious that they do. For instance, the ability to pursue purposes and act on reasons has clearly emerged over the course of evolution. Beyond that, it is also clear, at least to me, that some states of the world — especially states of extreme suffering — are truly more disvaluable than others, and hence, I would argue, that we have truly normative reasons to act so as to avoid the realization of such disvaluable states. Indeed, I would argue that this endeavor is our highest and ultimate purpose.

Darwinian Intuitions and the Moral Status of Death

“Nothing in biology makes sense except in the light of evolution”, wrote evolutionary biologist Theodosius Dobzhansky. And given that our moral psychology is, at least in large part, the product of our biology, one can reasonably make a similar claim about our moral intuitions: that we should seek to understand these intuitions in light of the evolutionary history of our species. This also seems important for our thinking about normative ethics, since such an understanding is likely to help inform our ethical judgments; by helping us better understand the origin of our intuitive moral judgments, and how they might be biased in various ways.

An Example: “Julie and Mark”

A commonly cited example that seems to demonstrate how evolution has firmly instilled certain moral intuitions into us is the following thought experiment, first appearing in a paper by Jonathan Haidt:

Julie and Mark are brother and sister. They are traveling together in France on summer vacation from college. One night they are staying alone in a cabin near the beach. They decide that it would be interesting and fun if they tried making love. At the very least it would be a new experience for each of them. Julie was already taking birth control pills, but Mark uses a condom too, just to be safe. They both enjoy making love, but they decide not to do it again. They keep that night as a special secret, which makes them feel even closer to each other. What do you think about that? Was it OK for them to make love?

According to Haidt: “Most people who hear the above story immediately say that it was wrong for the siblings to make love […]”. Yet most people also have a hard time explaining this wrongness, given that the risks of inbreeding are rendered moot in the thought experiment. But they still insist it is wrong. An obvious interpretation to make, then, is that evolution has hammered the lesson “sex between close relatives is wrong” into the core of our moral judgments. And given the maladaptive outcomes of human inbreeding, such an intuition would indeed make a lot of evolutionary sense. Indeed, in that context, given the high risk of harm, it even makes ethical sense. Yet in a modern context in which birth control has been invented and is employed, the intuition suddenly seems on less firm ground, at least ethically.

(It should be noted that the deeper point of Haidt’s paper cited above is to argue that “[…] moral reasoning is usually a post hoc construction, generated after a judgment has been reached.” And while it seems difficult to deny that there is a significant grain of truth to this, Haidt’s thesis has also been met with criticism.)

Moral Intuitions About Death: Biologically Contingent

With this idea in the back of our heads — that evolution has most likely shaped our moral intuitions significantly, and that we should perhaps not be that surprised if these intuitions are often difficult to defend within the realm of normative ethics — let us now proceed to look at the concrete issue of death. Yet before we look at the notional “human view of death”, it is perhaps worth first surveying some other species whose members are unlikely to view death in remotely the same way as we do, to see just how biologically contingent our view of death probably is.

For example, individuals belonging to species that practice sexual cannibalism — i.e. where the female eats the male prior to, during, or after copulation — seem most unlikely to view dying in this manner in remotely the same way as we humans would. Indeed, they might even find pleasure in it, both male and female (although in many cases, the male probably does not, especially when he is eaten prior to copulation, since it is not in his reproductive interest, which likely renders it yet another instance of the horrors of nature).

The same can likely be said of species that practice so-called matriphagy, i.e. where the offspring eat their own mother, sometimes while she is still alive. This behavior is also, at least in many cases, evolutionarily adaptive, and hence seems unlikely to be viewed as harmful by the mother (or at least the analogue of “viewed as harmful” found in the minds of these creatures). There may, of course, be many exceptions — cases in which the mother does indeed find herself harmed by, and disapproving of, the act. Yet it nonetheless seems clear that the beings who have evolved to practice this behavior do not view such a death in remotely the same way as a human mother would if her children suddenly started eating her alive.

The final example I wish to consider here is the practice of so-called filial cannibalism: when parents eat their own offspring. This practice is much more common, in terms of the number of species that practice it, compared to the other forms of cannibalism mentioned above, and also a clearer case of convergent evolution, as the species that practice it range from insects to mammals, including some cats, primates, birds, amphibians, fish (where it is especially prevalent), snails, and spiders. Again, we should expect individuals belonging to these species to view deaths of this kind very differently from the way we humans would view such, by any human standard, bizarre deaths. This is not to say that the younglings who are eaten do not suffer a great deal in these cases. They likely often do, as being eaten is often not in their reproductive interests (in terms of propagating their genes), although it may be in the case of some species: if it increases the reproductive success of their parents and/or siblings to a sufficient degree.

The deeper point, again, is that beings who belong to these species are unlikely to feel remotely the same way about these deaths as we humans would if such deaths were to occur within the human realm — i.e. if human parents ate their own children. And more generally: that the evolutionary history of a species greatly influences how it feels about deaths of various kinds, as well as how it views death in general.

Naturally, Most Beings Care Little About Most Deaths

It seems plausible to say that, in most animal species, individuals do not care the least about the death of unrelated individuals within their own species. And we should not be too starry-eyed about humans in this regard either, as it is not clear that we humans, historically, have cared much for people whom we did not view as belonging to our in-group, as the cruelties of history, as well as modern-day tribalism, testify. Only in recent times, it seems, have we in some parts of the world made all of humanity our in-group. Not all sentient beings, sadly, but not merely our own family or ethnic group either, fortunately.

So, both looking at other species, as well as across human history, we see that there appears to be a wide variety of views and intuitions about different kinds of deaths, and how “problematic” or harmful they are. Yet one regard in which there is much less disagreement is when it comes to “the human view of death”. Or more precisely: the natural moral intuitions humans have with respect to the death of someone in the in-group. And I would suspect this particular view to strongly influence — and indeed be the main template for — any human attempt to carve out a well-reasoned and general view of the moral status of death (of any morally relevant being). If this is true, it would seem relevant to zoom in on how we humans naturally view such an in-group death, and why.

The Human View of an In-group Death

So what is the human view of the death of someone belonging to our own group? In short: that it is tragic and something worth avoiding at great costs. And if we take our evolutionary glasses on, it seems easy to make sense of why we would be naturally inclined to think this: for most of our evolutionary history, we humans have lived in groups in which individuals collaborated in ways that benefitted the entire group.

In other words, the ability of any given human individual to survive and reproduce has depended significantly on the efforts of fellow group members, which means that the death of such a fellow group member would be very costly, in biological terms, to other individuals in that group. Something that is worth investing a lot to prevent for these other individuals. Something evolution would not allow them to be indifferent about in the least, much less happy about.

This may help resolve some puzzles. For example, many of us claim to hold a purely sentiocentric ethical view according to which consciousness is the sole currency of moral value: the presence and absence of consciousness, as well as its character, is what matters. Yet most people who claim to hold such a view, including myself, nonetheless tend to view dreamless sleep and death very differently, although both ultimately amount to an absence of conscious experience just the same. If the duration of the conscious experience of someone we care about is reduced by an early death, we consider this tragic. Yet if the duration of their conscious experience is instead reduced by dreamless sleep, we do not, for the most part, consider this tragic at all. On the contrary, we might even be quite pleased about it. We wish sound, deep sleep for our friends and family, and often view such sleep as something that is well-deserved and of great value.

On the view that the presence and absence of consciousness, as well as the quality of this consciousness, is all that matters, this evaluation makes little sense (provided we keep other things equal in our thought experiment: the quality of the conscious life is, when it is present, the same whether its duration is reduced by sleep or early death). Yet from an evolutionary perspective, it makes perfect sense why we would not only evaluate these two things differently, but indeed in completely opposite ways. For if a fellow group member is sleeping, then this is good for the rest of the group, as sleep is generally an investment that improves a person’s contribution to the group. Yet if the person is dead, they will no longer be able to contribute to the group. And if they are family, they will no longer be able to propagate the genes of the family. From a biological perspective, this is very sad.

(The hypothesis sketched out above — that our finding the death of an in-group member sad and worth avoiding at great costs is in large part due to their contribution to the success of our group, and ultimately our genes — would seem to yield a prediction: we should find the death of a young person who is able to contribute a lot to the group significantly more sad and worth avoiding compared to the death of an old person who is not able to contribute. And this is even more true if the person is also a relative, since the young person would have the potential to spread family genes, whereas a sufficiently old person would not.)

Implications

So what follows in light of these considerations about our “natural” view of the death of an in-group member? I would be hesitant to draw strong conclusions from such considerations alone. Yet it seems to me that they do, at the very least, give us reason to be skeptical with respect to our immediate moral intuitions about death (indeed, I would argue that we should be skeptical of our immediate moral intuitions in general). With respect to the great asymmetry in our evaluation of the ethical status of dreamless sleep versus death, two main responses seem available if one is seeking to make a pure sentiocentric position consistent (to take that fairly popular ethical view as an example).

Either one can view conscious life reduced by sleep as being significantly more bad, intrinsically, than what we intuitively evaluate it to be (classical utilitarians may choose to adopt this view, which could, in practice, imply that one should work on a cure for sleep, or at least to reduce sleep in a way that keeps quality of life intact). Or, one can view conscious life reduced by an early death as being significantly less bad, again intrinsically, than our moral intuitions hold. (One can, of course, also opt for a middleroad that maintains that we both intuitively underestimate the intrinsic badness of sleep while overestimating the intrinsic badness of death, and that we should bring our respective evaluations of these two together to meet somewhere in the middle.)

I favor the latter view: that we strongly overestimate the intrinsic badness of death, which is, of course, an extremely unpalatable view to our natural intuitions, including my own. Yet it must also be emphasized that the word “intrinsically” is extremely important here. For I would indeed argue that death is bad, and that we should generally view it as such. But I believe this badness is extrinsic rather than intrinsic: because death generally has bad consequences for sentient beings, including that the process of dying itself tends to involve a lot of suffering (where I would view this suffering as intrinsically bad, yet not the end of the life per se). And furthermore, I would argue that we should consider death a bad and harmful thing (as I indeed do) not just because this belief is accurate, but also because not doing so has bad consequences as well.

An Ethic of Survival

With respect to ethics and death, I recently encountered an interesting perspective in an exchange with Robert Daoust. He suggested, as I understood him, that the fundamental debate in ethics is ultimately one between an ethic of survival on the one hand, and an ethic of concern for sentience on the other. And he further noted that, even when we sincerely believe that we subscribe to the latter, we often in fact do support the survivalist ethic, for strong evolutionary reasons. A view according to which, even if life is significantly dominated by suffering, survival should still be our highest goal.

I find this view of Daoust’s interesting, and I certainly recognize strong survivalist intuitions in myself, even as I claim to hold, and publicly defend, values focused primarily on the reduction of suffering. And one can reasonably wonder what the considerations surveyed above, as well as similar considerations about the priorities and motives that evolution has naturally instilled in us, imply for our evaluation of such a (perhaps tacitly shared) survivalist ethic?

I would tentatively suggest that they imply we should view this survivalist ethic with skepticism. We should expect evolution to have given us a strong urge for survival at virtually any cost, and to view survival — if not of our own individual bodies, then at least of our own group and bloodline — as being intrinsically important; arguably even the most important thing of all. Yet I would argue that this is an implausible ethical view. Specifically, to accept continued survival at virtually any cost, including the cost of increasing the net amount of extreme suffering in the world, is, I would argue, highly implausible. Beyond that, one can argue that we, for evolutionary reasons, also wildly overestimate the ethical badness of an empty world, and grossly misjudge the value of the absence of sentience. Indeed, on a pure sentiocentric view, such an absence is just as good as deep, dreamless sleep. And what is so bad about that?

A Brief Note on Eternalism and Impacting the Future

Something I find puzzling is that many people in intellectual circles seem to embrace the so-called eternalist view of time, which holds that the past, present, and future all equally exist already, yet at the same time, in terms of practical ethics, these same people focus exclusively on impacting the future. These two positions do not seem compatible, and it is interesting that no one seems to take note of this, and that no attempt seems to be made at reconciling them, or otherwise examining this issue. 

For why, given an eternalist view of time, should one focus on impacting the future rather than the past? After all, the eternalist view of time amounts precisely to the rejection of the common sense view that the past is fixed while the future is not, which is the common sense view of time that seems to underpin our common sense focus on trying to impact the future rather than the past. So how can one reject the common sense view of time that seems to underlie our common sense practical focus, yet then still maintain this focus? If the past and the future equally exist already, why focus more on trying to impact one rather than the other?

The only attempted reply I have heard to this question so far, which came from Brian Tomasik, is that if, hypothetically, the present were different, then the future would be different, and hence it makes sense to focus on such changes that would render the future different. The problem, however, is that the same argument applies to the past: if, hypothetically, the present were different, then, for the equations of physics to be consistent, the past would also have to be different. Tomasik seemed to agree with this point. So I fail to see how this is an argument for focusing on impacting the future rather than the past given an eternalist view of time.

Possible Responses

There are various ways to respond to this conundrum. One can, for instance, try to argue that there is no conflict between eternalism and focusing only on impacting the future (which seems the prevailing assumption, but I have yet to see it defended). Another path one could take is to argue that we in fact should focus on impacting the past just as much as the future (something I find highly dubious). Alternatively, one could argue that it is just as senseless to try to change the future as it is to change the past (something few would be willing to accept in practice). Lastly, one could take the tension between these two widely esteemed views to imply that there may be something wrong with the eternalist view of time, and at the very least that we should lower our credence in eternalism, given its ostensible incompatibility with other, seemingly reasonable beliefs.

My Preferred Path: Questioning Eternalism

I would be curious to see attempts along any of the four paths mentioned above. I myself happen to lean toward the last one. I think many people display overconfidence with respect to the truth of eternalism. The fact that the equations of the theory of relativity, as they stand, do not demand an ontologically existing “now does not imply that no such thing exists (where this now, it must be noted, is not defined by “clocks all show the same”, as such a now clearly is impossible; yet there is no contradiction whatsoever in the existence of a unique, ontologically real “present” in which initially synchronized clocks show different times). In other words, although the equations of relativity do not demand the existence of such a now, they do not rule it out either. Yet it seems a widely entertained fallacy that they do, and people thus seem to accept that eternalist view as though it were a matter of logical certainty, when it is not. I think this is bad philosophy. And I think it is important to point this out, since false certainties can be dangerous in unexpected ways (for example, if the above-mentioned fallacy led us to falsely conclude that trying to impact the future is senseless).

Beyond that, as I have noted elsewhere, one can also question to what extent it makes sense to say — as eternalists often do, and as the name eternalism itself implies — that all moments exist “always”? After all, doesn’t “always” refer to something occurring over time? The meaning of claims of the sort that “every moment exists always” is, I believe, less obvious than proponents of eternalism appear to think, and seems in need of unpacking.

A General Note on Our Worldview

I think the tension explored here speaks to a more general point about our worldview, namely that we often do not derive the more practical views we hold (such as the view that we can influence the future but not the past), from our fundamental ontological theories of how the world works. Instead, such views are often derived mostly from tacit common sense notions and intuitions (which is not to say that they should necessarily be rejected, and certainly not on this ground alone). This means that sometimes — quite often, in fact — the views we hold on various subjects, such as the philosophy of time and practical ethics, are scarcely compatible. The project of bringing the various beliefs we hold across these different areas in concert is, I believe, an important and potentially fruitful one, for our theoretical views in themselves, as well as for our pratical efforts to act reasonably in the world.

Blog at WordPress.com.

Up ↑