The following is an excerpt from my book Reflections on Intelligence (2016/2024).

A question that is often considered open, sometimes even irrelevant, when it comes to “AGIs” and “superintelligences” is whether such entities would be conscious. Here is Nick Bostrom expressing such a sentiment:

By a “superintelligence” we mean an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills. This definition leaves open how the superintelligence is implemented: it could be a digital computer, an ensemble of networked computers, cultured cortical tissue or what have you. It also leaves open whether the superintelligence is conscious and has subjective experiences. (Bostrom, 2012, “Definition of ‘superintelligence’”)

Yet this is hardly true. If a system is “more capable than the best human brains in practically every field, including scientific creativity, general wisdom, and social skills”, the question of consciousness is highly relevant. Consciousness is integral to much of what we do and excel at, and thus if an entity is not conscious, it cannot outperform the best humans “in practically every field”, especially not in “general wisdom” and “scientific creativity”. Let us look at these in turn.

General Wisdom

A core aspect of “general wisdom” is to be wise about ethical issues. Yet being wise about ethical issues requires that one can consider and evaluate questions like the following in an informed manner:

• Is there anything about the experience of suffering that makes its reduction a moral priority
• Does anything about the experience of suffering justify the claim that reducing suffering has greater moral priority than increasing happiness (for the already happy)?
• Is there anything about states of extreme suffering that make their reduction an overriding moral priority?

It seems that one would have to be conscious in order to explore and answer such questions in an informed way. That is, one would have to know what such experiences are like in order to understand their experiential properties and significance. Knowing what a term like “suffering” refers to — i.e. knowing what actual experiences of suffering are like — is thus crucial for informed ethical reflection.

The same point holds true about other areas of philosophy that bear on wisdom, such as the philosophy of mind: without knowing what it is like to have a conscious mind, one cannot contribute much to the discussion about what it is like to have one and to the exploration of different modes of consciousness. Indeed, an unconscious entity has no genuine understanding about what the issue of consciousness is even about in the first place (Pearce, 2012a; 2012b).

So both in ethics and in the philosophy of mind, an unconscious entity would be less than clueless about many of the deepest questions at hand. If an entity not only fails to surpass humans in these areas, but fails to even have the slightest clue about what we are talking about, it hardly surpasses the best humans in practically every field. After all, questions about the phenomenology of consciousness are also relevant to many other fields, including psychology, epistemology, and ontology.

In short, experiencing and reasoning about consciousness is a key part of “human abilities”, and hence an entity that is unable to do this cannot be claimed to outperform humans in the most important, much less all, human abilities (see also Pearce, 2012a; 2012b).

Scientific Creativity

Another ability mentioned above that an unconscious entity could supposedly outdo humans at is scientific creativity. Yet scientific creativity must relate to all fields of knowledge, including the science of the conscious mind itself. This is also a part of the natural world, and a most relevant one at that.

Experiencing and accurately reporting what a given state of consciousness is like is essential for the science of mind, yet an unconscious entity obviously cannot do such a thing, as there is no experience it can report from. It cannot display any genuine scientific creativity, or even produce mere observations, in the direct exploration of consciousness.