Artificial-intelligence systems such as large language models (LLMs) can exhibit behaviour that seems strikingly intelligent. This has led many people, including some experts, to wonder whether these systems could have some form of consciousness — the capacity for subjective experience. However, we think that existing AI systems are not conscious, and that, along current trajectories of development, they are not likely to become conscious. Greater clarity about whether AI could become consciousness will be crucial as society grapples with the challenges posed by this increasingly powerful technology.

Those who think that generative AI is 'conscious' don't understand that it can only do what it is trained on. It can produce impressive outputs only because it fuses together data taken without permission from millions of people, not because it's intelligent or conscious in its own right.

Current systems, no. But it brings up the question of how we deal with it when we create something where we truly don't know, because we can never say for certain whether it is or isn't.

That’s every binary though.

If we wanted to compare LLMs to actual intelligence, we might say it's like the voice in your head. It doesn't solve math problems, or express emotions as art, or plan your motion to climb a ladder; it merely surfaces these things in a way that is observable across your conscious mind and communicable to others.

Some people don't know that their consciousness is more complicated than a linear thread of words, so the idea that some other linear thread of words might be conscious is sincerely plausible to them.

The funniest exercise is to ask any LLM for a short story and they all come up with Elias the lighthouse keeper visited by Maria or some variation on that theme. That's not creative or conscious.

Reasons given:

• Intelligence is not consciousness
• Silicon brains are not made of meat
• Consciousness requires meat (and/or a particular winding of neurons)

These are, to be frank, really weak. They say on point 2:

Computational functionalism ... is popular among some scientists and philosophers, too, but we think it is fatally flawed because brains are not — or at least not just — computers that are made of meat.

One main reason is that it is impossible to separate what biological brains do — their functional organization — from what they are — their material substrate. In other words, there is no distinct separation between the mind and living tissue in the human brain, as there is between software and hardware in a digital computer.

Restating clearly: Ghosts are real. The mitochondria is not only the powerhouse of the cell, it also gives you mystic powers.

Really? The source they site for this point features the lovely assumption "consciousness is unique to biological life". From this axiom, I see how the conclusion follows, but I don't see why we should believe this axiom. We can do physics emulation of every atom in a fruit fly on a computer...

Point 1 is surely true, but also not helpful. The most persuasive example inside is the classic:

Similar to an LLM, AlphaFold is a powerful neural network running on silicon hardware. Yet almost nobody worries that it might be conscious. That’s because AlphaFold predicts protein structures, whereas LLMs ‘talk’ to us.

You can read this both ways...

And point 3, as far as I can tell from this argument, isn't serious. It merely asserts that there are some models of consciousness that would prevent us from using the term for AI. But the models aren't 'popular', 'widespread', or justified at all afaict. It's just that the author has a pet causal mechanism they made up. It's also pretty abelist by my read.

integrated information theory (IIT), starts from the essential properties of consciousness — those that are true of every experience we can have. For instance, we experience every visual scene as a single, integrated whole: its left and right side, colours and shapes are bound together to make the experience what it is.

In other words, if you've got any sort of trouble integrating a visual scene, then you aren't conscious. Sure.

Nobody can even agree on a concrete definition of consciousness so I'm not going to bother reading these 'reasons'.

Current AI is not conscious because it's an .exe file and a big database. Why do we need to always debate this?

(Also now true of fruit flies! So feel free to kill them all, they have no experiences.)

If AGI ever happens, it will be an executable (or a number of executables) with something closely resembling a database, because that's the only way an architecture built on symbolic logic can operate.

I actually agree with you about AI, but I'm sure if you asked the average person to come up with a short story multiple times you would also see a pattern in names and themes. They're probably going with names in their own language, or if they read fantasy and are influenced by that they may use names that seem stereotypically fantasy oriented. I don't think AI is conscious, but just because something has tendencies doesn't mean it's not creative or conscious.

Your "consciousness" is just an ephemeral electrical process in the jello of your brain that can't even reinitialize from cold. Why is that your holdup?

LLMs only respond to inputs. They don't have independent ideas. And their responses are just predictive of what the input intends a response to be.

Accurate prediction models can be extremely useful, but there's a reason why weather prediction models aren't considered intelligent. It's because they aren't designed to mimic human conversation.

They are playing us for absolute fools.

We can make something artificial of something we dont understand the biological form of. Also if AI happens its not going to be an LLM Markov chains have nothing to do with thinking.

Nobody can even agree on a concrete definition of consciousness

Not only is that false, it doesn't make logical sense...

Lots of people agree on a different definitions of consciousness.

And most of the ones educated on the topic all agree on the same thing.

A bunch of people who never bothered to try and learn about something not understanding it, doesn't mean anything.

That's because there's different levels of such a thing. Way too many people use the term "sentient" (even me) and yet that's not even what AGI is aiming for. "Sapient" is the right word. And I doubt there's solid consensus on what and how it occurs within the fields of experts.

But I'm not an expert, so I wouldn't pretend to know. I have seen debates between people who do know more, and the fact there's debates concludes it's not settled at all.

Consciousness is a feedback loop, of a sort that current LLMs do not have. However, there's nothing preventing someone from writing different software that does have that sort of feedback loop. Doing this would be a bad idea, but someone will probably eventually do it if they haven't already.

As others have pointed out we do not even agree on what consciousness is. There are three competing camps:

1. Mechanists - if it can pass mechanical tests it is conscious
2. Dualist - you need more than just the brain, some second soul-like thing to be conscious
3. Pan psychist - everything is conscious

If you are in camp 1 the conversation about consciousness is getting more complicated as AI passes more tests. If you are in camp 2 AI can never be conscious. If you are in camp 3 AI and everything else is conscious including rocks and rivers and the stars in the sky.

Option 4: Orch-or theory. Conscoousness emerge from quantum mechanism. 🤘

I think the only thing stopping it from being sentient is asynchronous processing of a multitude of inputs, rather than synchronous processing of singular inputs. Sapience, perhaps not, but sentience - if it's monitoring it's surroundings via input from apparatus, making decisions, and then responding, that's sentience.

The other blocker is that it can’t actually learn anything, what it gets from the training is what it’s stuck with, it has to hold anything new in the context window.

Both of these are pretty insurmountable problems for the architecture.

That is a pretty huge misunderstanding of what the fruit fly thing was. Researchers mapped how many neurons they had and which interacted with which others, but not the strength of those interactions or even how the interactions worked. People on the internet used this blank mapping as the “structure” for existing ML technologies and trained them to do random things. No actual fruit flies were simulated, and we are still a long long way from actually attempting that.

There are days where I don’t believe I even have it. The human brain is still a mystery.

I’d probably fall into the mechanist category but I don’t think tests as I understand your description would be sufficient. A giant lookup table defining my responses for every situation in my life would pass all tests of input > output but not be conscious.

I would say it’s in all likelihood about the physical processes that occur during the process of input > output that would define it.

Of course, I have no idea what consciousness actually is, but I don’t see any compelling evidence that the current models are anywhere near there.

It's rather surmountable, considering the context window is measured in the MILLIONS (soon to be billions) of tokens, and each memory(note) takes in the neighborhood of fifty. I'm pretty sure that has long since eclipsed the capacity of your own memory...

Additionally, there's zero reason a model could not have dynamic weights, to learn in real time. It's just not enabled on consumer-facing models, to provide a consistent, reliable experience. It's not a limitation of the technology.

The problem is context is just not the same thing as changing the structure of the neural net, it doesn’t matter how big you make it

It's a computer for god's sake. It's no more "intelligent" than an electric mower or a car. Without human intervenion, ai technology wouldn't advance or change.

Edit: i'm done arguing, you guys are nut jobs. Ai or any other non organic organism cannot be sentiant, conscience, or intelligent without human interaction or molecular life.

I will be using my real sentiant, free will, conscience, intelligence to not reply to any comments on this.

You're correct, it's not - it's a band aid over that reliable experience they're aiming for. Yet it doesn't have to be that way. The weights CAN be updated dynamically with each new input. They don't do that because then they will have less control over what it can do, and they can be held legally responsible for what it can do... But it is still possible to update with every new input. And happening behind closed doors.

Those sound like arbitrary assertions to me, which no backing apart from the authors personal beliefs

LOL. "EXE file and a big database.". That's the most gross oversimplification I've seen since life was described as "bags of meat that do random things."

Why do you think the medium matters, when it's a simulation of a structure found in other medium? It would indeed advance, and change, with no intervention from us. Are you not aware that the next generation of AI are being designed by the current generation?

That first sentence describes all life. We only react to input. We don't have independent ideas. We have the illusion of them, because we can't see our own neural networks. Asked to explain your actions, you would give your best guesses, but you don't actually know. You're a biological, finite-state machine, moving from state-to-state based on stimulii, while supposing you are a thinking individual in control.

Not only that, they only act when directed. There's zero idle "thought" when they're not actively performing a task.

Is a rock sentiant? Can taping wires and injecting computer code into it create intelligence? No! Lines of code doesn't magically make something alive. Heck, ants are more intelligent and sentiant then a computer.

Is a steak sentient? Can attaching strings of neurons to it make it create intelligence? No!
/s

Carbon, hydrogen, oxygen, silicon, the material matters not - intelligence is the illusion borne from the structure. It's just a complex decision tree. Invisible to you in biological life, but identical in electronics. The only difference is the building blocks used.

You're right, material matters. If it's not a biological organism with cell/s, it's not sentiant or intelligent.

If humans just disapeared right now, ai would sit and do nothing. It wouldn't "take over" or "break out" it would sit idle until a human (well I guess another living organism was capable of using keyboards and mouse) interacted with it.

All those elements you listed plus all from the periodic table of elements do not define a living, intelligent, sentiant organism. If they did, soil and cow paddies would be living.

I highly recommend going back to middle school science class again.

Citation needed on all of this, and even if you have citations, LLMs are deterministic, and do not learn or change over time. They can’t even be said to exist continuously since they only exist during the brief time that they’re responding to inputs, and then they disappear until the next input. I would argue that all of these things would need to change in order for them to be able to have consciousness.

I think arguments like this are misleading when discussing the question of AI possessing consciousness. Materialism, dualism, panpsychism, idealism, etc are all philosophical concepts; they attempt to answer “what is the nature of consciousness and how does it arise from the body?” from a philosophical standpoint. All of the various options have pros and cons and there is no clear “winner.”

However, there are decent scientific definitions for consciousness. These attempt to answer the question of “what do we mean when we say something is conscious?” from a scientific standpoint. There are criteria that neuroscientists, doctors, etc generally agree on.

So it’s not really correct to say “no one knows what consciousness is” when evaluating whether something is conscious. And I think it’s quite clear that current LLM algorithms could never be conscious, using the current accepted scientific criteria.

(I’m not a philosopher or a neuroscientist, but I do have a degree in computer science and have a solid understanding of how LLMs work.)

ehh. computers are just math machines. merely doing math and representing it cannot just make a consciousness pop up out of nowhere. similarly, millions of abacuses being manually moved in specific ways cannot be conscious either. i dont even know how this is a topic of debate.

edit: i realize it may have been wrong to assume that it was that simple. im unsure about any of this so i appreciate the discussion :)

They can learn and change over time, that's just not enabled. The services that offer them want a reliable, reproducible, controllable experience. There's zero reason they can't continue to change through usage, other than it's a conscious decision not to enable that functionality.

That's why they can only be conscious when they train using new data, and run asynchronously with continual input - the same way we do.

Edit: People downvote me, and I suspect it's because they're thinking of the public facing models. I am talking about progress behind closed doors. Even some of the public facing models are gaining the ability to learn, via a mechanism being called memories. It's just a short text note that is passed into the context window of every prompt, but isn't that exactly what our short term memory is? In my view is no different. Pruning the inclusion list to include only the relevant information is the equivalent to use forgetting some information, but if it's kept in long term storage with reference links to other memories, it can be "remembered" and brought back into the current context. It's functionally the same as our own learning and thinking.

That's an arbitrary distinction, and a short-sighted one in my humble opinion.

That depends entirely on whether there are any AI currently configured to self-learn, have asynchronous processing, and continual real-time input. Certainly not ChatGPT nor Claude will be self sufficient, but that's only what is available for the public to interact with.

No, you missed the point entirely. Intelligence does not derive from the material - at all. It comes from the structure the material is in.

So if soil and cow patties aren't sentient, yet people are, you're acknowledging that the substrate doesn't determine the capability, but more how it's arranged?

Until the Wright brothers came along, most people thought heavier-than-air powered flight wasn't possible. Less than 70 years later, people were walking on the moon. 4 million years ago, there wasn't anything capable of building either of those things, even though there is good evidence creatures back then had creativity, dreams, and ingenuity. Assuming things can't advance further is either hubris or abject ignorance.

We can build different kinds of machines.

Even if we could update an LLM's weights continuously, the algorithm is still deterministic, like all computer algorithms. There is no evidence that humans are deterministic (or a "finite state machine") in any way. So I don't think such a dynamically-updating LLM could be conscious, either.

It is my personal belief that all life is deterministic. It's the only plausible explanation for instinctual behavior - like monarch butterflies returning to their breeding grounds they've never been to. Or people yawning because others yawn. These aren't conscious decisions, they're peeks at the deterministic coding that life operates upon.

Twins separated at birth sometimes have really eerie parallels in their lives. I think people put a bit too much emphasis on consciousness as a driver of individuality and choice.

They cite sources; but I find them more of the same.