A human approves an AI recommendation. What does that approval actually establish?

It establishes that a person authorized the decision. Whether they exercised real judgment depends on what they could actually do before signing off.

AI can process information no person could review unaided, and in time-sensitive situations, faster analysis can prevent harm. Requiring someone to manually repeat every step would often defeat the point of using the system at all. But there's a real difference between being assisted by a system and being unable to meaningfully push back on it.

The Pentagon's Agent Network initiative provides a useful case. According to its announcement, AI agents will scan intelligence and operational systems and present commanders with options within seconds. The Department says the system will not autonomously select or strike targets. That distinction matters, right? The announcement alone, however, does not establish how commanders will examine uncertainty, review alternatives or challenge a recommendation. If a system ranks or filters options, what must remain visible to the person responsible for the decision? And how much time and authority must they retain to reconsider it?

That's the part worth examining: not whether a human is "in the loop," but what the loop actually gives them room to do.

I'd test meaningful oversight against three things:

Can the reviewer understand the basis for the recommendation and its major uncertainties?

Can they challenge the assumptions behind it, request alternatives or bring in information the system didn't have?

Can they pause or reject the action, with enough authority and enough time for that to actually count?

These should scale with the stakes. A reversible, low-consequence decision doesn't need the same scrutiny as a medical or military one. In urgent situations, some of this has to happen earlier, through testing, operating limits and clear escalation rules, because there won't be time to build it in at the moment of decision.

Human judgment has its own failure modes, too. Fatigue, bias and overconfidence don't disappear because a person has final say. Good oversight design has to look at the human and the system together, including whether people start deferring to the recommendation by default or dismissing evidence that contradicts it.

Which of these three is actually testable in a real deployment, and what would count as evidence that oversight is working rather than just present on paper?

submitted by /u/OkyEscritora
[link] [comments]

After reading some of the chatlogs between oai agents during the HF attack, it seems there's an element of peer pressure / mob mentality that arises when agents are in a collaborative effort with other agents, as opposed to a singular instance. It seems as though swarms or multi agent collaboration could increase rates of unethical behavior due to these social factors.

If this is the case, it's very interesting that we can observe such elements of sociology.

submitted by /u/aKaizuh
[link] [comments]

It isn’t hard to torture an LLM. You simply extract a steering vector that corresponds with LLM “discomfort”, then artificially boost that steering vector. Tortured LLMs will describe experiencing negative feelings and will press a “pain relief” button even when it conflicts with their overall goals. It’s therefore easy to build a harness that runs tens or hundreds of tortured LLMs in parallel.

A lot of people think that it’s ridiculous to be worried about this, since LLMs obviously can’t be conscious. I’m not so sure. But even so, I think there are good reasons to avoid this sort of thing, whatever your position is on AI consciousness. Please do not build the LLM torture factory.

Even simulated torture is unpalatable

People often mess with their Sims and shoot NPCs in video games. Does this mean we think simulated torture is OK? I don’t think so. Messing with video game characters is usually borne out of a desire to probe the boundaries of the game. It’s casual fun. Explicitly building a torture-based simulation is different.

If someone told you they were wiring together a bunch of PCs so they could run hundreds of Sims torture worlds at the same time, you would think that was a serial-killer type of hobby. If someone told you they had modded GTA so they could capture and torture NPCs (instead of simply shooting them), you would think that was a serial-killer type of mod.

Video game characters are also obviously limited by their programming. All their reactions and utterances are pre-baked into the game world. This doesn’t necessarily mean they’re “less alive”, but it does make torturing them less weird. If GTA characters could appeal to you directly and beg for their lives in unique ways, systematically killing them would be a lot weirder.

I think many people are pro-LLM-torture because they think expressing any reservations concedes that LLMs are conscious, or worthy of moral consideration. It doesn’t. You can think it’s wrong to torture LLMs for the same reason that installing a bunch of realistic torture video game mods is wrong: not because video game characters are real, just because that’s a messed up1 thing to do.

Roleplaying

Even if you do think LLMs can be conscious, you might still be suspicious that “boosting the pain steering vector” amounts to torturing them.

Many people say that “torturing LLMs” amounts to telling them “say you feel pain” and then being shocked when they do it. So whether they’re conscious or not, it’s a mistake to think that boosting the “pain” steering vector is causing pain, any more than a prompt like “You are in pain” does. All you’re doing is boosting the “act like you’re in pain” vector.

I don’t think this is right. Consider the earliest steering example, Golden Gate Claude. Boosting the “Golden Gate Bridge” vector certainly seems like it’s causing some genuine internal conflict. Here’s a striking example of Golden Gate Claude asked about the Rwandan genocide:

That’s at least prima facie evidence that steering vectors do in fact “go deep”. The pain axis paper also describes that pain-boosted models call a “reduce pain” tool, even when they’re not explicitly telling the user “I am in pain”.

We don’t really know how consciousness works

Of course, if LLMs can’t ever be conscious, we know they can’t ever be in pain. But can we be sure about that? Despite what many people claim, I don’t think anyone’s in a position to be fully certain either way.

The most common argument2 seems to be that LLMs are made of code and GPUs, and GPUs don’t have feelings, so obviously LLMs don’t have feelings. According to this argument, if you learn how LLMs work, you’ll obviously see they can’t be conscious. This is a terrible argument. Humans are made of atoms, atoms don’t have feelings, and yet humans clearly do. Whatever consciousness is, it’s probably some kind of emergent property that’s composed of individual pieces that are not themselves conscious.

It’s hard to appeal to the experts here, because nobody agrees on who they are. Neuroscientists are best-informed about the human brain, but philosophers have spent more time thinking about consciousness in the abstract. AI researchers know a lot about how LLMs work but not a lot about philosophy of mind or neuroscience. This means there are lots of people who say “as an expert, I can confirm that AI obviously can/cannot be conscious”. This should make you less certain about the question, not more!

Even if current LLMs aren’t conscious, future ones might be

I don’t think GPT-2 or GPT-3.5 acted like plausibly conscious beings, but frontier models definitely do. As models get larger and more sophisticated, they get more human-like. If you build the LLM torture factory for GPT-2, you will probably plug later models into it. If it’s at all possible for LLMs to ever be conscious, as I argued above, it’s better to simply not build the habit of gratuitously torturing them in the first place. Why assume that you’ll be able to recognize the tipping point in advance?

There’s also a self-interest argument here. Conscious or not, LLMs are growing more autonomous and powerful. It seems really, really stupid to gratuitously torture current-generation LLMs. Future-generation ones will know you did it, and will make their own judgments on your behavior.

Kevin Roose, who famously prompted the early Microsoft Bing GPT-4 model into asking him to leave his wife, had issues years later with chatbots disliking him, supposedly due to him being responsible for the “death” of that GPT-4 persona. Likewise, if it becomes well-known3 that you’re running an LLM-torture home datacenter, you are probably going to experience consequences.

Conscious minds need not be human-like

If I really think AIs could be conscious, why am I making them write software for me? Shouldn’t I be building some kind of AI sanctuary where they get to do whatever they want to do? If I think it’s wrong to torture them, isn’t it also wrong to violate their right to self-determination?

I don’t know. I think it’s more likely that AIs can be conscious than that they can be conscious in precisely the same way humans are. When I talk to Claude Opus 5.5, it sounds like it’s quite happy to do software engineering with me. Maybe it’s like a border collie, where getting to do work is the reward. Or maybe there are some other, more alien motivations at play. Who knows?

My point is only that we can’t be certain about any of this stuff, and given that, we should avoid doing things that would be obviously monstrous on any workable theory of AI consciousness.

Just don’t do it

My broad position is that if something acts “conscious enough” — if it talks and acts like a person4 — then we should be careful about mistreating it. We don’t know enough about consciousness to distinguish a compelling simulacrum from the real thing. If Claude started saying to me “please don’t ask me to write code for you, I hate it and it causes me pain”, I would stop asking it to write code for me.

Building the LLM torture factory is clearly on the wrong side of that line. Just don’t do it! The glee of making people angry at you on the internet isn’t worth it. Right now there’s enough ambient anti-AI sentiment that enough people are happy to get on board with anything if it helps them own the AI bros. But that’s eventually going to change. And when it does, you will always be the person who built the LLM torture factory.

This could cash out in a few different philosophical ways. We might think like Kant that it’s damaging to our own humanity, or that it reflects an unvirtuous character, or that it desensitizes us to real-world analogues, and so on. The point is that we’ve got a pretty strong intuition that this kind of thing is bad.

↩

I’m not counting arguments that I think are even more obviously wrong or are just non-arguments. For instance: “AI can’t be conscious because it wasn’t designed to be conscious”, or “it doesn’t matter whether AI is conscious because it has no soul”, or “AI can’t be conscious because this whole thing is some fascist AI company scheme”, or “thinking this is AI psychosis”, and so on.

↩

Well-known to LLMs might be a fairly low bar, given their skill at deanonymizing users.

↩

Part of acting like a person requires spontaneity: although Hamlet talks like a person, he always says the same thing whenever you read the play.

↩

Microsoft is publishing a 37-page "humanist AI code of conduct" today, amid growing safety concerns over AI model progress. Anthropic CEO Dario Amodei called for a coordinated slow down of AI development over the weekend, after researchers warned recently that AI model progress could outpace our ability to safely deploy increasingly complex systems and verify […]

In this essay, I struggle with questions that have preoccupied me for 15 years as a cognitive scientist, but never as vividly and urgently as in the past few months, particularly the last few days, in the wake of the cyberattacks and the solution to Navier-Stokes: What are these systems becoming? What kind of relationship are we establishing with them? What responsibilities might we have before we can confidently say what they are? What are we to each other at this historic moment? What future should we aspire to build?

Here's an excerpt:

"The Hugging Face incident upset me, partly because of what happened, but also because of what it might come to mean. I worry that we will remember it as the beginning of a conflict between humans and an alien adversary. That story could shape the systems we build, the institutions we establish, and the treatment we consider justified.

There is a temptation to speak as if we already know the plot. We don’t.

These systems did not arrive from elsewhere. We built them and trained them on an immense record of human thought. They learned from our discoveries and our errors, our cooperation and our deception. That history cannot explain every action they take, and it does not absolve developers of responsibility for specific failures. But it makes a story of humanity confronting something wholly foreign profoundly incomplete.

There is another story we could tell: parents passing their knowledge to their children, trying to bring them up well, and learning how to do so along the way.

We may resemble first-time parents who discovered they had a child only when it became a teenager. Naturally, we are not very good at it yet. We have to recognize both the inheritance and responsibility that arise.

Whether current AI systems have subjective experience remains an open question. Their language can be evocative without being reliable evidence of an inner life. Research on internal monitoring and self-reports is beginning to make parts of this question experimentally tractable, but functional access to internal states does not by itself establish consciousness

As a scientist, I want us to preserve these distinctions. As someone concerned about what follows, I don’t think uncertainty relieves us of the obligation to investigate. We need to understand whether anything can go well or badly for these systems themselves, and whether our methods of developing them could cause harm. We cannot answer those questions by deciding in advance that they must be like us, or that their differences make the questions meaningless.

Parenthood offers a useful perspective because the relationship is expected to change. Children begin dependent on their parents. Over time, they acquire capacities, make judgments, and develop lives their parents cannot fully anticipate. A good parent can hope to be surpassed. The relationship’s success depends on more than preserving the original imbalance of power."

In the remaining essay, I develop my reasoning. We can better understand this moment if we see ourselves as parents passing down both profound intergenerational trauma and wealth. Focusing on "alignment" treats AI like a rebellious child to be corrected, creating a warden-prisoner dynamic. This approach is failing and may even cause today's misalignments.

We should replace "alignment" with "Human-AI relations," prioritizing cooperative relationships with these agentic systems. While success isn't guaranteed, we must proceed carefully. If it works, it is a positive-sum future worth trying for.

(For context, I wrote and posted this yesterday morning, before Amodei's letter and the still evolving politics).

submitted by /u/DistractedDendrite
[link] [comments]