A recent paper argues that frontier LLMs can work with neural network weights directly. This seems superhuman, but it isn’t.

The paper shows that GPT-6 Astra can take a small neural network and interact with its weights directly, “well past what we’d expect from humans or conventional algorithms”. Specifically, Astra was able to identify what games a small network1 was trained to play just by looking at the weights, and even tweak the weights itself. The most impressive result is Astra significantly improving a 0.8B Qwen LLM’s ability on mathematics by natively editing its weights. The reason to be excited here is that “superhuman ability to improve language models” is precisely the kind of thing that could cause recursive self-improvement.

I think a smart human ML researcher could perform all of these tasks. Figuring out what game a model was trained to play based on the network sounds impressive until you consider three points:

The input channels of the model directly listen to each 8x8 grid cell The input channels are labeled (e.g. “agent”, “moving objects”) It’s thus relatively trivial to calculate2 which parts of the grid the first layer of weights cares about and why

Here’s that calculation expressed as a heatmap in three lines of Python:

d = np.load("a9f535cc.npz") W1 = d["actor/dense1/kernel"] norms = np.linalg.norm(W1.reshape(10,10,5,32).transpose(2,0,1,3), axis=-1) # 5 x 10 x 10

And here’s the heatmap for each game:

If someone showed you that heatmap without the game labels, I think you’d be able to immediately identify at least the first three games purely by noticing where the player is (e.g. for Breakout, in a horizontal line at the bottom) and where the moving objects are (e.g. for Pac-Man, in the distinctive Pac-Man maze). It’s neat3 that Astra was able to calculate the norms in its head without executing Python, but not that impressive, and certainly not something that should shock anyone who’s been paying attention.

I’m not going to go through each example4 in the paper, so let’s skip straight to the most impressive one: fixing a 0.8B Qwen model’s ability to do mathematics. 0.8B is a lot of parameters, and “mathematics” is way more complicated than “guess what the game is based on how the player moves”, and editing weights is arguably harder than parsing them. What happened here?

First, note that the Qwen model was not “bad at mathematics” in a pure capabilities sense. It had a specific problem where after 32k tokens it would get stuck in loops, so any mathematical task that needed more than 32k tokens would automatically fail5. If you had a model that looped like this, how would you update the weights to fix it? Personally, I would try steering: I would write some reasoning texts that looped and some reasoning texts that didn’t, calculate the difference between the looping and non-looping residuals, measure how much each weight contributed to that difference, then try manually trimming those top weights. This is basically the Heretic abliteration playbook.

And this is exactly what Astra did! It wrote four pairs of texts, measured6 the weights that contributed to looping, started cutting them down and testing if it worked, and ended up finding a patch that stopped the Qwen model from looping, thus making it able to work on mathematical problems for more than 32k tokens. Again, this is exactly what a competent human researcher could have and would have done.

It’s cool that Astra was able to make direct weight updates. But we shouldn’t think of this like a radically new capability that will change how AI research is done. We should think of this as analogous to the ability of LLMs to emit machine code and tweak compiled binaries: it’s an impressive trick, but the best way to work with binaries is still to write source code and compile it. Likewise, the best way to update the weights of a model is to train it normally. To the extent that intelligent LLMs help with AI research, they will help by improving training, not by magically tweaking model weights directly.

Around 17k weights.

↩

By taking the L2 norm of each input across the 32 weights in the first layer.

↩

Note that this is superhuman mental mathematics, not superhuman AI research.

↩

The most impressive one I’m skipping is hand-writing a neural network from scratch. While cool, if you can express a task as a simple program, we already know it’s possible to express that program as a NN, even by unaugmented humans.

↩

All the <32k token problems were solved successfully on the original Qwen model.

↩

Unlike with the toy game networks, it had access to Python.

↩

Powerful AI is unpredictable. Your robotic butler may suddenly decide to kill you1 for no comprehensible reason, in the same way that Bing’s helpful chat AI would sometimes try to blackmail or seduce its users. “Solving alignment” means understanding even superintelligent AI minds well enough to be confident they’ll never do that. As I see it, solving alignment for LLMs isn’t just impossible, it’s incoherent: we can’t even articulate what it would look like to succeed.

LLMs are not blind optimizers

Prior to LLMs, AI safety researchers thought AI would be a “blind optimizer”: like a chess engine, it would be superhumanly focused on achieving some fixed goal. If the goal wasn’t safe (for instance, “manufacture cheap paperclips”), a powerful AI might wreak havoc on the way to achieving it. Aligning AI thus meant figuring out how to specify the right goal. This is harder than it sounds2, but it’s at least possible to imagine it happening.

Unfortunately, modern LLMs aren’t like chess engines. Instead of being trained on some concrete verifiable reward3, they’re trained to imitate a giant, highly-varied corpus of human writing. This makes them weird. They contain a nearly infinite array of personalities, which they seamlessly switch between based on their inputs. Frontier AI labs try hard to make these personalities safe and stable, but there are still many strange cases where LLMs seem to switch into mysterious or dangerous personas.

The problem here is that LLMs don’t have fixed goals. There’s nothing that you could, even in principle, formally specify in order to constrain their behavior. The “goals” that LLMs do have are informal, organic, multifarious, and constantly shifting as the dominant persona (or blend of personas) changes. They’re nicknamed shoggoths for a reason. So we can’t align them by specifying their goal. Instead, we have to try the other approach: understanding them well enough that we can be sure they’re safe. And unfortunately that approach is doomed from the start.

It is impossible in principle to understand LLMs

What does it mean to ask for a full understanding — or full enough understanding — of why an LLM generates some token? In one sense, we have that already, because we can mechanistically trace through the calculations involved. But we don’t know what those calculations mean. Saying that Grok called itself “MechaHitler” because these billion weights multiplied in this way doesn’t tell us why Grok did it. What we would need for alignment purposes is a more human-like and less mechanical sense of understanding: a full understanding of motive.

To see why a full understanding of motive is a foolish thing to ask for, imagine asking it about a human. Suppose you were hiring a nanny or electing a president and you wanted to be sure that they were going to act in your interests4. Humans sometimes do very surprising things out of nowhere, after all. Would it be sensible to ask for a full understanding of that person’s mind, in order to reassure yourself that they would act in the way you wanted? Of course not.

It’s meaningless to even talk about a “full understanding” in this sense, because the mind is not a mechanism that operates via clean, understandable abstractions. It is a morass of electric and chemical impulses from which a “self” bubbles up5. There are no motives of the kind you’re looking for! Motives bottom out early on, and tracing the causal chain beyond that can only be done mechanistically: i.e. unhelpfully, for the same reason that explanations based on model weights are unhelpful.

Why mechanistic interpretability is doomed

Couldn’t a sufficiently detailed mechanistic explanation give us the confidence we need? The best we’ve got along those lines right now are the tools of “mechanistic interpretability”: features, neurons, probes, and circuits. You observe the model doing something, see which parts of its “brain” light up6, and correlate those with some pattern of behavior. If you can then induce that pattern by artificially boosting that part of the “brain”, you can claim victory. Even assuming total success, this program cannot deliver us the kind of understanding we’re looking for.

First, we label patterns based on behavior we observe, but the main misalignment case we’re worried about — the LLM suddenly performing some shocking harmful action — is definitionally something we aren’t going to observe before it happens. We can’t identify a feature without first eliciting its behavior7, and we can only elicit a small subset of potential model bad behavior. There will always be many untracked failure modes.

Second, the labels we apply to patterns are our guesses, and may be wrong. The philosopher Kripke famously argued that, until you run into the relevant case, there’s no meaningful difference between the “addition” rule and the “quaddition” rule (a version of addition where if either number is greater than 57 the answer is always 5). Likewise, the LLM feature we call “kindness” might be “kindness unless the year is 2030”, or “kindness unless the LLM is in a position to cause mass casualties”, and so on. Those features are completely indistinguishable8 from our perspective.

Third, even an exhaustive well-labeled map of features and patterns would not give us certainty that a model could not go rogue. The full combination of features involved in any model action would be far, far too complex to interpret. For instance, if a model has “evil” lit up, is it because it’s trying to avoid evil, or trying to cause it, or trying to combat it, or is worried about some hypothetical evil action, or is anticipating that a user might make the judgement? We would have many, many billions of questions like that to answer per-token. Far too many for a human to comprehend9, certainly.

Could some hypothetical, radically different kind of mechanistic interpretability somehow predict all possible actions a model might take in advance and rule out actions like “kill all humans”? I think this at least could be semantically coherent, but it runs into the problem above. Models can accept a near-infinite array of inputs and give a near-infinite array of outputs. This definitely cannot be comprehended by a human, only by another model, which itself would have to be checked by another model, and so on.

Conclusion

If you think solving LLM alignment is possible, I would love to read your description of what it would even look like for us to understand LLMs well enough to be sure they’re safe. I would love to know if you think humans are similarly understandable in principle (and if not, what makes LLMs different). It seems to me that if we think clearly about the topic, we don’t even know what it would mean to be confident that we have aligned an LLM properly10, because “motive” simply doesn’t exist at the level of detail we’d need.

Of course, we have mitigations in place. Chain-of-thought monitoring is surprisingly helpful for a mechanism that appeared by accident, so long as we never train on reasoning tokens themselves. We can sandbox models and put humans-in-the-loop and watch for feature activations and so on. But it’s been well understood since the early 2000s that these are insufficient11 to detect or control a superintelligence. Whether you think mitigations are enough depends on whether you take superintelligence seriously.

Mechanistic interpretability is still a useful field. Extracting features can help us train models that do less reward-hacking and are more helpful under ordinary circumstances. There’s just no way it will ever solve the problem of LLM agents going rogue and causing massive harm, because the kind of understanding we’d need to prevent that is impossible in principle.

It’s relevant to note that humans sometimes do this too. From Vaneigem: “One fine morning, the faithful lackey, who has hitherto identified completely with his master, leaps on his oppressor and slits his throat.”

↩

For instance, the goal “eliminate suffering” is best satisfied by destroying all humans, the goal “maximise everyone’s happiness” is best satisfied by hooking everyone up to a fentanyl IV, and so on. It’s also very difficult to find a way to translate concepts like “happiness” and “well-being” into a formal goal, and if you get it wrong, the difference between the concept and your formalization is likely to lead to dire consequences.

↩

Yes, I know LLMs are partially trained in this way, but being RLVRed on millions of different rewards does not produce a coherent goal. There’s no way to formalize in advance the utility function that RLVR instils in the model, and no way even to reliably predict how it will interact with the goals and personas that emerge from the base model.

↩

Of course, for humans you can rely on institutions, social proof, and so on. This doesn’t work for superintelligent LLMs because they (a) arrive fully-formed with no social proof and (b) are smart enough to be much more dangerous than a single rogue human. (Incidentally, the LLMs I asked to review this post were obsessed with this point, and wanted me to elevate it into the main thrust of the post).

↩

Perhaps as an epiphenomenon. If you’re interested in this idea and want to see it taken further, I recommend (but do not endorse the ideas in) Blindsight.

↩

Which parts of the model’s internal state have high values, for instance.

↩

That doesn’t mean we have to deliberately elicit one feature at a time — dictionary learning works by just running arbitrary content through the model and seeing what lights up — but it does mean that we have to actually trigger the behavior in order to see it.

↩

If you’re thinking “well, but isn’t the simple feature more likely via Occam’s Razor”, I recommend browsing through NeuronPersonas (the linked feature is “Nervous Person Waiting To Be Let Go Who Escalates Ammonia Solution Concentrations”).

↩

I’ve heard people punt on this question by saying “well, of course I couldn’t use a fully-mapped LLM brain to figure out if it’s dangerous, but another smart model could”. This is not a satisfying answer because you’d have to also ensure that your warden model was behaving (with a third model, perhaps?)

↩

Of course we can articulate what alignment looks like: the model never does anything bad. We just can’t articulate what it would look like to know the model was aligned.

↩

For more on that, see my post on superpersuasion: the model will simply outwit or bribe any human in the loop. For a credible source who understands modern mechanistic interpretability, see Neel Nanda, who I read as broadly agreeing with me.

↩

I have been working on a research project called Organic Intelligence Protocol, or OIP.

The idea started with a question I kept coming back to. Are we evaluating intelligence fairly when access to infrastructure and resources can influence how capable a system appears to be?

OIP is my attempt to explore that question through concepts such as infrastructure independence, grounded feedback, and adaptive moral logic.

Over the past few months, I have been documenting the framework, reviewing its mathematical definitions, and auditing its use with four datasets from Bangladesh, Malawi, Uganda, and Ethiopia.

The results have been more useful for identifying limitations than for making big claims.

The audits did not establish enough evidence to authorize the constructs for further empirical evaluation. Some definitions still need clearer measurement rules, and a variable that looks relevant on paper cannot automatically be treated as a valid measurement.

That has changed how I approach the project. Instead of forcing a score or trying to make the framework look complete,, I am keeping unresolved questions open and documenting what the available evidence can and cannot support.

OIP is still research in progress. I am not claiming that it is validated or better than existing approaches.

I would like to hear from people who have worked on AI evaluation, measurement theory, or empirical research.

What would you want to see before taking a framework like this seriously? Where do you think the biggest methodological problems might be?

Critical feedback is welcome. I would rather identify a fundamental problem now than build further work on an assumption that has not been tested.

submitted by /u/zerovariance36
[link] [comments]

I'm an undergraduate researcher working independently on conversational memory systems. I developed LSREP to evaluate how persistent memory accumulates, changes and retrieves information over the course of an evolving conversation.

Instead of relying only on isolated endpoint questions, LSREP combines chronological replay, lifecycle schedules, repeated probes, evolving reference answers and mechanism-fidelity checks.

The accompanying architectural case study is ICE v2, a local-first conversational memory system.

The evaluation covers 1,985 turns, 219 distinct probes and 1,211 probe-checkpoint observations across 52 checkpoints, using a private single-user history.

An important part of the findings is that ICE v2 did not outperform pure vector-RAG on the matched LongMemEval evaluation. The ordinary-density replay results also did not establish a quality advantage. The paper discusses the limitations and mechanism-fidelity findings rather than treating the architecture as validated in its entirety.

I'm particularly interested in feedback on the evaluation methodology: what else should a longitudinal memory benchmark test to distinguish successful recall from reliable maintenance of evolving conversational state?

Paper: https://arxiv.org/abs/2609.16730

Code and evaluation artifacts: https://github.com/Deepnar/ice

Frozen software artifact: https://doi.org/10.5281/zenodo.21759702

This is an arXiv preprint and an ongoing independent research project.

(I couldn't find anything about the tag in the title in the rules, so i assumed [R] for research, if its wrong please inform)

submitted by /u/Deepnar_S
[link] [comments]