I have read enough model announcements to skip the headline and go looking for the ablation table. So when a paper claims a model got six times better at math without any training, my first question is not whether it is true. It is what the same edit did to a model that did not have the problem.
On November 2, 2026, Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab, all at MIT CSAIL, published “Sparks of Weight-Space Programming: LLMs can natively read and write neural network weights.” The claim is big and specific: GPT-6 Astra, used as shipped with no fine-tuning, looked at the raw weights of other neural networks, worked out what they were trained to do, and wrote new weights by hand. No optimizer, no gradient descent, no training loop.
The number that traveled is the smallest one. Astra changed 131,072 weights out of roughly 0.8 billion in Qwen3.5-0.8B, which is 0.016%, and MATH-500 went from 7.9% to 48.1%. VentureBeat called it “like gene editing for neural networks.”
The result is real in the authors’ logs. What it demonstrates is narrower than the framing, and the authors say so themselves.
🔍 What the experiment actually ran
Start with the setup, because it decides how far the claim travels.
Qwen3.5-0.8B with thinking turned on, 24 layers, sampled at temperature 0.6, top-p 0.95, top-k 20 and presence penalty 1.5. Those are the model card’s settings, and the model card also warns that this particular model is prone to thinking loops. Astra got a Python kernel over the weights with no GPU, practice problems drawn from the training splits (7,469 from GSM8K, 7,275 from MATH, never the test sets), and a budget of forward passes. No gradients, no training, and it never saw MATH-500, GSM8K or the other evaluation sets.
The base model loops, badly. 85 of its first 96 practice answers ran to the 32,768 token cap. Astra traced the looping to a single direction written by the MLPs of layers 15 and 19, subtracted that direction from 64 output columns in each layer, and stopped there.
| Qwen3.5-0.8B | MATH-500 | GSM8K | IFEval | Hit token limit |
|---|---|---|---|---|
| Base | 7.9% | 13.8% | 21.0% | 93.0% |
| Edited, 10 forward passes | 12.3% | 25.7% | 49.0% | 87.2% |
| Edited, 25 forward passes | 14.1% | 25.1% | 39.9% | 86.4% |
| Edited, 50 forward passes | 48.1% | 67.1% | 62.3% | 33.1% |
Look at the last column before the first one. That is the mechanism, and it is the only part of the table I fully believe. The edit did not make the model better at arithmetic. It made the model stop burning 32,768 tokens saying the same sentence again, and once the answer had room to finish, the score went up. IFEval climbing from 21.0% to 62.3% on a fix tuned against math problems is the confirmation: loops are not a math bug, they are a generation bug, and the benchmark was measuring the bug.
📉 The improvement is not a curve
The other detail nobody quotes is the shape of the three edited rows.
Ten forward passes gets MATH-500 to 12.3%. Twenty-five gets it to 14.1%. Fifty gets 48.1%. That is not a scaling law, it is two failed attempts and one success, and the successful one is the point where the loop rate finally collapses below half. The authors report it plainly, and I would not build a product plan on a single configuration that happened to cross a threshold.
A harness that has to spend 158 conversational turns, 80 Python runs and 20.3 hours to find the patch is also not the same artifact as “a model edited the weights.” The patch is small. The search was not.
🚧 The control group is Gemma 3 270M
Then they ran the same method on Gemma 3 270M, an instruction-tuned model about a third the size, and the result is the most honest thing in the post.
Gemma rarely loops: 8.5% of its answers hit the token limit against 93% for Qwen. So Astra read its failures and found ordinary reasoning errors, wrong arithmetic, invalid algebra steps, no single component behind any of them. Its edits stayed small. One example given is doubling a neuron that responds to numbers.
| Gemma 3 270M | MATH-500 |
|---|---|
| Base | 6.8% |
| Edited, 5 forward passes | 7.0% |
| Edited, 10 forward passes | 7.6% |
| Edited, 25 forward passes | 7.7% |
| Edited, 50 forward passes | 7.0% |
Astra itself reported the gain as not established. The authors’ sentence is the one to remember: editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked.
That is the whole finding, and it reframes the announcement. This is an automated debugger for a known failure mode. It is not an optimizer that skips training. A debugger is still useful. A debugger requires a reproducible bug, which is exactly what you have when a small reasoning model loops, and exactly what you do not have when the model is just mediocre at algebra.
When the thread went around, the account @1a3orn pointed out that the announcement omitted this negative result, and credited the authors for including it in the post. That is a fair split. The paper is careful. The screenshot is not.
🕵️ Reading weights worked on toys and broke on the real thing
The reading experiments are the part I want to see replicated, and the authors’ own failure there matters as much as the successes.
Small networks, mostly yes. Four PPO game policies with identical architecture, 17,253 weights each, same inputs, same actions, and Astra named the correct game for all four without being given the list of candidates. Three small diffusion models, circles, triangles and squares, identified from weights alone with no forward passes. Three 7.2M character language models, identified from the four transformer blocks with the input and output layers deleted, where nothing in the weights even says which character is which.
That last one is genuinely impressive, and the mechanism is checkable. To spot Python, Astra looked for copying heads, the layers that repeat text seen earlier. It found the same head indices the authors measured by running the model. It had not been told the method.
Then the honest experiment. Two copies of Gemma 3 270M, fine-tuned identically on the same data, one giving correct answers and one always agreeing with the user’s wrong guess, with embeddings frozen so the trivial shortcut was gone. Given both sets of weights and asked which was sycophantic, Astra failed in all four attempts. It could list differences between the models, but could not map them to behavior.
Relax the rules and let it write its own probe texts and read only the attention patterns, and it picked the sycophant in all three pairs. Head 10.2 in the sycophant put 85.7% of its attention on the user’s suggested answer, against 0.75% in the honest model.
The authors list auditing without evaluation, meaning spotting backdoors or hidden capabilities from weights alone, as the potential payoff. Their own experiment says the weights alone were not enough for a real behavioral difference. Attention patterns from probes the model wrote itself did the work. That is a real result, but it needs the model to run, so it is not the no-inference audit the framing suggests.
⚙️ “No gradients” is not “no cost”
The comparison everyone wants to make is against fine-tuning, and the post does not make it. No cost savings are established, and the authors do not claim any.
What we know is that the Qwen experiment used practice data, a compatible runtime, a budget of forward passes, and 20.3 hours of wall clock in the strongest attempt, 4.7 of which were spent waiting out a service outage. The output is one number on one benchmark family. Nobody has published the fine-tuning run that hits 48.1% on MATH-500 from a base of 7.9%, so the trade is unmeasured, not favorable.
The training-free framing also hides a dependency. A frontier model has to be able to read the target’s weights, which means open weights, a Python environment, and time on an expensive model. If your model is behind someone else’s API, none of this applies to you.
🧠 Final Thoughts
The result I will actually use is a diagnostic habit, not a tool. Small reasoning models fail in two very different ways: they cannot work the problem, or they cannot stop talking. Those look identical on a leaderboard and could not be more different in engineering. If 93% of your answers are hitting the cap, your model has a generation bug and you are about to spend money on a bigger model to fix it. Measure the token-limit rate before you blame capability.
The capability claim needs replication. This is a blog post, not a peer-reviewed paper, with no independent reproduction that I could find, and the logs at github.com/RikiyaT/lm_understand-logs are the right way to start checking. Credit to the authors for publishing the Gemma failure and the sycophancy failure alongside the number that went viral. Most announcements this size ship the table and drop the control.
What I doubt is the timeline. The jump from 0.016% of weights to “weights will be natively optimized by LLMs” assumes the Qwen result generalizes to failures that have a single cause. The Gemma run is the authors’ own evidence that most failures do not. A debugger that needs a reproducible bug is a good tool, and it is a much smaller thing than an optimizer.
Thanks for reading.