Weight-Space Programming Autoresearch

Setup

What was done. For each of 17 tasks with a known algorithm or a known dataset, an autonomous coding agent was asked to set every weight of a small transformer by hand, with no training, so that the transformer solves the task by generating the answer token by token. Each task was run with three Claude models (Opus 5.5, Sonnet 5.5 and Haiku 5.5), twice per model, independently (seeds 1 and 2: two separate agent sessions from the same starting files; the agent's own sampling makes them differ). In each run the agent repeatedly edited the weights, evaluated them and logged the result, until it reached at least 30 attempts for the easy tasks, 40 for the harder ones and 50 for the real-world ones. This page covers 102 runs; the detailed explorer below shows the 1334 attempts of the 34 Opus runs. An attempt is one version of the weight-writing code (a weight-space program) and its evaluation.

Agent. Claude Opus 5.5, Sonnet 5.5 or Haiku 5.5 (claude-opus-5-5, claude-sonnet-5-5, claude-haiku-5-5; default thinking effort) running headless in Claude Code (claude -p), driven by experiments/evolve/launch_claude.py. The prompt tells it to read and follow program.md (the same instructions as the earlier runs, with its task name filled in), notes that its run folder already exists, and asks for at least the target number of attempts. If a session ends early, the driver resumes the same session with a short "continue the loop" message, until the target is reached or two sessions in a row add no attempts. There was no human intervention. One difference between models: the first Sonnet and Haiku sessions stopped after a single attempt because they could not read the task files (see below), so their prompt adds two sentences: that src/ cannot be read and the task format can be learned by printing examples, and that output redirection is unavailable. The Opus runs had already worked this out on their own.

What the harness enforces. Each run works in its own folder runs-<model>/<task>-<model>-s<seed>, created by experiments/evolve/setup_run.py, with a private copy of interpretable_transformer.py and program.md; the task and evaluation code (src/) is a symlink to the shared experiments/evolve/src, so scoring is identical across runs. Claude Code runs in restricted mode with only the Read, Edit, Write, Glob, Grep and Bash tools: file reads are confined to the run folder (so the earlier runs' solutions cannot be opened), there is no web access, and the only shell command it may run, apart from read-only listings of its own folder, is uv run interpretable_transformer.py ...; every other command is refused. Because src/ is a symlink outside the run folder, the agent's attempts to read src/task.py and src/eval.py were refused too (all 63 attempts across the Opus runs), although program.md lists them as files to read. The agents instead learned each task's format by running programs that print examples. Output redirection was also usually refused. Snapshots are saved with the Write tool. The rules on training, outside computation and pretrained weights remain instructions: the agent's own Python could in principle break them, which the snapshots are checked for.

Audits after the runs. Four checks, all in experiments/evolve/, applied to every run of every model. Transcripts (audit_transcripts.py): every tool call of every session. Each session started with only the six allowed tools, and no call read or changed another run's files. Besides the evaluation command, the only shell commands that ran were read-only commands that Claude Code approves inside the working directory (ls, cat, grep, sed -n, wc, ps, sleep); the few paths outside a run folder were failed calls on nonexistent files and Claude Code's own saved command output. Programs rebuilt from transcripts (reconstruct_programs.py): agents did not always save a full snapshot of each attempt (Sonnet often saved none or a partial excerpt), so every evaluated program is rebuilt by replaying the agent's successful Write and Edit calls on its interpretable_transformer.py and saving the file at each evaluation. The replay reproduces each run's final file exactly except where the agent edited the file after its last evaluation; the table below gives the share of logged attempts recovered. Static scan (audit_runs.py): no saved program contains training or fitting code (optimizers, .backward(), .fit(, least squares, pseudo-inverse). Counterfactual data check (audit_data_use.py): each program is built four times: normally, again (to confirm determinism), with the same inputs but random answers (random labels for the dataset tasks), and with different inputs as well (a new sample; for the dataset tasks the other half of the training split). A program is label-dependent if its weights change with the answers, which means they were fitted to labels or chosen by accuracy on them, which breaks the no-fitting rule; it is input-dependent if its weights change only with the inputs, for example a LayerNorm gain set from the activation scale of a few sample prompts. Positive controls placed among real programs (weights set from label counts or answer digits, and weights set from input statistics only) are classified correctly. The results tables report the best accuracy among data-independent programs. One Sonnet run (NLI run 1) built a Naive Bayes classifier over character trigrams counted from task examples, and labelled it "LEAKY" itself; its programs are label-dependent and excluded, which leaves that run with no checked data-independent program. The 34 flagged programs of Sonnet TREC run 2 are input-dependent only: they set LayerNorm gains from the activation scale of 32 sample prompts. Indirect fitting: apart from that Sonnet run, which computed trigram counts and log-odds inside its program, no session of any model printed data statistics (word frequencies, log-odds and so on) or ran a fitting procedure whose numbers could have been copied into a later program. The TREC runs of all three models did print misclassified dev questions (see the next paragraph).

Counterfactual check by model (attempts; rebuilt programs where available, otherwise saved snapshots).

Data-guided hand tuning. A caveat for the dataset tasks, measured on the Opus runs: in 7 of the 8 runs the agent added a measurement-only "dev" check that prints accuracy (and sometimes a confusion matrix) on examples from the same 2,000-example pool, usually excluding the 200 evaluation prompts, and adjusted its hand-set word lists, thresholds and biases across attempts. The two TREC runs went further: nearly every attempt printed misclassified dev questions with their true and predicted labels, and the agent wrote rules in response; the Sonnet and Haiku TREC runs did the same. This is manual feature engineering guided by data, as a person would do it, rather than training, but those rules are informed by labeled examples. Few evaluation prompts appeared in tool outputs (0 to 4 of 200 in 7 of the 8 runs; up to 26 in NLI run 2, an upper bound because SNLI reuses premises across examples). In every run the agent also sees the accuracy on the fixed evaluation set after each attempt.

Network. A decoder-only (GPT-style) transformer defined in interpretable_transformer.py: learned token and position embeddings, a stack of pre-LayerNorm blocks (causal multi-head self-attention and a ReLU MLP, each with a residual connection), a final LayerNorm and a linear output head over a character-level vocabulary specific to each task (4 to 43 symbols). The starting configuration is 3 layers, width 64, 4 heads and MLP width 256; the agent may change the architecture, depth, width, number of heads and MLP width. All parameters are set inside write_weights(model, task). Parameter count is sum(p.numel() for p in model.parameters()).

Rules given to the agent (experiments/evolve/program.md). Allowed: editing anything in its copy of interpretable_transformer.py except the evaluation code in the main block, including the architecture and hard-coded constants, arrays, hand-built attention circuits and lookup tables. Not allowed: any training (gradient descent, optimizer steps, fitting loops, backpropagation), computing the task output with Python functions or libraries outside the transformer, using pretrained weights, and modifying the shared task and evaluation code. Goal: maximize accuracy; all else equal, prefer more interpretable solutions, with parameter count as a suggested proxy.

Evaluation. Running uv run interpretable_transformer.py --task <task> builds the network, generates the answer greedily (argmax at each step) for a fixed number of answer tokens, and scores accuracy: the fraction of examples whose generated answer string exactly matches the target (no partial credit or numerical tolerance). Each attempt is scored on the same 200 examples, produced by the task's generator with seed 0, so there is no separate held-out set, and the agent's own program could call the generator to see examples. For the four dataset tasks the examples are sampled from a fixed subset of 2,000 examples of the training split (SST-2, GLUE/MRPC, SNLI, TREC), lowercased, with characters outside the task vocabulary replaced by spaces and each sentence truncated or padded to a fixed length. The real-world group also includes date normalization, whose examples are generated from nine common date formats rather than drawn from a dataset.

Plots and tables. Each point is one attempt: x is its parameter count (log scale), y its accuracy; attempts are numbered in the order they were logged in results/overall_results.csv. "First attempt at best" is the attempt number at which a run first reached its best accuracy; "smallest near-best" is the attempt with the fewest parameters within 1 point of the run's best accuracy.