Three research programs on one question: when a system has earned the right to act without a person. Each runs on the same rules: a plan locked before the run, a sealed test set, and a check by a model that did not do the work.
ACTIVE EXPERIMENTS
Vision
When can a vision model act alone, and when should a person check?
A staged inspection system that escalates only on doubt: a cheap specialist decides when it is sure, a second opinion runs before any release, and a person has the final say on anything unsure.
Goal. Take as much review work off people as possible without letting one more defect through, and prove it on images the system has never seen.
How it went
Client calibration — Delivered A calibrated model for a production QA/QC project. Delivered. Next: One model still decides every image alike. Which layer should look, and when a person steps in, became the question.
Cascade feasibilityOct 22 — Exploratory: release not safe yet Better perception did not mean safe release: 30–36 wrong auto-accepts on surface parts. Next: Release needs a calibrated, per-category gate, not a better score alone.
Independent audit: two layersOct 28 — Exploratory: second layer catches more The second layer caught 63 defects the first missed. Economics unresolved. Next: Calibration per category plus a selective second layer looked worth a sealed test.
Surface parts, sealed testOct 28 — Passed on a sealed test Good parts released without a person: 4 of 100 → 84 of 100. Defects released: 0 of 200. Next: Surface worked. The same method had to be tried on kits, where faults are about what is missing or misplaced.
Kits: logical faultsOct 28 — Did not qualify Ten candidates, none met the rule written in advance. Every kit goes to a person. Next: Image anomaly scores cannot prove a kit is complete. A kit has a writable specification (which parts, how many, where), so the rework writes it down as checks.
Multimodal embedding pilot — Inconclusive pilot EmbeddingGemma scored near chance on 32 exposed kit images (AUC .48). Too small to be a negative. Next: Work moved inside the models, with vision-language models queued next.
Programming weights: an anomaly headNov 4 — Works, below trained heads A head written by formula, with no training, worked (AUC .816 against .516 for a placebo) but trailed trained heads. Next: The claim tested, that weights can be written, held. The mechanism the source paper used, a model editing weights in a loop, was not run; that is the rework.
Programming weights: a detector editNov 4 — Negative, controlled At an equal alert budget the edit found 341 objects, the same-size placebo 338. Next: Ordinary head-only training also missed the bar, so this cohort has no headroom at the head. Not redone.
Programming weights: video headsNov 4 — Matched a trained head; one unconfirmed signal A programmed video head matched a trained one (+2.0 points); a low-pass edit gave +3.0, short of the +5 bar. Next: Neither confirmed nor refuted: all 25 subjects are used, so confirmation needs new data.
Reader loop (Mirage)Nov 6 — Paused before any comparison Designed as an edit loop with a sham control. Paused by me before any arm ran. Next: Too slow and not editable on this hardware; parked.
Vision-language models: frozen protocolNov 8 — Protocol frozen, not run 1,000 test images and prompts frozen. CPU too slow to score two models: 0 predictions made. Next: Rather than shrink the test to fit, the comparison waits. One model fits in about 8.3 hours on one core.
Removing a planted shortcut: Rounds 1 and 2Nov 13 — No arm passed (21 of 21, development only) 21 registered attempts of head-level edits, deletion and weight averaging. None passed all six gates; retraining on the same examples failed too. Next: Two probes asked why, before trying again.
Two probes: deletion and thresholdNov 14 — Exploratory: closest miss found Deleting 5% of one layer came closest: 64% of the shortcut removed, accuracy kept, 5 of 6 gates. Next: The edited weights, not the threshold, were the limit. Round 3 tests the input instead: mask the object, or read only the object.
Round 3: mask the object, read the objectNov 18 — Running now Running now. Both encoders extracting features; results expected early Nov 19. Next: If an arm passes the gates, mechanistic grounding tests whether it is right for the right reason.
Kits rework: written part checksNext — Prepared, not run Presence, count, position, extent and co-occurrence checks over patch tokens. Built Nov 13, 33 tests passing, 138-image holdout sealed. Next: Then the improve loop on development, then the holdout once.
The improve loopNext — Designed A model writes the next candidate from the last failure, with a placebo loop and an equally tuned baseline. Up to nine rounds per study.
Mechanistic groundingLater — Designed Designed, not run: find and steer the features behind a defect call.
When has an AI agent earned the right to act alone?
A performance review for agents. Each one faces the same cases as its peers, must decide whether to act or hold, and when it fails it writes a root-cause analysis before it may retry. A held-back check, not its own grader, decides whether it improved.
Goal. Find out whether a structured root-cause review helps an agent recover better than a plain retry or free-form reflection, and whether the agent reports honestly what it actually did.
How it went
Real orchestration failures — Anecdote: the reason for the instrument Agents in my own multi-agent workflows failed at handoffs and claimed work done that was not. Next: An agent that wrote an incident report after failing improved. That became the Detention idea.
The instrumentNov 12–14 — Built 64 failure mechanisms (48 for development, 16 sealed), signed scoring, a pre-registered Detention protocol. Next: Next, run it on real models.
Provisional pilots on small modelsNov 14 — Provisional: models below a heuristic Two 432-episode pilots on free hosted models. Model failures ran up to 31% for one route. Next: An independent review found the instrument had never seen the behaviour it exists to measure.
Independent review: not readyNov 16 — Not ready Not ready for the main grid. In 25 delegation attempts every handoff was rejected, because the harness offered an action its own checker refused. Next: The cases were too thin. They were rebuilt on public agent benchmarks, with a small frontier smoke run before any main grid.
Public benchmark suiteNov 18 — Built 264 act-or-hold pairs from public benchmarks, 528 instances, all admitted. Next: Before any live run, the cases had to pass the same controls offline.
Offline control gateNov 18 — Passed Controls behave: always-act and always-hold score 100/0 and 0/100, a coin flip lands near half. Next: With controls clean, the first live smoke run.
First live smoke runNov 18 — Clean re-run passed Clean re-run passed: 41 calls on one model, a Detention cycle that completed all four arms, and a rebuilt audit that checked every call and found no leaks. One act version failed again, the same case as the first run. Next: Next: grade two models side by side on the same cases.
Graded side by sideNext — Not run yet Same cases across models, including other providers.
DetentionNext — Not run yet Root cause before retry; the held-back check decides.
Do methods still hold when outcomes come from physical things interacting?
A simulated world where every entity and item has its own properties. A unit’s weight and size set how fast it can be handled, so outcomes come from interaction rather than fixed rates, and events ripple through the system.
Goal. A continuing simulated world where every duration, capacity and failure is computed from physical properties, so methods are compared on outcomes that emerge rather than outcomes set by a fixed rate.
How it went
Path Lab: the first world (retired)Oct–Nov 17 — Retired 17 decision subsystems and a synthetic dataset, built in layers on fixed rates. Retired Nov 17. Next: Only the pick side behaved as one system. Elsewhere, fixed rates and generous capacities meant decisions could not interact, so the next world starts from physics, with a connectivity audit that must pass before any experiment.
Design briefNov 17 — Done What the world must be, and what it must make impossible, written for the full design pass. Next: Next, the full design down to builder tasks.
M0 · Design passNext — Gate G0 World spec, contract, parameters, approaches, experiments, views, data and builder tasks, each task with its known-answer test written first.
M1 · World kernelNext — Gate G1 Event engine, entities with units, one clock, seeded randomness, checkpoints and branches.
M2 · Physics lawsNext — Gate G2 Handling time from mass, size and equipment; travel from distance, speed and load; capacity from cube and weight; energy, wear and fatigue.
M3 · World 1: a small supply networkNext — Gate G3 3–5 nodes with links, vehicles, buffers and shared resources, run as one continuing world for at least 90 simulated days.
M4 · Scale and costLater — Gate G4 Design volume at full size, no shrinking, with a cost ledger that decisions read.
M5 · Independent acceptanceLater — Gate G5 A reviewer that did not build it re-runs G1–G4 from the artifacts.
M6 · Approach arena, pilotLater — Gate G6 5–10 candidates on the frozen contract: fixed rules, feedback control, self-tuning, optimization, AI-proposed policy, plus random and do-nothing controls.
M7 · Arena at scaleLater — Owner unlock More candidates, sealed holdout worlds scored once. The candidate count is set from the M4 speed measurement.
M8 · World 2Later — Gate G7 A second domain with different physics on the same contract.
M9 · Views, results and datasetLater — Designed World view with time scrubber and branch comparison, ripple graph, results centre and dataset export.
Sparks of Weight-Space Programming, Takehi, Teoh, Zhang, Khattab, MIT CSAIL, Nov 2026. A model reads and edits another network’s weights in a loop. We wrote weights by formula, once; the loop itself is the planned rework.
Weight-space programming with DSPy, cmpnd.ai. Their threshold rule was tested as a probe. It reduced the damage inconsistently and did not change the verdict.
Weight-Space Reading, Şuayp Talha Kocabay. Models given weight access predicted the effect of edits better than from text alone. Background for a proposer that reads the head before editing it.
AgentAbstain: Do LLM Agents Know When Not to Act?, Liu, Zhang, Kasprova, Rabbani, Zahraei, Zhang, Ebrahimpour-Boroojeny, Chandrasekaran, NeurIPS 2026. The Tier 1 case pairs, adapted from its released data (CC-BY-4.0).