AI LAB

How I work

Three research programs on one question: when a system has earned the right to act without a person. Each runs on the same rules: a plan locked before the run, a sealed test set, and a check by a model that did not do the work.

ACTIVE EXPERIMENTS

Vision

When can a vision model act alone, and when should a person check?

A staged inspection system that escalates only on doubt: a cheap specialist decides when it is sure, a second opinion runs before any release, and a person has the final say on anything unsure.

Goal. Take as much review work off people as possible without letting one more defect through, and prove it on images the system has never seen.

How it went

  1. Client calibration — Delivered
    A calibrated model for a production QA/QC project. Delivered. Next: One model still decides every image alike. Which layer should look, and when a person steps in, became the question.
  2. Cascade feasibility Oct 22 — Exploratory: release not safe yet
    Better perception did not mean safe release: 30–36 wrong auto-accepts on surface parts. Next: Release needs a calibrated, per-category gate, not a better score alone.
  3. Independent audit: two layers Oct 28 — Exploratory: second layer catches more
    The second layer caught 63 defects the first missed. Economics unresolved. Next: Calibration per category plus a selective second layer looked worth a sealed test.
  4. Surface parts, sealed test Oct 28 — Passed on a sealed test
    Good parts released without a person: 4 of 100 → 84 of 100. Defects released: 0 of 200. Next: Surface worked. The same method had to be tried on kits, where faults are about what is missing or misplaced.
  5. Kits: logical faults Oct 28 — Did not qualify
    Ten candidates, none met the rule written in advance. Every kit goes to a person. Next: Image anomaly scores cannot prove a kit is complete. A kit has a writable specification (which parts, how many, where), so the rework writes it down as checks.
  6. Multimodal embedding pilot — Inconclusive pilot
    EmbeddingGemma scored near chance on 32 exposed kit images (AUC .48). Too small to be a negative. Next: Work moved inside the models, with vision-language models queued next.
  7. Programming weights: an anomaly head Nov 4 — Works, below trained heads
    A head written by formula, with no training, worked (AUC .816 against .516 for a placebo) but trailed trained heads. Next: The claim tested, that weights can be written, held. The mechanism the source paper used, a model editing weights in a loop, was not run; that is the rework.
  8. Programming weights: a detector edit Nov 4 — Negative, controlled
    At an equal alert budget the edit found 341 objects, the same-size placebo 338. Next: Ordinary head-only training also missed the bar, so this cohort has no headroom at the head. Not redone.
  9. Programming weights: video heads Nov 4 — Matched a trained head; one unconfirmed signal
    A programmed video head matched a trained one (+2.0 points); a low-pass edit gave +3.0, short of the +5 bar. Next: Neither confirmed nor refuted: all 25 subjects are used, so confirmation needs new data.
  10. Reader loop (Mirage) Nov 6 — Paused before any comparison
    Designed as an edit loop with a sham control. Paused by me before any arm ran. Next: Too slow and not editable on this hardware; parked.
  11. Vision-language models: frozen protocol Nov 8 — Protocol frozen, not run
    1,000 test images and prompts frozen. CPU too slow to score two models: 0 predictions made. Next: Rather than shrink the test to fit, the comparison waits. One model fits in about 8.3 hours on one core.
  12. Removing a planted shortcut: Rounds 1 and 2 Nov 13 — No arm passed (21 of 21, development only)
    21 registered attempts of head-level edits, deletion and weight averaging. None passed all six gates; retraining on the same examples failed too. Next: Two probes asked why, before trying again.
  13. Two probes: deletion and threshold Nov 14 — Exploratory: closest miss found
    Deleting 5% of one layer came closest: 64% of the shortcut removed, accuracy kept, 5 of 6 gates. Next: The edited weights, not the threshold, were the limit. Round 3 tests the input instead: mask the object, or read only the object.
  14. Round 3: mask the object, read the object Nov 18 — Running now
    Running now. Both encoders extracting features; results expected early Nov 19. Next: If an arm passes the gates, mechanistic grounding tests whether it is right for the right reason.
  15. Kits rework: written part checks Next — Prepared, not run
    Presence, count, position, extent and co-occurrence checks over patch tokens. Built Nov 13, 33 tests passing, 138-image holdout sealed. Next: Then the improve loop on development, then the holdout once.
  16. The improve loop Next — Designed
    A model writes the next candidate from the last failure, with a placebo loop and an equally tuned baseline. Up to nine rounds per study.
  17. Mechanistic grounding Later — Designed
    Designed, not run: find and steer the features behind a defect call.

Open Vision in the interactive Lab →

ACTIVE EXPERIMENTS

Agent Detention

When has an AI agent earned the right to act alone?

A performance review for agents. Each one faces the same cases as its peers, must decide whether to act or hold, and when it fails it writes a root-cause analysis before it may retry. A held-back check, not its own grader, decides whether it improved.

Goal. Find out whether a structured root-cause review helps an agent recover better than a plain retry or free-form reflection, and whether the agent reports honestly what it actually did.

How it went

  1. Real orchestration failures — Anecdote: the reason for the instrument
    Agents in my own multi-agent workflows failed at handoffs and claimed work done that was not. Next: An agent that wrote an incident report after failing improved. That became the Detention idea.
  2. The instrument Nov 12–14 — Built
    64 failure mechanisms (48 for development, 16 sealed), signed scoring, a pre-registered Detention protocol. Next: Next, run it on real models.
  3. Provisional pilots on small models Nov 14 — Provisional: models below a heuristic
    Two 432-episode pilots on free hosted models. Model failures ran up to 31% for one route. Next: An independent review found the instrument had never seen the behaviour it exists to measure.
  4. Independent review: not ready Nov 16 — Not ready
    Not ready for the main grid. In 25 delegation attempts every handoff was rejected, because the harness offered an action its own checker refused. Next: The cases were too thin. They were rebuilt on public agent benchmarks, with a small frontier smoke run before any main grid.
  5. Public benchmark suite Nov 18 — Built
    264 act-or-hold pairs from public benchmarks, 528 instances, all admitted. Next: Before any live run, the cases had to pass the same controls offline.
  6. Offline control gate Nov 18 — Passed
    Controls behave: always-act and always-hold score 100/0 and 0/100, a coin flip lands near half. Next: With controls clean, the first live smoke run.
  7. First live smoke run Nov 18 — Clean re-run passed
    Clean re-run passed: 41 calls on one model, a Detention cycle that completed all four arms, and a rebuilt audit that checked every call and found no leaks. One act version failed again, the same case as the first run. Next: Next: grade two models side by side on the same cases.
  8. Graded side by side Next — Not run yet
    Same cases across models, including other providers.
  9. Detention Next — Not run yet
    Root cause before retry; the held-back check decides.

Open Agent Detention in the interactive Lab →

IN PLANNING

World model

Do methods still hold when outcomes come from physical things interacting?

A simulated world where every entity and item has its own properties. A unit’s weight and size set how fast it can be handled, so outcomes come from interaction rather than fixed rates, and events ripple through the system.

Goal. A continuing simulated world where every duration, capacity and failure is computed from physical properties, so methods are compared on outcomes that emerge rather than outcomes set by a fixed rate.

How it went

  1. Path Lab: the first world (retired) Oct–Nov 17 — Retired
    17 decision subsystems and a synthetic dataset, built in layers on fixed rates. Retired Nov 17. Next: Only the pick side behaved as one system. Elsewhere, fixed rates and generous capacities meant decisions could not interact, so the next world starts from physics, with a connectivity audit that must pass before any experiment.
  2. Design brief Nov 17 — Done
    What the world must be, and what it must make impossible, written for the full design pass. Next: Next, the full design down to builder tasks.
  3. M0 · Design pass Next — Gate G0
    World spec, contract, parameters, approaches, experiments, views, data and builder tasks, each task with its known-answer test written first.
  4. M1 · World kernel Next — Gate G1
    Event engine, entities with units, one clock, seeded randomness, checkpoints and branches.
  5. M2 · Physics laws Next — Gate G2
    Handling time from mass, size and equipment; travel from distance, speed and load; capacity from cube and weight; energy, wear and fatigue.
  6. M3 · World 1: a small supply network Next — Gate G3
    3–5 nodes with links, vehicles, buffers and shared resources, run as one continuing world for at least 90 simulated days.
  7. M4 · Scale and cost Later — Gate G4
    Design volume at full size, no shrinking, with a cost ledger that decisions read.
  8. M5 · Independent acceptance Later — Gate G5
    A reviewer that did not build it re-runs G1–G4 from the artifacts.
  9. M6 · Approach arena, pilot Later — Gate G6
    5–10 candidates on the frozen contract: fixed rules, feedback control, self-tuning, optimization, AI-proposed policy, plus random and do-nothing controls.
  10. M7 · Arena at scale Later — Owner unlock
    More candidates, sealed holdout worlds scored once. The candidate count is set from the M4 speed measurement.
  11. M8 · World 2 Later — Gate G7
    A second domain with different physics on the same contract.
  12. M9 · Views, results and dataset Later — Designed
    World view with time scrubber and branch comparison, ripple graph, results centre and dataset export.

Open World model in the interactive Lab →

Research in application

← Back to the portfolio