On July 18th, AGI House hosted the Auto-Research Summit & Build Session in Hillsborough — keynotes and interviews with Jerry Tworek (Core Automation, ex-OpenAI), Pushmeet Kohli (Google DeepMind / Google Cloud), and Kilian Lieret (Meta FAIR), followed by a build day of 18 projects. This is the first of three dispatches; see also our deep-dive interview with Jerry Tworek and the builder recap, plus the research brief we published before the event.

Consider who was in the room on the same day. The researcher who led the OpenAI team behind o1, o3, and the reasoning inside GPT-5, now betting a company on automating research itself. The founder of DeepMind's AI-for-Science unit — the group behind AlphaFold, a Nobel Prize, and AlphaEvolve. The creator of SWE-bench and SWE-agent, the evaluation substrate most coding agents are measured against. Plus founders out of xAI and Essential AI building the autoresearch stack, and 18 teams that spent the afternoon building against the open problems the speakers posed that morning.

What struck us wasn't the agreement — though there was a lot of it, and it converged on loops, test-time learning, and production signal. It was the one disagreement, between the two most senior people in the room, about what survives the transition: verification. This memo covers both.

Test-time compute is becoming experiment compute

Jerry Tworek's keynote (watch it here) made a five-beat argument that doubles as the field's new mental model:

  • Reasoning performance scales smoothly with test-time compute — the o1-era result.
  • Therefore autoresearch is about scaling test-time compute: the question is what to spend inference on.
  • The answer is loops — propose, run, verify, keep or revert — not longer monologues.
  • The famous loops already exist: GPU kernel generation contests, AlphaEvolve, and Karpathy's autoresearch repo, the 630-line artifact that set off this year's wave (and which our pre-event brief covers in depth).
  • Closing slide: "More inference can mean more experiments."

Here's the reframe: Test-time compute v1 meant a longer chain of thought on one problem. V2 means more iterations of the research loop — and the unit of inference spend shifts from the token to the experiment.

Tworek was candid about the current limits, which is what kept the keynote from being a pitch. He said his teams never found a meaningful ceiling on test-time compute itself; the binding constraints are latency — when a rollout takes a day, "there are only so many days in a year" to iterate on training — and imperfect long-context generalization. He was blunter still about parallel scaling: sub-agents currently burn tokens with worse marginal value, and users notice their usage vanishing into sub-agent work "that wasn't that useful."

Three speakers buried the static benchmark

The most striking convergence of the day was three people arriving at the same funeral from different directions.

Tworek, from the training floor: "The era of evals is dumb." Benchmarks now launch around 50% and get maxed within two or three model generations; with RL at scale and everyone building environments, "whatever benchmark you have, you will max it." His alternative comes from his years at an Amsterdam quant firm: evaluate the process, not the point. Backtest with strict causal splits — "every trader has made the mistake of leaking future information" — then run live A/B deployments. "The only test is the real world, and the only actual measurement is using those models for automating work."

Lieret, from inside the benchmark house: the man who built SWE-bench pointed out that SWE-bench is now as old as HumanEval was when SWE-bench launched — and that the field's responses (harder tasks, stricter grading) all saturate quickly. His claim: most benchmarks are stuck in a 2023 paradigm, and the real jump is from task-oriented evaluation (fix this bug; unit tests can grade it) to goal-oriented evaluation (earn money, cure a disease, prove a theorem). "How do you score high-level goals? That's the billion-dollar question." His full argument is in his research spotlight.

NeoSigma, from production: the hardest failures don't appear on public benchmarks at all — they live in the long tail of production traffic, which is why their improvement loops draw rewards directly from user feedback, telemetry, and business outcomes.

Three independent vantage points, one conclusion: point-in-time scoring against a public target no longer measures what matters.

The room seemed ready to bury evaluation altogether. Then Pushmeet Kohli pushed back.

But verification isn't dying — it's being promoted

Our moderator flagged the disagreement live, and Kohli took it head-on rather than smoothing it over. His dissent had two parts.

First, the complexity-theory argument. Some problems are fundamentally easier to verify than to solve, and that asymmetry is a lever, not a legacy: "if you're not utilizing it, you are leaving performance on the table." Note that this is not a defense of leaderboards — it's a claim about where verification belongs: inside the loop, as the reward signal. There's an irony worth naming: Tworek's own breakthrough — scaled RL on reasoning — ran on exactly this asymmetry. Verifiable code and verifiable math are why o1 exists.

Second, the argument nobody else made: verify the behavior, not just the solution. As agents gain agency, "it's not just about the final solution — it's about the way the agent is behaving." Then the analogy of the day:

"The emphasis in the whole community is on developing LLM performance — it's like engine development in Formula One. Everyone wants the fastest engine. But if you give a Formula One car to everyone on the planet, you know what's going to happen: crashes. It's not just about having the fastest car. You need seat belts."

Agent control, certainty, and trust, he argued, "require much more attention than they are currently getting."

Our read is that Tworek and Kohli are attacking different targets, and both are right about theirs. Tworek buries the static public benchmark as a measure of model quality. Kohli defends verification as a primitive — for training rewards, for safety, for trust — and worries the field is under-investing in it precisely as agents get more autonomous. Both relocate verification from the leaderboard into the loop. Where they genuinely differ is whether deployment signal alone is sufficient: Tworek says the real world is the only test; Kohli says unverified agency at scale is the Formula-One-for-everyone scenario.

Evals are dying. Verification is being promoted.

(Lieret's CodeClash sits exactly at the midpoint — competition itself as the verifier, unsaturable by construction. More on it in the builder recap.)

DeepMind picks moonshots with an algorithm — and you can steal it

Asked how AI for Science chose its bets across nine years, Kohli described not taste but an explicit algorithm, applied program after program. Four filters:

  1. Transformative by consensus. Solving it means a Nobel Prize or hundreds of billions of dollars in value. No ambiguity about whether it matters.
  2. Inevitable by consensus. No theoretical barrier to a solution. "We did not want to waste our time working on time travel."
  3. Believed impossible within five to ten years. "If a couple of professors at Berkeley or Stanford or MIT were working on that problem with their students, I did not want to touch it." This filter does double duty: it removes competition with academia and guarantees the problem isn't low-hanging fruit.
  4. Internal conviction of a 2–3x speedup — via multidisciplinary teams, data, compute, and, crucially, scaling behavior already observed in that specific problem. That's the tell that a moonshot is ripe.

The receipts: protein structure prediction passed all four filters and became AlphaFold 1 through 3, the 2024 Nobel Prize in Chemistry, and Isomorphic Labs. One or two new programs have launched every year since; some graduate into companies. Even SynthID, DeepMind's watermarking system now adopted well beyond Google, began as one of these moonshots.

For the founders and investors in our audience, translate the filters and you get a portfolio-construction algorithm: filters one and two are market size and technical feasibility; filter three is contrarian timing; filter four is founder-market fit stated as a falsifiable claim. It is the most operationally useful thing anyone said all day.

Four companies are independently building the same loop

Step back from the talks and a pattern emerges that no single speaker claimed, but every company on stage embodied: deploy → capture real-world signal → construct a reward or judge → improve → redeploy.

  • Core Automation (Tworek, ex-OpenAI): test-time training as the product. His diagnosis of today's memory scaffolds — markdown files, retrieval, context tricks — is that they rhyme with AutoGPT in 2023: scaffolding on a model that was never trained end-to-end for the job. "Those are machine learning problems, not markdown-file problems." His thought experiment: how useful would today's frozen models be if every training cluster shut down tomorrow? The existence of "freshness fine-tunes" is the symptom.
  • Introspection (ex-xAI): a dual loop — an inner loop turns signals into work; an outer loop distills learnings back into the agent's recipe. Traces become patterns, patterns become judges, judges grade experiments, experiments promote new versions to prod.
  • NeoSigma (ex-Essential AI): agents that improve from experience, hill-climbing the production failure long tail with rewards drawn from real outcomes.
  • Daytona, underneath all of it: when an RL run cycles thousands of sandboxes, spin-up latency is wasted cluster compute. Infrastructure is the loop's clock speed.

Notice that every one of these loops has a judge inside it. The convergence is quiet evidence for Kohli's position.

And the business-model implication got said out loud — twice, by senior people from competing orgs. Kohli's hot take of the day: "We have developed this amazing source of superhuman intelligence, and the best we can do is sell it by the token... I think it's not a reasonable option." Tworek made the same argument from the demand side: enterprises want a model that compounds on their own data without that data ever leaving — "continual learning is solving personalization the way it should be solved" — and "I don't think anyone actually wants to be fine-tuning open-weight models." Frozen intelligence priced per token versus intelligence that learns on your deployment: two of the day's most senior speakers agreed the token-API era is a waypoint, not an endpoint.

The honest bottleneck is idea diversity

Our pre-event brief argued that the constraint on autoresearch isn't experiment throughput — models execute tirelessly — but proposing what's worth trying. The day kept confirming it:

  • Tworek's own open-problems slide listed diversity/exploration first, ahead of better models and test-time training.
  • Lieret's CodeClash — a PvP benchmark where models write codebases that compete in arenas over multiple rounds — produced the most sobering datapoint of the day: relatively simple human solutions beat roughly 90% of model submissions, even from frontier models. Out of distribution, "models just kind of flop around."
  • The winning build-day project, Vibes Lab for Auto Research, attacked the problem directly by injecting human research judgment into scientific autoresearch loops. (All four winners are covered in the builder recap.)

Collate the caveats the speakers themselves supplied — sub-agents burning usage without value, models losing to simple human baselines, unverified agency at scale — and you get the field's counter-melody, sung by its own leaders. The loop works. The loop is not yet wise.

"Ask the right questions"

Kohli's parting advice to the builders wasn't about using the technology at all. "You have a front-row seat to what is possible. The real challenge will be to take this technology and ask it the right questions — what are the questions that will completely transform society, which society needs?" That, plus agent security, were his two nominations for the most neglected problems in the room.

And his answer to where the next breakthroughs land was the day's most expansive image. The genome: "There are three billion tokens — three billion base pairs — that define you and me and everyone in this room. And we don't know the meaning of those tokens. If you change one token, what does it mean? What will happen?" And materials: our whole history is named for the materials we mastered, from the Stone Age to the Silicon Age — "imagine what will be the material of the next age, and how it will be discovered."

That discovery, he's certain, will have AI in the loop. The question the summit left open is only how much of the loop will be human.

This memo is part of our Auto-Research Summit series. Continue with the deep-dive interview with Jerry Tworek — the inside history of scaling RL at OpenAI — and the builder recap covering all 18 projects and the autoresearch stack. Recordings: Tworek's keynote, Tworek's interview, and Lieret's research spotlight. More recordings and demos are coming to the AGI House platform.