Inherent’s Faraday points to replication, not raw benchmarks, as a training path for AI scientist agents

Known model/system details. Inherent’s paper says Faraday is produced by post-training Qwen3.6-27B and uses Codex GPT-5.5 as a coding-agent tool in later training/evaluation; earlier training stages used GPT-5.4 mini as the tool. The Replica task environment gives the agent a container, the redacted paper PDF, research libraries, internet access, and a one-seventh MIG slice of an H200 GPU under a one-hour task limit. Training AI Scientists to Replicate Research

Access status. Faraday does not appear to be a generally released public model. In the interview, Hughes says Inherent does not think the model is “general or reliable enough” for arbitrary release, while inviting selected collaborators with use cases (01:46:41). That access limitation matters: outside researchers cannot yet fully reproduce end-to-end claims unless code, model weights, task set, judge prompts, and infrastructure are made available or independently rebuilt.

Architecture: “scientist layer” over coding agents

Faraday’s key design pattern is coding agent as tool: a smaller outer model acts as the research planner/supervisor, while a stronger coding agent executes implementation-heavy work. In the paper’s setup, Faraday can invoke a non-interactive Codex CLI wrapper, receive transcripts of commands and outputs, resume or reset sessions, and run multiple coding agents in parallel. This is not a multi-agent debate system or a hard-coded scientific workflow; it is a relatively simple container harness plus a trained policy for delegation. Training AI Scientists to Replicate Research

The practical implication is that “agent capability” may increasingly reside in orchestration policies: deciding what experiment to run, when to scale down, how to check a result, and when to stop or redirect a coding assistant. In the interview, Hughes contrasts coding agents as “pretty good engineers” with the missing layer of scientific question-asking and judgment (01:19:30).

Replica: replication as an underspecified training distribution

Replica converts papers into tasks by redacting a results plot and asking the agent to recreate the experimental result without seeing the original plot. The paper reports 242 training tasks and 68 test tasks drawn from 100 well-known ML and AI-for-science papers; training tasks are ML papers from 1990–2026, while test tasks are AI-for-science papers from 2012–2026. Gemini 2.5 Pro is used in a pipeline to identify and redact main-text plots, followed by human filtering. Training AI Scientists to Replicate Research

This is a deliberate construct-validity choice. Exact visual reproduction is not the full target; the reward asks whether the agent implements the underlying experiment, supports the paper’s scientific claim, uses resources sensibly, and avoids cheating. Training AI Scientists to Replicate Research In the interview, Hughes calls this “deep replication,” the first step in a “curriculum of underspecification” from replication toward innovation (01:07:39, 01:09:19).

Evaluation and RL recipe

The reward is an LLM-based judge using task-specific rubrics. Claude Opus 4.7 generates rubrics from the redacted paper context, while Codex GPT-5.5 evaluates rollouts against rubric dimensions. The judge can inspect the generated code, outputs, git history, transcript, workspace and gold plot. The paper reports that rubrics cover visual fidelity, claim reproduction, experimental implementation, resource use and scientific integrity. Training AI Scientists to Replicate Research

Post-training uses a modified GRPO recipe: LoRA rank 128, alpha 128, adapters on all linear projections, 128K context, constant learning rate of 6e-6, batches of 10 tasks with eight rollouts each, and multi-sample judge aggregation. The authors report that long-horizon non-verifiable RL was unstable until they added multi-sample judging and turn-level credit assignment, where the judge assigns weights to agent turns and those weights scale per-token advantages. Training AI Scientists to Replicate Research

For implementers, the most transferable engineering idea is not the specific benchmark but the recipe: combine a process-aware judge, repeated judge samples, and credit assignment over long trajectories, rather than treating an hour-long rollout as one undifferentiated reward event.

Claim: Faraday outperforms Claude Opus 4.8 and GPT-5.5 on Replica. This is author-reported, based on Inherent’s own benchmark and rubric judge. The paper reports Faraday outperforming both Claude and Codex on 73% of in-distribution ML tasks and 60% of held-out AI-for-science tasks; average test improvements are reported as 6% over Claude and 8% over Codex. These numbers should not be treated as independently verified performance. Training AI Scientists to Replicate Research

Claim: human experts partially validate the judge and Faraday advantage. The paper’s human evidence is mixed. For judge comparison, 19 participants produced 76 rankings, and humans sided with the rubric judge on 63% of disputed pairs, but the result was not statistically significant at conventional thresholds. For agent comparison, 11 participants produced 41 rankings on tasks selected where the rubric judge already gave Faraday a large advantage; humans preferred Faraday over both baselines in 71% of those rankings. The authors explicitly say this supports only a conditional claim, not an average human preference across the benchmark. Training AI Scientists to Replicate Research

Claim: Faraday shows more rigorous scientific behavior. This is primarily author qualitative analysis. The paper gives examples where Faraday implements mechanisms rather than hard-coding expected outputs, runs more faithful scale-downs, or reports uncertainty where a baseline used one seed. These examples are useful diagnostics but not independent proof of general scientific judgment. Training AI Scientists to Replicate Research

Claim: replication can lead toward innovation. This is a research hypothesis. The paper tests 20 “imagined” variants and reports Faraday beating Codex on 19/20 according to its rubric judge, while warning that the judge was not validated for these imagined tasks. Training AI Scientists to Replicate Research Hughes makes the same argument in the interview by describing replication as a way to recover tacit experimental decisions and then alter constraints to generate new results (01:15:31).

The episode sits in a fast-moving line of AI-for-research systems. Sakana AI’s 2024 AI Scientist paper presented a framework that generates ideas, writes code, executes experiments, visualizes results, writes papers and runs simulated review; it also used automated reviewing, which drew attention to the difficulty of evaluating AI-generated science. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery Google DeepMind’s AlphaEvolve uses LLM-guided evolutionary search over code, but it is strongest where candidates can be automatically evaluated with scalar metrics. AlphaEvolve: A coding agent for scientific and algorithmic discovery Hughes’s 2024 open-endedness position paper, co-authored while at DeepMind, argued that open-ended systems should produce artifacts that are both novel and learnable to an observer, and that open-endedness is central to artificial superhuman intelligence. Open-Endedness is Essential for Artificial Superhuman Intelligence

Inherent’s differentiation is therefore clear: instead of fully automating paper production or optimizing known programmatic objectives, it trains on replication under ambiguity. The economic motivation also echoes Bloom, Jones, Van Reenen and Webb’s finding that research effort is rising while measured research productivity is declining in multiple domains. Are Ideas Getting Harder to Find?

The largest limitation is evaluator circularity: Faraday is trained using a rubric judge and then mostly evaluated using that same evaluation paradigm. Third-party analysis from Pith Review characterizes the contribution as real but “partly circular,” noting weak human-validation evidence and dependence on the judge. Training AI Scientists to Replicate Research · Pith Review Emergent Mind’s summary likewise flags limited benchmark scope, incomplete contamination analysis, baseline-control issues, weak validation of larger-scale experiments and unvalidated innovation claims; this is third-party analysis, not an independent reproduction. Training AI Scientists to Replicate Research

The paper also acknowledges pretraining contamination risk: source papers and figures may be in frontier models’ training data, though the authors argue that the unpublished process data behind figures is not. Training AI Scientists to Replicate Research Faraday failures do not prove original papers are wrong, and humans still need to inspect agent replication results. Training AI Scientists to Replicate Research On safety, the authors selected in-silico tasks they judged unlikely to cause harm, constrained time and compute, and gave Faraday no physical lab access, though it did have internet access. Training AI Scientists to Replicate Research

For business leaders, Inherent’s thesis is that value may shift from chatbots and coding copilots toward AI-accelerated R&D systems that combine models, tools, human expertise, memory, permissioning and organizational design. In the interview, Hughes describes a “recursive company” where agents share enough company context and affordances to become proactively useful (01:47:20). He argues the next era is “collective intelligence” rather than one-to-one agent use (01:50:35).

For practitioners, near-term adoption should focus on bounded workflows: replication checks, experimental ablation planning, literature-to-code reconstruction, internal benchmark generation, and research-review assistance. Do not delegate scientific sign-off to agents. Build audit logs, reproducible containers, human review gates, contamination checks, and explicit norms around acceptable scale-downs, stopping rules and negative results.