Publications

We introduce a task-completion time horizon metric to benchmark frontier AI on software engineering tasks, finding that AI time horizons have been doubling approximately every seven months since 2019.

We present HCAST, a benchmark of 189 tasks with over 1500 hours of human baselines, finding that current AI agents succeed 70-80% on tasks taking humans less than one hour but under 20% on tasks taking more than 4 hours.

We present the first case study in rigorously compressing nonlinear feature-maps, discovering that the MLP in modular addition models can be understood as evaluating a quadrature scheme.

We formally study what sparse circuits neural networks can represent and compute in superposition – that is, using sparse features respresented near-orthogonally in a high-dimensional space. We show that not only can ReLU MLPs perform more computation at a fixed given width by placing features in superposition, but also that they can maintain superposition through multiple layers.

We use mechanistic interpretability to general compact formal guarantees on model performance on a toy Max-of-K task. We find that shorter proofs seem to require more mechanistic understanding and more faithful mechanistic understanding leads to tighter performance bounds. We identify compounding structureless noise as a key challenge for the proofs approach.

We create four agents from Claude and GPT-4 to investigate the ability of frontier language models to perform autonomous replication and adaptation.

We reverse engineer small transformers trained on group composition and use our understanding to explore the universality hypothesis.

We reverse engineer 32 small transformers trained on modular addition and use this understanding to study grokking.

We compare humans to small language models on next-token prediction tasks, and find that even relatively small language models consistently outperform humans.

We argue that AGIs trained in similar ways as today’s most capable models could learn to act deceptively to receive higher reward; learn internally-represented goals which generalize beyond their training distributions; and pursue those goals using power-seeking strategies.

We introduce a more principled approach for evaluating the quality of mechanistic interpretations via behavior-preserving resampling ablations—converting hypotheses into classes of activations inside a neural network can be resampled without affecting behavior.

We study the limits of adversarial training on a injury classification task using a variety of attacks including a new saliency map–based snippet rewrite tool to assist our human adversaries. While advesarial training was not able to eliminate all in-distribution failures, it did increase robustness to the adversarial attacks that we trained on.

We study the effect of human irrationality on reward inference. We find that irrationality can both help and hurt reward inference, depending on the type of irrationality. We also find that the effect of irrationality on reward inference is not monotonic in the degree of irrationality.

Many robotics algorithms use model predictive control (MPC) for planning, which optimizes a cost function over a finite time horizon to determine the next action. Typically, the cost function for MPC is identical to the ground-truth task cost used for evaluation. In this work, we challenge this common practice, and propose that a different cost function for MPC to optimize can yield better task performance. This is because MPC is an imperfect planner – it has a limited horizon and uses an imperfect model. We formalize this as an optimal cost design problem, and propose solving it with zeroth-order optimization. We test our approach in a few autonomous driving scenarios in simulation, where the learned costs lead to qualitatively interesting emergent driving behaviors.

We illustrate the benefits of agents that try to assist humans, over agents that learn a reward during training and then maximize said reward after deployment.

Inverse reinforcement learning (IRL) is a powerful tool for learning reward functions from demonstrations. However, standard IRL assumes that the human demonstrator is stationary; that is, the human’s policy does not change during the demonstrations. In this paper, we study IRL when the human is learning – that is, the human’s policy improves over the course of the demonstrations. We show that observing a learning human can be more informative about their preferences than observing a human with a fixed policy, and that standard IRL techniques perform poorly when the human is learning in an unfamiliar environment.

We study the problem of learning to assist a human who is themselves learning about their preferences.