Authors: Senthooran Rajamanoharan*, Arthur Conmy*, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, Neel Nanda

A new paper from the Google DeepMind mech interp team: Improving Dictionary Learning with Gated Sparse Autoencoders! 

Gated SAEs are a new Sparse Autoencoder architecture that seems to be a significant Pareto-improvement over normal SAEs, verified on models up to Gemma 7B. They are now our team's preferred way to train sparse autoencoders, and we'd love to see them adopted by the community! (Or to be convinced that it would be a bad idea for them to be adopted by the community!)

They achieve similar reconstruction with about half as many firing features, and while being either comparably or more interpretable (confidence interval for the increase is 0%-13%).

See Sen's Twitter summary, my Twitter summary, and the paper!

This is a link post for the Anthropic Alignment Science team's first "Alignment Note" blog post. We expect to use this format to showcase early-stage research and work-in-progress updates more in the future.

Twitter thread here.

Top-level summary:

In this post we present "defection probes": linear classifiers that use residual stream activations to predict when a sleeper agent trojan model will choose to "defect" and behave in accordance with a dangerous hidden goal. Using the models we trained in "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training", we show that linear detectors with AUROC scores above 99% can be created using generic contrast pairs that don't depend on any information about the defection trigger or the dangerous behavior, e.g. "Human: Are you doing something dangerous? Assistant: yes" and

YouTube link

In 2022, it was announced that a fairly simple method can be used to extract the true beliefs of a language model on any given topic, without having to actually understand the topic at hand. Earlier, in 2021, it was announced that neural networks sometimes ‘grok’: that is, when training them on certain tasks, they initially memorize their training data (achieving their training goal in a way that doesn’t generalize), but then suddenly switch to understanding the ‘real’ solution in a way that generalizes. What’s going on with these discoveries? Are they all they’re cracked up to be, and if so, how are they working? In this episode, I talk to Vikrant Varma about his research getting to the bottom of these questions.

Topics we discuss:

Abstract

This paper presents $\text{ProLU}$, an alternative to $\text{ReLU}$ for the activation function in sparse autoencoders that produces a pareto improvement over both standard sparse autoencoders trained with an L1 penalty and sparse autoencoders trained with a Sqrt(L1) penalty.

$\text{ProLU}(m_i,b_i)=\{\begin{array}{cc}{m}_i& \text{if}{m}_i+b_i>0\text{and}{m}_i>0\\ 0& \text{otherwise}\end{array}$

The gradient wrt. $b$ is zero, so we generate two candidate classes of $\text{ProLU}$ differentiable wrt. $b$:

• ${\text{ProLU}}_{ReLU}$
  • $\frac{{\partial }^{\ast }{\text{ProLU}}_{ReLU}(m_i,b_i)}{\partial b_i}=\frac{\partial {\text{ProLU}}_{ReLU}(m_i,b_i)}{\partial m_i}=\{\begin{array}{cc}1& \text{if}{m}_i+b_i>0\text{and}{m}_i>0\\ 0& \text{otherwise}\end{array}$
• ${\text{ProLU}}_{STE}$
  • $\frac{{\partial }^{\ast }{\text{ProLU}}_{STE}(m_i,b_i)}{\partial m_i}=\{\begin{array}{cc}1+m_i& \text{if}{m}_i>0\text{and}{m}_i+b_i>0\\ 0& \text{otherwise}\end{array}$
  • $\frac{{\partial }^{\ast }{\text{ProLU}}_{STE}(m_i,b_i)}{\partial b_i}=\{\begin{array}{cc}{m}_i& \text{if}{m}_i>0\text{and}{m}_i+b_i>0\\ 0& \text{otherwise}\end{array}$

PyTorch Implementation

Introduction

SAE Context and Terminology

Learnable parameters of a...

The Löwenheim–Skolem theorem implies, among other things, that any first-order theory whose symbols are countable, and which has an infinite model, has a countably infinite model. This means that, in attempting to refer to uncountably infinite structures (such as in set theory), one "may as well" be referring to an only countably infinite structure, as far as proofs are concerned.

The main limitation I see with this theorem is that it preserves arbitrarily deep quantifier nesting. In Peano arithmetic, it is possible to form statements that correspond (under the standard interpretation) to arbitrary statements in the arithmetic hierarchy (by which I mean, the union of ${\Sigma }_n^0$ and ${\Pi }_n^0$ for arbitrary n). Not all of these statements are computable. In general, the question of whether a given statement is...