Good point, I buy it

It doesn't seem that surprising, conditioned on everything being linear

Eg Wood Labs + eval prompt is 1.0 eval-ness Wood Labs + deployment prompt is 0.7 eval-ness

A steering vector should be able to amplify this, if the noise isn't too bad, to get it below 0

despite the fact that the model does not believe you.

Why do you think the model doesn't believe us? We get this from the original model, and these are the reflexive activations formed with no reasoning, so it seems like it should be pretty dumb

On your broader point, I agree, but I think what Tim really meant was that it was unlikely to be smuggling in a given specific other concepts like "be evil"

My guess is that our work provides little evidence on that question, and is fairly consistent with both worlds. I would personally guess that models with deep self control are fairly far off, but would love further work there

Imo a significant majority of frontier model interp would be possible with the ability to cache and add residual streams, even just at one layer. Though caching residual streams might enable some weight exfiltration if you can get a ton out? Seems like a massive pain though

I disagree. Some analogies: If someone tells me, by the way, this is a trick question, I'm a lot better at answering trick questions. If someone tells me the maths problem I'm currently trying to solve was put on a maths Olympiad, that tells me a lot of useful information, like that it's going to have a reasonably nice solution, which makes it much easier to solve. And if someone's tricking me into doing something, then being told, by the way, you're being tested right now, is extremely helpful because I'll pay a lot more attention and be a lot more critical, and I might notice the trick.

That seems reasonable, thanks a lot for all the detail and context!

Can you say anything about what METR's annual budget/runway is? Given that you raised $17mn a year ago, I would have expected METR to be well funded

Interesting. A related experiment That seems fun: Do subliminal learning but rather than giving a system prompts or fine-tuning you just add a steering vector for some concept and see if that's transfers the concept/if there's a change in activations corresponding to that vector. Then try this for an arbitrary vector. If it's really about concepts or personas then interpretable steering vectors should do much better. You could get a large sample size by using SAE decoder vectors

Speaking as someone who does not mentor for the program, I agree! Seems like a high calibre of mentors and fellows