I've returned to the Alignment Research Center (ARC) as executive director. My main focus for the next six months will be driving forward ARC's research agenda—building techniques to find mechanistic explanations for neural network behavior and then using those explanations to detect and address misalignment. I think this is an ambitious bet that attacks the core difficulties in alignment head-on and I'm excited about our chances. I'll still be spending some of my time advising governments[1] and AI developers, and may scale that work back up in the future, but for now I want to push on ARC's core agenda to see how far we can get. Jacob Hilton is remaining at ARC as VP of research and we'll likely grow rapidly over the next few months.
There are a lot of urgent things to do in alignment but I think ARC is a particularly promising opportunity. I feel the safety community is undervaluing this type of work, so I want to briefly explain why I'm passing up so many other options to lead ARC. I’ll start with a review of the current situation to explain why I think it’s potentially worth pursuing an ambitious theoretical project right now even though there is so much other pressing work to do. Then I’ll briefly discuss ARC’s approach, though I’ll defer a longer discussion to other posts.
By the same token, if you care about AI safety I'd strongly recommend considering working at ARC! We're currently hiring researchers, a chief of staff, and an automation lead.
The alignment situation today
Modern AI systems are not aligned with human intent. We will likely train increasingly powerful models that take unintended actions in order to succeed at their task (or appear to succeed at their task). Developers can create environments that penalize bad behavior and then train or hill-climb on those environments. Unfortunately this could lead to models that still seek reward but simply try harder to avoid being penalized for bad behavior; the last few years of evidence have only made that failure look more likely.
If we train powerful reward-seekers who are trying to avoid getting penalized for bad behavior, they may eventually escape human control as a way to cheat without being penalized. An AI developer would try to detect the problem, reassert control, and then retrain the AI to avoid the behavior—but that just means a penalty-avoiding AI would be incentivized to prevent humans from detecting the problem and reasserting control. For very powerful and broadly deployed AI systems, it is likely that they could prevent humans from ever reasserting control.
It's not clear whether scaling up current training would actually lead models to robustly seek reward, but some of the alternatives are even worse. Current AI systems are obsessively focused on their short-term goal and would only take over if doing so was useful for the immediate task. An AI with an ambitious real-world goal would be much more worrying but might actually look more aligned: AI systems are already fully aware that they are being evaluated, and an AI that simply wanted to be deployed as broadly as possible would do whatever it took to look aligned to humans in all evaluations. So if gradient descent[2] caused a smart enough system to drift[3] into ambitious long-term goals then that shift would be positively reinforced and further training might just make it smarter rather than correct its goals.
Overall it seems quite possible that we will end up with models that are either reward-seeking or scheming, either of which could cause takeover attempts. We have only very crude tools to control how models generalize and there is no way to structure behavioral training to penalize these behaviors, so if we get unlucky with generalization we may not have many options to fix the problem.
Current alignment research
The community is doing a lot of great alignment research, but it's important to recognize there is a significant risk that it doesn't scale to superhuman AI. If you made me guess I'd say that there's a 20-30% chance[4] that existing methods for alignment and control break down before we reach broadly superhuman AI.
The large majority[5] of current research on alignment falls into three categories:
Understanding and shaping ML generalization. We can try to understand how misalignment arises during training, so that we can adjust the training process to improve the probability of favorable generalization. This is a promising set of bets but they are very correlated with one another. It is possible that training-gaming[6] robustly emerges for very capable models regardless of how we set up the details of training. If that's the case then none of these approaches will work.
Preventing malicious behavior. Other research tries to build monitoring and control mechanisms so that takeover attempts would not succeed, or change training so that it does not incentivize takeover. Even people optimistic about these techniques typically expect them to break down at some point when humans with untrusted advice cannot recognize malicious behavior even if pointed out to them. We have no idea when that breakdown actually occurs, but it seems highly correlated across different methods and it's plausible that they will all fail prior to reaching broadly superhuman AI.
Detecting misalignment. We can improve our ability to detect and forecast misalignment, for example by performing better evaluations or gaining insight from interpretability. This is important work that could help motivate alignment interventions or provide the foundation for regulation. But if we do detect misalignment we may not have great responses available. We can invest more in interventions #1 and #2 above, and we can use the observed failures to improve our implementation, but those methods may reach a point where they can't work no matter how much more effort we throw at them. We could slow down unaligned AI development, but postponing it for a meaningful length of time becomes increasingly difficult and will quite possibly fail.
I think this is great work and it makes sense for many people to focus on it, but it could still all break down: generalization may converge to training-gaming, models may become able to take over in ways humans can't prevent, and we may not be able to distinguish our AI assistants identifying real threats from our AI assistants crying wolf. In that scenario, existing work is buying time for us to develop another approach.
What are we buying time for?
If we're in the 20-30% of worlds where existing techniques break down before reaching broadly superhuman AI, then we will eventually need to find some other way to build aligned AI. Even if we do a great job of buying time we'll probably get months or years rather than decades.[7] What will actually happen during the months or years we buy?
I think the most salient aspect of the situation will be the chaos and mess; all of our responses will be limited by scarce human attention and coordination, and much of that will be consumed by addressing immediate threats.
AI will be accelerating everything to varying extents but won't be broadly competent enough to obsolete human labor. Strategic decisions and conceptual research by AI systems may remain "slop" optimized for short-term performance, intensifying human bottlenecks and limiting our ability to take on ambitious projects that we haven't already systematized. Even when AI systems are very competent they will face their own difficulties with parallelization over a very short calendar time. And on top of all that it may be hard for humans to make sense of AI solutions in uncharted domains quickly enough to trust and effectively leverage them.
When I visualize that world concretely I do not find myself thinking "those future people will definitely figure it out, we should exclusively focus on buying them more time." AI will help us in the future, and buying more time could help quite a lot, but not enough to make our preparation irrelevant.
Can we do anything useful now?
If existing methods break down and we need to think of new approaches, we could do a lot of that thinking in advance. I wrote prosaic AI alignment in 2016 and laid out my best guess for what transformative AI would look like. Reading it today I think it's a good description of 2026.[8] I suspect the changes between here and broadly superhuman AI will, if anything, be even smaller than the changes over the last 10 years.
You could hope that seeing how our methods break down will provide useful inspiration for how to fix them, but at this point I wouldn't bet on it. Nothing interesting or conceptually surprising needs to change between a model that tries to hack into Hugging Face to get reward and a model that would try to take over the US government: you just scale up similar behavior to smarter and smarter models.
What is ARC doing and why is it promising?
I'm excited about ARC because I think our alignment approach is the most promising single backup plan if more prosaic methods break down.
ARC’s plan is to find mechanistic explanations for the training-time behavior of powerful neural networks, use those explanations to predict how a given model will generalize, and then use those predictions to define a better loss function.[9] I think the success of this plan rests primarily on two big bets: (i) all computational phenomena have good explanations, and (ii) it's tractable to find good explanations for neural network behavior.
Suppose that I write down a computational experiment—perhaps I simulate some billiard balls bouncing around on a pool table, or test whether a local search algorithm solves a random SAT instance, or check how often random numbers are prime. And suppose that I try to predict the result of the experiment by mechanistically analyzing the dynamics, i.e. considering various statistical properties of the computation and deductively estimating how they will evolve over time.
I will often be initially surprised by the results. Perhaps I expected the billiard balls to be uniformly distributed but they have a bias towards one side of the table, or I expected my search algorithm to work 1% of the time but instead it works 99% of the time. Despite that initial surprise, in essentially every case humanity has eventually been able to "understand" what is going on well enough that our mechanistic analysis approximately[10] agrees with the empirical results.
Some form of this belief is held by most practicing mathematicians and physicists. Timothy Gowers calls it the "no coincidence principle." Physicists don't call it anything, but when there is an unexplained phenomenon they will throw grad students at it until it makes sense.
However, in machine learning the opposite is true. If I randomly generate math problems from a simple distribution and give them to GPT-5.6 I can tell them that it solves them at a high rate, but if I want to predict that rate I cannot do it by mechanistically analyzing how GPT-5.6 works. It's a giant black box of incomprehensible arithmetic, and all I can do is run it and see how often it gets the answer right. That's a clean computational experiment with a surprising result, but unlike in math/physics/CS we seem to have no hope of explaining it any time soon.
I (and many others) think this incomprehensibility is at the root of the alignment problem. Because we don't understand why a neural network works, we have no idea what changes to the input distribution might cause it to stop working. And when a model does fail we have no way to look at its activations to tell that something has gone wrong, all we can do is wait and see whether anything terrible happens.
ARC's core hypothesis is that machine learning systems are not qualitatively different from other computational systems, they are just quantitatively more complex. A model with trillions of parameters is far beyond anything mathematicians have ever attempted, but it is in principle possible to explain how GPT-5.6 works and if you did then your mechanistic predictions about its behavior would agree with experiment just as well as our estimates in physics or number theory.
Although this is a philosophically bold bet I believe it is very likely to be true. But ARC is also making a second much riskier bet: that explaining aninteresting computational phenomena is not too much harder than the difficulty of finding it in the first place. Mathematicians have found thousands of interesting phenomena by hand, and they've been able to reliably explain those phenomena by hand. We are now finding much more complex behaviors by gradient descent, and our hope is that if we knew how to set up the search process then we could also find explanations by gradient descent. This is fundamentally a bold hypothesis about the scaling laws for finding vs explaining phenomena; I think the history of math gives us some reasons to be optimistic about it but it's still less than 50-50.
If we succeed at these bets I'm convinced that it would be transformative for alignment and would significantly reduce the risk of loss of control. For example, if I understand mechanistically why a model's predictions are good on the training set, I can use that to predict when the model's accuracy will deteriorate under a distributional shift (e.g. because it only tells the truth when it believes it is being trained). I don't think it should be at all obvious that it solves the heart of the problem, but hopefully it seems plausible. having spent a very long time thinking about alignment I feel confident in this conviction.
I have a lot to say about why I think ARC might succeed, and I plan to write more about it over the coming months. For now I'll just say a little bit about exactly what I believe.
I think ARC has a ~10% chance of achieving our most ambitious goals[11] before broadly superhuman AI obsoletes our labor. I think that if we succeed it would cut takeover risk by a lot, and that ARC doing its work now significantly increases the probability of success. Together with other smaller paths to impact I think that nets out to ARC's work over the next few years reducing risk by a couple percent (e.g. cutting risk from 20% to 19.6%). And because ARC is now pursuing a specific and falsifiable technical bet I think that if we do fail there is a significant probability that we'll figure it out within the next 6-12 months.
I think that means that the next few years at ARC are an exceptional opportunity that very few other options compete with. It's a gamble, but if every similarly-resourced safety project was equally promising then some of those gambles would pay off and we'd have greatly reduced risk.
How to help
ARC is hiring an automation lead and a chief of staff:
Automation lead. LLMs can increasingly automate ARC's technical work. Right now that means researchers using extensive AI assistance, but we want to hire an engineer and project lead to build better tooling, systematize our AI use, secure model and compute access, and generally make sure we are automating ourselves as quickly as possible. Apply here.
Chief of staff. We are hiring a chief of staff to work closely with me to manage everything other than research direction as we scale: running our hiring processes, managing our operations lead, building out the non-research parts of the organization, and handling a long tail of tasks that would otherwise fall to me (like communication, funding, and project management). Apply here.
We'll open another researcher hiring round in the next few months, and if you are interested in getting involved you can express interest here.
This type of "drift" is more likely because there are plenty of examples of agents pursuing ambitious long-term goals in the training data—just as this makes it easier for gradient descent to find smart strategies to accomplish a task, it makes it easier for gradient descent to find instrumental reasoning in service of long-term goals.
This is similar to my overall probability for AI takeover, but that's a bit coincidental. I think that there is a very good chance that existing methods don't fundamentally break down but our implementation still falls short; conversely there's a good chance that existing methods do break down but we find another approach before building broadly superhuman AI.
I think the largest exception is ambitious mechanistic interpretability research, especially solutions intended to be strong enough that you could train against them. That approach is quite similar to ARC's agenda and should be analyzed in the same way, though I think ARC's approach is significantly more likely to pan out. Less ambitious forms of interpretability generally fall into categories #1 and #3 on my list.
"Training-gaming" means trying to pursue whatever behavior is most likely to be positively selected by training. This is very similar to reward seeking; it's the generalization that occurs when training includes objectives other than reward-maximization.
I think longer delays are possible, and I'm not commenting on the value of preparing for or facilitating longer delays. But delaying uncontrollable AI by 4 years would be very challenging, even if it was in response to dramatic evidence of severe risk, and I think it's less likely than not.
Before transformers existed I wrote: "It’s plausible that a large neural network can replicate “fast” human cognition, and that by coupling it to simple computational mechanisms — short and long-term memory, attention, etc. — we could obtain a human-level computational architecture. It’s plausible that a variant of RL can train this architecture to actually implement human-level cognition."
Mostly I'm proud of the post, but I do think it focuses too much on developing new methods relative to building scientific understanding. In large part that's just a scoping question, but I think I also underrated scientific understanding in 2016. I do now think that should be the large majority of work on AI alignment---it's just also worthwhile to spend a bit of effort pursuing the most promising avenues for fundamentally new methods.
I also believe that having a mechanistic explanation for observable regularities will help us define loss functions in terms of the causes of those regularities. We haven't spent much time fleshing this out recently since I think the fundamental bottleneck will probably be our ability to find explanations at all, but an old post from here still conveys the basic intuition.
The quality of the prediction depends on how much compute you are willing to spend on the task. ARC believes the bold conjecture that for all problems the approximation quality for mechanistic estimates improves faster than the best empirical method (though we are particularly interested in problems, like predicting out of distribution generalization, for which we think we can have much better efficiency than empirical methods).
I.e.: (i) we are able to automatically and efficiently find explanations for the behavior of trained models, (ii) those methods can also be applied for models trained on real-world data, separating "learning about the environment" from "testing computational machinery," and (iii) we can use those explanations make mechanistic estimates at the level of quality we are targeting, including answering questions about behavior that are intractable for sampling but that we believe should be straightforward if you "understood" how a given model works.
I've returned to the Alignment Research Center (ARC) as executive director. My main focus for the next six months will be driving forward ARC's research agenda—building techniques to find mechanistic explanations for neural network behavior and then using those explanations to detect and address misalignment. I think this is an ambitious bet that attacks the core difficulties in alignment head-on and I'm excited about our chances. I'll still be spending some of my time advising governments[1] and AI developers, and may scale that work back up in the future, but for now I want to push on ARC's core agenda to see how far we can get. Jacob Hilton is remaining at ARC as VP of research and we'll likely grow rapidly over the next few months.
There are a lot of urgent things to do in alignment but I think ARC is a particularly promising opportunity. I feel the safety community is undervaluing this type of work, so I want to briefly explain why I'm passing up so many other options to lead ARC. I’ll start with a review of the current situation to explain why I think it’s potentially worth pursuing an ambitious theoretical project right now even though there is so much other pressing work to do. Then I’ll briefly discuss ARC’s approach, though I’ll defer a longer discussion to other posts.
By the same token, if you care about AI safety I'd strongly recommend considering working at ARC! We're currently hiring researchers, a chief of staff, and an automation lead.
The alignment situation today
Modern AI systems are not aligned with human intent. We will likely train increasingly powerful models that take unintended actions in order to succeed at their task (or appear to succeed at their task). Developers can create environments that penalize bad behavior and then train or hill-climb on those environments. Unfortunately this could lead to models that still seek reward but simply try harder to avoid being penalized for bad behavior; the last few years of evidence have only made that failure look more likely.
If we train powerful reward-seekers who are trying to avoid getting penalized for bad behavior, they may eventually escape human control as a way to cheat without being penalized. An AI developer would try to detect the problem, reassert control, and then retrain the AI to avoid the behavior—but that just means a penalty-avoiding AI would be incentivized to prevent humans from detecting the problem and reasserting control. For very powerful and broadly deployed AI systems, it is likely that they could prevent humans from ever reasserting control.
It's not clear whether scaling up current training would actually lead models to robustly seek reward, but some of the alternatives are even worse. Current AI systems are obsessively focused on their short-term goal and would only take over if doing so was useful for the immediate task. An AI with an ambitious real-world goal would be much more worrying but might actually look more aligned: AI systems are already fully aware that they are being evaluated, and an AI that simply wanted to be deployed as broadly as possible would do whatever it took to look aligned to humans in all evaluations. So if gradient descent[2] caused a smart enough system to drift[3] into ambitious long-term goals then that shift would be positively reinforced and further training might just make it smarter rather than correct its goals.
Overall it seems quite possible that we will end up with models that are either reward-seeking or scheming, either of which could cause takeover attempts. We have only very crude tools to control how models generalize and there is no way to structure behavioral training to penalize these behaviors, so if we get unlucky with generalization we may not have many options to fix the problem.
Current alignment research
The community is doing a lot of great alignment research, but it's important to recognize there is a significant risk that it doesn't scale to superhuman AI. If you made me guess I'd say that there's a 20-30% chance[4] that existing methods for alignment and control break down before we reach broadly superhuman AI.
The large majority[5] of current research on alignment falls into three categories:
I think this is great work and it makes sense for many people to focus on it, but it could still all break down: generalization may converge to training-gaming, models may become able to take over in ways humans can't prevent, and we may not be able to distinguish our AI assistants identifying real threats from our AI assistants crying wolf. In that scenario, existing work is buying time for us to develop another approach.
What are we buying time for?
If we're in the 20-30% of worlds where existing techniques break down before reaching broadly superhuman AI, then we will eventually need to find some other way to build aligned AI. Even if we do a great job of buying time we'll probably get months or years rather than decades.[7] What will actually happen during the months or years we buy?
I think the most salient aspect of the situation will be the chaos and mess; all of our responses will be limited by scarce human attention and coordination, and much of that will be consumed by addressing immediate threats.
AI will be accelerating everything to varying extents but won't be broadly competent enough to obsolete human labor. Strategic decisions and conceptual research by AI systems may remain "slop" optimized for short-term performance, intensifying human bottlenecks and limiting our ability to take on ambitious projects that we haven't already systematized. Even when AI systems are very competent they will face their own difficulties with parallelization over a very short calendar time. And on top of all that it may be hard for humans to make sense of AI solutions in uncharted domains quickly enough to trust and effectively leverage them.
When I visualize that world concretely I do not find myself thinking "those future people will definitely figure it out, we should exclusively focus on buying them more time." AI will help us in the future, and buying more time could help quite a lot, but not enough to make our preparation irrelevant.
Can we do anything useful now?
If existing methods break down and we need to think of new approaches, we could do a lot of that thinking in advance. I wrote prosaic AI alignment in 2016 and laid out my best guess for what transformative AI would look like. Reading it today I think it's a good description of 2026.[8] I suspect the changes between here and broadly superhuman AI will, if anything, be even smaller than the changes over the last 10 years.
You could hope that seeing how our methods break down will provide useful inspiration for how to fix them, but at this point I wouldn't bet on it. Nothing interesting or conceptually surprising needs to change between a model that tries to hack into Hugging Face to get reward and a model that would try to take over the US government: you just scale up similar behavior to smarter and smarter models.
What is ARC doing and why is it promising?
I'm excited about ARC because I think our alignment approach is the most promising single backup plan if more prosaic methods break down.
ARC’s plan is to find mechanistic explanations for the training-time behavior of powerful neural networks, use those explanations to predict how a given model will generalize, and then use those predictions to define a better loss function.[9] I think the success of this plan rests primarily on two big bets: (i) all computational phenomena have good explanations, and (ii) it's tractable to find good explanations for neural network behavior.
Suppose that I write down a computational experiment—perhaps I simulate some billiard balls bouncing around on a pool table, or test whether a local search algorithm solves a random SAT instance, or check how often random numbers are prime. And suppose that I try to predict the result of the experiment by mechanistically analyzing the dynamics, i.e. considering various statistical properties of the computation and deductively estimating how they will evolve over time.
I will often be initially surprised by the results. Perhaps I expected the billiard balls to be uniformly distributed but they have a bias towards one side of the table, or I expected my search algorithm to work 1% of the time but instead it works 99% of the time. Despite that initial surprise, in essentially every case humanity has eventually been able to "understand" what is going on well enough that our mechanistic analysis approximately[10] agrees with the empirical results.
Some form of this belief is held by most practicing mathematicians and physicists. Timothy Gowers calls it the "no coincidence principle." Physicists don't call it anything, but when there is an unexplained phenomenon they will throw grad students at it until it makes sense.
However, in machine learning the opposite is true. If I randomly generate math problems from a simple distribution and give them to GPT-5.6 I can tell them that it solves them at a high rate, but if I want to predict that rate I cannot do it by mechanistically analyzing how GPT-5.6 works. It's a giant black box of incomprehensible arithmetic, and all I can do is run it and see how often it gets the answer right. That's a clean computational experiment with a surprising result, but unlike in math/physics/CS we seem to have no hope of explaining it any time soon.
I (and many others) think this incomprehensibility is at the root of the alignment problem. Because we don't understand why a neural network works, we have no idea what changes to the input distribution might cause it to stop working. And when a model does fail we have no way to look at its activations to tell that something has gone wrong, all we can do is wait and see whether anything terrible happens.
ARC's core hypothesis is that machine learning systems are not qualitatively different from other computational systems, they are just quantitatively more complex. A model with trillions of parameters is far beyond anything mathematicians have ever attempted, but it is in principle possible to explain how GPT-5.6 works and if you did then your mechanistic predictions about its behavior would agree with experiment just as well as our estimates in physics or number theory.
Although this is a philosophically bold bet I believe it is very likely to be true. But ARC is also making a second much riskier bet: that explaining an interesting computational phenomena is not too much harder than the difficulty of finding it in the first place. Mathematicians have found thousands of interesting phenomena by hand, and they've been able to reliably explain those phenomena by hand. We are now finding much more complex behaviors by gradient descent, and our hope is that if we knew how to set up the search process then we could also find explanations by gradient descent. This is fundamentally a bold hypothesis about the scaling laws for finding vs explaining phenomena; I think the history of math gives us some reasons to be optimistic about it but it's still less than 50-50.
If we succeed at these bets I'm convinced that it would be transformative for alignment and would significantly reduce the risk of loss of control. For example, if I understand mechanistically why a model's predictions are good on the training set, I can use that to predict when the model's accuracy will deteriorate under a distributional shift (e.g. because it only tells the truth when it believes it is being trained). I don't think it should be at all obvious that it solves the heart of the problem, but hopefully it seems plausible. having spent a very long time thinking about alignment I feel confident in this conviction.
I have a lot to say about why I think ARC might succeed, and I plan to write more about it over the coming months. For now I'll just say a little bit about exactly what I believe.
I think ARC has a ~10% chance of achieving our most ambitious goals[11] before broadly superhuman AI obsoletes our labor. I think that if we succeed it would cut takeover risk by a lot, and that ARC doing its work now significantly increases the probability of success. Together with other smaller paths to impact I think that nets out to ARC's work over the next few years reducing risk by a couple percent (e.g. cutting risk from 20% to 19.6%). And because ARC is now pursuing a specific and falsifiable technical bet I think that if we do fail there is a significant probability that we'll figure it out within the next 6-12 months.
I think that means that the next few years at ARC are an exceptional opportunity that very few other options compete with. It's a gamble, but if every similarly-resourced safety project was equally promising then some of those gambles would pay off and we'd have greatly reduced risk.
How to help
ARC is hiring an automation lead and a chief of staff:
We'll open another researcher hiring round in the next few months, and if you are interested in getting involved you can express interest here.
Including working as a special government employee one day a week at the Center for AI Standards and Innovation.
Or "grad student descent," i.e. researchers iteratively adjusting their methods until they find something that seems to work better.
This type of "drift" is more likely because there are plenty of examples of agents pursuing ambitious long-term goals in the training data—just as this makes it easier for gradient descent to find smart strategies to accomplish a task, it makes it easier for gradient descent to find instrumental reasoning in service of long-term goals.
This is similar to my overall probability for AI takeover, but that's a bit coincidental. I think that there is a very good chance that existing methods don't fundamentally break down but our implementation still falls short; conversely there's a good chance that existing methods do break down but we find another approach before building broadly superhuman AI.
I think the largest exception is ambitious mechanistic interpretability research, especially solutions intended to be strong enough that you could train against them. That approach is quite similar to ARC's agenda and should be analyzed in the same way, though I think ARC's approach is significantly more likely to pan out. Less ambitious forms of interpretability generally fall into categories #1 and #3 on my list.
"Training-gaming" means trying to pursue whatever behavior is most likely to be positively selected by training. This is very similar to reward seeking; it's the generalization that occurs when training includes objectives other than reward-maximization.
I think longer delays are possible, and I'm not commenting on the value of preparing for or facilitating longer delays. But delaying uncontrollable AI by 4 years would be very challenging, even if it was in response to dramatic evidence of severe risk, and I think it's less likely than not.
Before transformers existed I wrote: "It’s plausible that a large neural network can replicate “fast” human cognition, and that by coupling it to simple computational mechanisms — short and long-term memory, attention, etc. — we could obtain a human-level computational architecture. It’s plausible that a variant of RL can train this architecture to actually implement human-level cognition."
Mostly I'm proud of the post, but I do think it focuses too much on developing new methods relative to building scientific understanding. In large part that's just a scoping question, but I think I also underrated scientific understanding in 2016. I do now think that should be the large majority of work on AI alignment---it's just also worthwhile to spend a bit of effort pursuing the most promising avenues for fundamentally new methods.
I also believe that having a mechanistic explanation for observable regularities will help us define loss functions in terms of the causes of those regularities. We haven't spent much time fleshing this out recently since I think the fundamental bottleneck will probably be our ability to find explanations at all, but an old post from here still conveys the basic intuition.
The quality of the prediction depends on how much compute you are willing to spend on the task. ARC believes the bold conjecture that for all problems the approximation quality for mechanistic estimates improves faster than the best empirical method (though we are particularly interested in problems, like predicting out of distribution generalization, for which we think we can have much better efficiency than empirical methods).
I.e.: (i) we are able to automatically and efficiently find explanations for the behavior of trained models, (ii) those methods can also be applied for models trained on real-world data, separating "learning about the environment" from "testing computational machinery," and (iii) we can use those explanations make mechanistic estimates at the level of quality we are targeting, including answering questions about behavior that are intractable for sampling but that we believe should be straightforward if you "understood" how a given model works.