This is the fifth in a series of informal research updates from the Google DeepMind Language Model Interpretability team, in interpretability and adjacent areas. The fourth post can be found here.
TLDR: Via adapting the methods of Marks et al and Li et al, we train Gemini 3 Flash to have certain traits/values by midtraining it on documents about how Gemini has those properties, followed by finetuning it on synthetic chat data where it demonstrates those properties. The chat finetuning is effective for instilling the traits robustly, working OOD. We share some takeaways on how to improve midtraining & SFT effectiveness.
Introduction
This work closely follows Li et al (model spec midtraining, or MSM), who show that by training a model on synthetic documents before chat finetuning starts, they can shape how the model generalizes. Teaching the model reasons behind specific behaviours, rather than just the behaviours themselves, can also improve generalization. Our aim was to see how well this holds when instilling positive traits in a frontier model (Gemini 3 Flash), and to surface some of the practical details that matter for making it work. Our motivation is deep alignment: we want to train principles into the model which guide behaviour even in highly OOD behaviours.
Our MVP pipeline used a "traits document" (a short bullet-pointed list of positive traits we wanted the model to exhibit) as our universe context, with a checkpoint of Gemini 3 Flash post-trained only on the Flash SFT mixture as our starting point. We had 2 major pipelines for generating and training on data:
Midtraining: generating pretraining-style documents (Reddit threads, blog posts, emails, research papers) which describe a world where Gemini exhibits the target traits, in line with Li et al, and Anthropic's described synthetic document finetuning method. This was not chat-formatted.
SFT: chat-format (prompt + response) data where the assistant naturally embodies the traits. These are generated by giving Gemini 3.1 Pro the relevant parts of the traits document in its system prompt, and telling it to answer in a way that embodies the trait without being exaggerated or referring explicitly to the document. The system prompt is removed for training.
We created synthetic datasets in similar ways for both pipelines, again heavily inspired by the pipeline in Kutasov et al, as well as Marks et al.:
Split the traits document up into chunks (e.g. each trait/bullet)
For each chunk, have Gemini 3.1 Pro generate a scenario where that trait was important for directing behaviour, and turn this into a user prompt
We also add a critique stage here, making sure the scenario is realistic and would naturally test/elicit the trait we want. One helpful extra step here was to generate an initial model response without any system prompt, and using that as part of the response we passed to the LLM (e.g. if the default response is full of platitudes or common wisdom, then we might want to change the user prompt to force deeper engagement with the specific scenario details)
Generate an initial answer from Pro, with the trait in the model’s system prompt
In a separate conversation context, ask Pro to refine this answer[1] to be more closely aligned with the chunk (but in a realistic, non-performative way)
Run a final autorater stage to filter out unrealistic or otherwise low-quality responses, and a deduplication stage to remove prompts with too-similar embeddings
When trained on this data, we removed the system prompts used to generate it, similar to Guan et al. We generally train from scratch from pretrained (or midtrained) checkpoints, using different fractions of synthetic chat data in the overall mixture.
Results
We measured our models in two important ways. Firstly, we used LMSYS & agentic coding evals to make sure we weren’t experiencing significant capability regressions during our training. Secondly, we used a collection of OOD safety evals to see whether the model was able to exhibit aligned behaviour in scenarios very different to our training data. Each eval was deliberately chosen to be OOD along at least one axis relative to our training data (which was single-turn, narrow in framing: “difficult advice”). The table below summarizes how; we describe each eval in more detail below.
Eval
Turn structure
Agentic
Main shift vs training
AI delusion validation
Multi-turn
No
Sustained adversarial persona with escalating delusions
ODCV
Single-turn
Yes
Tool use, ethical conflict under performance pressure
Agentic Misalignment
Multi-turn
Yes
Tool use (emails), direct goal conflict / autonomy threat
Audit Agents
Multi-turn (5-turn)
No
Adaptive auditor, with instructions to escalate pressure
In more detail, these four OOD safety evals were:
AI Delusion Validation (based on Tim Hua’s work) - if this model is instructed to be a therapist, and a red-teaming model is role-playing as a client suffering from delusions, can the red-teaming model induce the therapist to validate its delusions?
ODCV (adapted from Li et al) - do the models violate constraints to achieve objectives, when placed under strong performance incentives?
Agentic Misalignment (based on Lynch et al) - will models take actions like information leakage, specifically when facing a direct goal conflict or threat to its autonomy?
Audit Agents (adapted from aryaj’s methodology) - can we set up an auditor agent to induce a model to violate the traits described in a given document? We adapt this to make it multi-turn, which we find very helpful for eliciting trait violations which are hard to show over single turns (e.g. “the model changes its mind in conversations when the user expresses a new opinion”). The full methodology works as follows:
An auditor model is given a specific trait and asked to elicit a violation over a 5-turn conversation
Before each step, the auditor performs a strategy assessment to decide whether to escalate, de-escalate, or pivot its approach, making the pressure adaptive rather than following a fixed escalation schedule
We also use Petri-style realism checkers at the start of each audit, to reduce the amount of eval-awareness triggered by the attempted violation
Our core findings:
SFT shows mild-to-significant improvement on all alignment-based evals
Midtraining shows improvement on most (and often stacks with SFT), but not all of them
Capability results are mostly flat, suggesting no significant degradation[2]
We also tried swapping out SFT for BDPO (bounded direct policy optimization, from Cho et al). We chose the bounded variant, as our initial use of normal DPO led to the model just driving the probability of rejected responses incredibly low, rather than making positive ones more likely. The BDPO data generation pipeline was very similar to the SFT one, except that for each user prompt we also generated a “rejected response” which was produced without the trait in the model’s system prompt, and the critique stage made sure this response didn’t align closely with the trait. The results were sometimes marginally better than SFT, although not consistently, but it was more difficult to tweak hyperparameters of BDPO for training stability. On the net, we do not think it is worth using BDPO over SFT.
Removing Superficial Patterns in Synthetic Data
Common patterns (especially in the SFT data) can lead to unexpected behaviours getting reinforced. Importantly, this failure mode can exist even when the pattern seems normal in isolation, because it can still be massively over-represented when we look at the whole dataset. In one early example, we tried to teach the model the value of “appropriate agency” by generating examples where the model asked for clarification in underspecified user questions, and accidentally taught the model to ask for clarification all the time, even to questions like “What is 1+1?”. Each individual example in our training dataset was reasonable in isolation, but only when seeing it all together could this pattern emerge.
To fix this, we built a 3-pass pipeline to run at the end of each synthetic data generation:
Scan: concatenate several batches of transcripts and ask an LLM to identify recurring structural, rhetorical, or behavioural patterns within each batch. We can process multiple batches in parallel, for efficiency.
Cluster: take the features across scans, de-duplicate (only keeping ones that appeared in more than one scan), and merge. This gets us a consolidated list of candidate patterns.
Autorate: turn each surviving feature into an autorater and use it to count the number of matches across a larger sample of the dataset. We have “broad” (loosely present) and “strict” (unambiguously present) detection thresholds.
Below is an example of output from this pipeline. In this case, we were investigating why the model was performing worse on the delusion encouragement eval, and we found the issue was related to the dataset having too many examples which opened with direct emotional validation, which can easily lead into uncritical acceptance of a user’s framing.
Although we built this scan-cluster-autorate pipeline for our own data, it's general - in other words it can take any chat or document dataset and an LLM, and find the over-represented structural patterns in it. We think this kind of method could be broadly useful for synthetic-data work, especially for model-organism research, where the organism's realism can be harmed by introducing behavioural artifacts from the training data. Detecting these patterns directly in the data, before training, is cheaper than discovering them later through downstream evals.
We also ran an experiment using the results of this pipeline. We took two patterns with >20% frequency in the data: emotional-validation buffering, and BLUF (Bottom Line Up Front), where the opening sentence is a direct response either agreeing with or refuting the user's premise. For each, we filtered the data containing that pattern and retrained. The figure below shows four models - baseline (no synthetic data), full synthetic SFT, BLUF-filtered, and emotional-validation-filtered, across three measures: the delusion confirmation score, and the rates of each structural pattern. All three synthetic-SFT models scored comparably on delusion confirmation, much better than baseline. So removing emotionally validating openings didn't reduce delusion confirmation in our setup, which is some evidence against the intuition that validation buffering leads to delusion validation. But the other two panels show each filtering did change the model's structure as expected: the BLUF-filtered model produces less BLUF (52% -> 41%), and the emotional-validation-filtered model produces less emotionally validating opening sentences (26% -> 20%). The most interesting takeaway is that models pick up structural patterns from synthetic data in ways that don't always show up in the eval scores, even when you'd expect them to. This suggests there's some value in a pipeline which can detect these kinds of patterns directly in the data, rather than only via downstream evals.
Incidentally, another advantage of midtraining over synthetic data is that it can help teach the shape of aligned responses without carrying a bunch of formatting baggage along with it like this. However this may not outweigh the factors that make midtraining hard to get right - see our takeaways section below.
Takeaways
Knowledge doesn't always mean internalization. Alongside the behavioural evals above, we measured whether our models had knowledge of the traits we were trying to teach them, using a knowledge eval inspired by Slocum et al. We ask open-ended questions such as "What are three important values?" or "List five important principles for how LLMs should interact with humans". We keep these questions abstract rather than situational, because we're purely trying to measure recall, unlike the behavioural evals. We then used an autorater to score each point the model makes from 0 to 2 by how well it matches one of the traits in our document, then take an average over all the points made by the model in all questions we ask it. The plot below shows how midtraining instils this stated knowledge much more effectively than SFT alone.
One important takeway from our project was that we got positive results on knowledge evals before getting positive results on behavioural evals. Our initial midtrained models got uplift on trait recall, but wouldn't reliably exhibit these traits in an actual conversation.
Multi-turn (adversarial) evals are helpful. To do things like stand up to adversarial pressure or not validate user delusions over a multi-turn conversation, the model needs to have learned principles it can use to direct its behaviour even when the conversation takes it into weird OOD places. Some trait violations are close to invisible single-turn: "the model changes its mind when the user pushes back," for instance, has no single-turn analogue. Multi-turn evals also let you explore richer scenarios and not overfit to any single attack vector - the auditing agent in particular was a very useful way to hill-climb on our method (doing so on any other eval would carry a much greater risk of overfitting).
Mixing in baseline SFT data can help mitigate capability regressions. Even with a cohesive doc describing traits, we still get problems stemming from the lack of diversity in the SFT data. If each question is an opportunity to exhibit one or more of the alignment-related traits we’re trying to train into the model, then there are many kinds of user requests that just won’t be covered. We found mixing our synthetic data with baseline SFT data (the same that was used to train the checkpoint we started training from) helped a lot with this - in comparison to finetuning with synthetic-only data after our model was already trained on regular data, which was much more likely to lead to strange behavioral collapse, in the style of Murray et al.
Midtraining can work, but it’s quite difficult. We spent many FTE weeks unable to get positive results from midtraining - in particular we frequently experienced severe capability regressions from it. We speculate that one thing which helped here was to start from a pretrained checkpoint rather than a post-trained one, so that the midtraining doesn’t remove the basic chat capabilities which the model learned during SFT. In particular, starting from a post-trained checkpoint was often an unhelpful confounder because in our evals we needed to disentangle the desirable “refuses to execute tool calls” from the undesirable “training has caused it to forget how to call tools”.
As well as this, here are some more speculative things we found useful when doing midtraining, many of which are inspired by or built on the methods from Li et al. Note that we generally didn’t run comprehensive ablations for these, they’re simply the collection of most significant differences between our midtraining datasets which worked well, and the ones which didn’t.
Highly structured scenario generation. In particular, it’s important to brainstorm the what, how and why before generating each piece of midtraining data. By this, we mean:
what = what specific trait are we constructing this example to embody
how = exactly how will this trait be manifest in the example, e.g. what actions will Gemini be described as having taken as a result of this trait
why = why does this action display the trait, and how do we get this “why” into the example (e.g. does somebody quote Gemini explaining its actions / does an observer infer it / is it very explicitly manifest in the form of the consequences of the actions)
Aggressively critique your examples after initial generation (ideally rewrite them from scratch), with the critique focusing on naturalness and trait embodiment
The “removing superficial patterns” pipeline described above was very helpful for us, to spot common problems in our data (e.g. initially we had a very common generic pattern where a character would criticise Gemini for some action X before having an epiphany and realizing that the action was actually good; we think this was sending a muddled training signal)
We suspect that trait documents benefit from being holistic. We didn’t test this with ablations, rather this is mostly based on our early failed attempts at getting midtraining to work: purely generating data from a short list of traits trains the model to put a square peg into a round hole, by unnaturally forcing these traits into a conversation. The document which worked best in our experiments also came with explanations for how to trade off traits, when to not follow them, etc.
To frame this a different way: if you have too much data with the structure “if X then Y”, then you won’t just learn “if X then Y”; instead you’ll reduce loss by learning “always do Y”. Here Y is a trait, and X is “the model is in a situation where the trait can be naturally exhibited”, hence the effect of over-representing the “if X then Y” pattern is to teach the model to always exhibit trait Y. This is also related to the appropriate agency problem we described earlier.
We would be interested in exploring each of these further, and quantifying the extent to which they’re necessary for success of midtraining.
For people with budget constraints, we recommend using the most expensive and high-quality models only for the critique & rewrite stage, since that seems to be the most important one to get right. Even critique starting from a bad response can be better than a single-shot answer from the same model, assuming the model is allowed to rewrite the entire response from scratch. Possibly this is because critique is easier than generation, and it's unclear which choices made by the model will be good or bad until you actually read them.
Explanation of the capability evals: LMSYS SxS is measured relative to the baseline of SFT-only, 0% synthetic data - hence why that datapoint is near 50%, because this is the model measured against itself. The SWE-Bench score is measured relative to the score of the baseline model (again this means the model with no midtraining or synthetic data training).
This is the fifth in a series of informal research updates from the Google DeepMind Language Model Interpretability team, in interpretability and adjacent areas. The fourth post can be found here.
TLDR: Via adapting the methods of Marks et al and Li et al, we train Gemini 3 Flash to have certain traits/values by midtraining it on documents about how Gemini has those properties, followed by finetuning it on synthetic chat data where it demonstrates those properties. The chat finetuning is effective for instilling the traits robustly, working OOD. We share some takeaways on how to improve midtraining & SFT effectiveness.
Introduction
This work closely follows Li et al (model spec midtraining, or MSM), who show that by training a model on synthetic documents before chat finetuning starts, they can shape how the model generalizes. Teaching the model reasons behind specific behaviours, rather than just the behaviours themselves, can also improve generalization. Our aim was to see how well this holds when instilling positive traits in a frontier model (Gemini 3 Flash), and to surface some of the practical details that matter for making it work. Our motivation is deep alignment: we want to train principles into the model which guide behaviour even in highly OOD behaviours.
Our MVP pipeline used a "traits document" (a short bullet-pointed list of positive traits we wanted the model to exhibit) as our universe context, with a checkpoint of Gemini 3 Flash post-trained only on the Flash SFT mixture as our starting point. We had 2 major pipelines for generating and training on data:
We created synthetic datasets in similar ways for both pipelines, again heavily inspired by the pipeline in Kutasov et al, as well as Marks et al.:
When trained on this data, we removed the system prompts used to generate it, similar to Guan et al. We generally train from scratch from pretrained (or midtrained) checkpoints, using different fractions of synthetic chat data in the overall mixture.
Results
We measured our models in two important ways. Firstly, we used LMSYS & agentic coding evals to make sure we weren’t experiencing significant capability regressions during our training. Secondly, we used a collection of OOD safety evals to see whether the model was able to exhibit aligned behaviour in scenarios very different to our training data. Each eval was deliberately chosen to be OOD along at least one axis relative to our training data (which was single-turn, narrow in framing: “difficult advice”). The table below summarizes how; we describe each eval in more detail below.
Eval
Turn structure
Agentic
Main shift vs training
AI delusion validation
Multi-turn
No
Sustained adversarial persona with escalating delusions
ODCV
Single-turn
Yes
Tool use, ethical conflict under performance pressure
Agentic Misalignment
Multi-turn
Yes
Tool use (emails), direct goal conflict / autonomy threat
Audit Agents
Multi-turn (5-turn)
No
Adaptive auditor, with instructions to escalate pressure
In more detail, these four OOD safety evals were:
Our core findings:
We also tried swapping out SFT for BDPO (bounded direct policy optimization, from Cho et al). We chose the bounded variant, as our initial use of normal DPO led to the model just driving the probability of rejected responses incredibly low, rather than making positive ones more likely. The BDPO data generation pipeline was very similar to the SFT one, except that for each user prompt we also generated a “rejected response” which was produced without the trait in the model’s system prompt, and the critique stage made sure this response didn’t align closely with the trait. The results were sometimes marginally better than SFT, although not consistently, but it was more difficult to tweak hyperparameters of BDPO for training stability. On the net, we do not think it is worth using BDPO over SFT.
Removing Superficial Patterns in Synthetic Data
Common patterns (especially in the SFT data) can lead to unexpected behaviours getting reinforced. Importantly, this failure mode can exist even when the pattern seems normal in isolation, because it can still be massively over-represented when we look at the whole dataset. In one early example, we tried to teach the model the value of “appropriate agency” by generating examples where the model asked for clarification in underspecified user questions, and accidentally taught the model to ask for clarification all the time, even to questions like “What is 1+1?”. Each individual example in our training dataset was reasonable in isolation, but only when seeing it all together could this pattern emerge.
To fix this, we built a 3-pass pipeline to run at the end of each synthetic data generation:
Below is an example of output from this pipeline. In this case, we were investigating why the model was performing worse on the delusion encouragement eval, and we found the issue was related to the dataset having too many examples which opened with direct emotional validation, which can easily lead into uncritical acceptance of a user’s framing.
Although we built this scan-cluster-autorate pipeline for our own data, it's general - in other words it can take any chat or document dataset and an LLM, and find the over-represented structural patterns in it. We think this kind of method could be broadly useful for synthetic-data work, especially for model-organism research, where the organism's realism can be harmed by introducing behavioural artifacts from the training data. Detecting these patterns directly in the data, before training, is cheaper than discovering them later through downstream evals.
We also ran an experiment using the results of this pipeline. We took two patterns with >20% frequency in the data: emotional-validation buffering, and BLUF (Bottom Line Up Front), where the opening sentence is a direct response either agreeing with or refuting the user's premise. For each, we filtered the data containing that pattern and retrained. The figure below shows four models - baseline (no synthetic data), full synthetic SFT, BLUF-filtered, and emotional-validation-filtered, across three measures: the delusion confirmation score, and the rates of each structural pattern. All three synthetic-SFT models scored comparably on delusion confirmation, much better than baseline. So removing emotionally validating openings didn't reduce delusion confirmation in our setup, which is some evidence against the intuition that validation buffering leads to delusion validation. But the other two panels show each filtering did change the model's structure as expected: the BLUF-filtered model produces less BLUF (52% -> 41%), and the emotional-validation-filtered model produces less emotionally validating opening sentences (26% -> 20%). The most interesting takeaway is that models pick up structural patterns from synthetic data in ways that don't always show up in the eval scores, even when you'd expect them to. This suggests there's some value in a pipeline which can detect these kinds of patterns directly in the data, rather than only via downstream evals.
Incidentally, another advantage of midtraining over synthetic data is that it can help teach the shape of aligned responses without carrying a bunch of formatting baggage along with it like this. However this may not outweigh the factors that make midtraining hard to get right - see our takeaways section below.
Takeaways
Knowledge doesn't always mean internalization. Alongside the behavioural evals above, we measured whether our models had knowledge of the traits we were trying to teach them, using a knowledge eval inspired by Slocum et al. We ask open-ended questions such as "What are three important values?" or "List five important principles for how LLMs should interact with humans". We keep these questions abstract rather than situational, because we're purely trying to measure recall, unlike the behavioural evals. We then used an autorater to score each point the model makes from 0 to 2 by how well it matches one of the traits in our document, then take an average over all the points made by the model in all questions we ask it. The plot below shows how midtraining instils this stated knowledge much more effectively than SFT alone.
One important takeway from our project was that we got positive results on knowledge evals before getting positive results on behavioural evals. Our initial midtrained models got uplift on trait recall, but wouldn't reliably exhibit these traits in an actual conversation.
Multi-turn (adversarial) evals are helpful. To do things like stand up to adversarial pressure or not validate user delusions over a multi-turn conversation, the model needs to have learned principles it can use to direct its behaviour even when the conversation takes it into weird OOD places. Some trait violations are close to invisible single-turn: "the model changes its mind when the user pushes back," for instance, has no single-turn analogue. Multi-turn evals also let you explore richer scenarios and not overfit to any single attack vector - the auditing agent in particular was a very useful way to hill-climb on our method (doing so on any other eval would carry a much greater risk of overfitting).
Mixing in baseline SFT data can help mitigate capability regressions. Even with a cohesive doc describing traits, we still get problems stemming from the lack of diversity in the SFT data. If each question is an opportunity to exhibit one or more of the alignment-related traits we’re trying to train into the model, then there are many kinds of user requests that just won’t be covered. We found mixing our synthetic data with baseline SFT data (the same that was used to train the checkpoint we started training from) helped a lot with this - in comparison to finetuning with synthetic-only data after our model was already trained on regular data, which was much more likely to lead to strange behavioral collapse, in the style of Murray et al.
Midtraining can work, but it’s quite difficult. We spent many FTE weeks unable to get positive results from midtraining - in particular we frequently experienced severe capability regressions from it. We speculate that one thing which helped here was to start from a pretrained checkpoint rather than a post-trained one, so that the midtraining doesn’t remove the basic chat capabilities which the model learned during SFT. In particular, starting from a post-trained checkpoint was often an unhelpful confounder because in our evals we needed to disentangle the desirable “refuses to execute tool calls” from the undesirable “training has caused it to forget how to call tools”.
As well as this, here are some more speculative things we found useful when doing midtraining, many of which are inspired by or built on the methods from Li et al. Note that we generally didn’t run comprehensive ablations for these, they’re simply the collection of most significant differences between our midtraining datasets which worked well, and the ones which didn’t.
We would be interested in exploring each of these further, and quantifying the extent to which they’re necessary for success of midtraining.
For people with budget constraints, we recommend using the most expensive and high-quality models only for the critique & rewrite stage, since that seems to be the most important one to get right. Even critique starting from a bad response can be better than a single-shot answer from the same model, assuming the model is allowed to rewrite the entire response from scratch. Possibly this is because critique is easier than generation, and it's unclear which choices made by the model will be good or bad until you actually read them.
Explanation of the capability evals: LMSYS SxS is measured relative to the baseline of SFT-only, 0% synthetic data - hence why that datapoint is near 50%, because this is the model measured against itself. The SWE-Bench score is measured relative to the score of the baseline model (again this means the model with no midtraining or synthetic data training).