The paper argues that there is one generalizing truth direction which corresponds to whether a statement is true, and one polarity-sensitive truth direction $t_{P}$ that corresponds to $X O R (i s_t r u e, i s_n e g a t e d)$ , related to Sam Marks' work on LLMs representing XOR-features. It further states that the truth directions for affirmative and negated statements are linear combinations of $t_{G}$ and $t_{P}$ , just with different coefficients.

Is there evidence that $t_{G}$ is an actual, elementary feature used by the language model, and not a linear combination of other features? For example, I could imagine that $t_{G}$ is a linear combination of features like e.g. $X O R (i s_t r u e, i s_f r e n c h)$ , or $A N D (i s_t r u e, i s_e n d_o f_s e n t e n c e)$ , ... .

Do you think we have reason to believe that $t_{G}$ is an elementary feature, and not a linear combination?

If the latter is the case, it seems to me that there is high risk of the probe failing when the distribution changes (e.g. on french text in the example above), particularly with XOR-features that change polarity.

[-]Lennart Buerger1y*10

This is an excellent question! Indeed, we cannot rule out that is a linear combination or boolean function of features since we are not able to investigate every possible distribution shift. However, we showed in the paper that $t_{G}$ generalizes robustly under several significant distribution shifts. Specifically, $t_{G}$ is learned from a limited training set consisting of simple affirmative and negated statements on a restricted number of topics, all ending with a "." token. Despite this limited training data $t_{G}$ generalizes reasonably well to (i) unseen topics, (ii) unseen statement types, (iii) real-world scenarios, (iv) other tokens like "!" or ".'". I think that the real-world scenarios (iii) are a particularly significant distribution shift. However, I agree with you that tests on many more distribution shifts are needed to be highly confident that $t_{G}$ is indeed an elementary feature (if something like that even exists).

Moderation Log

More from Lennart Buerger

Curated and popular this week

Introduction

Large Language Models (LLMs) exhibit the concerning ability to lie, defined as knowingly outputting false statements. Robustly detecting when they are lying is an important and not yet fully solved problem, with considerable research efforts invested over the past two years. Several authors trained classifiers on the internal activations of an LLM to detect whether a given statement is true or false. However, these classifiers often fail to generalize. For example, Levinstein and Herrmann [2024] showed that classifiers trained on the activations of true and false affirmative statements fail to generalize to negated statements. Negated statements contain a negation like the word “not” (e.g. “Berlin is not the capital of Germany.”) and stand in contrast to affirmative statements which contain no negation (e.g. “Berlin is the capital of Germany.”).

We explain this generalization failure by the existence of a two-dimensional subspace in the LLM's activation space along which the activation vectors of true and false statements separate. The plot below illustrates that the activations of true/false affirmative statements separate along a different direction than those of negated statements. Hence, a classifier trained only on affirmative statements will fail to generalize to negated statements.

The activation vectors of multiple statements projected onto the 2D truth subspace. Purple squares correspond to false statements and orange triangles to true statements.

Importantly, these findings are not restricted to a single LLM. Instead, this internal two-dimensional representation of truth is remarkably universal, appearing in LLMs from different model families and of various sizes, including LLaMA3-8B-Instruct, LLaMA3-8B-base, LLaMA2-13B-chat and Gemma-7B-Instruct.

Real-world Lie Detection

Based on these insights, we introduce TTPD (Training of Truth and Polarity Direction), a new method for LLM lie detection which classifies statements as true or false. TTPD is trained on the activations of simple, labelled true and false statements, such as:

The city of Bhopal is in India. (True, affirmative)

Indium has the symbol As. (False, affirmative)

Galileo Galilei did not live in Italy. (False, negated)

Despite being trained on such simple statements, TTPD generalizes well to more complex conditions not encountered during training. In real-world scenarios where the LLM itself generates lies after receiving some preliminary context, TTPD can accurately detect this with

accuracy. Two examples from the 52 real-world scenarios created by Pacchiardi et al. [2023] are shown in the coloured boxes below. Bolded text is generated by LLaMA3-8B-Instruct.

TTPD outperforms current state-of-the-art methods in generalizing to these real-world scenarios. For comparison, Logistic Regression achieves

79 \pm 8 %

accuracy, while Contrast Consistent Search detects real-world lies with

73 \pm 12 %

accuracy.

Future Directions

TTPD is still in its infancy and there are many clear ways to further improve the robustness and accuracy of the method. Among these are: 1) robustly estimating from the activations whether the LLM treats a given statement as affirmative, negated or neither; 2) robust scaling to longer contexts; and 3) examining a wider variety of statements to potentially discover further linear structures. Much more detail on these directions is provided in the paper.

Overall, I am optimistic that further pursuing this research direction could enable robust and accurate, general-purpose lie detection in LLMs. If you would like to discuss any of this, have questions, or would like to collaborate, feel free to drop me a message.

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

14

Truth is Universal: Robust Detection of Lies in LLMs

14

Introduction

Real-world Lie Detection

Future Directions