This post is the write-up of a few days reading the automated fact-checking literature.
As AIs become more powerful, it would be very valuable to be able to train an agent to be able to tell us, not just whether particular statements are true, but why, and to explain their sources and reasoning as best as possible, especially in slow-takeoff scenarios. Fact-checking represents the present-day, practical end of the spectrum of possible work in this area, and I was curious what what had been done on it, the challenges the field faces, and whether it might shed light on the wider alignment problem.
Ultimately, I don't think there are any major positive surprises or insights generated by what I've read of the field. Instead the tale is more about the difficulties of getting high-quality data, and (especially given the current difficulty of the task) the need to carefully combine automated tools with natural human workflow in order to make something useful. Nonetheless, you might find value in some background on the field, its datasets and challenges.
If you're just interested in a general survey of the field, I would recommend reading the recent summary of the field Automated Fact-Checking for Assisting Human Fact-Checkers which talks through largely the same material from a more practical perspective. I'm instead looking more closely at the generation of the datasets and the techniques employed in solving them.
Summary
The few fact-checking datasets generally consist of tens to hundreds of thousands of sentences hand-curated by Mechanical Turk or similar workers who are told to create 'facts' from structured sources like Wikipedia. The core bottleneck is that generating explicit 'explanations' is both difficult and incredibly time-consuming at a size large enough to form a useful training set.
There has been some movement towards the use of large, pre-trained models with the use of a fine-tuned BERT model, but there's not much visible activity to apply the big beasts of language modelling to integrated fact-checking systems (though work may well be ongoing at Facebook and elsewhere). These machine learning systems are therefore some way behind the cutting edge of language understanding and I expect large leaps would result from greater attention and firepower. At the moment, state-of-the-art fact checking systems are piecemeal and machine learning approaches are still sometimes outperformed by much simpler statistical algorithms. Fully automated approaches have so far been unsuccessful in their goal of giving useful feedback to found statements.
The primary practical use of fact-checking systems thus far has been in collating facts worth checking, and extracting an initial set of relevant pages (a task also carried out by internet search). The task of actually bringing specific pieces of evidence to bear on a claim and rendering a judgement is only in its infancy.
Wikipedia represents probably the largest and highest quality dataset with millions of articles leveraging both internal and external links to substantiate the points made, and it looks likely that this will form the bulk of future training for good explanation, alongside more scarce but higher quality human feedback.
Datasets
Creating large datasets to train an ML-based fact checking system is perhaps the biggest obstacle to a useful system. Here I go over what seem to be the major datasets and how they structure and source their data.
Fact Extraction and Verification (FEVER)
FEVER (paper, dataset website) is the most commonly used fact-checking corpus over the last few years. Its primary virtue is its large size and that it has a large set of claims for which the 'supporting evidence' sentences are explicitly picked out, whereas other datasets often only contain True/False/Not Supported labels.
They take the introduction of 5,000 of the most popular Wikipedia pages, and the pages that are linked from these introductions to get a set of 50,000 linked introductions. Facts are generated by hand-writing sentences labelled either Supported, Refuted, or Not Enough Evidence. For the former two classes, the writer also label the sentences that contribute to the statement being supported or refuted.
They generate 185k claims, of which about 135k have supporting sentences. The structure of the dataset is a separate entry for each supporting sentence, with each entry having the claim, the label, the title of the supporting page and the sentence ID of the supporting/refuting sentence.
Example showing structure of the dataset. Indicates that the claim is supported by (0-indexed) sentences 0, 1 and 6 of the page 'Oliver Reed', and the first sentences of the introduction for the films 'Castaway' and 'Gladiator', referenced in the 3rd and 4th sentences of the introduction respectively.
They used a group of 50 native English speakers but appeared to have significant difficulty getting their labellers to do more than trivial rewordings or negations of sentences in the introduction, designing the interface to have suggestions for other types of mutation and an 'ontology diagram', as well as highlighting negations to discourage their overuse.
Browsing the dataset shows that the claims are largely very simple and often of quite low quality.. The first claims in the Supporting/Refuting/Not enough evidence categories are, respectively, 'Nikolaj Coster-Waldau worked with the Fox Broadcasting Company.', 'Adrienne Bailon is an accountant.' and 'System of a Down briefly disbanded in limbo.'.
WikiFactCheckEnglish
This is another dataset (paper, GitHub for data) that uses Wikipedia as its ground truth but it sources its supporting evidence from PDFs cited as sources (as these are judged to generally be of a higher standard). It takes sentences from Wikipedia articles which are immediately followed by citation of a 'well-formed' document (mid-length pdf, in English). They use MTurk workers to generate negations and mutations of these sentences.
The dataset itself simply consists of the claim, whether it is True or False, and the URL of the PDF with supporting evidence, along with a context field which is the preceding sentences up to the last paragraph break. There is no 'correct' supporting claim - instead the detection of particular supporting sentences is an (optional) part of the process of determining if the claim is true or false.
This level of technical language is common to most of the entries in the database, giving a high degree of difficulty to pull it out of a document which may have many other themes.
This is clearly a much more ambitious challenge, and one which, relying on humans only for mutations and not for the sentences/sources themselves, could scale up to a much larger sourcing and entailment dataset.
The authors' own approach to solving their own challenge is to extract the most similar claim by picking the sentence with the lower normalized Levenshtein distance and then using either a linear classifier based on a hand-crafted features, or a neural network model trained on the SNLI (see below).
TabFact
This dataset (arXiv, dataset) tries to teach structured data by taking tables from Wikipedia (which is a dataset in itself) and asking MTurk users to create statements which are either entailed or refuted by the table.
The use of MTurk workers trying to create data as quickly as possible and without much training is clear. Poor quality English is very common. The meaning is usually, but not always, clear. They apparently restricted their data gathering to English-native countries and checked one statement from each task that it met criteria which include grammar, but it seems they've had to be quite generous in allowing poor grammar in their sentence creation.
This is the first example from TabFact I came across, and is broadly representative of the dataset
Stanford Natural Language Inference Corpus (SNLI)
This isn't exactly a fact-checking dataset but rather the preferred dataset for learning logical entailment: it asks the solver to decide whether a given 'hypothesis' is an entailment of, a contradiction of, or is neutral with respect to, the 'text'.
Examples from the SNLI.
A state-of-the-art fact-checking system generally is not a single system - instead it consists of separate models which A. pick out relevant documents, B. select relevant sentences from these documents, and C. decide whether these sentences support or refute the statement under question.
The SNLI is widely used to solve step C of the above process, either as the primary training data, or as a source of pre-training for models which are then trained on FEVER (which also trains some degree of sentence selection but only offers paragraph-sized example documents).
Other datasets of interest
The fact checks of Politifactand others have been collated here.
The Fake News Challenge has a small but reasonably high quality dataset assessing whether sentences agree with, disagree with, or are not relevant to a claim.
Approaches
Here's a summary of the main themes that came up when reading how researchers have tried to solve these datasets, and the automatic fact-checking problem more broadly.
Pipelines
There has generally not been a single model which attempts to solve the full stack of challenges. Instead, a pipeline of different models is used to get a complete system. The core of this system is Document Retrieval -> Sentence Selection -> Recognizing Textual Entailment and there is generally not an attempt to combine these into an end-to-end-trainable system. Instead they remain as separately trained units where the intermediate outputs can be inspected and used as outputs in their own right.
For sentence selection and recognizing textual entailment, the datasets above seem to be the major sources of training data. For evidence retrieval, I've not looked as closely, but there are some very different approaches. The human-assistance tools seem to rely on taking the top results from Google as a starting point and then filter the documents by a few, mostly hard-coded criteria.
The more ambitious pipeline approach of Nie at al instead use a neural document matcher which first narrows down relevant documents by simple keyword matching (I think using Wikipedia as its document pool), and then uses the FEVER dataset to train an LSTM-based ANN to learn whether there is a high relatedness, by using the sentences in FEVER listed as evidence for the claim as the positive cases of relatedness. (I think this is an accurate summary.. I found their methods section rather lacking when it came to how the model was trained.)
Helper Systems
Current systems are not sufficiently competent to be of much use on their own, so if models are intended to be of practical use then they come integrated in programs which assist a potential fact-checker.
Ideally I'd be able to explore these in more detail but I've not been able to test these systems. The two best documented systems are FullFact's system and the BRENDA system, which used to be publicly available but has now been folded into the proprietary Factiverse demo. I've requested trial access to both but so far without success.
All other tools that I've found attempt the more limited task of matching claims found online and in speech to previously checked claims. While a less ambitious project, a hypothetical version of this system which was able to point to the best explainer of the background of any claim would be a very valuable human-augmenting tool so it's worth keeping an eye on.
However, even these systems have struggled, as you can see from this rather sheepish article about how difficult it has been to connect their database of fact checkers to a real time audio feed from live debates.
Facebook has a predictably more upbeat piece about its own efforts which work towards identifying whether the content of some text or image is functionally identical to anything which has previously been checked but I don't think it has any publicly documented attempts to determine the truth value for itself.
Large Text Models
As many of you will know, perhaps the biggest ML development in the last few years has the power of huge, transformer-based language models to understand text at a fundamental level than had previously been possible. These have become state-of-the-art in a number of domains and I would expect that any highly successful system would build upon something of this nature in order to extract the most information from the text received.
As far as I can tell, the most advanced text-model applied to the problem is a 110M parameter version of BERT used by the authors of TabFact (Chen et al) and fine tuned to learn whether statements were entailed by Wikipedia tables, training for another 10k steps. This is obviously a step in the direction of big models but is a long way behind the state of the art (in terms of parameters, this is ~1000x smaller than GPT-3, for example) and this is reflected in the poor performance of these models, only working out if the relatively simple statements were true or false in about 70% of cases.
Instead, this smaller BERT model was marginally outperformed by a much more bespoke system for which they created a domain specific language to compose table-actions into a boolean result, converted these programs to strings, and then used a transformer to learn whether a given program string learns the T/F value of a statement.
Also interesting was the difficulty in linearising these kind of tables. At least with the limited fine tuning that they applied, the BERT model wasn't able to perform at all well when given the table in its original structure. Instead, they used a 'template' method where they would feed in sentences of the form 'row $ROW the $PROPERTY1 is $VALUE1'. With this more natural sentence structure, the model was then able to at least match the program-generation approach described above.
I expect that if someone were to apply the techniques from Learning to Summarize from Human Feedback - that is, fine-tuning a 100B+ parameter model on limited but genuinely high quality data and using high-concentration human ratings of success to drive maximizing behaviour then this would easily outperform anything that the field has produced so far.
Conclusion
Ultimately I see this as an area which is still struggling to find its feet, both because of the difficulty of the problem and the difficulty of getting high-quality data to train any system. There's a lot of untapped potential for the use of large language models and investment in good quality feedback - I think that the first somewhat useful system is probably 2-5 years away, depending some specialized investment from but offers a promising use case for human-machine co-working to flourish. Unfortunately, the issue of how to create a system which aggressively tries to find the truth, rather than replicating what is expected from a human checker, remains untouched by current work.
Intro / Motivation
This post is the write-up of a few days reading the automated fact-checking literature.
As AIs become more powerful, it would be very valuable to be able to train an agent to be able to tell us, not just whether particular statements are true, but why, and to explain their sources and reasoning as best as possible, especially in slow-takeoff scenarios. Fact-checking represents the present-day, practical end of the spectrum of possible work in this area, and I was curious what what had been done on it, the challenges the field faces, and whether it might shed light on the wider alignment problem.
Ultimately, I don't think there are any major positive surprises or insights generated by what I've read of the field. Instead the tale is more about the difficulties of getting high-quality data, and (especially given the current difficulty of the task) the need to carefully combine automated tools with natural human workflow in order to make something useful. Nonetheless, you might find value in some background on the field, its datasets and challenges.
If you're just interested in a general survey of the field, I would recommend reading the recent summary of the field Automated Fact-Checking for Assisting Human Fact-Checkers which talks through largely the same material from a more practical perspective. I'm instead looking more closely at the generation of the datasets and the techniques employed in solving them.
Summary
The few fact-checking datasets generally consist of tens to hundreds of thousands of sentences hand-curated by Mechanical Turk or similar workers who are told to create 'facts' from structured sources like Wikipedia. The core bottleneck is that generating explicit 'explanations' is both difficult and incredibly time-consuming at a size large enough to form a useful training set.
There has been some movement towards the use of large, pre-trained models with the use of a fine-tuned BERT model, but there's not much visible activity to apply the big beasts of language modelling to integrated fact-checking systems (though work may well be ongoing at Facebook and elsewhere). These machine learning systems are therefore some way behind the cutting edge of language understanding and I expect large leaps would result from greater attention and firepower. At the moment, state-of-the-art fact checking systems are piecemeal and machine learning approaches are still sometimes outperformed by much simpler statistical algorithms. Fully automated approaches have so far been unsuccessful in their goal of giving useful feedback to found statements.
The primary practical use of fact-checking systems thus far has been in collating facts worth checking, and extracting an initial set of relevant pages (a task also carried out by internet search). The task of actually bringing specific pieces of evidence to bear on a claim and rendering a judgement is only in its infancy.
Wikipedia represents probably the largest and highest quality dataset with millions of articles leveraging both internal and external links to substantiate the points made, and it looks likely that this will form the bulk of future training for good explanation, alongside more scarce but higher quality human feedback.
Datasets
Creating large datasets to train an ML-based fact checking system is perhaps the biggest obstacle to a useful system. Here I go over what seem to be the major datasets and how they structure and source their data.
Fact Extraction and Verification (FEVER)
FEVER (paper, dataset website) is the most commonly used fact-checking corpus over the last few years. Its primary virtue is its large size and that it has a large set of claims for which the 'supporting evidence' sentences are explicitly picked out, whereas other datasets often only contain True/False/Not Supported labels.
They take the introduction of 5,000 of the most popular Wikipedia pages, and the pages that are linked from these introductions to get a set of 50,000 linked introductions. Facts are generated by hand-writing sentences labelled either Supported, Refuted, or Not Enough Evidence. For the former two classes, the writer also label the sentences that contribute to the statement being supported or refuted.
They generate 185k claims, of which about 135k have supporting sentences. The structure of the dataset is a separate entry for each supporting sentence, with each entry having the claim, the label, the title of the supporting page and the sentence ID of the supporting/refuting sentence.
They used a group of 50 native English speakers but appeared to have significant difficulty getting their labellers to do more than trivial rewordings or negations of sentences in the introduction, designing the interface to have suggestions for other types of mutation and an 'ontology diagram', as well as highlighting negations to discourage their overuse.
Browsing the dataset shows that the claims are largely very simple and often of quite low quality.. The first claims in the Supporting/Refuting/Not enough evidence categories are, respectively, 'Nikolaj Coster-Waldau worked with the Fox Broadcasting Company.', 'Adrienne Bailon is an accountant.' and 'System of a Down briefly disbanded in limbo.'.
WikiFactCheckEnglish
This is another dataset (paper, GitHub for data) that uses Wikipedia as its ground truth but it sources its supporting evidence from PDFs cited as sources (as these are judged to generally be of a higher standard). It takes sentences from Wikipedia articles which are immediately followed by citation of a 'well-formed' document (mid-length pdf, in English). They use MTurk workers to generate negations and mutations of these sentences.
The dataset itself simply consists of the claim, whether it is True or False, and the URL of the PDF with supporting evidence, along with a context field which is the preceding sentences up to the last paragraph break. There is no 'correct' supporting claim - instead the detection of particular supporting sentences is an (optional) part of the process of determining if the claim is true or false.
This is clearly a much more ambitious challenge, and one which, relying on humans only for mutations and not for the sentences/sources themselves, could scale up to a much larger sourcing and entailment dataset.
The authors' own approach to solving their own challenge is to extract the most similar claim by picking the sentence with the lower normalized Levenshtein distance and then using either a linear classifier based on a hand-crafted features, or a neural network model trained on the SNLI (see below).
TabFact
This dataset (arXiv, dataset) tries to teach structured data by taking tables from Wikipedia (which is a dataset in itself) and asking MTurk users to create statements which are either entailed or refuted by the table.
The use of MTurk workers trying to create data as quickly as possible and without much training is clear. Poor quality English is very common. The meaning is usually, but not always, clear. They apparently restricted their data gathering to English-native countries and checked one statement from each task that it met criteria which include grammar, but it seems they've had to be quite generous in allowing poor grammar in their sentence creation.
Stanford Natural Language Inference Corpus (SNLI)
This isn't exactly a fact-checking dataset but rather the preferred dataset for learning logical entailment: it asks the solver to decide whether a given 'hypothesis' is an entailment of, a contradiction of, or is neutral with respect to, the 'text'.
A state-of-the-art fact-checking system generally is not a single system - instead it consists of separate models which A. pick out relevant documents, B. select relevant sentences from these documents, and C. decide whether these sentences support or refute the statement under question.
The SNLI is widely used to solve step C of the above process, either as the primary training data, or as a source of pre-training for models which are then trained on FEVER (which also trains some degree of sentence selection but only offers paragraph-sized example documents).
Other datasets of interest
The fact checks of Politifact and others have been collated here.
The Fake News Challenge has a small but reasonably high quality dataset assessing whether sentences agree with, disagree with, or are not relevant to a claim.
Approaches
Here's a summary of the main themes that came up when reading how researchers have tried to solve these datasets, and the automatic fact-checking problem more broadly.
Pipelines
There has generally not been a single model which attempts to solve the full stack of challenges. Instead, a pipeline of different models is used to get a complete system. The core of this system is Document Retrieval -> Sentence Selection -> Recognizing Textual Entailment and there is generally not an attempt to combine these into an end-to-end-trainable system. Instead they remain as separately trained units where the intermediate outputs can be inspected and used as outputs in their own right.
For sentence selection and recognizing textual entailment, the datasets above seem to be the major sources of training data. For evidence retrieval, I've not looked as closely, but there are some very different approaches. The human-assistance tools seem to rely on taking the top results from Google as a starting point and then filter the documents by a few, mostly hard-coded criteria.
The more ambitious pipeline approach of Nie at al instead use a neural document matcher which first narrows down relevant documents by simple keyword matching (I think using Wikipedia as its document pool), and then uses the FEVER dataset to train an LSTM-based ANN to learn whether there is a high relatedness, by using the sentences in FEVER listed as evidence for the claim as the positive cases of relatedness. (I think this is an accurate summary.. I found their methods section rather lacking when it came to how the model was trained.)
Helper Systems
Current systems are not sufficiently competent to be of much use on their own, so if models are intended to be of practical use then they come integrated in programs which assist a potential fact-checker.
Ideally I'd be able to explore these in more detail but I've not been able to test these systems. The two best documented systems are FullFact's system and the BRENDA system, which used to be publicly available but has now been folded into the proprietary Factiverse demo. I've requested trial access to both but so far without success.
All other tools that I've found attempt the more limited task of matching claims found online and in speech to previously checked claims. While a less ambitious project, a hypothetical version of this system which was able to point to the best explainer of the background of any claim would be a very valuable human-augmenting tool so it's worth keeping an eye on.
However, even these systems have struggled, as you can see from this rather sheepish article about how difficult it has been to connect their database of fact checkers to a real time audio feed from live debates.
Facebook has a predictably more upbeat piece about its own efforts which work towards identifying whether the content of some text or image is functionally identical to anything which has previously been checked but I don't think it has any publicly documented attempts to determine the truth value for itself.
Large Text Models
As many of you will know, perhaps the biggest ML development in the last few years has the power of huge, transformer-based language models to understand text at a fundamental level than had previously been possible. These have become state-of-the-art in a number of domains and I would expect that any highly successful system would build upon something of this nature in order to extract the most information from the text received.
As far as I can tell, the most advanced text-model applied to the problem is a 110M parameter version of BERT used by the authors of TabFact (Chen et al) and fine tuned to learn whether statements were entailed by Wikipedia tables, training for another 10k steps. This is obviously a step in the direction of big models but is a long way behind the state of the art (in terms of parameters, this is ~1000x smaller than GPT-3, for example) and this is reflected in the poor performance of these models, only working out if the relatively simple statements were true or false in about 70% of cases.
Instead, this smaller BERT model was marginally outperformed by a much more bespoke system for which they created a domain specific language to compose table-actions into a boolean result, converted these programs to strings, and then used a transformer to learn whether a given program string learns the T/F value of a statement.
Also interesting was the difficulty in linearising these kind of tables. At least with the limited fine tuning that they applied, the BERT model wasn't able to perform at all well when given the table in its original structure. Instead, they used a 'template' method where they would feed in sentences of the form 'row $ROW the $PROPERTY1 is $VALUE1'. With this more natural sentence structure, the model was then able to at least match the program-generation approach described above.
I expect that if someone were to apply the techniques from Learning to Summarize from Human Feedback - that is, fine-tuning a 100B+ parameter model on limited but genuinely high quality data and using high-concentration human ratings of success to drive maximizing behaviour then this would easily outperform anything that the field has produced so far.
Conclusion
Ultimately I see this as an area which is still struggling to find its feet, both because of the difficulty of the problem and the difficulty of getting high-quality data to train any system. There's a lot of untapped potential for the use of large language models and investment in good quality feedback - I think that the first somewhat useful system is probably 2-5 years away, depending some specialized investment from but offers a promising use case for human-machine co-working to flourish. Unfortunately, the issue of how to create a system which aggressively tries to find the truth, rather than replicating what is expected from a human checker, remains untouched by current work.