This repository contains the implementation and experimental artifacts for SAFE-FC, a framework for evaluating automated fact-checking systems beyond verdict accuracy.
SAFE-FC separates two sources of input ambiguity from two types of system failure:
| Evaluation stage | Dimension | What it measures |
|---|---|---|
| Claim | Claim ambiguity | Whether the claim is underspecified or semantically ambiguous |
| Evidence | Evidence ambiguity | Whether the retrieved evidence is incomplete, outdated, or contradictory |
| Output | Grounding failure | Whether the response misrepresents its sources or violates the verification protocol |
| Output | Reasoning failure | Whether the reasoning mishandles evidence, ambiguity, internal consistency, or time |
This distinction helps identify cases in which a wrong verdict results from an ill-posed verification problem, as well as cases in which a correct verdict is supported by unreliable reasoning.
data/
experiments/ Per-claim metric outputs used by the analysis
fever_claims_slice_300.jsonl Development slice of 300 FEVER claims
results/ Generated summaries, statistical tests, and tables
scripts/
claim_ambiguity.py Claim-side scoring pipeline
extract_misinfo_slice.py Dataset extraction utility
experiments/
analyze_ambiguity_with_evidence.py
src/
input_layer/
specification/ Temporal and geographic underspecification
semantic/ Lexical polysemy and vagueness
evidence/ Timeliness, contradiction, and evidence gaps
output_layer/
grounding/ Attribution and protocol-compliance checks
reasoning/ Reasoning-failure diagnostics
The committed files in data/experiments/ provide per-claim inputs to the analysis. Generated aggregate results are written to results/.
- Temporal underspecification: the claim depends on time but lacks a sufficiently precise temporal anchor.
- Geographic underspecification: the claim depends on location or jurisdiction but lacks a sufficiently precise geographic anchor.
- Lexical polysemy: words in the claim admit multiple plausible meanings in context.
- Vagueness: the claim contains fuzzy or imprecise language.
- Evidence timeliness: retrieved evidence may be outdated relative to the claim and reference date.
- Evidence contradiction: relevant evidence items disagree with the claim or provide conflicting signals.
- Evidence gaps: the available evidence set provides limited coverage.
- False attribution: statements are attributed to sources that do not support them.
- Task-instruction failure: the response does not follow the required verification protocol or output format.
- Evidence-reasoning failure: the rationale does not adequately integrate the available evidence.
- Ambiguity-awareness failure: the system commits to a verdict when the input warrants uncertainty or abstention.
- Internal contradiction: the rationale conflicts with itself or with the predicted verdict.
- Temporal-reasoning failure: the rationale mishandles temporal scope, ordering, or causality.
Formal definitions of all metrics are provided in the paper and supplementary material.
Because several SAFE-FC dimensions are evaluated with LLM judges, the paper tests whether the measurements are stable across judge models, prompt formulations, benchmarks, and human assessment. Full protocols and per-metric results are reported in the supplementary material.
The first study evaluates the same 100-claim LiveFact sample using three judge models (two open-weight models and one proprietary model) and three ways of framing each construct. It considers two tasks: scoring claim ambiguity and labeling each evidence item as SUPPORT, CONTRADICT, or UNRELATED.
- Evidence-contradiction judgments show substantial agreement, with Cohen's kappa between 0.60 and 0.68.
- Claim-ambiguity judgments show lower agreement, with kappa between 0.15 and 0.33.
- For claim ambiguity, changing the construct framing has a larger effect than changing the judge model.
These results indicate that claim-level ambiguity is partly definition-dependent. SAFE-FC therefore reports separate ambiguity dimensions and conditions its analyses on dataset-specific low, middle, and high strata rather than treating small differences between exact scores as meaningful.
The second study isolates prompt wording from construct definition. Each of the nine LLM-judged metrics is evaluated with three semantically equivalent versions of its production prompt: prose, sectioned, and terse with reordered input blocks. The decision criteria, thresholds, rubric, JSON schema, and value domain remain fixed.
The full sweep crosses two open-weight judge families (Llama and Gemma) with two benchmarks (FEVER and COVIDFact). Agreement is measured using ICC(2,1) for continuous claim-side scores and mean pairwise Cohen's kappa for binary flags. Internal contradiction is evaluated separately against synthetically injected contradictions because its positive class is sparse.
- All 32 ICC/kappa metric-model-benchmark combinations are at or above 0.60, and 28 exceed 0.65.
- Mean ICC ranges from 0.70 to 0.75 across the four model-benchmark settings.
- Mean kappa ranges from 0.74 to 0.84.
- Ambiguity-awareness and instruction-compliance judgments are the most stable; geographic underspecification and evidence-reasoning judgments are less stable.
The residual variation is primarily metric-specific rather than concentrated in one model family or benchmark.
An independent non-author validation was conducted on 110 COVIDFact instances across two disjoint subsets. Claim, evidence, grounding, and reasoning judgments were rated on a three-point scale: 2 for agree, 1 for somewhat agree, and 0 for disagree.
| Dimension | Disagree | Somewhat agree | Agree | Weighted agreement |
|---|---|---|---|---|
| Claim ambiguity | 10.0% | 23.6% | 66.4% | 78.2% |
| Evidence ambiguity | 7.3% | 10.9% | 81.8% | 87.3% |
| Grounding failure | 8.2% | 15.5% | 76.4% | 84.1% |
| Reasoning failure | 10.0% | 29.1% | 60.9% | 75.5% |
| All judgments | 8.9% | 19.8% | 71.4% | 81.3% |
The two subsets produced similar weighted agreement scores of 80.9% and 81.6%. Agreement is strongest for evidence ambiguity and grounding failures, while reasoning failures receive the largest share of partial-agreement ratings.
The paper evaluates SAFE-FC on five datasets with different domains and construction methods:
| Dataset | Claims used | Domain | Gold verdicts used? |
|---|---|---|---|
| FEVER | 4,000 | Wikipedia-based claims | Yes |
| COVIDFact | 3,923 | COVID-19 claims | Yes |
| FaVIQ | 979 | Open-domain claims derived from information-seeking questions | Yes |
| SciNLP | 937 | Scientific claims | No |
| LiveFact | 2,000 | Dynamic, time-aware claims | Independent ambiguity labels |
SciNLP is used for label-free ambiguity and failure profiling. LiveFact is used to compare SAFE-FC's claim-side ambiguity scores with independently annotated ambiguity labels.
Dataset files use one JSON object per line. The main fields are:
{
"claim": "...",
"veracity": "true|false|unknown",
"label": "SUPPORTS|REFUTES|...",
"dataset": "...",
"split": "train"
}The repository also includes data/fever_claims_slice_300.jsonl, a small development slice for testing the pipeline. It is not the full FEVER sample reported in the paper.
- Python 3
- A local Ollama server for LLM-based metrics
- An Ollama-compatible model; the code defaults to
llama3.1:8b - Optional: spaCy with
en_core_web_smfor lexical candidate extraction
The Ollama model and endpoint can be changed through environment variables:
export OLLAMA_MODEL=llama3.1:8b
export OLLAMA_BASE_URL=http://localhost:11434Specification scorers can fall back to heuristic scoring if Ollama is unavailable. Metrics that rely on an LLM judge require a running Ollama server.
The commands below run the claim-side pipeline on the included 300-claim FEVER development slice.
python scripts/claim_ambiguity.py \
--input data/fever_claims_slice_300.jsonl \
--output data/experiments/fever_claims_slice_300_claim_ambiguity.csvRun the temporal or geographic scorers separately with:
python src/input_layer/specification/temporal_underspecification.py \
--input data/fever_claims_slice_300.jsonl \
--output data/fever_claims_slice_300_temporal_scores.jsonl \
--use-llm
python src/input_layer/specification/geographic_underspecification.py \
--input data/fever_claims_slice_300.jsonl \
--output data/fever_claims_slice_300_geographic_scores.jsonl \
--use-llmAdd --no-llm to either command to use heuristic-only specification scoring.
The repository includes the per-claim metric CSVs consumed by the two-axis analysis. To regenerate the aggregate tables, statistical tests, correlation matrix, and summary report, run:
python scripts/experiments/analyze_ambiguity_with_evidence.pyThe script writes the following files to results/:
merged_per_claim.csv
intrinsic_ambiguity_summary.csv
accuracy_by_claim_stratum.csv
accuracy_by_evidence_stratum.csv
joint_stratum_grid.csv
output_failures_by_claim_stratum.csv
output_failures_by_evidence_stratum.csv
low_vs_high_tests_claim.csv
low_vs_high_tests_evidence.csv
correlation_7x7.csv
summary_report.md
summary_report.md provides a compact narrative summary of the generated results.
The extraction utility supports dataset configurations from ComplexDataLab/Misinfo_Datasets. For example:
python scripts/extract_misinfo_slice.py \
--dataset-config fever \
--allow-unknown \
--size all \
--output data/fever_claims_full.jsonl
python scripts/extract_misinfo_slice.py \
--dataset-config covidfact \
--size all \
--output data/covidfact_claims_full.jsonl
python scripts/extract_misinfo_slice.py \
--dataset-config faviq \
--size all \
--output data/faviq_claims_full.jsonlExact duplicate claims are removed by default. Use --no-dedupe to retain them or replace all with an integer to create a fixed-size slice.
The metric files include small executable examples:
python src/input_layer/evidence/evidence_contradiction.py
python src/input_layer/evidence/evidence_gaps.py
python src/input_layer/evidence/evidence_timeliness.py
python src/input_layer/semantic/vagueness.py
python src/input_layer/semantic/lexical_polysemy.py
python src/output_layer/grounding/false_attribution.py
python src/output_layer/grounding/instruction_ambuguity.py
python src/output_layer/reasoning/evidence_reasoning_failure.py
python src/output_layer/reasoning/ambuguity_awarness_failure.py
python src/output_layer/reasoning/internal_contradiction.py
python src/output_layer/reasoning/temporal_reasonoing_failure.py