Skip to content

Protocols

The protocol layer defines the core abstractions for the truthfulness evaluation pipeline. Each protocol represents a distinct responsibility in the workflow:

  • ClaimExtractor: Extracts factual claims from input documents
  • EvidenceGatherer: Gathers supporting or refuting evidence for claims
  • ClaimVerifier: Verifies claims against collected evidence
  • ReportFormatter: Formats verification results into human-readable reports

These protocols enable pluggable strategies, allowing you to mix and match different implementations for each stage of the pipeline.

ClaimExtractor

truthfulness_evaluator.core.protocols.ClaimExtractor

Bases: Protocol

Protocol for claim extraction strategies.

Implementations extract factual claims from document text. Different implementations handle different document types and domains.

Source code in src/truthfulness_evaluator/core/protocols.py 
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
@runtime_checkable
class ClaimExtractor(Protocol):
    """Protocol for claim extraction strategies.

    Implementations extract factual claims from document text.
    Different implementations handle different document types and domains.
    """

    async def extract(
        self,
        document: str,
        source_path: str,
        *,
        max_claims: int | None = None,
    ) -> list[Claim]:
        """Extract claims from document text.

        Args:
            document: The document text to analyze.
            source_path: Path or URL of the source document.
            max_claims: Optional limit on number of claims to extract.

        Returns:
            List of extracted Claim objects.
        """
        ...

extract(document, source_path, *, max_claims=None) async

Extract claims from document text.

Parameters:

Name Type Description Default
document str

The document text to analyze.

 required
source_path str

Path or URL of the source document.

 required
max_claims int | None

Optional limit on number of claims to extract.

 None

Returns:

Type Description
list[Claim]

List of extracted Claim objects.
Source code in src/truthfulness_evaluator/core/protocols.py 
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
async def extract(
    self,
    document: str,
    source_path: str,
    *,
    max_claims: int | None = None,
) -> list[Claim]:
    """Extract claims from document text.

    Args:
        document: The document text to analyze.
        source_path: Path or URL of the source document.
        max_claims: Optional limit on number of claims to extract.

    Returns:
        List of extracted Claim objects.
    """
    ...

Usage Note: Concrete implementations include SimpleExtractor (general-purpose) and TripletExtractor (subject-predicate-object triples).

EvidenceGatherer

truthfulness_evaluator.core.protocols.EvidenceGatherer

Bases: Protocol

Protocol for evidence gathering strategies.

Implementations search for evidence that supports or refutes claims. Different implementations search different sources (web, filesystem, API endpoints, git history, etc.).

Source code in src/truthfulness_evaluator/core/protocols.py 
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
@runtime_checkable
class EvidenceGatherer(Protocol):
    """Protocol for evidence gathering strategies.

    Implementations search for evidence that supports or refutes claims.
    Different implementations search different sources (web, filesystem,
    API endpoints, git history, etc.).
    """

    async def gather(
        self,
        claim: Claim,
        context: dict[str, Any],
    ) -> list[Evidence]:
        """Gather evidence for a claim.

        Args:
            claim: The claim to find evidence for.
            context: Workflow context (root_path, config, etc.).

        Returns:
            List of Evidence objects found.
        """
        ...

gather(claim, context) async

Gather evidence for a claim.

Parameters:

Name Type Description Default
claim Claim

The claim to find evidence for.

 required
context dict[str, Any]

Workflow context (root_path, config, etc.).

 required

Returns:

Type Description
list[Evidence]

List of Evidence objects found.
Source code in src/truthfulness_evaluator/core/protocols.py 
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
async def gather(
    self,
    claim: Claim,
    context: dict[str, Any],
) -> list[Evidence]:
    """Gather evidence for a claim.

    Args:
        claim: The claim to find evidence for.
        context: Workflow context (root_path, config, etc.).

    Returns:
        List of Evidence objects found.
    """
    ...

Usage Note: Implementations range from WebSearchGatherer (web-based evidence) to FilesystemGatherer (local file evidence) to CompositeGatherer (combines multiple gatherers).

ClaimVerifier

truthfulness_evaluator.core.protocols.ClaimVerifier

Bases: Protocol

Protocol for claim verification strategies.

Implementations judge whether evidence supports, refutes, or is insufficient for a claim. Different implementations use different judgment methods (single LLM, consensus, deterministic).

Source code in src/truthfulness_evaluator/core/protocols.py 
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
@runtime_checkable
class ClaimVerifier(Protocol):
    """Protocol for claim verification strategies.

    Implementations judge whether evidence supports, refutes,
    or is insufficient for a claim. Different implementations use
    different judgment methods (single LLM, consensus, deterministic).
    """

    async def verify(
        self,
        claim: Claim,
        evidence: list[Evidence],
    ) -> VerificationResult:
        """Verify a claim against gathered evidence.

        Args:
            claim: The claim to verify.
            evidence: Evidence gathered for this claim.

        Returns:
            VerificationResult with verdict, confidence, and explanation.
        """
        ...

verify(claim, evidence) async

Verify a claim against gathered evidence.

Parameters:

Name Type Description Default
claim Claim

The claim to verify.

 required
evidence list[Evidence]

Evidence gathered for this claim.

 required

Returns:

Type Description
VerificationResult

VerificationResult with verdict, confidence, and explanation.
Source code in src/truthfulness_evaluator/core/protocols.py 
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
async def verify(
    self,
    claim: Claim,
    evidence: list[Evidence],
) -> VerificationResult:
    """Verify a claim against gathered evidence.

    Args:
        claim: The claim to verify.
        evidence: Evidence gathered for this claim.

    Returns:
        VerificationResult with verdict, confidence, and explanation.
    """
    ...

Usage Note: Choose between SingleModelVerifier (fast, single LLM), ConsensusVerifier (multi-model voting), or InternalVerifier (code-documentation alignment).

ReportFormatter

truthfulness_evaluator.core.protocols.ReportFormatter

Bases: Protocol

Protocol for report formatting strategies.

Implementations produce output in different formats (JSON, Markdown, HTML, custom domain-specific formats).

Source code in src/truthfulness_evaluator/core/protocols.py 
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
@runtime_checkable
class ReportFormatter(Protocol):
    """Protocol for report formatting strategies.

    Implementations produce output in different formats
    (JSON, Markdown, HTML, custom domain-specific formats).
    """

    def format(self, report: TruthfulnessReport) -> str:
        """Format a truthfulness report.

        Args:
            report: The report to format.

        Returns:
            Formatted report as a string.
        """
        ...

    def file_extension(self) -> str:
        """Return the default file extension for this format.

        Returns:
            Extension string (e.g., ".json", ".md", ".html").
        """
        ...

file_extension()

Return the default file extension for this format.

Returns:

Type Description
str

Extension string (e.g., ".json", ".md", ".html").
Source code in src/truthfulness_evaluator/core/protocols.py 
114
115
116
117
118
119
120
def file_extension(self) -> str:
    """Return the default file extension for this format.

    Returns:
        Extension string (e.g., ".json", ".md", ".html").
    """
    ...

format(report)

Format a truthfulness report.

Parameters:

Name Type Description Default
report TruthfulnessReport

The report to format.

 required

Returns:

Type Description
str

Formatted report as a string.
Source code in src/truthfulness_evaluator/core/protocols.py 
103
104
105
106
107
108
109
110
111
112
def format(self, report: TruthfulnessReport) -> str:
    """Format a truthfulness report.

    Args:
        report: The report to format.

    Returns:
        Formatted report as a string.
    """
    ...

Usage Note: Built-in formatters support JSON, Markdown, and HTML output. Implement this protocol to add custom report formats.