Installation

Requirements

• Python 3.11+
• OpenAI API key (or Anthropic for multi-model)
• Typer 0.24+ (CLI framework)

Install

pip install truthfulness-evaluator

Or with Poetry:

poetry add truthfulness-evaluator

Set API Keys

export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."  # Optional, for multi-model

Or use a .env file:

cat > .env << EOF
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
EOF

Verify Installation

truth-eval --version

Should output: Truthfulness Evaluator v0.1.0

Optional Dependencies

Web search is enabled by default using duckduckgo-search (bundled with the package).

For development:

pip install truthfulness-evaluator[dev]

Note: refchecker is optional and NOT required. The tool works without it by falling back to LLM-based extraction.

Troubleshooting

ImportError: No module named 'refchecker'

RefChecker is optional and NOT required. The tool automatically falls back to LLM-based extraction. You can safely ignore this warning.

If you want to use triplet extraction:

pip install refchecker  # Optional, for triplet extraction

DuckDuckGo search not working

pip install -U duckduckgo-search

Rate limits

Use cheaper models for extraction:

export TRUTH_EXTRACTION_MODEL=gpt-4o-mini