Claim Verification Kit

An agent workflow for fact-checking, domain vetting, claim verification, and source citation. This repo packages the working components: skill files that encode the verification procedures, copy-paste prompt snippets extracted from them, the citation ledger script that mechanizes sourcing, and a complete real report produced by the pipeline.

The design goal: an LLM agent that verifies claims instead of asserting them. Every factual statement that leaves the system carries a source, a tier rating, and a verdict from a fixed taxonomy. The pipeline is model-agnostic: it works with any agent framework that can search the web, fetch URLs, and follow a structured output contract.

One target application this kit was assembled for: a utility website that takes a Wikipedia article, finds uncited claims in it, researches reliable sources backing each claim (or notes that none could be found), and hands an itemized report to a human editor who makes the final call. Example 2 below demonstrates exactly that output.

Pipeline at a Glance

                      ┌─────────────────────────────────────┐
                      │  claim text in (statement, or a     │
                      │  full article URL)                  │
                      └──────────────────┬──────────────────┘
                                         │
              ┌──────────────────────────┼──────────────────────────┐
              ▼                          ▼                          ▼
    ┌───────────────────┐    ┌────────────────────┐    ┌─────────────────────┐
    │ claim-verify      │    │ domain-intel       │    │ grounded-citations  │
    │ parse assertions, │    │ WHOIS/DNS/SSL/ASN  │    │ citation ledger:    │
    │ gather evidence,  │    │ on the SOURCE,     │    │ every URL gets an   │
    │ tiered sources,   │    │ credibility score, │    │ id from retrieval,  │
    │ verdict           │    │ bias assessment    │    │ verbatim quotes     │
    └─────────┬─────────┘    └─────────┬──────────┘    └──────────┬──────────┘
              │                        │                          │
              ▼                        │                          ▼
    ┌───────────────────┐              │              ┌─────────────────────┐
    │ wikidata          │              │              │ verify --evidence   │
    │ structured entity │              │              │ gate: fails drafts  │
    │ facts (Q-ID/P-ID) │              │              │ citing unevidenced  │
    └───────────────────┘              │              │ sources             │
                                       │              └─────────────────────┘
              ┌────────────────────────┘
              ▼
    ┌───────────────────┐
    │ wayback           │  dead/changed/paywalled sources →
    │ archived snapshots│  archived copies with timestamps
    └───────────────────┘

                      ┌─────────────────────────────────────┐
                      │  verdict + confidence + tiered      │
                      │  citations out                      │
                      └─────────────────────────────────────┘

Flow: parse the claim into discrete assertions, gather corroborating and contradicting evidence, assess the source before trusting it, attach citations mechanically, emit a fixed-format report. For batches of claims, factcheck-kanban dispatches one worker per claim on a board.

Component Catalog

claim-verify — the core verification pipeline

File: skills/claim-verify.md

The main skill. Four phases: parse the claim into verifiable assertions, gather evidence, evaluate quality and consensus, synthesize a verdict in a mandatory report format. Handles two claim classes differently:

• Simple fact claims go through parallel web searches: direct verification, academic sources (site:.edu, site:.gov), reputable news, fact-checking organizations, expert consensus, and a deliberate counter-evidence search ("{claim}" false debunked hoax myth).
• Comparative/numeric claims require programmatic analysis. If a claim says a country ranks 7th by some ratio, the agent must fetch both datasets, compute the ratio for every entity, sort, and compare the computed rank to the claimed one. Guessing from memory is prohibited.

Trigger phrases: "Fact-check: ...", "Verify this claim: ...", "Is this true: ..."

Key machinery (all verbatim in the skill file):

• Six-verdict taxonomy with consensus thresholds (see snippets/verdict-taxonomy.md)
• Four-tier source reliability ladder (see snippets/source-tiers.md)
• Anti-hallucination rules that bind the verifying agent (see snippets/anti-hallucination-rules.md)
• Mandatory output template with length limits (see snippets/output-format-template.md)
• Report persistence convention: completed reports land in fact-check-reports/[YYYYMMDD_HHMMSS]/completed/ with a naming scheme

domain-intel — source vetting

File: skills/domain-intel.md

Runs in parallel with claim verification: before trusting a source, investigate the source. Four technical checks via the WhoisJSON API (WHOIS, DNS, SSL, domain availability) plus ASN identification, then six web searches for reputation: MBFC ratings, editorial transparency, bias indicators, content-mill signals (freelance postings demanding "minimum X articles per day"), and traffic estimates.

Scoring model: factuality 40%, transparency 25%, balanced reporting 20%, bias direction 15%. Special-category penalties for conspiracy or pseudoscience content. Four possible outcomes:

• ✓ LEGIT — safe to cite with normal editorial verification
• ? MEDIUM — verify independently before citing
• ☠ BOGUS — do not cite
• ✗ CONFIRMED BOGUS — domain unregistered, SSL subject mismatch, or no A records

Confirmed-bogus is a stop condition: if the domain is fake, the article claims do not get the benefit of corroboration.

For a full article URL it also fact-checks the article itself, including an internal-consistency check: contradictory figures across the same outlet's own content are a sourcing red flag.

Requires WHOISJSON_API_KEY (free tier: 1,000 requests/month). CLI fallbacks (dig, openssl, Wayback CDX) cover DNS, TLS, and domain-age estimation when the API is unavailable.

grounded-citations — the citation spine

File: skills/grounded-citations.md Script: scripts/sources.py (stdlib-only Python 3)

The mechanism that makes hallucinated citations structurally impossible. A ledger script owns the url → [n] mapping. The model only ever emits small integers it was handed by retrieval; it never retypes a URL.

S=scripts/sources.py
python "$S" reset                                # fresh ledger per task
python "$S" add <url> --title "Title"            # prints: [1]
python "$S" quote 1 --text "exact sentence" --from page.txt
python "$S" render --cited-in draft.md           # write the Sources block
python "$S" verify draft.md --evidence --min-coverage 0.5

Three properties worth stealing for any LLM pipeline:

1. Verbatim-quote evidence. quote is rejected unless the text appears exactly in the fetched page. A paraphrase cannot pose as evidence.
2. [unverified] markers. Claims that cannot be sourced get an explicit flag, never a fake citation. The verify gate counts them as declared provenance.
3. The evidence gate. verify --evidence fails any draft whose cited sources carry no attached quotes. Non-zero exit on unknown ids, on a Sources block that disagrees with the ledger, or on prose below the coverage threshold.

wikidata — structured entity facts

File: skills/wikidata.md

Tier 1 source for claims about people, places, organizations, dates, and positions. No API key. Three-step pattern:

1. wbsearchentities → resolve name to Q-ID
2. wbgetclaims      → retrieve statement for property (rank=preferred)
3. wbgetentities    → resolve Q-ID values to readable labels (batch up to 50)

Includes the property cheat-sheet (P569 birth date, P19 birthplace, P39 position held with P580/P582 qualifier windows, P1082 population, ...), rank semantics (preferred/normal/deprecated), and SPARQL patterns for multi-hop queries that per-entity lookups cannot express.

Best for: ages, death claims, birthplaces, citizenship, current positions, capitals, populations. Weak for: breaking news (community-edited lag).

wayback — archived sources (and a live demonstration of why it matters)

File: skills/wayback.md

One API call, four uses: original URL returns 404, content was modified, you need what a page said at a specific time, or the source went behind a paywall after capture.

http://archive.org/wayback/available?url={url}&timestamp={YYYYMMDDhhmmss}

Citation format for recovered sources: cite the original with the archive URL and snapshot date. The skill also documents its own limits: not everything is archived, JS-heavy pages may not render, and some sites block crawlers.

This is not a theoretical capability — the worked examples below use it. The Columbia Journalism Review piece that sources the Michael Kelly quote is dead at its original URL (archives.cjr.org); the pipeline located the Internet Archive snapshot (captured 2004-02-15), extracted the article, and verified the quote verbatim against the archived text. The citation in the report carries both the original URL and the archive URL with the snapshot date, so a reader can check either. For historical work (the Hardy/Titanic chronology example) the same pattern applies to 1912 newspaper scans whose hosting pages are fragile.

factcheck-kanban — batch orchestration

File: skills/factcheck-kanban.md

Hermes-specific. Scales the pipeline from one claim to a queue: a dispatcher picks up ready cards every 60 seconds and spawns a worker profile per claim. Each card carries the claim text, instructions to load claim-verify, the report output path, and the expected output contract. Includes the failure catalog: skills not found (worker profile isolation), workers that never dispatch (missing profile or model config), and the status lifecycle (triage → todo → ready → running → done/blocked).

The generalizable part is the task body template: claim verbatim, verification pipeline pointer, report path, and expected output format. Any queue system can use that shape.

Prompt Snippets

Ready-to-paste fragments extracted from the skill files, for building subagent prompts without shipping the full skill text:

• snippets/output-format-template.md the mandatory report skeleton with length limits
• snippets/verdict-taxonomy.md six verdicts with consensus thresholds
• snippets/source-tiers.md the four-tier reliability ladder and evidence-quality bands
• snippets/anti-hallucination-rules.md binding rules for any agent handling factual claims
• snippets/citation-rules.md URL-mandatory citations, no-weasel rules, snippet-vs-page discipline
• snippets/source-cred-evaluator.md compact subagent prompt: URL in, strict JSON credibility rating out (pairs with the wiki-uncited-claims demo as the per-source filter)

Worked Examples

Three complete outputs produced by this pipeline, unedited:

1. Winnie the Pooh real-bear claim

examples/winnie-the-pooh-real-bear-claim-20260828.md

The classic single-claim fact-check. A seven-part claim about the real bear behind Winnie-the-Pooh, parsed into assertions, verified against a 1914 war diary entry ("Left Port Arthur, 7am, On train, bought bear, $20"), Veterans Affairs Canada, and the Canadian War Museum. Core verdict TRUE at HIGH confidence, with two precision corrections the pipeline caught: the source image misspelled Winnipeg, and "Winnie-the-Pooh was a real bear" is shorthand worth tightening.

2. Wikipedia uncited-claims report (the utility-website use case)

examples/wiki-uncited-claims-fact-checking-article-2026-10-15.md

The output shape for a tool that takes a Wikipedia article, finds uncited claims, and researches them for a human editor. Run against the live Wikipedia article "Fact-checking": three uncited claims extracted (one explicitly tagged {{citation needed}}, two unreferenced paragraphs), each receiving a verdict, verbatim supporting text, suggested reference markup where sourcing succeeds, and an explicit statement of the human decision that remains. Includes the WP:ReliableSources judgment-call case (an authentic quote whose only source is the subject's own article in a denominational magazine) surfaced with options rather than silently resolved, and a rejected source class (student blogs) documented.

3. The "august event" chronology (Hardy, "The Convergence of the Twain")

examples/convergence-of-the-twain-august-chronology-2026-10-15.md

A literary-historical claim decided by chronology: a hypothesis that the lowercase "august" in stanza X nods to the Allan liner Corsican, which struck an iceberg in August 1912. The pipeline built a dated timeline (manuscript April 24, 1912; first publication May 14, 1912; Corsican iceberg August 12, 1912), showing the poem predates the incident by almost four months, then ran a proper counter-evidence search against the critical literature and reported the absence of any contrary reading. Demonstrates chronology-as-verification, verbatim quoting from 1912 period sources, and clear separation of fact (TRUE/HIGH) from interpretation (LIKELY TRUE, typographical evidence).

Pitfalls We Actually Hit

Earned during runs of this pipeline, documented in the skill files:

• Cite the page, not the snippet. A search-result description supports only what it literally says. Extract the page, then cite. Evidence quotes from snippets are the classic verification failure.
• API keys get masked in sandboxed execution. Environment variables containing secrets can arrive at a Python sandbox as ***, producing 401s with a valid key. Call the API from a shell that sources the env file directly, and batch the calls.
• Rate limits triggered by failed auth. Repeated 401s from masked keys can trip a provider's rate limiter (HTTP 429). Wait it out or switch to CLI fallbacks (dig, openssl, curl) rather than hammering.
• Cloudflare 1010 is not a broken site. A 403 with a cf-mitigated: challenge header on a curl probe means normal bot protection. Set a descriptive User-Agent; rely on DNS/SSL checks for Cloudflare-proxied domains instead of HTTP probes.
• Registering sources after writing the draft. The citation ledger must be populated from tool output as retrieval happens. Reconstructing it from the finished draft reintroduces exactly the hallucinated-URL risk it exists to remove.
• Never hand-edit citation ids. Ids are ledger identities. Renumbering mid-task silently disconnects claims from their sources.
• Server-side filters return empty on stale keys. When a filter parameter yields zero results, retry without the filter before concluding there is no data. Fetching everything and filtering locally is sometimes both faster and more correct.

Repo Layout

claim-verification-kit/
├── README.md            this document
├── skills/              complete skill files (the machinery, verbatim)
├── snippets/            copy-paste prompt fragments
├── examples/            three complete real reports
├── scripts/             citation ledger (sources.py, _hermes_home.py)
└── LICENSE              MIT

The skill files are used verbatim: they are the actual operating documents agents load. Paths referencing the original Hermes Agent skill directories were preserved where they document provenance.

Setup

• claim-verify, wikidata, wayback, grounded-citations: no API keys. wikidata and wayback are plain HTTP APIs; sources.py is stdlib-only Python 3.
• domain-intel: needs WHOISJSON_API_KEY (https://whoisjson.com, free 1,000 requests/month). CLI fallbacks documented in the skill cover most checks without a key.
• factcheck-kanban: needs the Hermes Agent Kanban system (https://hermes-agent.nousresearch.com/docs) with a worker profile.
• Web search: any search backend works; the skills were run against Brave Search.

No keys, tokens, or credentials are stored in this repo.

License

MIT. See LICENSE.