Skip to content

About

Checks whether the references in a bibliography actually exist, against OpenAlex, Crossref, arXiv and five more indexes.

Topics

Resources

Contributing

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

7 Commits

Folders and files

Repository files navigation

citeverify

Checks whether the references in a bibliography actually exist, against OpenAlex, Crossref, arXiv and five more indexes.

What this is

citeverify reads a reference list and looks up each entry in eight free bibliographic indexes (large online catalogues of published papers, books and datasets). For every reference it reports whether a matching record exists, and flags the ones it could not find or that disagree on the year or authors.

It exists because made-up references take seconds to produce and are hard to spot by eye. A reference copied wrongly, half-remembered, or invented by an AI chatbot can look exactly like a real one. citeverify gives you a list of the entries worth checking by hand.

It reads BibTeX, RIS, CSL-JSON, LaTeX .bbl, Word .docx, PDF and plain text. It covers journals, preprints, books, computer science conference proceedings, biomedical literature and datasets.

For the Extended Project Qualification (EPQ)

If you are writing an EPQ or any essay with a reference list, you can point citeverify at your Word document. It finds the heading "References", "Bibliography" or "Works Cited", reads the list below it, and checks each entry. A NOT FOUND result does not prove a source is fake. It tells you which entries to look at again before you hand in.

What you can learn from it

  • How academic sources are catalogued, and what a DOI (Digital Object Identifier, a permanent ID for a published work) is for (Research skills, EPQ).
  • How a program can query a web API (an address that returns data for programs to read) and combine answers from several services (Computer Science).
  • How to measure how similar two titles are with Jaccard overlap: shared words divided by total distinct words (Computer Science, Mathematics).
  • How to choose thresholds that balance false alarms against missed problems (Computer Science, Statistics).
  • How automated tests check code without touching the network (Computer Science).

Run it

On your own computer

You will need:

  • Python 3.9 or later.
  • An internet connection, because the checks query online indexes.

Tested on macOS with Python 3. The project's automatic tests run on Ubuntu with Python 3.9, 3.11 and 3.13.

  1. Open a terminal (Terminal on macOS, PowerShell on Windows).

  2. Install the latest version from GitHub:

    pip install https://github.com/godofecht/citeverify/archive/refs/heads/main.zip

    The version on PyPI, installed with pip install citeverify, is older (0.1.0). It reads plain text only and checks three indexes. The commands below need 0.2.0 from GitHub.

  3. Check it installed:

    citeverify --version

    You should see citeverify 0.2.0.

  4. Check your essay. Replace essay.docx with the name of your file:

    citeverify check essay.docx

When it works you see a summary like this one, from a test essay with two real references and one invented one:

format: docx   references: 3
Checked 3 references
----------------------------------------------------
  verified              2   ( 66.7%)
  NOT FOUND             1   ( 33.3%)

1 reference(s) need attention:

  [NOT FOUND] Smith, J. (2019). Quantum entanglement in recurrent transformer memory lattices for sixth form biolo
      closest: Deep learning, pro-and con, mathematics vs physics, spiritualists vs physicists (2023)

Reading the results

Result What it means What to do
verified A matching record was found Nothing
year mismatch The title matches, the year does not Check the year. Old books often match a later edition
author mismatch The title matches, no author surnames do Check the author names
partial Something similar was found Check the title is copied exactly
NOT FOUND Nothing close was found Find the source again by hand. If you cannot, remove it
error Every index failed to answer Check your internet connection and try again

Other ways to use it

citeverify check refs.txt                      # plain text, one reference per line
citeverify check refs.bib                      # BibTeX
citeverify check library.ris                   # Zotero / EndNote / Mendeley export
citeverify parse refs.txt                      # show how the references were read, no network calls

If you keep your sources in Zotero or Mendeley, export them as .bib or .ris and check that file. Those formats give citeverify exact fields, so the results are more reliable than reading free text.

How it works

  1. A reader turns your file into a list of references. Each reference has a title, authors, a year and, where present, an identifier such as a DOI.
  2. For each reference, citeverify picks an order for the indexes. A book goes to Open Library first, a reference with a PubMed ID goes to Europe PMC first, and a conference paper goes to DBLP first.
  3. It searches each index in turn and scores every result by how many title words it shares with your reference.
  4. It stops at the first strong match, then compares year and authors to give the final result.
File or folder What is in it
citeverify/cli.py The citeverify command and its options
citeverify/readers.py Readers for each file format, including Word and PDF
citeverify/parse.py Splits free text into references and finds titles, years and identifiers
citeverify/normalize.py Cleans up titles and names before comparing them
citeverify/match.py Title scoring and the rules for each result
citeverify/sources.py One class per index, and the order they are tried in
citeverify/verify.py Runs the search across indexes and builds the results
tests/ Automated tests, run without network access

Ideas to try

  1. Run citeverify parse on a reference list and compare its idea of each title with yours. Entries with confidence: low are the ones most likely to be misread.
  2. Make the checks stricter or looser with --verified-threshold and --partial-threshold, and see how the results change. The defaults are set in citeverify/cli.py.
  3. Open citeverify/normalize.py and read STOPWORDS, the list of common words dropped from titles before scoring. Add a word and see whether pytest still passes.
  4. Write a test. Add a case to tests/test_match.py for two titles you think should or should not match, and run pytest.
  5. Add a new index. Each index is a class in citeverify/sources.py. Read OpenLibrarySource and write one for another free catalogue. CONTRIBUTING.md says a new index adapter is welcome.

Technical details

Cross-checks the references in a bibliography against eight free bibliographic indexes, and reports whether each one resolves to an indexed record.

Install

pip install citeverify

Requires Python 3.9 or later. The core has no third-party dependencies. See "Project status" for the PyPI version gap.

CLI

citeverify check refs.bib                      # BibTeX
citeverify check library.ris                   # Zotero / EndNote / Mendeley export
citeverify check refs.json                     # CSL-JSON
citeverify check paper.docx                    # reads the reference section
citeverify check paper.pdf                     # needs: pip install 'citeverify[pdf]'
citeverify check refs.txt                      # plain text, one per line
citeverify check refs.txt --json out.json      # structured results
citeverify check refs.txt --cache .cv.json     # memoise source responses
citeverify check refs.txt --strict             # exit 1 on any suspicious verdict
citeverify parse refs.txt                      # parse only, no network calls
citeverify check - < refs.txt                  # read stdin
Flag Default Effect
--format detected Force an input format: bibtex, ris, csl-json, latex, docx, pdf, document, text
--mailto unset Sent as a contact address to OpenAlex and Crossref, which routes the request to their polite pool
--cache PATH unset JSON memo of source responses, keyed by source and identifier
--delay 0.12 Seconds between calls that hit the network. Cached lookups do not sleep
--verified-threshold 0.60 Title score at or above which a candidate can be verified
--partial-threshold 0.35 Title score at or above which a candidate is partial
--year-tolerance 1 Permitted absolute difference between cited and indexed year
--strict off Exit 1 if any reference is suspicious. error verdicts do not trigger it
--quiet off Suppress progress output

Exit codes: 0 completed, 1 suspicious references found under --strict, 2 usage or input error.

Input formats

Detected from the extension, or from the content when reading stdin. Override with --format.

Format Extension Notes
BibTeX .bib Brace-matched, so nested braces in titles survive. LaTeX accents ({\'E}mile) are folded to Unicode so they match indexed records
RIS .ris Zotero, Mendeley, EndNote, Web of Science, Scopus. Soft-wrapped fields are rejoined
CSL-JSON .json Zotero and Pandoc. Accepts a bare array or a wrapped export
LaTeX .bbl, .tex Splits on \newblock, which delimits author/title/venue explicitly and beats sentence segmentation
Word .docx Paragraph boundaries are explicit, so hanging-indent bibliographies survive intact
PDF .pdf Requires pip install 'citeverify[pdf]'
Document any Finds the References heading in a longer text and parses only what follows, stopping at Appendix
Plain text any One reference per line. The heuristic fallback

The reference-section finder accepts the headings References, Bibliography, Works Cited, Literature Cited and Reference List, optionally numbered. If several appear, the last one wins. It stops at Appendix, Appendices, Supplementary, Acknowledgements, Author Contributions, About the Author(s) or Index.

Structured formats set parse_confidence to high because field boundaries are given rather than inferred. That matters: a mis-segmented title is the main cause of a false unverified.

Verdicts

Verdict Condition
verified Title score at or above the verified threshold, year within tolerance, author overlap present or uncheckable
year_mismatch Title score at or above the verified threshold, year outside tolerance
author_mismatch Title score at or above the verified threshold, no surname overlap with the indexed record
partial Title score between the partial and verified thresholds
unverified No candidate above the partial threshold, or no title or identifier to search on
error Every configured source failed to respond

Result.suspicious is true for year_mismatch, author_mismatch, partial and unverified. It is false for error. The command-line summary prints unverified as NOT FOUND.

Resolution semantics

Transport failures are separated from negative results. If every source raises, the verdict is error and suspicious is false. Failed lookups are not written to the cache, so they are retried on the next run.

Missing author data does not lower a verdict. author_overlap is tri-state: true, false, or None when either side lists no authors. None is treated as passing.

Candidate selection uses title score alone. Author and year are recorded as corroborating evidence and applied in classify, outside candidate ranking. A reference to a real work carrying an incorrect year therefore resolves to that work and returns year_mismatch rather than unverified.

Title scoring is Jaccard overlap of content tokens. Stopwords and single-character tokens are dropped before comparison.

Containment is a guarded escape hatch. If either normalised title contains the other the score is 1.0, but only when the contained title carries at least 4 content tokens and accounts for at least 70% of the longer one. Without that guard, containment is exactly the shape a fabricated citation takes: an invented title opening with a real, short one. "Quantum entanglement" is contained in "Quantum Entanglement in Recurrent Transformer Memory Lattices", and scoring that 1.0 reports a fabrication as verified. Pairs that fail the guard fall through to Jaccard and are scored on the evidence actually shared.

An exact identifier outranks any title score. When a citation and a candidate carry the same DOI, PMID, arXiv id or ISBN, the match is identity rather than similarity and scores 1.0 regardless of title. A reference that cites by PMID alone, with no title, still verifies.

Python API

from citeverify import Cache, parse_references, summarize, verify_all

refs = parse_references(open("references.txt").read())
results = verify_all(refs, cache=Cache(".cv.json"))

summarize(results)          # {'verified': 4, 'unverified': 1, ...}
[r for r in results if r.suspicious]

verify() and verify_all() accept dicts directly, bypassing the parser:

verify({"title": "...", "authors": "...", "year": 2017, "doi": "10.1/x"})

Recognised keys: title, authors, year, doi, arxiv_id, isbn, pmid, entry_type, venue. authors accepts a string or a list. entry_type (book, inproceedings, dataset, ...) steers source ordering.

Thresholds

from citeverify import Thresholds, verify_all

verify_all(refs, thresholds=Thresholds(verified=0.7, partial=0.4, year_tolerance=0))

Thresholds validates 0 <= partial <= verified <= 1 and year_tolerance >= 0 at construction.

Transport

All network access goes through one callable with signature (url, headers) -> str. No other module performs I/O.

from citeverify import OpenAlexSource, verify

def fetcher(url, headers):
    return proxy.get(url, headers=headers).text

verify(citation, sources=[OpenAlexSource(fetcher=fetcher)])

Sources are tried in order and the search stops at the first verified result. Each source exposes last_error after a failed call.

default_sources() returns all eight, none of which require an API key:

Source Covers
OpenAlex Broadest general index, queried first
Crossref DOI-registered publisher records
arXiv Preprints
Europe PMC Life sciences and everything in PubMed; the only source that resolves a bare PMID
Semantic Scholar Computer science, and preprints that never acquired a DOI
DBLP CS conference proceedings, patchily covered by DOI-centric indexes
Open Library Books, which have no DOI and are the largest blind spot in DOI-based checking
DataCite Datasets, software and theses

sources_for(citation) reorders that list by what the reference looks like, so a book reaches Open Library first rather than after seven journal databases have each spent a request failing to find it. Since verification stops at the first confirmation, this turns eight requests into one. verify() applies it automatically.

Parser

parse_references splits on newlines, rejoins soft-wrapped entries, and strips list markers ([1], 1., (1)). Each entry yields a Citation with parse_confidence set to:

  • high: title found via a quoted string, or via an author/parenthesised-year/title split
  • low: title inferred by sentence segmentation. Recorded in notes

DOIs match 10.\d{4,9}/.... arXiv ids match both the NNNN.NNNNN and the older archive/NNNNNNN forms. A parenthesised year is preferred over a bare four-digit number, which may be a volume or page range.

Limits

  • Title matching is lexical and does not model meaning.
  • Coverage is bounded by the eight indexes. Standards, grey literature, and much non-English and pre-1970 work remain thinly indexed, so unverified on those often reflects absence from the index rather than fabrication.
  • The free-text parser is heuristic. Structured input (.bib, .ris, CSL-JSON) is read exactly and should be preferred when available; it is also what sets parse_confidence to high.
  • PDF extraction is only as good as the PDF. Two-column layouts and ligature-heavy fonts still produce broken reference strings. Export a .bib where you can.
  • The tool determines whether a matching record exists. It does not evaluate whether a citation supports the claim it is attached to.

Development

pip install -e ".[dev]"
pytest

No network access or API key required; sources are exercised through injected fetchers. See CONTRIBUTING.md before opening a pull request.

Project status

The test suite (115 tests at the last run) passes on Python 3.9, 3.11 and 3.13 in the project's automatic checks.

  • The latest release on PyPI is 0.1.0. The code on GitHub is 0.2.0, which adds the file format readers and five of the eight indexes. Until 0.2.0 is published, install from GitHub as shown in "Run it".
  • index.html in the repository loads scripts from /assets/ paths that do not exist in this repository, so parts of it will not work when opened on its own.

Licence

You can use, copy and change this code for any purpose, as long as you keep the copyright notice. See LICENSE (MIT).

Credits

Written by Abhishek Shivakumar. citeverify queries these free services: OpenAlex, Crossref, arXiv, Europe PMC, Semantic Scholar, DBLP, Open Library and DataCite. PDF reading uses pypdf. To cite this work, see CITATION.cff.

About

Checks whether the references in a bibliography actually exist, against OpenAlex, Crossref, arXiv and five more indexes.

Topics

Resources

Contributing

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages