Checks whether the references in a bibliography actually exist, against OpenAlex, Crossref, arXiv and five more indexes.
citeverify reads a reference list and looks up each entry in eight free bibliographic indexes (large online catalogues of published papers, books and datasets). For every reference it reports whether a matching record exists, and flags the ones it could not find or that disagree on the year or authors.
It exists because made-up references take seconds to produce and are hard to spot by eye. A reference copied wrongly, half-remembered, or invented by an AI chatbot can look exactly like a real one. citeverify gives you a list of the entries worth checking by hand.
It reads BibTeX, RIS, CSL-JSON, LaTeX .bbl, Word .docx, PDF and plain text. It covers journals, preprints, books, computer science conference proceedings, biomedical literature and datasets.
If you are writing an EPQ or any essay with a reference list, you can point citeverify at your Word document. It finds the heading "References", "Bibliography" or "Works Cited", reads the list below it, and checks each entry. A NOT FOUND result does not prove a source is fake. It tells you which entries to look at again before you hand in.
- How academic sources are catalogued, and what a DOI (Digital Object Identifier, a permanent ID for a published work) is for (Research skills, EPQ).
- How a program can query a web API (an address that returns data for programs to read) and combine answers from several services (Computer Science).
- How to measure how similar two titles are with Jaccard overlap: shared words divided by total distinct words (Computer Science, Mathematics).
- How to choose thresholds that balance false alarms against missed problems (Computer Science, Statistics).
- How automated tests check code without touching the network (Computer Science).
You will need:
- Python 3.9 or later.
- An internet connection, because the checks query online indexes.
Tested on macOS with Python 3. The project's automatic tests run on Ubuntu with Python 3.9, 3.11 and 3.13.
-
Open a terminal (Terminal on macOS, PowerShell on Windows).
-
Install the latest version from GitHub:
pip install https://github.com/godofecht/citeverify/archive/refs/heads/main.zip
The version on PyPI, installed with
pip install citeverify, is older (0.1.0). It reads plain text only and checks three indexes. The commands below need 0.2.0 from GitHub. -
Check it installed:
citeverify --version
You should see
citeverify 0.2.0. -
Check your essay. Replace
essay.docxwith the name of your file:citeverify check essay.docx
When it works you see a summary like this one, from a test essay with two real references and one invented one:
format: docx references: 3
Checked 3 references
----------------------------------------------------
verified 2 ( 66.7%)
NOT FOUND 1 ( 33.3%)
1 reference(s) need attention:
[NOT FOUND] Smith, J. (2019). Quantum entanglement in recurrent transformer memory lattices for sixth form biolo
closest: Deep learning, pro-and con, mathematics vs physics, spiritualists vs physicists (2023)
| Result | What it means | What to do |
|---|---|---|
verified |
A matching record was found | Nothing |
year mismatch |
The title matches, the year does not | Check the year. Old books often match a later edition |
author mismatch |
The title matches, no author surnames do | Check the author names |
partial |
Something similar was found | Check the title is copied exactly |
NOT FOUND |
Nothing close was found | Find the source again by hand. If you cannot, remove it |
error |
Every index failed to answer | Check your internet connection and try again |
citeverify check refs.txt # plain text, one reference per line
citeverify check refs.bib # BibTeX
citeverify check library.ris # Zotero / EndNote / Mendeley export
citeverify parse refs.txt # show how the references were read, no network callsIf you keep your sources in Zotero or Mendeley, export them as .bib or .ris and check that file. Those formats give citeverify exact fields, so the results are more reliable than reading free text.
- A reader turns your file into a list of references. Each reference has a title, authors, a year and, where present, an identifier such as a DOI.
- For each reference, citeverify picks an order for the indexes. A book goes to Open Library first, a reference with a PubMed ID goes to Europe PMC first, and a conference paper goes to DBLP first.
- It searches each index in turn and scores every result by how many title words it shares with your reference.
- It stops at the first strong match, then compares year and authors to give the final result.
| File or folder | What is in it |
|---|---|
citeverify/cli.py |
The citeverify command and its options |
citeverify/readers.py |
Readers for each file format, including Word and PDF |
citeverify/parse.py |
Splits free text into references and finds titles, years and identifiers |
citeverify/normalize.py |
Cleans up titles and names before comparing them |
citeverify/match.py |
Title scoring and the rules for each result |
citeverify/sources.py |
One class per index, and the order they are tried in |
citeverify/verify.py |
Runs the search across indexes and builds the results |
tests/ |
Automated tests, run without network access |
- Run
citeverify parseon a reference list and compare its idea of each title with yours. Entries withconfidence: loware the ones most likely to be misread. - Make the checks stricter or looser with
--verified-thresholdand--partial-threshold, and see how the results change. The defaults are set inciteverify/cli.py. - Open
citeverify/normalize.pyand readSTOPWORDS, the list of common words dropped from titles before scoring. Add a word and see whetherpyteststill passes. - Write a test. Add a case to
tests/test_match.pyfor two titles you think should or should not match, and runpytest. - Add a new index. Each index is a class in
citeverify/sources.py. ReadOpenLibrarySourceand write one for another free catalogue.CONTRIBUTING.mdsays a new index adapter is welcome.
Cross-checks the references in a bibliography against eight free bibliographic indexes, and reports whether each one resolves to an indexed record.
pip install citeverifyRequires Python 3.9 or later. The core has no third-party dependencies. See "Project status" for the PyPI version gap.
citeverify check refs.bib # BibTeX
citeverify check library.ris # Zotero / EndNote / Mendeley export
citeverify check refs.json # CSL-JSON
citeverify check paper.docx # reads the reference section
citeverify check paper.pdf # needs: pip install 'citeverify[pdf]'
citeverify check refs.txt # plain text, one per line
citeverify check refs.txt --json out.json # structured results
citeverify check refs.txt --cache .cv.json # memoise source responses
citeverify check refs.txt --strict # exit 1 on any suspicious verdict
citeverify parse refs.txt # parse only, no network calls
citeverify check - < refs.txt # read stdin| Flag | Default | Effect |
|---|---|---|
--format |
detected | Force an input format: bibtex, ris, csl-json, latex, docx, pdf, document, text |
--mailto |
unset | Sent as a contact address to OpenAlex and Crossref, which routes the request to their polite pool |
--cache PATH |
unset | JSON memo of source responses, keyed by source and identifier |
--delay |
0.12 |
Seconds between calls that hit the network. Cached lookups do not sleep |
--verified-threshold |
0.60 |
Title score at or above which a candidate can be verified |
--partial-threshold |
0.35 |
Title score at or above which a candidate is partial |
--year-tolerance |
1 |
Permitted absolute difference between cited and indexed year |
--strict |
off | Exit 1 if any reference is suspicious. error verdicts do not trigger it |
--quiet |
off | Suppress progress output |
Exit codes: 0 completed, 1 suspicious references found under --strict, 2 usage or input error.
Detected from the extension, or from the content when reading stdin. Override with --format.
| Format | Extension | Notes |
|---|---|---|
| BibTeX | .bib |
Brace-matched, so nested braces in titles survive. LaTeX accents ({\'E}mile) are folded to Unicode so they match indexed records |
| RIS | .ris |
Zotero, Mendeley, EndNote, Web of Science, Scopus. Soft-wrapped fields are rejoined |
| CSL-JSON | .json |
Zotero and Pandoc. Accepts a bare array or a wrapped export |
| LaTeX | .bbl, .tex |
Splits on \newblock, which delimits author/title/venue explicitly and beats sentence segmentation |
| Word | .docx |
Paragraph boundaries are explicit, so hanging-indent bibliographies survive intact |
.pdf |
Requires pip install 'citeverify[pdf]' |
|
| Document | any | Finds the References heading in a longer text and parses only what follows, stopping at Appendix |
| Plain text | any | One reference per line. The heuristic fallback |
The reference-section finder accepts the headings References, Bibliography, Works Cited, Literature Cited and Reference List, optionally numbered. If several appear, the last one wins. It stops at Appendix, Appendices, Supplementary, Acknowledgements, Author Contributions, About the Author(s) or Index.
Structured formats set parse_confidence to high because field boundaries are given rather than inferred. That matters: a mis-segmented title is the main cause of a false unverified.
| Verdict | Condition |
|---|---|
verified |
Title score at or above the verified threshold, year within tolerance, author overlap present or uncheckable |
year_mismatch |
Title score at or above the verified threshold, year outside tolerance |
author_mismatch |
Title score at or above the verified threshold, no surname overlap with the indexed record |
partial |
Title score between the partial and verified thresholds |
unverified |
No candidate above the partial threshold, or no title or identifier to search on |
error |
Every configured source failed to respond |
Result.suspicious is true for year_mismatch, author_mismatch, partial and unverified. It is false for error. The command-line summary prints unverified as NOT FOUND.
Transport failures are separated from negative results. If every source raises, the verdict is error and suspicious is false. Failed lookups are not written to the cache, so they are retried on the next run.
Missing author data does not lower a verdict. author_overlap is tri-state: true, false, or None when either side lists no authors. None is treated as passing.
Candidate selection uses title score alone. Author and year are recorded as corroborating evidence and applied in classify, outside candidate ranking. A reference to a real work carrying an incorrect year therefore resolves to that work and returns year_mismatch rather than unverified.
Title scoring is Jaccard overlap of content tokens. Stopwords and single-character tokens are dropped before comparison.
Containment is a guarded escape hatch. If either normalised title contains the other the score is 1.0, but only when the contained title carries at least 4 content tokens and accounts for at least 70% of the longer one. Without that guard, containment is exactly the shape a fabricated citation takes: an invented title opening with a real, short one. "Quantum entanglement" is contained in "Quantum Entanglement in Recurrent Transformer Memory Lattices", and scoring that 1.0 reports a fabrication as verified. Pairs that fail the guard fall through to Jaccard and are scored on the evidence actually shared.
An exact identifier outranks any title score. When a citation and a candidate carry the same DOI, PMID, arXiv id or ISBN, the match is identity rather than similarity and scores 1.0 regardless of title. A reference that cites by PMID alone, with no title, still verifies.
from citeverify import Cache, parse_references, summarize, verify_all
refs = parse_references(open("references.txt").read())
results = verify_all(refs, cache=Cache(".cv.json"))
summarize(results) # {'verified': 4, 'unverified': 1, ...}
[r for r in results if r.suspicious]verify() and verify_all() accept dicts directly, bypassing the parser:
verify({"title": "...", "authors": "...", "year": 2017, "doi": "10.1/x"})Recognised keys: title, authors, year, doi, arxiv_id, isbn, pmid, entry_type, venue. authors accepts a string or a list. entry_type (book, inproceedings, dataset, ...) steers source ordering.
from citeverify import Thresholds, verify_all
verify_all(refs, thresholds=Thresholds(verified=0.7, partial=0.4, year_tolerance=0))Thresholds validates 0 <= partial <= verified <= 1 and year_tolerance >= 0 at construction.
All network access goes through one callable with signature (url, headers) -> str. No other module performs I/O.
from citeverify import OpenAlexSource, verify
def fetcher(url, headers):
return proxy.get(url, headers=headers).text
verify(citation, sources=[OpenAlexSource(fetcher=fetcher)])Sources are tried in order and the search stops at the first verified result. Each source exposes last_error after a failed call.
default_sources() returns all eight, none of which require an API key:
| Source | Covers |
|---|---|
| OpenAlex | Broadest general index, queried first |
| Crossref | DOI-registered publisher records |
| arXiv | Preprints |
| Europe PMC | Life sciences and everything in PubMed; the only source that resolves a bare PMID |
| Semantic Scholar | Computer science, and preprints that never acquired a DOI |
| DBLP | CS conference proceedings, patchily covered by DOI-centric indexes |
| Open Library | Books, which have no DOI and are the largest blind spot in DOI-based checking |
| DataCite | Datasets, software and theses |
sources_for(citation) reorders that list by what the reference looks like, so a book reaches Open Library first rather than after seven journal databases have each spent a request failing to find it. Since verification stops at the first confirmation, this turns eight requests into one. verify() applies it automatically.
parse_references splits on newlines, rejoins soft-wrapped entries, and strips list markers ([1], 1., (1)). Each entry yields a Citation with parse_confidence set to:
high: title found via a quoted string, or via an author/parenthesised-year/title splitlow: title inferred by sentence segmentation. Recorded innotes
DOIs match 10.\d{4,9}/.... arXiv ids match both the NNNN.NNNNN and the older archive/NNNNNNN forms. A parenthesised year is preferred over a bare four-digit number, which may be a volume or page range.
- Title matching is lexical and does not model meaning.
- Coverage is bounded by the eight indexes. Standards, grey literature, and much non-English and pre-1970 work remain thinly indexed, so
unverifiedon those often reflects absence from the index rather than fabrication. - The free-text parser is heuristic. Structured input (
.bib,.ris, CSL-JSON) is read exactly and should be preferred when available; it is also what setsparse_confidencetohigh. - PDF extraction is only as good as the PDF. Two-column layouts and ligature-heavy fonts still produce broken reference strings. Export a
.bibwhere you can. - The tool determines whether a matching record exists. It does not evaluate whether a citation supports the claim it is attached to.
pip install -e ".[dev]"
pytestNo network access or API key required; sources are exercised through injected fetchers. See CONTRIBUTING.md before opening a pull request.
The test suite (115 tests at the last run) passes on Python 3.9, 3.11 and 3.13 in the project's automatic checks.
- The latest release on PyPI is 0.1.0. The code on GitHub is 0.2.0, which adds the file format readers and five of the eight indexes. Until 0.2.0 is published, install from GitHub as shown in "Run it".
index.htmlin the repository loads scripts from/assets/paths that do not exist in this repository, so parts of it will not work when opened on its own.
You can use, copy and change this code for any purpose, as long as you keep the copyright notice. See LICENSE (MIT).
Written by Abhishek Shivakumar. citeverify queries these free services: OpenAlex, Crossref, arXiv, Europe PMC, Semantic Scholar, DBLP, Open Library and DataCite. PDF reading uses pypdf. To cite this work, see CITATION.cff.