# How to Fix Incomplete References in Your Reference Manager By the end of this guide you will be able to audit an entire reference library for missing fields, repair records with DOI and PubMed lookups, pull metadata out of PDFs, undo encoding damage from RIS imports, merge duplicates without losing data, and verify that every citation renders correctly in your target style. The walkthrough covers Zotero, Mendeley Reference Manager, and EndNote, the three managers most life science students and labs use. To follow along you need one of those installed, a plain text editor, and Python 3 if you want the scripted audit. Keep a fresh export of your library on hand, along with the DOIs or PMIDs of the papers you cite most. A reference manager (also called a citation manager or, in software catalogs, reference management software) stores bibliographic records, attaches PDFs, and inserts formatted citations into Word, Google Docs, or LaTeX. Every citation it produces is only as complete as the record behind it, and reference data quality is a measurable scientific variable. A 2025 study in mSystems built new nucleotide databases for metagenomic classification and showed that explicit quality control on reference data, including reference decontamination and filtering, dramatically reduced spurious Plasmodium annotations when published metagenomic datasets were reanalyzed [1]. A bibliography is reference data too. A missing volume number or a truncated author list is the same class of defect at a smaller scale, and it compounds across every document you write. ## The Six-Step Workflow to Fix Incomplete References in Your Reference Manager ![Six numbered steps in order: audit the library, repair with identifiers, repair from PDFs, fix import damage, merge duplicates, verify every repair.](/article-images/illustrations/fix-incomplete-references-reference-manager-six-step-workflow.webp) Here is the step-by-step version. Each step has its own section below. 1. Audit the library and list every record with a missing field. 2. Repair records with DOI, PMID, and ISBN lookups. 3. Repair PDF-only records with automatic metadata retrieval. 4. Fix import mapping and encoding damage in RIS and BibTeX files. 5. Merge duplicates without losing data. 6. Verify every repair before you cite. ## What an Incomplete Reference Is and Where It Comes From A record is one bibliographic entry in your library. A reference is incomplete when any field a citation style needs is missing, empty, or wrong. For a journal article the critical fields are authors, year, article title, journal name (the container), volume, issue, page range, and DOI. For a book chapter the container is the book title, plus editors, publisher, and page range. Item type matters as much as fields. A record typed as a journal article when it is really a preprint or a thesis will render wrong even when every field is filled. Incompleteness has a short list of causes, and knowing the cause tells you the fix: - **PDF-first imports.** Dragging in a PDF without retrievable metadata creates a record whose title is a filename. - **Thin exports.** Conference programs, theses, government reports, and some preprint servers publish records with no DOI, no pages, or partial author lists. - **Format conversion gaps.** RIS and BibTeX map fields differently across tools, so a journal name stored in one tag can land in the wrong field or vanish. - **Stale lookups.** Managers that auto-fill records pull whatever the upstream service returns at that moment, staleness included. - **Asynchronous updates.** Public resources drift out of sync with each other. The mSystems team found that asynchronous updates between public sequence and taxonomy databases create inconsistencies in taxonomic assignments, and they designed their pipeline to minimize that drift [1]. The same dynamic leaves a citation record carrying an old journal name after a rename. - **Merge and sync damage.** Merging two versions of a record can drop fields that existed in only one copy, and sync conflicts can resurrect stale versions. The consequences are visible in your documents. In APA styles a missing date prints as n.d. A missing container-title leaves a blank where the journal name belongs. A record holding only the first three authors will print et al. after three names no matter what the style file says, because a style can only use the data it has. Journals check reference lists at submission, and incomplete entries come back as technical corrections that cost days. ## Before You Start: Back Up Your Library Bulk repairs touch many records at once, so create the restore point first. - **Zotero.** File → Export Library, choose CSL JSON, tick the options to include notes and files, and set the encoding to UTF-8. For the most faithful backup, close Zotero and copy the whole data folder, which you can locate with the Show Data Directory button under Settings → Advanced → Files and Folders. - **Mendeley Reference Manager.** Your library syncs to your Mendeley account, and the export option in the File menu saves a local BibTeX or RIS copy. - **EndNote.** File → Compressed Library (.enlx) writes a single archive containing the library and its attachments. Store the export outside the library folder with the date in the filename. If a bulk edit goes wrong, you can delete the damaged records and re-import. ## Step 1: Audit Your Library for Incomplete Records Three methods, fastest first. ### Method A: Built-in searches and sorting In Zotero, right-click My Library in the left pane and choose New Saved Search. Name it Missing DOI. Add a condition with the field set to DOI, the operator set to is, and the value left blank, which matches records where the field has no value. Build the same search for publicationTitle and date for journal items. If your Zotero build does not match blank values, use the JavaScript console in Method D. In Mendeley Reference Manager, look for the Needs review filter, which collects documents whose details are incomplete. In EndNote, click a column header such as Year or Journal to sort, which groups empty cells together, and handle overlaps with Library → Find Duplicates. ### Method B: Export to CSL JSON and run a script ![Worked calculation showing 214 records minus 37 missing DOIs leaves 177, which is 82.7 percent complete.](/article-images/illustrations/fix-incomplete-references-reference-manager-completeness-percentage.webp) Export your library as CSL JSON (Zotero: File → Export Library, format CSL JSON) and save it as library.json next to this script: ```python # audit_library.py # Usage: python audit_library.py library.json import json import sys with open(sys.argv[1], encoding="utf-8") as fh: lib = json.load(fh) checks = { "DOI": lambda it: it.get("DOI"), "container-title": lambda it: it.get("container-title"), "issued (year)": lambda it: it.get("issued", {}).get("date-parts"), "author": lambda it: it.get("author"), } total = len(lib) print(f"Records: {total}") for label, has_value in checks.items(): bad = [it for it in lib if not has_value(it)] pct = 100 * (total - len(bad)) / total if total else 0 print(f"{label}: {len(bad)} missing, {pct:.1f}% complete") for it in bad[:5]: print(" -", (it.get("title") or "UNTITLED")[:70]) ``` Expected output for a 214-record library: ``` Records: 214 DOI: 37 missing, 82.7% complete container-title: 12 missing, 94.4% complete issued (year): 4 missing, 98.1% complete author: 2 missing, 99.1% complete ``` The percentages come from a simple calculation. With 214 records and 37 missing DOIs, 177 records carry a DOI, and 100 × 177 ÷ 214 = 82.7 percent. The checks dictionary defines what counts as missing, so add volume or page checks by extending it. CSL JSON stores the journal or book title under container-title and the DOI under an uppercase DOI key. Fix the largest missing bucket first. Two errors you may hit at this stage: - `FileNotFoundError: [Errno 2] No such file or directory: 'library.json'` means the path is wrong. Run the script from the folder holding the export, or pass the full path. - `json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)` means the file is empty or you exported BibTeX or RIS instead of CSL JSON. Re-export in the right format. ### Method C: BibTeX users, read the compile log BibTeX styles check required fields at compile time and write one warning per empty field into the .blg log file: ``` grep "Warning" paper.blg ``` Each warning names the empty field and the entry key, for example an empty journal warning for the entry smith2021gut. If the compiled PDF shows `Citation \`smith2021gut' on page 4 undefined`, the key is misspelled or the entry is missing entirely. Those warnings and undefined citations are your repair list. ### Method D: The Zotero JavaScript console Open Tools → Developer → Run JavaScript, tick Run as async function, and paste: ```javascript var items = await Zotero.Items.getAll(Zotero.Libraries.userLibraryID) var counts = { DOI: 0, journal: 0, date: 0 } for (let item of items) { if (!item.isRegularItem()) continue if (!item.getField('DOI')) counts.DOI++ if (!item.getField('publicationTitle')) counts.journal++ if (!item.getField('date')) counts.date++ } var msg = 'DOI missing: ' + counts.DOI msg += ', journal missing: ' + counts.journal msg += ', date missing: ' + counts.date return msg ``` Expected output: ``` DOI missing: 37, journal missing: 12, date missing: 4 ``` The isRegularItem() check filters out attachments and notes so the counts describe citable records. Change the field names inside getField() to audit anything else, such as pages or volume. ## Step 2: Repair Records With DOI and PMID Lookups An identifier is the fastest repair key. A DOI (digital object identifier) is a permanent string that resolves through doi.org, and a PMID is the numeric identifier PubMed assigns to a biomedical record. One identifier pulls the complete publisher or NLM record into your manager. - **Zotero.** Click the Add Item by Identifier button (the magic wand) in the toolbar, paste a DOI, PMID, ISBN, or arXiv ID, and press Enter. Zotero creates a fully populated item. - **Mendeley Reference Manager.** Use Add new → Add entry manually and trigger the DOI lookup on the entry form, or search the catalog by title and confirm the match. - **EndNote.** With a reference selected, run Find Reference Updates (check the References or Library menu depending on your version) to refresh the record against publisher data. Alternatively switch to Online Search mode, choose PubMed, and search by PMID or keyword to import a complete record. ### Pull records straight from PubMed PubMed hands any manager a complete MEDLINE-format record. On pubmed.ncbi.nlm.nih.gov, tick the checkboxes for your results, open the Send to menu, and choose Citation manager. PubMed downloads a .nbib file that every major manager imports without field loss. A MEDLINE record is plain text with two-letter tags: ``` PMID- 40111052 TI - Addressing the dynamic nature of reference data: a new nucleotide database for robust metagenomic classification. AU - Martí JM AU - Kok CR JT - mSystems DP - 2025 ER - ``` That is the tagged source record for the mSystems nucleotide database paper [1]. TI is the title, each AU line is one author, JT is the journal title, DP is the publication date, and ER marks the end of the record. The DOI travels in the AID tag, so an import from this file fills every field you were missing. ### Fetch metadata with NCBI E-utilities For scripted repairs, NCBI's E-utilities API returns structured metadata. The esummary endpoint takes three parameters: db (the database, here pubmed), id (a comma-separated list of PMIDs), and retmode (the response format, here json): ``` curl -s "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=40111052&retmode=json" ``` Trimmed response for the mSystems paper [1] (volume, issue, pages, and the full DOI omitted for space): ```json { "header": { "type": "esummary", "version": "0.3" }, "result": { "uids": ["40111052"], "40111052": { "uid": "40111052", "pubdate": "2025", "source": "mSystems", "fulljournalname": "mSystems", "title": "Addressing the dynamic nature of reference data: a new nucleotide database for robust metagenomic classification.", "authors": [ { "name": "Martí JM", "authtype": "Author" }, { "name": "Kok CR", "authtype": "Author" } ], "articleids": [ { "idtype": "pubmed", "value": "40111052" }, { "idtype": "doi", "value": "10.1128/..." } ] } } } ``` The full record also carries volume, issue, pages, and the complete author list. Copy each value into the matching field of your broken record and the repair is done. ### Fetch metadata from the DOI system For non-biomedical or very recent records, query the DOI registration data directly. The Crossref REST API returns the full metadata record for any registered DOI: ``` curl -s "https://api.crossref.org/works/" | python -m json.tool ``` Replace `` with the DOI string from your record. The response includes title, container-title, author, issued, volume, issue, and page. If you prefer BibTeX, content negotiation returns a formatted entry in one line: ``` curl -LH "Accept: application/x-bibtex" "https://doi.org/" ``` Paste the result into a .bib file or import it into your manager. ## Step 3: Repair Records From PDFs PDF-first workflows create the worst records, and they are also the easiest to fix in bulk. In Zotero, select one or more PDFs in your library, right-click, and choose Retrieve Metadata for PDF. Zotero runs the files through GROBID, a machine-learning PDF parser, extracts identifiers such as the DOI, and queries them online. Successful PDFs convert from bare attachments into full items with authors, journal, year, and DOI. When Zotero reports that no metadata could be found, which happens with scanned or non-English PDFs, open the first page, copy the DOI, use Add Item by Identifier, and drag the PDF onto the new item to attach it. In Mendeley Reference Manager, drag PDFs into the library and Mendeley runs an automatic details lookup that fills the Details pane from its catalog. Records it cannot match stay thin, so review them the same way. In EndNote, use File → Import → File or Folder with the PDF import option to create records from a batch of PDFs, then select the new records and run Find Reference Updates to fill the gaps from the publisher record. After any retrieval, confirm the DOI resolves. Paste `https://doi.org/` followed by the DOI into a browser. A working DOI lands on the article page. If it lands somewhere else, the record carries the wrong DOI, which is worse than no DOI because every citation inherits the error. ## Step 4: Fix Import Mapping and Encoding Problems Many incomplete libraries are import casualties. Two file formats carry most bibliographic exchanges, and both have failure modes you can fix at the file level. ### RIS field tags RIS is the tagged format most databases export. These tags decide completeness: ``` TY - record type (JOUR journal article, CHAP book chapter, RPRT report) AU - one author per AU line TI - title T2 - secondary title (the journal name for articles in most mappings) JO - journal name (alternate tag some exporters use) VL - volume IS - issue SP - start page EP - end page PY - publication year DO - DOI UR - URL ER - end of record ``` A common failure: an exporter writes the journal name into T2 while your importer reads only JO, so every imported article loses its journal. Open the RIS file in a text editor, check which tag actually holds the journal name, and either correct the tag or import into a tool whose mapping matches the file. Delete the damaged batch before re-importing so you do not create duplicates. ### BibTeX entry anatomy BibTeX is the LaTeX bibliography format. A record built from the PubMed data above looks like this: ```bibtex @article{marti2025nucleotide, author = {Martí, J.M. and Kok, C.R. and Thissen, J.B. and others}, title = {Addressing the dynamic nature of reference data: a new nucleotide database for robust metagenomic classification}, journal = {mSystems}, year = {2025} } ``` This entry is still incomplete. BibTeX styles need volume, number, pages, and doi for a clean journal citation, so copy them from the publisher page or the esummary output in Step 2. BibTeX splits author lists on the word and and expects Last, First order. Corporate authors need double braces so the parser treats the whole name as one unit: `author = {{World Health Organization}}`. ### Encoding damage Accented names are where import damage shows first. The author name Martí stored as UTF-8 but read as Windows-1252 displays as Martí. If a Python read of your exported file throws this error: ``` UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 421: invalid start byte ``` the file is not UTF-8. Byte 0x92 is the Windows-1252 right single quotation mark, common in titles copied from word processors. Convert the file and re-import: ``` iconv -f cp1252 -t utf-8 refs.ris > refs-utf8.ris ``` The -f flag is the source encoding and -t is the target. In Python, open the file with `encoding="cp1252"` instead. After conversion, spot-check one accented name and one title with an apostrophe before re-importing. Mojibake (garbled text produced by an encoding mismatch) that has already entered your library needs record-by-record correction, which is the main reason to fix encoding at the file stage. ## Step 5: Merge Duplicates Without Losing Data Duplicates are incomplete records in disguise. Each copy holds part of the truth, citations split across copies, and styles miscount. Merge first, then repair the survivor, because repairs applied to one copy vanish when another copy wins a later merge. - **Zotero.** Open Duplicate Items in the left pane, select a group, and click Merge Items. Zotero merges fields from the copies and keeps attachments and notes. Review the survivor, because a field blank in the base copy can stay blank. - **Mendeley Reference Manager.** Open the Duplicates pane, review each proposed pair and the details each copy holds, then confirm the merge. - **EndNote.** Library → Find Duplicates, then apply the Merge References command (menu location varies by version) and check the surviving record field by field. ## Step 6: Verify Every Repair Verification is a pipeline stage, not an afterthought. The mSystems team treats validation as an explicit stage with its own quality controls rather than assuming a database is clean because it is large [1]. Apply the same standard with four checks. 1. **Re-run the audit.** The script from Step 1 should report 0 missing for every field you targeted and 100.0% complete. If the numbers did not move, the repairs landed in a different library or in a duplicate copy. 2. **Spot-check against the source.** For every tenth record, compare authors, year, journal, volume, pages, and DOI against the PubMed record or the publisher page. Watch author order, which retrieval tools occasionally scramble. While you are there, glance at the PubMed record for retraction or correction notices that an incomplete record can hide. 3. **Resolve every DOI.** Each DOI in the repaired set should land on the correct article through doi.org. 4. **Render a test bibliography.** Insert three repaired citations into a scratch document in your target style and read the bibliography line by line. A record that passes the audit but prints wrong in the style still needs work. Sensible next steps: import with identifiers first and add PDFs second, keep one canonical import path per source, and schedule a full audit before every submission so incompleteness never accumulates silently. ## Citation Manager Comparison for Repair Work | Manager | Identifier lookup | PDF metadata retrieval | Duplicate cleanup | Repair extras | |--|--|--|--|--| | Zotero | Add Item by Identifier accepts DOI, PMID, ISBN, and arXiv ID | Retrieve Metadata for PDF, powered by a machine-learning parser | Duplicate Items pane with one-click merge | Free and open source, CSL styles, scriptable audit console | | Mendeley Reference Manager | DOI lookup on the manual entry form | Automatic details lookup when PDFs are added | Duplicates pane with confirm-and-merge dialog | Free with a Mendeley account, Needs review filter, BibTeX export | | EndNote | Online Search of PubMed by PMID or keyword | PDF import with automatic metadata capture | Find Duplicates plus Merge References | Paid license, Find Reference Updates, deep Word field-code integration | All three managers complete the repair loop. Zotero is the most scriptable and costs nothing, which makes it the default for students building a library from PDFs. Mendeley Reference Manager suits researchers who want automatic PDF matching with minimal setup. EndNote suits clinical and large-institution workflows built around Word field codes and publisher-record refreshes. The repair features differ less than the audit features, which is why the scripted audit in Step 1 matters regardless of your choice. ## Common Mistakes and Limitations - **Trusting auto-populated metadata.** Lookups return whatever the upstream service holds at that moment, and upstream services age. The Centrifuge metagenomic classifier had no current nucleotide index for years, and the last public version dated to 2018, so anyone relying on it worked with stale reference data until the 2025 rebuild [1]. A record auto-filled years ago can carry the same staleness. - **Editing the bibliography text instead of the record.** In Word, citations are field codes tied to the manager. Direct edits vanish on the next refresh. Fix the record, then refresh. - **Fixing one citation instead of the record.** A repaired record corrects every citation that draws on it. A repaired sentence fixes one. - **Merging without review.** A merge can drop fields that existed only in the discarded copy. Check the survivor. - **Assuming a DOI proves correctness.** A wrong DOI attached at import propagates into every citation. Resolve it before you trust it. - **Mixing preprint and published versions.** Cite the version you read. If you switch to the version of record, update the record and the item type together. - **Wrong item type.** A book chapter typed as a journal article has nowhere to store the book title, so the chapter title prints alone. - **Bulk edits without a backup.** The export from Before You Start is your undo button. - **Skipping the text-layer check.** Retrieval tools cannot read scanned PDFs without a text layer, and they struggle with theses, abstracts-only records, and non-English sources. Those need manual completion. Automated retrieval only reaches records that exist upstream. Older literature, conference abstracts, government reports, and non-English publications often have no DOI, so identifier lookups return nothing and manual completion is the only route. For a stubborn record, a subject librarian or the software's official documentation is the next stop. The maintenance burden never reaches zero, even for large curated resources. The [NCBI nucleotide database](/blog/guides/ncbi-nucleotide-database) referenced by the mSystems study exceeds 10¹² nucleotides (over a trillion) and grows exponentially, which is why the team built and maintains their own Centrifuge-compatible indices on high-performance computing resources [1]. A personal library of a few thousand records is far easier to maintain, but the principle is identical. Schedule the audit, or the incompleteness returns. ## Frequently Asked Questions ### What counts as an incomplete reference? A reference is incomplete when any field a citation style needs is missing, empty, or wrong: authors, year, title, journal or book title, volume, issue, pages, DOI, or item type. A missing date is the easiest to spot, because APA styles print n.d. in place of the year. ### How do I find every incomplete reference at once? Export the library to CSL JSON and run a short audit script that counts records missing a DOI, container title, year, or author list, or build a saved search in Zotero that matches a blank field. The audit reports a missing count and a completeness percentage per field so you can repair the biggest gap first. ### Why does my citation show n.d. or a blank journal name? The record behind the citation is missing the date or the container-title field, so the style has nothing to print. Correct the record in the reference manager and refresh, and the citation and bibliography update everywhere that record is used. ### Can I fix incomplete references automatically? Identifier lookups and PDF metadata retrieval repair most modern journal records automatically once you supply a DOI or PMID. Scanned PDFs, conference abstracts, theses, and non-English sources usually still need manual completion. ### How do I fix garbled characters such as Martí after importing RIS? The file was exported in one encoding and read in another. Re-save the RIS file as UTF-8 in a text editor or convert it with iconv, then delete the damaged records and re-import. ### Should I cite the preprint DOI or the journal DOI? Cite the version you actually read, and keep the item type set to preprint or journal article so the style renders it correctly. When you switch to the published version, update the record to the version of record. ### Which reference manager is easiest for repairing records? Zotero is the most repair-friendly of the three, because it combines identifier lookup, PDF metadata retrieval, duplicate merging, and a scriptable audit console, and it is free and open source. Mendeley and EndNote cover the same core repairs, and EndNote adds Find Reference Updates for refreshing records against publisher data. ### How often should I audit my library? Run a full audit before every thesis, dissertation, or manuscript submission, and after any large PDF import. A quick saved-search check after each import session keeps the backlog from building up. ## Sources 1. [Addressing the dynamic nature of reference data: a new nucleotide database for robust metagenomic classification.](https://pubmed.ncbi.nlm.nih.gov/40111052/)