# How to Conduct a Reference Library Audit By the end of this guide you will be able to run a full reference library audit on your own research library, produce a scored report showing exactly which records are healthy and which are broken, fix the most common defects, and set up a repeatable audit schedule. If you work with DNA barcoding data, you will also learn how to run an automated audit and grading pass on a COI reference library using the BAGS tool. Here is what you need on hand before you start: - A citation manager with your library loaded. Zotero is the free, open-source option used in the walkthrough below. - A spreadsheet application or Python 3 with the pandas and requests packages installed. - A backup location for your exported library file. - If you audit molecular reference libraries, an R installation (version 4.0 or newer) and a GitHub account. ## What a Reference Library Audit Is A reference library audit is a systematic review of the records in a reference collection, done to find and fix errors before those errors propagate into manuscripts, pipelines, or downstream analyses. In clinical and laboratory settings, an audit of any kind is understood as a systematic process used to identify problems and prevent them from recurring [1]. The same discipline applies to reference libraries, in two distinct senses of the term. The first sense is bibliographic. Your reference library is the collection of papers, books, datasets, and standards stored in reference management software such as Zotero, Mendeley, or EndNote. Auditing it means checking every record for duplicate entries, incomplete metadata, broken identifiers, and citations that do not actually exist. The second sense is molecular. In biodiversity research, a reference library is a curated set of reference sequences, most commonly cytochrome c oxidase subunit I (COI) barcode sequences stored in the Barcode of Life Data System (BOLD), used to assign species identities in DNA metabarcoding studies. The accuracy of species-level assignment depends entirely on the quality of that library, and errors introduced during barcode generation can compromise it [2]. Auditing means checking each species entry for data quality and for agreement between the species name and the sequences grouped in its barcode cluster. This article walks through both. The principles, template, and scoring rubric are shared, and the tooling differs. ## What Typically Goes Wrong in a Reference Library Audits exist because reference libraries degrade in predictable ways. These are the defects you should expect to find: 1. **Duplicate records.** Imports from PDFs, web captures, and coauthor libraries create multiple entries for the same paper, often with conflicting metadata. 2. **Incomplete or wrong metadata.** Missing publication years, mangled author lists, and absent DOIs are the most common gaps. 3. **Dead or mistyped identifiers.** DOIs that resolve to the wrong article or fail to resolve at all. 4. **Fabricated or unverifiable references.** This problem has grown sharply with the use of large language models. A scoping review of artificial intelligence in otolaryngology identified hallucination and reference fabrication as a principal challenge domain, with one evaluated study reporting a 61.6 percent rate of references being irrelevant to the prompt that generated them [3]. Any citation produced by a chatbot is unverified until you have opened the source yourself. 5. **Out-of-alignment reference values.** Reference libraries are not only collections of papers. A systematic review of the AUDIT alcohol screening instrument found that cut-off scores in routine use across low- and middle-income countries were frequently not aligned with the World Health Organization recommendations, and that most studies had never run a psychometric evaluation of the values they used [4]. If your work depends on a library of thresholds, reference ranges, or standard values, an audit asks whether each entry still matches its authoritative source. 6. **Sequence-to-name mismatches in molecular libraries.** In barcode reference libraries, a species entry may contain sequences that do not cluster together, which signals that at least one record is mislabeled or misidentified [2]. A good reference library audit format catches all six defect classes with a single pass through the collection. ## The Reference Library Audit Template ![A triage path that asks whether a record exists, whether it is usable, and whether the problem is only cosmetic, ending in Critical, Major or Minor.](/article-images/illustrations/reference-library-audit-severity-triage.webp) A reusable reference library audit template has two parts: a tracking spreadsheet and a severity scheme. The tracking file is a simple CSV with one row per record. Start it with this header row: ```csv record_id,title,first_author,year,doi_or_accession,item_type,source_of_entry,duplicate_flag,identifier_resolves,metadata_complete,verified_against_publisher,severity,action,auditor,date_audited ``` The severity scheme borrows from how clinical pre-audit systems classify triggered problems by level of severity so that the most dangerous issues get fixed first [5]. A three-tier scheme is enough for most libraries: | Tier | Definition | Examples | Required action | |, - |, - |, - |, - | | Critical | The citation points to something that does not exist or is the wrong source | Fabricated DOI, chatbot-generated reference, wrong article attached to a DOI | Fix or delete before the library is used again | | Major | The record is real but unusable or ambiguous | Missing year, duplicate entry, unresolved author list | Fix within the audit cycle | | Minor | Cosmetic or formatting problems | Inconsistent capitalization, missing abstract | Fix opportunistically | Copy this table into the first sheet of your audit workbook. Every flagged record gets exactly one tier and one action, dated, so the next auditor can see what changed and why. ## Walkthrough 1: Auditing a Bibliographic Library in Zotero This is a complete reference library audit example for a personal or lab library of a few hundred to a few thousand records. The steps assume Zotero, but the logic transfers to any citation manager. ### Step 1. Back up and freeze the library Export a full copy before touching anything. In Zotero, go to File, then Export Library, choose the CSV format, and check the box to export notes if you use them. Save the file somewhere with a date-stamped name such as `library_export_2026-11-11.csv`. If an audit step goes wrong, you restore from this file. In a well-run biobank or specimen library, the same principle is implemented as full lifecycle traceability and audit trails, so that every change to a record can be traced back to who made it and when [6]. Your exported snapshot plays that role at a smaller scale. ### Step 2. Build the inventory Load the export and count what you have. This becomes the denominator for every metric that follows. ```python import pandas as pd df = pd.read_csv("library_export_2026-11-11.csv") print(len(df), "total records") print(df["Item Type"].value_counts()) ``` Expected output for a mid-sized lab library: ``` 513 total records journalArticle 401 bookSection 52 conferencePaper 28 thesis 18 dataset 14 ``` ### Step 3. Detect duplicates two ways Zotero has a built-in duplicate pane. Click Duplicate Items in the left sidebar, review each candidate pair side by side, and merge. The built-in matcher relies on titles, identifiers, and years, and it works well for exact duplicates. It will not catch near-duplicates such as a preprint and its published version with different titles. Run a second, scriptable pass on the exported CSV: ```python import pandas as pd df = pd.read_csv("library_export_2026-11-11.csv") df["match_key"] = ( df["Title"].str.lower().str.replace(r"[^a-z0-9 ]", "", regex=True).str.strip() + "|" + df["Publication Year"].astype(str) ) dupes = df[df.duplicated("match_key", keep=False)].sort_values("match_key") groups = dupes["match_key"].nunique() print(f"{groups} duplicate groups, {len(dupes)} records involved") dupes.to_csv("duplicates_to_review.csv", index=False) ``` Expected output on the example library: ``` 14 duplicate groups, 31 records involved ``` Review every group manually before merging. Automated tools propose merges, and a human confirms them. Merging two genuinely distinct records, for example two papers with near-identical titles from the same group, is worse than leaving both in place. ### Step 4. Score metadata completeness ![A worked calculation showing 513 records, the share with all required fields filled, and the resulting 87.4 percent completeness score.](/article-images/illustrations/reference-library-audit-completeness-score.webp) Define the required fields for your workflow. For a journal-article-driven library, a reasonable set is title, first author, year, and DOI. The completeness score is the share of records that have all required fields filled: ```text metadata completeness = (records with all required fields filled / total records) x 100 ``` Compute it directly: ```python required = ["Title", "Publication Year", "Author", "DOI"] complete = df[required].notna().all(axis=1).mean() * 100 print(f"Metadata completeness: {complete:.1f}%") ``` Expected output: ``` Metadata completeness: 87.4% ``` Every record below 100 percent lands in your audit spreadsheet with severity Major or Minor depending on which field is missing. ### Step 5. Verify that identifiers resolve A DOI is only useful if it points to the right article. Check each one against the DOI resolver. Keep the request rate low and pause between requests so you do not hammer the resolver. ```python import pandas as pd import requests import time df = pd.read_csv("library_export_2026-11-11.csv").dropna(subset=["DOI"]) broken = [] for doi in df["DOI"]: r = requests.head("https://doi.org/" + str(doi).strip(), allow_redirects=True, timeout=10) if r.status_code != 200: broken.append((doi, r.status_code)) time.sleep(0.5) print(f"{len(broken)} of {len(df)} DOIs failed to resolve") for doi, code in broken: print(doi, code) ``` Expected output: ``` 3 of 448 DOIs failed to resolve 10.1234/bad.suffix 404 10.9999/never.issued 404 10.1037/typo.in.press 404 ``` Each failure becomes a Critical or Major row depending on whether the correct record can be located. A DOI that resolves but points to a different article than the one on your record is the most dangerous case, so spot-check a sample by opening the resolved page and comparing the title. ### Step 6. Screen for fabricated and unverifiable references Sort your library by entry source, or add a `source_of_entry` column if you do not have one. Set aside every record that entered the library through an AI assistant, an unpublished reading list, or a coauthor's unverified bibliography. For each of those, find the article in the publisher's database or PubMed yourself. If it cannot be found, it is Critical and it comes out. The 61.6 percent irrelevance figure cited above came from an evaluation of large language model platforms in a clinical specialty, and the broader review concluded that current tools are not accurate enough for unsupervised use [3]. Treat machine-generated citations as leads, never as records. ### Step 7. Score the library and write the report Roll your findings into four headline numbers. These targets are a sensible starting rubric for a working research library, and you should tighten them once your baseline is established. | Metric | Formula | Starting target | |, - |, - |, - | | Metadata completeness | Records with all required fields filled / total records x 100 | 95% or higher | | Duplicate rate | Records involved in duplicate groups / total records x 100 | Under 2% | | Broken identifier rate | DOIs that fail to resolve / DOIs tested x 100 | Under 1% | | Unverified source rate | Records never confirmed against the publisher / total records x 100 | 0% for anything cited in a manuscript | For the example library, the report reads: 513 records, 87.4 percent metadata completeness, 6.0 percent duplicate rate, 0.7 percent broken DOI rate, and 11 records from unverified sources. That single paragraph is the deliverable most supervisors and collaborators want. ### Step 8. Fix, log, and close the loop Apply fixes in severity order. Log every change in the audit spreadsheet with the date and your initials, which creates the same audit trail discipline used in governed specimen management systems [6]. Then schedule the next audit. The evidence from health care implementation research is that audit and feedback interventions produce real but often short-lived changes in behavior, which is why one-off audits underperform [7]. Put a recurring audit on your calendar, quarterly for an active manuscript pipeline, annually for an archival library, and consider a lightweight dashboard that tracks the four headline numbers over time so that decay is visible between audits. ## Walkthrough 2: Auditing a DNA Barcode Reference Library with BAGS If your reference library is a set of COI barcode sequences rather than a set of papers, the audit question changes. You are no longer asking whether metadata is complete. You are asking whether each species entry would actually assign the correct species in a metabarcoding run. Some vocabulary first. DNA barcoding uses a short, standardized gene region to identify species, and DNA metabarcoding applies that idea at scale to mixed environmental samples, which makes it a widely used identification tool in biomonitoring and conservation programs [2]. COI, cytochrome c oxidase subunit I, is the standard barcode region for animals. The Barcode of Life Data System (BOLD) is the public repository that hosts barcode records. A barcode index number, or BIN, is a cluster of sequences that the system infers to correspond to a single species. When the species names attached to sequences in one BIN disagree, something is wrong in the library. ### Step 1. Install the audit tool BAGS, short for Barcode, Audit and Grade System, is an open-source R application built specifically to automate this audit for COI libraries of animal taxa drawn from BOLD [2]. ```r install.packages("devtools") devtools::install_github("tadeu95/BAGS") library(BAGS) ``` The tool is developed and documented on GitHub at the repository named in its publication, so open the package documentation there for the current function signatures and input format before running your data through it. ### Step 2. Prepare your library extract Download the COI records for your taxonomic group from BOLD. BAGS audits one animal taxonomic group per run, so scope each run to a single group, for example a family or order, rather than passing the whole kingdom at once [2]. ### Step 3. Run the automated audit and read the grades BAGS performs an automated audit and annotation of the library and then assigns each species one of five grades, A through E [2]. The grades are a qualitative ranking based on the attributes of the underlying data and on how congruent the species names are with the sequences clustered in their BINs. Grade A indicates the most reliable entries and grade E the least, with records whose names are incongruent with their barcode clusters segregated and highlighted so you can separate usable records from suspect ones [2]. The precise criteria for each grade are defined in the tool's documentation, and you should read them before interpreting your output, because the grade of a species entry tells you how much weight its identifications can carry in a downstream analysis. ### Step 4. Act on the low grades The purpose of the grading pass is not to delete data but to segregate it. Low-graded species entries are candidates for taxonomic review, re-sequencing from voucher specimens, or exclusion from conservative species-level assignments. The tool was built to fill a gap in the DNA barcoding toolset by screening reference libraries quickly and flagging where the data support a species assignment and where they do not [2]. Re-run the audit after corrections so the grade distribution of your library becomes a tracked metric, the same way the four bibliographic metrics above are tracked between audit cycles. ## Choosing Tools: A Citation Manager Comparison The bibliographic half of an audit goes faster in the right tool. Here is a brief citation manager comparison on the features that matter for auditing. All four options below are established tools, and the characterizations are qualitative. | Tool | Cost model | Audit-relevant strengths | Watch out for | |, - |, - |, - |, - | | Zotero | Free, open source | Built-in duplicate pane, clean CSV and BibTeX export, browser connector, large plugin ecosystem | Duplicate matcher misses near-duplicates, so run a script pass too | | Mendeley | Free with an account | Strong PDF annotation and reference extraction from PDFs | Less scriptable export, so plan on manual review | | EndNote | Paid | Handles very large libraries and journal styles well | Less open export formats, which slows scripted audits | | JabRef | Free, open source | Native BibTeX, good for LaTeX-driven workflows | No web connector, entry relies on manual or imported BibTeX | The single most important feature for audit purposes is a lossless export to a plain format, because that is what makes the scripted checks in the walkthrough above possible. ## Common Mistakes and Limitations - **Merging duplicates automatically.** Every proposed merge should be eyeballed. Near-identical titles describe different papers more often than new auditors expect, especially in taxonomic literature where species epithets repeat. - **Auditing without a frozen backup.** Merge operations are hard to reverse. Export first, always. - **Trusting chatbot-provided citations.** The reference fabrication problem documented for large language models is severe enough that every machine-generated reference needs independent confirmation against the publisher record [3]. - **Auditing the bibliography but not the reference values.** If your project relies on a set of thresholds or reference standards, verify each one against its current authoritative source. The systematic review of AUDIT cut-off scores shows how widely used reference values can drift out of alignment with the body that defined them [4]. - **Reading a barcode grade as ground truth.** A grade summarizes the attributes and congruency of the data on record [2]. It does not replace taxonomic judgment on a contested species. - **One auditor, one pass.** A second reviewer catching discrepancies is standard practice in audit-based reviews, where independent screening and a third-party tiebreaker are used to resolve disagreements [1]. For a lab library, having a labmate re-check your Critical findings is a cheap version of the same control. - **Auditing once and never again.** Libraries decay. Audit and feedback effects fade without a recurring cycle [7], so the calendar entry in Step 8 is not optional polish, it is the mechanism that makes the audit stick. - **Overcorrecting the library.** Deleting every record that fails a strict check can strip out data that are merely imperfect but useful. Segregate and annotate questionable records rather than destroying them, which is the same philosophy BAGS applies to low-graded species entries [2]. One limitation belongs up front. This guide covers the general method. Individual libraries, especially institutional ones with regulatory obligations, may require role-based access controls, standardized data dictionaries, and formal audit trails of the kind implemented in governed biobank information systems [6], and those requirements come from your institution, not from a template. ## Frequently Asked Questions ### How often should I run a reference library audit? Quarterly for a library that feeds active manuscripts, annually for an archival library. Recurrence matters because the improvements from a single audit and feedback cycle tend to fade over time [7]. ### Is there a reference library audit example I can follow? Yes. The Zotero walkthrough in this article is a complete worked example, from a 513-record export through duplicate detection, DOI verification, and a four-metric scored report. ### What does a standard reference library audit format look like? A CSV tracking file with one row per record, columns for the identifier, duplicate flag, verification status, severity tier, action, auditor, and date, plus a three-tier severity scheme of Critical, Major, and Minor findings. ### Which citation manager is best for auditing? Zotero is the strongest choice for auditability because it is free, open source, and exports losslessly to CSV and BibTeX, which makes scripted checks possible. See the comparison table above for the alternatives. ### Can I trust references generated by AI chatbots? No. Reference fabrication and irrelevance are documented failure modes of large language models, including one evaluated study reporting a 61.6 percent reference-to-prompt irrelevance rate [3], so verify every machine-generated citation against the publisher yourself. ### How do I audit a DNA barcode reference library? Use an automated tool built for the job. BAGS audits COI reference libraries from BOLD for a chosen animal taxonomic group and assigns each species a reliability grade based on data attributes and BIN congruency [2]. ### What do the BAGS grades A through E mean? They are a qualitative ranking from most reliable (A) to least reliable (E), driven by data attributes and by whether species names are congruent with the sequences clustered in their BINs. The exact criteria per grade are documented in the tool's repository [2]. ### How do I know whether my audit actually worked? Re-run the same four metrics after fixes and compare. A falling duplicate rate, rising metadata completeness, and a cleared unverified-source list across scheduled audits show the cycle is working, and a dashboard tracking those numbers keeps the trend visible between audits [7]. ## Sources 1. [Implementation of maternal and perinatal death reviews: a scoping review protocol.](https://pubmed.ncbi.nlm.nih.gov/31780590/) 2. [BAGS: An automated Barcode, Audit & Grade System for DNA barcode reference libraries.](https://pubmed.ncbi.nlm.nih.gov/33000878/) 3. [Turing problems in otolaryngology: a scoping review of the principal challenges of artificial ıntelligence and large language models.](https://pubmed.ncbi.nlm.nih.gov/42747528/) 4. [Auditing the AUDIT: A systematic review of cut-off scores for the Alcohol Use Disorders Identification Test (AUDIT) in low- and middle-income countries.](https://pubmed.ncbi.nlm.nih.gov/31349205/) 5. [Optimization and impact of an evidence-based pre-audit prescription decision system in primary healthcare settings.](https://pubmed.ncbi.nlm.nih.gov/40297139/) 6. [Construction and Application of a Biological Sample Library Information Management System Based on a Data Governance System in China.](https://pubmed.ncbi.nlm.nih.gov/41681004/) 7. [Effectiveness of interactive dashboards to optimise prescribing in primary care: a protocol for a systematic review.](https://pubmed.ncbi.nlm.nih.gov/39931386/)