# How to Migrate Historical Lab Notebooks to an ELN By the end of this guide, you will be able to take a shelf of finished paper lab notebooks and turn it into a searchable, verified electronic archive inside an electronic [lab notebook](/knowledge/molecular-biology/lab-notebook) (ELN). You will build a manifest of every notebook, scan at settings that keep handwriting legible, prove that no file changed in transit, add a searchable text layer with optical character recognition (OCR), import everything into a reusable lab notebook template, and quality control the result. You need the notebooks themselves, a scanner, a computer with Python 3 plus the free OCR tools Tesseract and OCRmyPDF, a spreadsheet application, and an ELN account with permission to create projects and upload files. Every software tool in this pipeline is free. Migrating historical lab notebooks to an ELN is a [data management](/blog/guides/data-management-basics-principles-processes-and-best-practices) project more than a scanning project. The scan is the easy part. The inventory, the file naming, the integrity checks, and the second-person review are what make the archive trustworthy five years from now. ## What You Need on Hand - The notebooks, in date order if you can arrange it. - A flatbed scanner for bound pages, ideally paired with a document scanner that has an automatic document feeder (ADF), a tray that pulls loose sheets through one at a time. The flatbed handles bound and fragile pages. The ADF handles anything you are allowed to unbind. - A workstation with Python 3.10 or newer, Tesseract, OCRmyPDF, and poppler-utils, a command-line toolkit for inspecting and splitting PDFs. Install them with: ```bash # Debian or Ubuntu sudo apt install tesseract-ocr tesseract-ocr-eng ocrmypdf poppler-utils # macOS with Homebrew brew install tesseract ocrmypdf poppler # Windows: run the Tesseract installer, install Ghostscript, then: pip install ocrmypdf ``` - A spreadsheet application for the manifest, with the ID and date columns formatted as text before you type anything. - Disk space. A page scanned at 300 dpi (dots per inch, a measure of image detail) and saved as a compressed PDF usually lands between 0.2 and 1 MB. Budget 1 MB per page. Twenty notebooks of roughly 200 pages each then need about 4 GB for the working copy, and about 12 GB once you keep three copies as Step 9 requires. - An ELN account that can create projects, upload attachments, and define custom templates. ## Choosing an ELN Before You Migrate If the lab has not picked a platform, decide before scanning, because import tools and limits differ. An electronic lab notebook comparison comes down to a short list of criteria: - **Export.** Can you export every entry, attachment, and metadata field in open formats such as PDF, CSV, or JSON? A system you cannot leave is a system you will migrate twice. - **Audit trail.** The ELN should log who changed what and when, in a record users cannot edit. - **Signatures.** If your work follows electronic records rules such as 21 CFR Part 11, confirm the platform supports electronic signatures and a validated audit trail. - **Search depth.** Test whether search finds words inside attached PDFs, not just entry titles. - **Templates.** You need custom fields for the migrated records described in Step 3. - **Limits.** Check per-file upload caps and total storage quotas before you commit to one large PDF per notebook. - **Permissions.** Migrated notebooks should be read-only for everyone except the archive owner. Commercial platforms such as Benchling, LabArchives, RSpace, and SciNote, and open-source options such as eLabFTW, differ mainly in these areas. Run one pilot notebook through import in your leading candidate before you scan anything else. ## Migrating Historical Lab Notebooks to an ELN, Step by Step ![Flowchart of the migration pipeline from inventory through scanning, checksums, OCR, import and quality control to backup and retirement.](/article-images/illustrations/migrating-historical-lab-notebooks-to-eln-migration-pipeline.webp) Steps 1 through 3 are planning. Steps 4 through 6 are the pipeline. Steps 7 through 9 are import, review, and retirement. Each step includes the settings that matter, the checks that catch errors, and tips drawn from the failures labs hit most often. ### Step 1: Inventory every notebook and build a manifest A manifest is a spreadsheet listing every notebook and the facts you will need later. It is the chain of custody for the whole migration. Give each notebook an ID such as NB-001, then record the page count, the first and last entry dates, the owner, the project, and the shelf. ```text notebook_id,page_start,page_end,date_start,date_end,owner,project,shelf NB-001,1,212,2018-02-05,2018-11-30,a.rivera,pKG207,shelf-2 NB-002,1,198,2018-12-01,2019-07-15,a.rivera,pKG207,shelf-2 NB-003,1,176,2019-08-01,2020-01-20,j.osei,pKG207,shelf-3 ``` Two formatting rules save hours of cleanup. First, format the ID and page columns as text so the spreadsheet does not strip the leading zeros from NB-001. Second, write dates in ISO 8601 format (year-month-day, as in 2018-02-05), which spreadsheet applications leave alone. A cell typed as 03/04/2019 can be silently reinterpreted as April 3, and a date shift inside a lab record is a real problem. Check: walk the shelf and confirm the manifest row count equals the notebook count. ### Step 2: Choose the migration approach for each notebook ![Comparison table of image archive, curated transcription and hybrid approaches by what moves, searchability and effort.](/article-images/illustrations/migrating-historical-lab-notebooks-to-eln-migration-approaches.webp) Three approaches exist, and most labs do best mixing them. An image archive scans every page and transcribes nothing. A curated transcription retypes selected experiments into structured ELN entries and skips the page images. The hybrid scans everything and adds curated entries for the experiments the lab still cites. | Approach | What moves | Searchability | Effort per notebook | Best for | |--|--|--|--|--| | Image archive | Every page scanned, no transcription | OCR text only, weak on handwriting | Low | Cold storage, retention, rarely cited work | | Curated transcription | Key experiments retyped into structured entries | Strong, structured fields | High | Active projects, shared protocols | | Hybrid | Full page scans plus curated entries for key experiments | Strong | Moderate | Most labs, and the default this guide assumes | Decide per notebook using the manifest. Notebooks from active projects get the hybrid treatment. Cold archives get image-only. Full scans protect everything you did not anticipate needing, and curated entries make the small fraction of experiments that matter most actually findable. ### Step 3: Create the lab notebook template in the ELN Create two templates before any upload. The first is a migration record template, used once per notebook or per page range. Give it these fields: - Title, formatted as NB-001 p.14-15, 2018-02-05, PCR assembly - Original notebook ID (text) - Page range (text, so leading zeros survive) - Experiment date (date field) - Author (text or user field) - Project (dropdown matching the manifest) - Scan file (attachment) - OCR text (attachment or pasted body text) - Transcription status (dropdown: archived, summarized, or transcribed) - Reviewed by (user field, filled at quality control) The second is a forward-looking experiment template so new work matches the old records: objective, protocol reference, reagent lots, instrument settings, observations, links to raw data, conclusion, and a signature line. One lab notebook template for old and new entries keeps search results coherent. ### Step 4: Scan the pages Scan in page order, front to back, and include blank pages so each image position matches the printed page number. Note any blank or missing pages in the manifest. Photograph the cover, spine, and any loose inserts first so the archive records the physical object. Sticky notes and loose sheets get scanned separately and logged against the page they were stuck to. Bound notebooks go on the flatbed, one page per pass, pressing gently near the spine. Never force a binding flat enough to crack it. Tight bindings and fragile pages photograph well from above with a phone stand and even light. Foldouts, oversized gel prints, and charts get scanned in overlapping tiles and stitched later, or photographed square-on. Match settings to the page: | Page type | Resolution | Color mode | Master format | Why | |--|--|--|--|--| | Typed or neatly printed text | 300 dpi | Grayscale | TIFF or PNG | Standard baseline, small files, crisp text | | Handwriting, pencil, faint ink | 400 to 600 dpi | Color | TIFF | Extra detail keeps faint strokes legible | | Taped-in photos of gels, plates, blots | 400 to 600 dpi | Color | TIFF, plus a JPEG copy for ELN display | Color preserves staining and labels | | Foldouts and oversized charts | 300 dpi | Color | TIFF in tiles, stitched later | Flatbeds cannot take oversized pages in one pass | | Graph paper with colored inks | 300 dpi | Color | TIFF or PDF/A | Color separates pen over pencil | TIFF (Tagged Image File Format) and PNG are lossless formats, meaning every pixel is preserved. JPEG is compressed and smaller, which is fine for display copies but not for masters. Name files with zero-padded page numbers so NB-001_p0002.tif sorts before NB-001_p0010.tif. Without padding, p10 sorts before p2 and the archive lands out of order. If the scanner names files its own way, fix them in one pass: ```bash cd scans/NB-001 i=1 for f in *.tif; do mv "$f" "$(printf 'NB-001_p%04d.tif' "$i")" i=$((i+1)) done ``` Run this once, from a folder holding only raw scans. Run it twice and the files rename again, so keep originals in a raw subfolder if you might re-run. Check: open the first, middle, and last page of each notebook at full zoom. Confirm orientation, order, and that handwriting stays legible when you zoom in. ### Step 5: Record checksums before anything moves A checksum is a fixed-length fingerprint computed from a file's bytes. SHA-256, the standard choice here, produces a 64-character hexadecimal value. Change one pixel in a scan and the fingerprint changes completely, which is what makes checksums the integrity proof for the migration. Hash every file before it leaves the scanning workstation: ```bash cd scans/NB-001 sha256sum *.tif > NB-001.sha256 ``` The checksum file looks like this (example values): ```text 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 NB-001_p0001.tif 2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824 NB-001_p0002.tif ``` Verify at any time with: ```bash sha256sum -c NB-001.sha256 ``` Expected output, one line per file: ```text NB-001_p0001.tif: OK NB-001_p0002.tif: OK ``` On Windows, PowerShell does the same job: ```powershell Get-ChildItem *.tif | ForEach-Object { certutil -hashfile $_.FullName SHA256 } ``` with output in this shape: ```text SHA256 hash of NB-001_p0001.tif: 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 CertUtil: -hashfile command completed successfully. ``` For a whole archive at once, this Python script walks the scan folders and writes a manifest with sizes and hashes: ```python import hashlib, csv, datetime, pathlib root = pathlib.Path("scans") rows = [] for f in sorted(root.rglob("*.tif")): h = hashlib.sha256() with f.open("rb") as fh: for chunk in iter(lambda: fh.read(1 << 20), b""): h.update(chunk) rows.append({ "file": str(f), "bytes": f.stat().st_size, "sha256": h.hexdigest(), "recorded": datetime.date.today().isoformat(), }) with open("manifest_hashes.csv", "w", newline="") as fh: writer = csv.DictWriter(fh, fieldnames=["file", "bytes", "sha256", "recorded"]) writer.writeheader() writer.writerows(rows) print(f"wrote {len(rows)} rows to manifest_hashes.csv") ``` Expected output: ```text wrote 586 rows to manifest_hashes.csv ``` The script reads files in 1 MiB chunks (1 << 20 bytes) so large scans never load into memory whole. Check: the row count in manifest_hashes.csv equals the total page count from the Step 1 manifest. ### Step 6: Run OCR and build the PDF/A copies OCR converts pictures of text into actual text, and it is what makes a scan searchable. Tesseract handles single images: ```bash tesseract NB-001_p0014.png NB-001_p0014 -l eng --dpi 300 ``` The `-l eng` flag selects the English language model, and `-dpi 300` tells Tesseract the image resolution when the file itself does not record it, which PNG files often do not. The command writes NB-001_p0014.txt. For a whole folder: ```bash for f in *.png; do tesseract "$f" "${f%.png}" -l eng --dpi 300 done ``` Typical output for a lab page looks like this: ```text 2018-02-05 NB-001 p.14 PCR assembly for pKG207 insert Master mix 25 uL, primer F 1 uL, primer R 1 uL Template 1 ng, cycles 30, anneal 58 C ``` Notice what OCR got wrong. The degree symbol dropped from the annealing temperature, and units drift. Accuracy on typeset text is high, on neat block printing it is moderate, and on cursive handwriting it is poor. Treat OCR as a search index over the scans, never as a transcript. Transcribe experiments that matter by hand into the curated entries from Step 2. If you scanned straight to PDF, OCRmyPDF adds a text layer to the whole file and can output PDF/A, an ISO-standardized PDF variant (ISO 19005) built for long-term storage: ```bash ocrmypdf --deskew --output-type pdfa NB-001.pdf NB-001_ocr.pdf ``` `-deskew` straightens pages that scanned at a slight angle. If a PDF already contains a text layer, OCRmyPDF refuses by default with the error shown in the fixes section below, and `-skip-text` or `-redo-ocr` resolves it. Check that the text layer landed: ```bash grep -l "pKG207" NB-001/*.txt ``` Expected output: ```text NB-001_p0014.txt NB-001_p0015.txt ``` ### Step 7: Import into the ELN Import in this order: 1. Build the project tree in the ELN first, one project per research project, matching the manifest's project column. 2. Upload the OCR'd PDF/A for each notebook into a dedicated migration archive section, or attach page-range files to individual entries if the ELN supports attachments at that granularity. 3. Attach or paste the OCR text so search can index it. Whether an ELN searches inside attachments varies by platform, so test with one entry: search a distinctive term such as pKG207 and confirm the right entry surfaces. 4. Fill the migration template fields from the Step 1 manifest. Many ELNs import metadata in bulk from CSV, with required column names defined in the vendor documentation, so read that page before scripting the import. 5. Set migrated entries read-only for everyone except the archive owner. If a 200-page PDF exceeds a per-file upload cap, split it by page ranges and upload the parts, or upload one file per experiment. Poppler's pdfseparate and pdfunite handle splitting and rejoining from the command line. ### Step 8: Quality control the migration Quality control has three layers. First, integrity. Download one file back from the ELN, hash it again, and compare against the manifest. Then run a full verification pass on the local copies with `sha256sum -c`. Every line must read OK. Second, sampling. Review at least 10 pages or 10 percent of each notebook, whichever is greater. Check page order, orientation, legibility at full zoom, and a spot check of the OCR text against the image. Record the sample in the manifest. Third, review. A second person signs off on entries that will support publications, patents, or regulated work. The person who scanned misses what they missed while scanning, so the reviewer must not be the scanner. Log the QC result as a signed entry in the ELN and update the manifest status to qc_passed. Confirm page counts survived import: ```bash pdfinfo NB-001_ocr.pdf | grep Pages ``` Expected output: ```text Pages: 212 ``` ### Step 9: Back up, then retire the paper Keep three copies of the archive, on two kinds of media, with one copy offsite. This is the 3-2-1 rule, and it exists because any single location eventually fails. A workstation drive, an external drive, and an institutional server or cloud storage satisfy it. The ELN itself is not a backup. Accounts lapse, exports fail, vendors change. Keep the archive independent of the platform. The paper notebooks stay put until two conditions hold: quality control has passed, and your institutional records policy allows disposal. Many retention policies require originals for set periods, and a scan is a copy, not the signed original. Until then, store the notebooks dry, dark, flat, and off the floor. When disposal is authorized, update the manifest status to retired and record the authorization in the ELN so the decision itself has a record. ## Common Errors and Their Exact Messages These are the failures you will actually hit, with their exact messages and fixes. **Checksum mismatch after transfer.** ```text NB-001_p0137.tif: FAILED sha256sum: WARNING: 1 computed checksum did NOT match ``` A file changed between hashing and verification, usually a truncated upload or a partial copy. Re-copy or re-scan that page from the original notebook. Never regenerate the checksum to make the warning disappear, because that erases the only evidence something went wrong. **Tesseract cannot find its language data.** ```text Error opening data file /usr/share/tesseract-ocr/5/tessdata/eng.traineddata Please make sure the TESSDATA_PREFIX environment variable is set to your "tessdata" directory. Failed loading language 'eng' Tesseract couldn't load any languages! ``` The English language pack is missing or installed somewhere unusual. Install it with the package manager (on Debian or Ubuntu, the tesseract-ocr-eng package), or point Tesseract at the right folder with `-tessdata-dir /path/to/tessdata`. **OCRmyPDF finds existing text.** ```text PriorOcrFoundError: page already has text! ``` The PDF already has a text layer, often from the scanner's own bundled OCR. Add `-skip-text` to keep the existing layer, or `-redo-ocr` to replace it with Tesseract's output. **Python cannot find a file.** ```text FileNotFoundError: [Errno 2] No such file or directory: 'scans/NB-001_p0001.tif' ``` The path is wrong or the working directory is not where you think. Print the exact path in the script, list the folder with `ls` (or `dir` on Windows), and fix the mismatch. This error most often appears when a file was renamed after the manifest was built. **Silent spreadsheet mangling.** No error message appears, which is the problem. IDs lose leading zeros, 03/04/2019 becomes a different date, and long numbers turn into scientific notation. Prevent it by formatting columns as text before typing and by using ISO 8601 dates. Audit the finished manifest against the physical shelf. **Upload rejected by the ELN.** The platform reports an oversized file or an unsupported type, with wording that varies by vendor. Split the PDF by page ranges, convert to a supported type, and re-upload. Check the per-file cap in the vendor documentation before choosing a file size strategy. ## Common Mistakes and Limitations - **Scanning too lightly to save space.** Low-resolution text looks acceptable on screen and collapses when you zoom in to read handwriting or a margin note. Disk space is cheaper than re-scanning. - **Treating OCR as a transcript.** OCR is an index. Cursive handwriting rarely survives it, and numbers in tables drift. Anything cited or handed to a collaborator gets transcribed by a person. - **Giving every notebook equal effort.** Curating a cold archive wastes days, and image-only treatment of an active project's notebooks hides the best experiments. Match the Step 2 approach to how the notebook gets used. - **Skipping the pilot.** Settings chosen on notebook one get repeated across the whole shelf. Run one notebook end to end, including import and QC, before batch processing. - **Reviewing your own work.** The scanner is the wrong QC reviewer. Use a second person for anything that matters. - **Treating the ELN as the backup.** Keep the 3-2-1 copies independent of the platform. - **Uploading without checksums.** Truncation during transfer is invisible until someone needs page 137. - **Forgetting to test export.** Test the ELN's export in week one with the pilot notebook, not in year five when the lab changes platforms. Migration preserves what exists. OCR cannot recover handwriting that was already illegible, ink that faded years ago, or pages removed before the migration. Compliance obligations also differ between academic labs and labs working under Good Laboratory Practice (GLP) or related frameworks, so confirm retention periods, signature requirements, and disposal rules with your institutional compliance or quality office before acting. ## Next Steps Schedule a quarterly spot check of a few random pages against their checksums. Train the lab on the forward-looking template so every new experiment starts in the ELN. Once new entries flow, the paper archive stays frozen and finite, and the migration is done. ## Frequently Asked Questions ### How long does it take to migrate historical lab notebooks to an ELN? Plan on roughly half a working day per 200-page notebook for a careful one-person migration. Flatbed scanning with careful handling runs about 20 to 30 seconds per page, so a 200-page notebook takes around 100 minutes of scanning alone. OCR, import, and quality control add a comparable amount of time. ### What resolution should I scan lab notebook pages at? Use 300 dpi as the baseline for typed and neatly printed pages, and 400 to 600 dpi for handwriting, pencil, faint ink, or taped-in photos. Higher resolution keeps faint strokes legible when zoomed. Scan in color whenever color carries meaning, such as highlighter marks or colored pen over pencil. ### Can OCR read handwritten lab notes? Sometimes, but never count on it. OCR engines are trained mostly on typeset text, so neat block printing comes through reasonably well while cursive handwriting usually does not. Treat OCR as a search index over the scans and transcribe important experiments by hand. ### Should I scan lab notebooks to PDF or TIFF? Do both, in sequence. Keep lossless TIFF or PNG files as master images, then build a PDF/A copy with an OCR text layer for reading and upload. TIFF preserves every pixel for the archive, while PDF/A keeps pages, text, and images together in one file an ELN can display. ### How much storage space do I need for scanned notebooks? Budget about 1 MB per page for color PDF scans and roughly a third of that for grayscale, then multiply by three for the backup copies. Twenty 200-page notebooks therefore need about 4 GB working and about 12 GB with backups. A single external drive covers most labs. ### What should the lab notebook template for migrated entries include? Include fields for the original notebook ID, page range, experiment date, author, project, the attached scan, the OCR text, a transcription status such as archived, summarized, or transcribed, and a reviewer. Reuse the same template for new entries so old and new records stay coherent in search. ### Can I throw away the paper notebooks after scanning? Not until your institutional records policy allows it and quality control has passed. Many retention policies require originals for set periods, and a scan is a copy, not the signed original. Ask your compliance or quality office before any disposal. ### How do I compare ELNs before migrating historical lab notebooks? Judge candidates on export formats, audit trails, electronic signatures, template support, whether search indexes attached PDFs, storage limits, and API access. Migrating into a system you cannot export from recreates today's problem in digital form. Run one pilot notebook through import in your top choice before committing the archive.