# How to Migrate Data Between Repositories: A Step-by-Step Guide By the end of this guide, you will be able to move a complete research dataset from one repository to another without losing files, breaking metadata, or destroying the provenance trail that makes your data citable and reusable. You will run an inventory of the source dataset, generate a checksum manifest, map your metadata to the destination schema, transfer files through a repository API, verify every byte arrived intact, and publish a record that links back to the original. To follow the walkthrough you need a computer with command line access, Python 3.9 or later with the `requests` library installed, the `sha256sum` utility (bundled with Linux and macOS, available through Git Bash on Windows), an API token for the destination repository, and the documentation pages for both the source and destination repositories. Budget a full working day for a dataset of a few thousand files, and more if you need institutional sign-off on sensitive data. ## Why Researchers Migrate Data Between Repositories Researchers move data between repositories for practical reasons. A lab closes or changes institutions. A domain repository adds curatorial services or better search. A funder or journal updates its data sharing policy, and the current home no longer qualifies. A project outgrows an ad hoc storage folder and needs a real repository with a persistent identifier. The pressure to deposit is growing. In a global review of shorebird tracking studies, only 26.9 percent of 351 peer-reviewed publications reported that data were archived in an online repository, though that share increased over time, and the authors identified improved archiving practices as key to getting more value from existing data [1]. Journals, funders, and repositories increasingly treat deposition as the norm rather than the exception, which pushes many older datasets toward their first real repository home. Migration also happens at scale. Genomic and related health data from government funded research have undergone mass migration to cloud based repositories, where they can be stored, accessed, and analyzed inside shared computing environments. That shift brings new stewardship duties around privacy, integrity, and appropriate access throughout the data lifecycle [2]. Whether you are moving one dataset or an entire program of work, the same core discipline applies: know what you have, verify what moves, and document what changed. There is a reuse argument as well. Public repositories now hold vast archives of legacy data that researchers mine for new questions, including large collections of microarray data that recent benchmarking work shows remain analytically useful with the right methods [3]. A clean migration makes your dataset part of that reusable pool. A sloppy migration buries it. ## Choosing a Destination: A Practical Data Repository Comparison Before you touch a single file, compare candidate repositories on the criteria that affect long term usability. Domain specific repositories offer curated metadata schemas and community recognition. Generalist repositories accept any file type and issue [persistent identifiers](/knowledge/bioinformatics/persistent-identifiers-in-bioinformatics-selection-criteria-and-implementation-workflow) quickly. Hybrid platforms handle ingestion, standardization, and processing in one place, such as the neuroimaging platform NeuroStage, which combines automatic upload, anonymization, organization into the Brain Imaging Data Structure (BIDS) format, and a collaborative repository with secure transfer protocols [4]. Use this comparison checklist when you evaluate destinations. | Criterion | What to check | Why it matters | |--|--|--| | Persistent identifier | Does the dataset receive a DOI or equivalent that survives at the file level? | Citations and grants point to the identifier, not a URL that can rot | | Metadata schema | Does the schema match your data type, and can you export metadata in standard formats? | Rich, structured metadata is what makes your dataset discoverable | | License and access model | Can you apply your intended license, and does the repository support embargo or controlled access? | License conflicts can invalidate the migration | | Access controls | Can you restrict sensitive records while publishing the rest? | Human subject and clinical data usually require tiered access | | API and bulk upload | Is there a documented API with batch endpoints and checksum support? | Manual web uploads do not scale and resist verification | | Preservation commitment | Is there a stated retention policy and a sustainability plan? | Migrating twice because a repository folded is a real risk | | Curation and validation | Does the repository run metadata validation or format checks at ingest? | Early validation catches broken files before they become the public record | | Integration with standards | Does it interoperate with community standards and terminologies? | Standards such as LOINC and protocol libraries like PhenX exist to make cross study comparison possible [5] | Repository architecture documents are useful reading during comparison because they tell you what a well built destination actually requires. The framework for building large scale sleep data repositories, for example, names data intake and processing, centralized storage, external data integration, and secure study environments as core components, with data standardization requirements that include European Data Format parsing, channel harmonization, and metadata validation [6]. If a candidate repository cannot describe its ingest pipeline in those terms, expect friction. ## How to Migrate Data Between Repositories, Step by Step The walkthrough below assumes a source repository you can download from and a destination repository with a REST API. Replace the example endpoint with your destination's real one, and read its API documentation before running anything. ### Step 1. Inventory the source dataset Count every file and record the directory structure before you change anything. Run this from the top folder of the dataset. ```bash cd /path/to/source-dataset find . -type f | wc -l find . -type f | sed 's|/[^/]*$||' | sort | uniq -c ``` The first command prints a total file count. The second prints each subdirectory with its file count, which gives you a structure snapshot. Expected output looks like this: ``` 1284 612 ./raw/2019 440 ./raw/2020 180 ./processed 38 ./docs 14 ./metadata ``` Save both outputs to a text file. You will compare against them after the transfer. Note the total size on disk as well, because destination repositories often enforce per dataset quotas. ```bash du -sh . ``` ### Step 2. Generate a checksum manifest A checksum is a fixed length fingerprint computed from a file's bytes. If even one byte changes, the fingerprint changes. SHA-256 is the standard choice for verification work. ```bash find . -type f -print0 | sort -z | xargs -0 sha256sum > manifest.sha256 wc -l manifest.sha256 head -3 manifest.sha256 ``` Expected output: ``` 1284 manifest.sha256 2b1f9c8e41d7aa63f0c2e5b19d84a763c1f4e2b8d90a5c7f31e6b4d8a2c9f107 ./metadata/README.md 7c3d5e91a2f8b406c1d9e7a35f2b8c64d1e0a9b7c3f5e2816d4a7c9b3e1f8d20 ./raw/2019/tag_0147.csv ``` The manifest is your ground truth. Keep a copy outside the dataset folder, on a drive or in a second location, so a mistake during migration cannot destroy it along with the data. ### Step 3. Map metadata to the destination schema Metadata is the structured description of a dataset: title, creators, dates, methods, variable definitions, and provenance. Every repository has its own required fields. Download the destination's metadata template or schema, then build a mapping table that links each source field to its destination field. A minimal mapping for a tracking dataset might look like this. | Source field | Destination field | Action | |--|--|--| | project_title | title | Copy verbatim | | pi_name | creators | Split into family and given names, one entry per person | | collection_dates | temporal_coverage | Convert to ISO 8601 format, for example 2019-04-01/2023-10-31 | | species_code | taxonomic_coverage | Expand codes to scientific names | | sensor_type | methods | Move into the methods description with device details | | internal notes | (none) | Move anything archival into README, discard nothing silently | Write the mapped values into a structured file you can submit through the API. YAML works well for review because humans can read it. ```yaml title: "Shorebird tracking data, Delaware Bay stopover sites, 2019-2023" creators: - family_name: "Rivera" given_name: "Ana" - family_name: "Okafor" given_name: "Chidi" temporal_coverage: "2019-04-01/2023-10-31" methods: "GPS tags deployed at two stopover sites, fix interval 2 h" ``` Two cautions from the literature. First, use community protocols where they exist. The PhenX Toolkit maintains consensus measurement protocols for genomic, clinical, and epidemiological research and collaborates with data standards such as LOINC and with data repositories to promote interoperability, so mapping your variables to established protocols pays off in cross study reuse [5]. Second, harmonization is real work. A methods protocol for a global Human Internal Migration Database had to extend existing smoothing methods specifically to impose a harmonized, homogeneous format across more than 50 countries, which shows how much effort a common format can demand even for well defined variables [7]. ### Step 4. Standardize file formats before transfer Migration is the cheapest moment to fix format problems, because you are already touching every file. Convert proprietary or fragile formats to open, documented ones. Where a community standard exists for your data type, adopt it. Neuroimaging offers the clearest example. BIDS specifies a directory layout and naming convention for MRI data, and platforms built around it can organize incoming scans into BIDS automatically, which is what makes those datasets usable by standard [preprocessing](/blog/data-analysis/what-is-data-preprocessing) pipelines without per file hand editing [4]. Sleep research has an equivalent in the European Data Format (EDF) for physiological signals, where repository builders treat EDF parsing and channel harmonization as baseline ingestion requirements [6]. If your data combines tables and time series, decide on one internal convention per variable type and document it in the data dictionary. If your dataset will later be merged with others, check how comparable efforts handle alignment. Work that integrates animal tracking databases with trait databases found that limited taxonomic and spatial overlap between sources is a central obstacle, which is a direct consequence of inconsistent metadata across repositories [8]. ### Step 5. Screen for restrictions and prepare access terms Before anything leaves your control, classify each file. Open, restricted, and sensitive data need different handling, and the destination must support the access model you need. Start with a field level review. Map which variables could identify individuals or carry legal restrictions. The Irish health information systems audit mapped 128 national data repositories and found that only 28, about 22 percent, recorded migration related variables such as country of birth or citizenship, and it examined each for secondary access procedures and linkage potential [9]. That style of audit, repository by repository and variable by variable, is exactly what you should run on your own dataset at smaller scale. If your data are human genomic or clinical data, plan for controlled access from the start. Responsible stewardship of genomic data in cloud repositories means managing accuracy, privacy, integrity, and appropriate access throughout the data lifecycle, and repositories increasingly treat these as core outcomes they are expected to demonstrate [2]. Confirm the destination supports user authentication, data use agreements, and access logging before you commit. ### Step 6. Package the dataset A migrated dataset should arrive as a self contained package a stranger can understand. The standard contents are the data files, a README, a data dictionary, and method documentation. Good repositories model this structure. The SHoRT-CA smart home repository, for example, ships data files together with anomaly descriptions and figures of sensor values with highlighted anomalies, so users can interpret the data without contacting the authors [10]. A practical package layout: ``` dataset/ README.md data_dictionary.csv methods.md manifest.sha256 raw/ processed/ metadata/ ``` The README should state, in plain language, what was measured, with what instruments, under what conditions, who collected it, how it was processed, what each file contains, and under what license it is released. The data dictionary defines every column and coded value. Write these documents before transfer, because the destination repository may run metadata validation at ingest and reject a package with undefined fields. ### Step 7. Transfer the files and verify integrity Now the actual move. For datasets beyond a handful of files, script the upload through the repository's API. The following example hashes every file, uploads it, and records what the server reports back. Substitute your destination's endpoint, item identifier, and parameter names per its documentation. ```python import hashlib import json import pathlib import requests BASE = "https://repository.example.org/api/v1" ITEM_ID = "YOUR_DATASET_ID" TOKEN = "YOUR_API_TOKEN" headers = {"Authorization": "Bearer " + TOKEN} def sha256(path): h = hashlib.sha256() with open(path, "rb") as f: for chunk in iter(lambda: f.read(1 << 20), b""): h.update(chunk) return h.hexdigest() # Hash everything before upload manifest = {} for path in pathlib.Path("dataset").rglob("*"): if path.is_file(): manifest[str(path)] = sha256(path) with open("manifest_before.json", "w") as f: json.dump(manifest, f, indent=2) print(len(manifest), "files hashed") # Upload each file for path in pathlib.Path("dataset").rglob("*"): if not path.is_file(): continue with open(path, "rb") as f: r = requests.post( BASE + "/items/" + ITEM_ID + "/files", headers=headers, files={"file": (str(path.relative_to("dataset")), f)}, ) r.raise_for_status() print(path.name, r.json().get("checksum", "no checksum returned")) ``` Expected output: ``` 1284 files hashed README.md e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 data_dictionary.csv 4f2a9c7e31b8d605a1f4c2e9b7d3a8c5e1f6b0d9a4c7e3f2b8d1a5c9e6f3b0d4 raw/2019/tag_0147.csv 9b1c8e4f2a7d3b6c0e5f9a1d8c4b7e2f6a3d9c0b5e8f1a4d7c2b9e6f3a0d5c8b ``` Compare every server reported checksum against your local manifest. If the destination computes checksums at ingest, that comparison is your byte level proof of a faithful transfer. ### Fixing common transfer errors These are the errors you will actually hit, with their typical messages and fixes. **`HTTPError: 413 Client Error: Payload Too Large for url`**. The endpoint rejected a file because it exceeded a size limit. Check the repository's documented maximum, split the offending file, or use a multipart or resumable upload endpoint if the API provides one. **`requests.exceptions.SSLError: HTTPSConnectionPool ... certificate verify failed`**. The client could not verify the server's TLS certificate. Update your operating system's certificate store or Python's `certifi` package. Do not work around this by disabling verification, because you would be sending research data over an unauthenticated channel. **`sha256sum: ./raw/2019/tag_0147.csv: No such file or directory`**. Your manifest lists a file that no longer exists at that path, usually because a file was renamed or a subfolder moved after the manifest was generated. Regenerate the manifest from the current tree and investigate why the file disappeared before continuing. **`pandas.errors.ParserError: Error tokenizing data. C error: Expected 12 fields in line 847, saw 15`**. A CSV has ragged rows. Open the file at the reported line, decide whether the extra fields are corrupted rows or legitimate nested values, and normalize the file before upload. Fixing this after migration means the public copy is broken. **`HTTPError: 422 Client Error: Unprocessable Entity for url`**. The API accepted the file but rejected the metadata payload, usually for a missing required field or a malformed date. Read the error body, which typically names the failing field, correct the mapping table from Step 3, and resubmit. **`OSError: [Errno 28] No space left on device`**. Your local staging disk filled during hashing or packaging. Hashing streams file contents, so free staging space by removing intermediate copies, or point temporary directories at a larger volume. ### Step 8. Publish, link, and audit the result ![Worked example using the 1284 files in the manifest and the count of OK checksum results after the published dataset is downloaded again.](/article-images/illustrations/migrate-data-repository-checksum-audit-count.webp) Once every checksum verifies, publish the dataset. The repository issues a persistent identifier, most commonly a DOI. Then close the loop in three ways. First, update the source side. If the old repository allows it, mark the record as superseded and point to the new identifier. If it does not, at minimum record the migration in the new dataset's README, including the date, the reason, and the identifier of the source record. Provenance is exactly what suffers when records drift, and commentary on AI era research integrity warns that erosion of provenance erodes transparency in the knowledge base, with governance mechanisms increasingly expected to live inside databases and repositories themselves [11]. Your migration notes are your dataset's provenance of movement. Second, update every citation. Papers, preprints, and data statements that cite the old repository location should be corrected where possible or annotated in the new record's related resources. Third, audit. Download the published dataset fresh, recompute checksums, and compare against `manifest_before.json`. ```bash sha256sum -c manifest.sha256 | grep -c ": OK" ``` Expected output: ``` 1284 ``` A count equal to your original file total, with zero failures, means the migration preserved the data exactly. Then open a random sample of files, load them in your analysis environment, and confirm the data dictionary matches the actual columns. Finally, test the access path a stranger would use: find the dataset through the repository's search, follow the identifier, and download one file. Some projects deliberately maintain two public copies. The ethical PROMISE NFR dataset was released on both IEEE DataPort and Mendeley Data, which is a legitimate pattern for visibility and redundancy when both copies carry the same identifier relationships and stay synchronized [12]. If you do this, designate one as canonical so future updates have a single source of truth. ## Common Mistakes and Limitations **Migrating files without metadata.** Files without their metadata are a warehouse of numbers, not a dataset. Bring the data dictionary, methods description, and collection context. Repository scale work in metabolomics shows the payoff: an expanded bile acid spectral library built across three major metabolomics repositories succeeded because it carried harmonized pan repository metadata alongside the spectra [13]. **Regenerating files instead of copying them.** Re-exporting from a database or re-saving a spreadsheet during migration can silently change encoding, precision, or column order. Migrate the original bytes, and verify with checksums. If a format conversion is genuinely needed, keep the originals in a raw folder and document the conversion. **Assuming checksum failure means data loss.** A mismatched checksum means the bytes differ, which could be a truncation, an encoding change, or an intentional transformation. Diagnose before re-uploading, and never "fix" a mismatch by regenerating the manifest on the moved copy without understanding the difference. **Ignoring the license.** Data licensed for noncommercial use cannot simply be republished under a broader license at the new repository, and third party components inside the dataset may carry their own terms. Resolve rights before transfer, not after publication. **Skipping the sensitive data screen.** Identifiers hide in unexpected places, including free text fields, filenames, and device serial numbers. Run the field level review in Step 5 even when you believe the dataset is fully de-identified. **Trusting a single copy at any point during the move.** Keep the source intact until the destination is verified and published. Delete the source record only after the new identifier resolves and the audit passes. **Migrating manually through a web browser.** Browser uploads give you no checksums, no retry logic, and no log. For anything more than a few dozen files, use the API. **One limitation on scope.** This guide covers open and internally held research data. Individual datasets carrying regulatory restrictions, clinical identifiers, or third party agreements need review by your institution's data governance or compliance office before any migration, and the timeline will be set by that review rather than by the transfer itself. ## Frequently Asked Questions ### What does it mean to migrate data between repositories? It means moving a complete dataset, including files, metadata, and documentation, from one data repository to another while preserving integrity and provenance, then updating identifiers and citations so users can find the data at its new home. ### When should I migrate a dataset to a new repository? Migrate when your funder or journal's data sharing requirements change, when your current repository is being retired, when you need curatorial services, stronger access controls, or a domain specific metadata schema the current home cannot provide, or when consolidating a lab's outputs after a move. ### How do I choose between a domain repository and a generalist one? Use the comparison criteria that matter for your case: persistent identifiers, metadata schema fit, license support, access controls, API quality, and preservation commitment. Domain repositories offer community recognized schemas and curation, generalists offer speed and breadth, and the best choice is the one whose ingest requirements your data can genuinely meet. ### Do I have to reformat my data before migrating? No, but migration is the cheapest time to do it. Copy original files unchanged, and separately convert anything that needs an open format or a community standard such as BIDS for neuroimaging or EDF for physiological signals, documenting each conversion in the data dictionary. ### How do I prove nothing was corrupted during the transfer? Generate a SHA-256 checksum manifest before the move, compare server reported checksums during upload, then download the published copy and recompute checksums against the original manifest. A matching count of OK results across all files is your verification. ### What metadata do I need to bring to the new repository? At minimum, title, creators, date range, methods, license, and a data dictionary defining every variable. Map each source field to the destination schema before transfer, and adopt community protocols and terminologies where they exist so cross study reuse stays possible. ### Can I migrate sensitive or human subject data the same way? You can move it, but the destination must support the access model your data requires, such as controlled access with authentication and data use agreements. Classify every file and variable before transfer, and route anything with regulatory restrictions through your institution's compliance review first. ### Can a migration be automated, and how long does it take? Yes, the hashing and upload steps in this guide are scripts you can rerun and scale to tens of thousands of files. A small dataset takes an afternoon, while large or restricted collections are governed by metadata mapping, compliance review, and validation cycles rather than by transfer speed. ## Sources 1. [Global review of shorebird tracking data to identify research gaps and conservation priorities.](https://pubmed.ncbi.nlm.nih.gov/41536012/) 2. [Characterizing core outcomes of responsible stewardship for human genomic data in the cloud.](https://pubmed.ncbi.nlm.nih.gov/42092915/) 3. [Benchmarking of cell type deconvolution techniques for microarray data.](https://pubmed.ncbi.nlm.nih.gov/42997613/) 4. [Democratizing data sharing and processing of routine MRI data acquired for neuroimaging research.](https://pubmed.ncbi.nlm.nih.gov/42991042/) 5. [PhenX Toolkit: a knowledgebase that enables consistent data collection and facilitates cross-study analysis and data reuse.](https://pubmed.ncbi.nlm.nih.gov/42973429/) 6. [Building a Sleep Data Repository: Architecture, Data Sources, and Governance.](https://pubmed.ncbi.nlm.nih.gov/42977354/) 7. [Estimating complete migration probabilities from grouped data: A methods protocol for developing a global Human Internal Migration Database.](https://pubmed.ncbi.nlm.nih.gov/39656704/) 8. [Integrating animal tracking and trait data to facilitate global ecological discoveries.](https://pubmed.ncbi.nlm.nih.gov/39973193/) 9. [The collection and integration of data on migrants in health information systems: evidence from Ireland.](https://pubmed.ncbi.nlm.nih.gov/41275258/) 10. [SHoRT-CA: Smart home data repository of time series with contextual anomalies.](https://pubmed.ncbi.nlm.nih.gov/42999780/) 11. [Scientific slop: generative AI, research incentives, and the integrity of the genomic knowledge base.](https://pubmed.ncbi.nlm.nih.gov/43012003/) 12. [The ethical PROMISE NFR dataset: data for ethical concern classification in software requirements.](https://pubmed.ncbi.nlm.nih.gov/42985665/) 13. [The Evolving Bileome: A Pan-Repository Landscape of Expanded Bile Acid and Related Steroidal Signatures.](https://pubmed.ncbi.nlm.nih.gov/42999269/)