# Metadata Creation for Data Repositories: A Field-by-Field Guide By the end of this guide you will be able to build a complete, validated metadata record for a research dataset, choose the right format for your field, fill in every field a repository reviewer will check, and catch the errors that make datasets invisible in search. You will need a text editor or spreadsheet, a JSON schema validator (the Python package `jsonschema` works well), an ORCID iD for each author, the metadata template or schema published by your target repository, and about an afternoon for a first deposit. Metadata is [structured data](/blog/data-analysis/what-is-structured-data) about data. It records who created a dataset, what it measures, how it was produced, under what license it can be reused, and where it connects to papers, protocols, and samples. Repositories use these fields to index, filter, and display your dataset. When the fields are vague or missing, the dataset becomes practically invisible to other researchers, even though the underlying files are perfectly good. ## Why Metadata Decides Whether Your Data Gets Found An evaluation of six Canadian open health data repositories found that every one of them relied on metadata-based filtering to let users reach open health data, and all of them exposed subject facets for filtering plus title and description on the results list [1]. In plain terms, the subject terms, title, and description you write are the fields a searcher actually queries. If your subject terms do not match the vocabulary your audience uses, your dataset will not appear in their filtered results no matter how complete the rest of the record is. The same evaluation found concrete gaps. Use case testing suggested improvements including advanced search, health-specific search terms, and links to related publications, which tells you that depositors who add publication links and precise subject terms by hand are compensating for what repositories still do poorly [1]. This matters beyond any one field. The Research Data Alliance COVID-19 working group built cross-disciplinary data sharing guidelines during the pandemic and highlighted a recurring theme: data must adhere to FAIR principles (Findable, Accessible, Interoperable, Reusable) while respecting legal and ethical considerations, and the Findable part starts with metadata [2]. The FAIR principles are community guidelines stating that research outputs should be described with rich metadata, assigned [persistent identifiers](/knowledge/bioinformatics/persistent-identifiers-in-bioinformatics-selection-criteria-and-implementation-workflow), and retrievable through standardized protocols. The cost of thin metadata shows up at reuse time. A systematic review of publicly available human metabolomic datasets related to COVID-19 identified 110 unique datasets and assessed each against 15 variables covering availability, accessibility, collection methodology, sample size, and participant metadata. The review concluded that information about accessibility, quality, and usability remains scattered and inconsistent, which directly limits how many of those datasets can actually be reused [3]. Missing metadata is what turns a published dataset into a dead dataset. ## The Core Field-by-Field Checklist Most repository schemas, whether specialized or generalist, ask for the same backbone of fields. A survey of data repositories used to build the Metadatasheet standard for biomedical scientists found that relevant information lives at different stages of the data lifecycle, and that defining it only at publication time leads to data loss and extra work [4]. Use the table below as your master checklist, then fill in the field groups that follow. | Field group | What it answers | Common pitfall | |, - |, - |, - | | Identifier and title | What is this dataset called | Vague titles with no method, organism, or condition | | Description | What was measured, where, and why | One-sentence abstracts with no sampling design | | Creators | Who did the work | No ORCID iDs, author order missing | | Subject and keywords | What topics does it cover | Free-text synonyms instead of controlled terms | | Dates and versions | When was it collected and released | Confusing collection date with publication date | | License and access rights | How can others reuse it | No license, which legally means "no reuse" | | Methods and instruments | How was the data produced | Missing instrument model, software, and versions | | Sample and provenance | What biological or physical material is behind each record | Missing organism, tissue, or sample identifiers | | Related outputs | Which paper or protocol does it support | No DOI link, so citation loops never close | ### Identifier and Title The title is the highest-weight field in almost every repository search index. A strong title names the data type, the organism or system, the condition or treatment, and the timeframe. Compare "Coral survey data" against "Photoquadrat benthic cover surveys of Caribbean reef sites, 2019 to 2021". The second title survives keyword filtering, full-text search, and human browsing. Write for someone who has never heard of your project, because that person is your future reuser. ### Description Write the description as a compact methods paragraph, not an abstract. State what was measured, with what instrument or assay, at how many sites or time points, and any [preprocessing](/blog/data-analysis/what-is-data-preprocessing) applied. Repositories index this field, and so do external harvesters that crawl repository APIs. Three to eight specific sentences beats a vague paragraph every time. ### Creators and Persistent IDs List every creator in a consistent order with an ORCID iD where possible. ORCID is a persistent identifier service that gives each researcher a unique 16-digit number, which prevents the classic problem of three "J. Smith" entries being merged or split incorrectly. Repository APIs, including those used by institutional Dataverse installations, return creator fields to harvesters, so an ORCID iD makes your authorship resolvable machine to machine. ### Subject Terms and Controlled Vocabulary This is where creating metadata that gets your data found is won or lost. Controlled vocabularies are fixed lists of approved terms, such as MeSH terms in health research or ontology terms in biomedicine. Because all six Canadian repositories evaluated relied on subject facets for filtering, depositing under free-text phrases that nobody else uses will strand your dataset outside the main browse path [1]. Look up how comparable datasets in your target repository describe their subjects and match that vocabulary, then add two or three free-text keywords for specificity. ### Dates and Versions Most schemas distinguish the date the data was collected, the date it was deposited, and the date it was published. Fill all of them separately. If you re-release the dataset after corrections, create a new version with a change note rather than silently overwriting files. Version history is what lets a reuser cite exactly the data state behind their analysis. ### License and Access Rights A dataset with no license statement is legally closed by default in most jurisdictions, which defeats the purpose of research data sharing. Choose a standard license (CC0 or a Creative Commons license for most open scientific data) unless funder or ethics agreements restrict reuse. For restricted data, state the access conditions and the contact or application process in the access rights field rather than leaving it blank. ### Methods, Instruments, and Software Record the instrument model, assay or protocol identifiers, and software with version numbers. This field group is where reuse lives or dies in computational fields. In molecular dynamics, for example, a tool called GROMACS MetaDump extracts the full set of simulation parameters (software version, force field, water model, box geometry, temperature) from simulation output files precisely because fragmented metadata across repositories hinders reuse [5]. The lesson generalizes: parameters you think are obvious from context are the ones a reuser cannot reconstruct. ### Sample-Level Metadata and Provenance Each biological sample behind your dataset needs its own description: organism, tissue, condition, and a stable identifier. The BioSamples database at EMBL-EBI exists exactly for this, serving as the centralized store of sample metadata for repositories including the European Nucleotide Archive, ArrayExpress, and the European Phenome-Genome Archive, which makes samples connectable across multi-omics studies [6]. If your target repository submits sample metadata to BioSamples or a similar hub, structure your sample sheet to match their attribute format from the start. Provenance also has an ethical dimension. For data derived from genetic resources connected to Indigenous Peoples and Local Communities, provenance metadata determines whether benefit sharing under the Nagoya Protocol framework can function at all. The Traditional Knowledge and Biocultural Labels initiative and its Biocultural Labels provide a practical mechanism for communities to attach origin and authority information to data held in research collections [7]. If your samples have community provenance, ask the community how they want that recorded before you deposit. ### Related Publications, Funding, and Protocols Link the dataset to the paper DOI, grant numbers, and any protocol registry entries. Closing this loop is not cosmetic. Repositories and knowledgebases use publication links to build citation networks, and the proteomics community's ProteomeXchange consortium showed how standardized submission and dissemination workflows enable large-scale reanalysis of public mass spectrometry data, in part because deposition connects cleanly to publications [8]. ## Choosing a Metadata Format ![Decision path asking whether the repository accepts a native template, then whether the data is omics, then whether web indexing matters.](/article-images/illustrations/metadata-creation-for-data-repositories-format-choice.webp) There is no single universal format. A crosswalk project between FHIR, OMOP, CDISC, and openEHR, the four major clinical data standards, concluded that no single predefined standard meets all requirements, and that a convergence format covering the maximum set of metadata items is the workable solution [9]. The same pattern holds across science. Your job is to match format to destination, and the table below summarizes the main options. | Format | Typical domain | Machine-readable | Best use | |, - |, - |, - |, - | | JSON Schema forms | Generalist, institutional repositories | Yes | Guided web-form entry, validation before deposit | | ISA-Tab | Metabolomics, omics assays | Yes (tabular) | Investigation-study-assay structure for MS pipelines | | YAML with validation | Omics research [data management](/blog/guides/data-management-basics-principles-processes-and-best-practices) | Yes | Dialog-driven creation, export to repository formats | | Schema.org (JSON-LD) | Web-indexed research outputs | Yes | Making datasets discoverable through search engines | | Repository-native templates | Domain repositories (GEO, ENA, ProteomeXchange) | Varies | The format the repository will actually accept | JSON Schema is a vocabulary for describing the structure and constraints of JSON documents. Tools built on it, such as Adamant, turn any valid schema into an interactive web form so researchers can collect metadata from the conception of an experiment onward, with an API for integration with electronic lab notebooks and repositories [10]. Schema.org is a metadata vocabulary originally built for web pages that has become a route for making research outputs findable through ordinary search engines, and tools like Schema Playground let researchers browse Schema.org classes, extend them for biomedical use, and register custom classes so others can reuse them [11]. YAML is a human-readable text format for structured data. The FRED toolkit uses a machine-readable YAML format for omics metadata, providing guided dialog-based creation, semantic validation, cross-file search, and export directly to NCBI GEO submission format, with a demonstrated workflow on a published single-nucleus RNA-seq dataset [12]. ISA-Tab is a tabular standard that organizes metadata as Investigation, Study, and Assay files, and metabolomics pipelines store experimental and biological metadata in it so that data processing flows straight through to public repository submission [13]. Two principles help you choose. First, the format that matters is the one your destination repository accepts, so check its submission documentation before writing anything. Second, open-source development of standards has real advantages. An analysis drawing on astronomy, high-energy physics, earth science, and neuroscience argues that the open-source software model provides significant benefits for standard creation and adaptation, while flagging the tension between flexibility and stability that depositors should expect from any evolving standard [14]. ## Worked Walkthrough: Building, Validating, and Depositing a Metadata Record ![Six-step flow: get the repository schema, draft the record, validate, auto-extract, deposit, then verify the record after deposit.](/article-images/illustrations/metadata-creation-for-data-repositories-deposit-walkthrough.webp) This walkthrough creates a metadata record for a [small RNA-seq](/knowledge/molecular-biology/small-rna-seq) dataset, validates it against a JSON schema, and deposits it. The same logic applies to any domain. The example doubles as a metadata creation for data repositories example you can adapt. ### Step 1: Get the Repository Schema Download the submission template from your target repository. If your repository supports JSON Schema, save it as `schema.json`. If it only offers a spreadsheet, still work through the same fields, because validation logic transfers. For omics data destined for NCBI GEO, the FRED toolkit generates its YAML files and exports them to GEO submission format, which removes a whole layer of manual reformatting [12]. ### Step 2: Draft the Record Create `record.json` with the core fields: ```json { "title": "Bulk RNA-seq of murine hippocampus under sleep deprivation, 2024", "description": "RNA-seq of hippocampal tissue from 12 adult mice, 6 sleep-deprived and 6 controls. Libraries prepared with polyA selection and sequenced on an Illumina NovaSeq 6000, 150 bp paired-end. Reads aligned to the GRCm39 reference. Raw FASTQ files and count matrix included.", "creators": [ {"name": "Rivera, Ana", "orcid": "0000-0002-0000-0000"}, {"name": "Okafor, Chidi", "orcid": "0000-0003-0000-0000"} ], "subjects": ["neuroscience", "[transcriptomics](/knowledge/bioinformatics/modern-transcriptomics-bulk-single-cell-spatial)", "sleep biology"], "dates": {"collected": "2024-03-01", "deposited": "2026-11-11"}, "license": "CC0-1.0", "methods": { "instrument": "Illumina NovaSeq 6000", "library_prep": "polyA selection", "reference_genome": "GRCm39" }, "samples": [ {"sample_id": "SD-01", "organism": "Mus musculus", "tissue": "hippocampus", "condition": "sleep deprived"}, {"sample_id": "CT-01", "organism": "Mus musculus", "tissue": "hippocampus", "condition": "control"} ], "related_publications": [] } ``` Notice the title carries data type, organism, tissue, condition, and year. The description states sample size, library prep, sequencer, read length, and reference genome. Those specifics are what a reuser needs to decide within thirty seconds whether this dataset fits their meta-analysis. ### Step 3: Validate Against the Schema Install the validator and run it: ```bash pip install jsonschema jsonschema --instance record.json schema.json ``` A clean record prints nothing and exits with status 0. If your schema requires a `creators` field and you deleted it, you will see: ```text ValidationError: 'creators' is a required property Failed validating 'required' in schema: {'properties': {...}, 'required': ['title', 'creators', 'license']} On instance: {'title': 'Bulk RNA-seq ...', 'description': '...'} ``` The fix is to re-add the missing property. The three validation errors you will hit most often are missing required properties, wrong types (a date written as plain text where the schema expects a string with a date pattern), and enum mismatches (a license written as "CC0" where the schema expects the exact string "CC0-1.0"). Read the validator message top to bottom. It names the failed keyword, the schema rule, and the offending instance value. If your repository instead provides a spreadsheet template, an alternative is the Metadata Workbook approach developed for biomedical scientists, which records metadata in Microsoft Excel with automation, dynamic adaptation, metadata integrity checks, and export options to multiple standards [4]. That fits labs that already run their sample tracking in spreadsheets. ### Step 4: Auto-Extract What You Can Manual typing introduces errors, so extract machine-readable metadata where the software already knows it. The GROMACS MetaDump tool, available as a web application, API service, and command-line utility, generates human- and machine-readable metadata directly from simulation input files [5]. The general habit transfers: pull parameters from instrument export files, run logs, and configuration files rather than retyping them. Automated drafting with language models is also becoming practical, with an important caveat. The Pre-Meta pipeline augments retrieval-augmented generation with prior metadata tags and ontologies, and in validation across five metadata fields sampled from 1500 papers it improved annotation accuracy by 23% for GPT-4o mini, 72% for Llama 8B, and 75% for Mistral 7B compared with conventional retrieval approaches [15]. Those are measured gains over a weak baseline, not guarantees. Treat model-generated metadata as a draft that a domain expert verifies field by field, especially for sample conditions and organism names. ### Step 5: Deposit and Time It Prepare your data files, upload them with the validated metadata, and submit. Realistic timing is encouraging. A librarian-led project at McGill University created a Dataverse collection for knowledge synthesis search strategies, spending about six hours on the initial data management plan and documentation templates, after which each subsequent deposit takes roughly 5 to 10 minutes on average [16]. The first deposit is the expensive one. Templates you build for your lab pay for themselves by the second dataset. Some platforms add structured metadata entry at upload. The Biologging intelligent Platform (BiP) lets data owners interactively upload sensor data and input metadata associated with individual animals, devices, and deployments, then standardizes the data to internationally recognized formats and offers open or private sharing settings [17]. Platforms designed around metadata capture (animal, device, deployment as separate linked entities) are worth preferring over platforms that treat metadata as one free-text box. ### Step 6: Verify the Record After Deposit Do not assume the upload worked. Check four things. 1. Search the repository for your title and confirm the record appears. 2. Filter by your subject terms and confirm the record survives the filter, because subject facets are the main browse path in evaluated repositories [1]. 3. Download your own files through the public interface and confirm they open. 4. Check that your sample-level metadata resolved to the right entries if your repository feeds a hub such as BioSamples [6]. For domain-specific records, verify against a domain library where possible. The WFSR Food Safety Mass Spectral Library, for instance, provides comprehensive per-compound metadata including common names, CAS registry numbers, SMILES, InChIKeys, retention times, and compound classes alongside its 6993 manually curated spectra from 1001 compounds, and it reached GNPS and other repositories in a form that allowed direct comparison [18]. Records rich enough to be cross-compared field by field are the standard to aim for. ## Common Mistakes and Limitations **Writing metadata only at publication time.** The strongest documented failure mode is collecting metadata information only when the dataset is about to be published, which causes data loss and extra workload because early-stage context (which sample went into which run, which batch, which operator) is gone by month six [4]. Capture fields at the experiment's conception instead, which is the design goal of schema-driven form tools built for exactly this stage [10]. **Free-text keywords instead of controlled terms.** Because repository filtering runs on subject facets, idiosyncratic keyword choices put your dataset outside the browse paths searchers actually use [1]. Copy vocabulary from comparable deposited records before inventing your own terms. **Describing the samples but not the deployment, or vice versa.** In sensor-based biology, the useful unit of metadata is often the deployment (which device on which animal over which interval), not just the animal [17]. Similarly, omics studies need assay-level context, not just sample-level. Think about the unit of reuse in your field and make sure a metadata entity exists for it. **Assuming automation replaces judgment.** Language model annotation shows measurable accuracy gains when augmented with priors, but the Pre-Meta results are averages across selected fields and models, and no reported configuration removes the need for human review of biomedical annotations [15]. Automated extraction from files, as done for simulation parameters [5], is safer than generative drafting because it copies facts rather than predicting them. **One-off schemas that fragment the landscape.** Custom schemas per repository create the discoverability problems that motivated crosswalk projects in clinical metadata [9] and the open-source, community-maintained standard development argued for across four disciplines [14]. Before writing a schema from scratch, check whether an existing standard covers your case, and extend rather than fork when you can. Tools such as Schema Playground exist to extend standard classes while keeping them registered and reusable by others [11]. **Ignoring ethical provenance fields.** For data with Indigenous or community provenance, omitting origin and authority information is not just incomplete, it forecloses the benefit sharing arrangements that frameworks like the Nagoya Protocol anticipate for genetic resources and related data [7]. Include provenance labels when the community has specified them. **Expecting perfect comparability across repositories.** Even well-built repositories differ in search features, metadata element counts, and filtering mechanisms, as the six-repository Canadian evaluation documented [1]. A data repository comparison should weigh metadata schema richness, controlled vocabularies, API access, license support, and whether sample-level metadata integrates with cross-repository hubs [6]. Choose based on where your reusers already search, not only on where submission is easiest. ## Frequently Asked Questions ### What is metadata creation for data repositories? Metadata creation is the process of writing the structured descriptive fields (title, description, creators, subjects, dates, license, methods, sample details, and links) that a data repository needs to index, display, filter, and license your dataset for reuse. ### Can you give an example of a filled-in metadata record? A complete example would include a title naming data type, organism, condition, and year, a methods-style description with sample size and instrument model, creators with ORCID iDs, controlled subject terms, separate collection and publication dates, a license such as CC0-1.0, per-sample entries with organism and tissue, and a DOI link to the related paper. The JSON block in the walkthrough above is a working template you can copy and edit. ### What format should my metadata be in? Use whatever format your target repository's submission system accepts, because that is the format that gets ingested. Common options are JSON Schema-driven forms, ISA-Tab for omics assays, YAML with validation for omics workflows, and Schema.org for web-indexed research outputs. Well-designed tools can convert between formats, so pick the one your destination requires and capture fields early in that structure. ### Is there a template I can reuse across projects? Yes. Build a lab-level template once, covering the backbone fields in the checklist table plus your domain's instrument and sample fields, then fill it per dataset. Spreadsheet-based metadata standards designed for biomedical labs let you record metadata continuously through the project and export to repository formats at submission time. ### How long does a repository deposit actually take? The first deposit takes longer because you design your documentation and workflow. After templates exist, reported real-world deposits take on the order of minutes per dataset, so the upfront investment pays off quickly across a lab's output. ### How do I make my dataset findable in repository search? Match your subject terms to the controlled vocabulary or facet terms the repository uses, write a title rich in data type, organism, and condition, write a specific description, and link the related publication. Repository search and filtering run on these fields, so precision there drives discoverability. ### Can AI tools generate my metadata for me? AI-assisted annotation can help, and studies show accuracy gains when language models are augmented with prior tags and ontologies, but treat the output as a draft requiring field-by-field expert review. Automated extraction of parameters directly from instrument or software files is generally more reliable than generative drafting. ### What should I do if my data has provenance or ethical restrictions? Record access conditions explicitly in the access rights field, apply any community-specified provenance or biocultural labels requested by source communities, and never publish restricted data under an open license. If ethics approvals limit sharing, describe the dataset with metadata while directing reusers to the documented access procedure. ## Sources 1. [Challenges with organization, discoverability and access in Canadian open health data repositories.](https://pubmed.ncbi.nlm.nih.gov/35949503/) 2. [Fostering global data sharing: highlighting the recommendations of the Research Data Alliance COVID-19 working group.](https://pubmed.ncbi.nlm.nih.gov/33501381/) 3. [Metabolomic datasets in COVID-19 research: a systematic literature review of availability, characteristics, and methodologies.](https://pubmed.ncbi.nlm.nih.gov/42438709/) 4. [From Planning Stage Towards FAIR Data: A Practical Metadatasheet For Biomedical Scientists.](https://pubmed.ncbi.nlm.nih.gov/38778016/) 5. [Gromacs MetaDump: a tool for extracting GROMACS simulation metadata.](https://pubmed.ncbi.nlm.nih.gov/41131648/) 6. [BioSamples database: the global hub for sample metadata and multi-omics integration.](https://pubmed.ncbi.nlm.nih.gov/41231754/) 7. [Benefit sharing: Why inclusive provenance metadata matter.](https://pubmed.ncbi.nlm.nih.gov/36212139/) 8. [Proteomic repository data submission, dissemination, and reuse: key messages.](https://pubmed.ncbi.nlm.nih.gov/36529941/) 9. [Harvesting metadata in clinical care: a crosswalk between FHIR, OMOP, CDISC and openEHR metadata.](https://pubmed.ncbi.nlm.nih.gov/36307424/) 10. [Adamant: a JSON schema-based metadata editor for research data management workflows.](https://pubmed.ncbi.nlm.nih.gov/35707001/) 11. [Schema Playground: a tool for authoring, extending, and using metadata schemas to improve FAIRness of biomedical data.](https://pubmed.ncbi.nlm.nih.gov/37081398/) 12. [FRED enables standardized FAIR metadata generation and management for omics research.](https://pubmed.ncbi.nlm.nih.gov/42498726/) 13. [MetaDB a Data Processing Workflow in Untargeted MS-Based Metabolomics Experiments.](https://pubmed.ncbi.nlm.nih.gov/25566535/) 14. [Open-source models for development of data and metadata standards.](https://pubmed.ncbi.nlm.nih.gov/40926966/) 15. [Pre-Meta: priors-augmented retrieval for LLM-based metadata generation.](https://pubmed.ncbi.nlm.nih.gov/40973196/) 16. [Searches as data: archiving and sharing search strategies using an institutional data repository.](https://pubmed.ncbi.nlm.nih.gov/38911529/) 17. [Biologging intelligent Platform (BiP): an integrated and standardized platform for sharing, visualizing, and analyzing biologging data.](https://pubmed.ncbi.nlm.nih.gov/40170100/) 18. [Creation of an Open-Access High-Resolution Tandem Mass Spectral Library of 1000 Food Toxicants.](https://pubmed.ncbi.nlm.nih.gov/41070935/)