Zubair Khalid

Virologist/Molecular Biologist | Veterinarian | Bioinformatician

Conventional & Molecular Virology • Vaccine Development • Computational Biology

Dr. Zubair Khalid is a veterinarian and virologist specializing in conventional and molecular virology, vaccine development, and computational biology. Dedicated to advancing animal health through innovative research and multi-omics approaches.

Dr. Zubair Khalid - Veterinarian, Virologist, and Vaccine Development Researcher specializing in Computational Biology, Multi-omics, Animal Health, and Infectious Disease Research

Category: Guides

Version Control Systems for Research: A Comparison

Researchers in the life sciences and other fields increasingly depend on computational workflows that generate data, code, and documents. A version control system (VCS) tracks iterative changes to files, allowing you to experiment with new ideas while retaining the ability to revert to a specific past version used to produce particular results. This article compares three widely used version control tools, Git, Mercurial, and Subversion, with a focus on features relevant to research projects. The comparison is intended for students, researchers, life-science professionals, and informed general readers who need to choose a tool for tracking changes in research code, data, and manuscripts.

Why Research Projects Need Version Control

Scientific computing has become essential across all branches of science, yet most researchers are never taught the equivalent of basic lab skills for research computing. As a result, data can get lost, analyses can take much longer than necessary, and researchers are limited in how effectively they can work with software and data. Computing workflows need to follow the same practices as lab projects and notebooks, with organized data, documented steps, and the project structured for reproducibility.

Version control addresses several problems that arise when iteratively developing code. Without a VCS, researchers often make copies of scripts before making new edits, which clutters file systems with uninformative filenames such as analysis.sh, analysis_02.sh, and analysis_03.sh. It becomes difficult to remember the differences between versions and, more importantly, which version produced specific results, especially when returning to the code months later. When code is shared with multiple lab mates or collaborators, emailing files to multiple people forces manual incorporation of all the changes each person sends.

A version control system solves these issues by tracking iterative changes to code and other files. Researchers can record messages as they save each successive version so that anyone reviewing the development history can understand the rationale for given edits. This capability directly supports research reproducibility, which has been the subject of extensive debate and policy attention from funders, publishers, and research institutions.

At a Glance: Git, Mercurial, and Subversion Compared

The table below summarizes the key differences between the three tools for research use.

Feature Git Mercurial Subversion
Repository model Distributed, each clone has full history Distributed, each clone has full history Centralized, single server holds history
Offline commits Yes, commits are local until pushed Yes, commits are local until pushed No, commits require server connection
Learning curve Steep, many commands and concepts Moderate, simpler command set Moderate, familiar to those used to centralized systems
Branching and merging Powerful but complex, many workflows Simpler branching model Branching is directory-based, merging can be difficult
Binary file handling Works but repository grows quickly Works but repository grows quickly Works, central server stores all versions
Large data support Poor without extensions like Git LFS Poor without extensions Poor, server storage grows with every version
Ecosystem and hosting GitHub, GitLab, Bitbucket, extensive tooling Bitbucket, Heptapod, smaller ecosystem Apache infrastructure, smaller modern ecosystem
Best fit for research Individual or team projects with code focus Teams wanting distributed control with simpler commands Legacy projects or environments requiring centralized control

Core Principles of Version Control

Tracking Changes and Commit History

A version control system records snapshots of files at specific points in time. Each saved snapshot is called a commit, and each commit carries a message describing what changed and why. This history allows researchers to understand how a project evolved and to identify which version of the code generated a particular result. The importance of this capability is well documented in scientific computing guidance, which emphasizes that computing workflows need organized data and documented steps structured for reproducibility.

Branching and Merging

Branching creates a separate line of development from the main project. This allows researchers to experiment with new ideas without risking the currently working code. If the experiment succeeds, the branch can be merged back into the main line. If it fails, the branch can be discarded without affecting the main project. The ability to experiment safely is one of the primary reasons researchers adopt version control.

Collaboration and Conflict Resolution

When multiple researchers work on the same files, version control systems manage the integration of changes. Each collaborator works on their own copy of the project, and the system helps merge changes when they are shared. When two people modify the same lines of a file, a conflict occurs, and the system requires manual resolution. This process is more reliable than emailing files back and forth, which risks losing changes or overwriting someone else's work.

Reproducibility and Provenance

Version control provides a direct link between a specific result and the exact code and data that produced it. By recording the commit identifier associated with an analysis run, researchers can reconstruct the computational environment at any later time. This practice aligns with broader reproducibility efforts in the research community, where funders, publishers, and research institutions have adopted policies aimed at increasing research reproducibility.

Practical Workflow for Research Projects

Step 1: Initialize the Repository

The first step is to create a repository in the project directory. In Git, this is done with the init command. In Mercurial, the equivalent is hg init. In Subversion, you create a repository on a server and check out a working copy. The choice of tool affects where the repository lives and how collaborators access it.

Step 2: Structure the Project Directory

Before tracking files, decide on a project structure. Research on data analysis project structures shows that templates show little overlap, but many distinct practices can be highlighted. A consensus structure typically includes folders for raw data, processed data, code, figures, manuscripts, and documentation. The Research Data Framework from the National Institute of Standards and Technology provides guidance on managing research data throughout its lifecycle.

Step 3: Track Files and Make Commits

Add files to version control and make commits at meaningful points. Each commit should represent a logical unit of work, such as adding a new analysis script, fixing a bug, or updating a figure. Write clear commit messages that explain what changed and why. This practice makes the history understandable to anyone reviewing the project later.

Step 4: Use Branches for Experiments

Create a branch when experimenting with new ideas or major changes. This keeps the main line of development stable while allowing exploration. In Git, branching is a common workflow. In Mercurial, branching is simpler but less flexible. In Subversion, branches are directories within the repository, which can make merging more complex.

Step 5: Share and Collaborate

Push commits to a shared remote repository or pull changes from collaborators. The distributed model of Git and Mercurial allows each researcher to work independently and share changes when ready. Subversion requires a central server, so all commits go directly to the shared repository.

Step 6: Tag Releases and Key Results

Use tags to mark important points in the project history, such as versions of a manuscript submitted to a journal or versions of code used for a published analysis. Tags make it easy to return to a specific state of the project.

Step 7: Document the Workflow

Record the version control commands used and the commit identifiers associated with each analysis run. This documentation supports reproducibility by allowing others to reconstruct the exact computational steps.

Git: The Dominant Tool in Research

Strengths for Research Use

Git is the most widely used version control system in scientific computing. Its distributed model means every clone contains the full history of the project, allowing researchers to work offline and commit locally. This is particularly valuable for researchers who travel or work in environments with unreliable internet access.

The ecosystem around Git is extensive. Hosting platforms such as GitHub and GitLab provide web interfaces, issue tracking, and collaboration features that integrate with the version control workflow. Many scientific journals and repositories accept code hosted on these platforms, and researchers can share analysis code with reviewers and readers through them.

Git's branching and merging capabilities are powerful, supporting complex workflows that involve multiple collaborators working on different features simultaneously. The ability to create experimental branches and merge them back into the main line is essential for research projects that evolve iteratively.

Limitations for Research Use

Git has a steep learning curve. The command set is large, and concepts such as staging areas, remotes, and rebasing can be confusing for new users. Researchers who are not trained in software development may find Git intimidating, and the risk of making mistakes that are difficult to undo can be a barrier to adoption.

Git handles binary files poorly. Every version of a binary file is stored in the repository, so repositories grow quickly when tracking images, data files, or other non-text content. Extensions such as Git Large File Storage (LFS) can mitigate this problem, but they add complexity and require additional setup.

For researchers working with large datasets, Git is not an ideal solution. The repository size grows with every version of every file, and cloning a large repository can be slow. Researchers working with big data may need to keep data files outside the repository and track only code and small configuration files.

When to Choose Git

Choose Git when working on code-intensive projects, collaborating with others who already use Git, or planning to share code through public repositories. Git is also the best choice when you need to work offline and want the full history available on every clone.

Mercurial: A Simpler Distributed Alternative

Strengths for Research Use

Mercurial offers the same distributed model as Git, with each clone containing the full history of the project. The key difference is that Mercurial has a simpler command set and a gentler learning curve. Researchers who find Git confusing may find Mercurial more approachable.

Mercurial's branching model is more straightforward than Git's. Branches are permanent and named, which makes the history easier to understand. This can be an advantage for research projects where clarity of history is more important than flexibility.

Mercurial handles merges well, and the tool is known for its stability and reliability. The simpler design means fewer ways to make mistakes, which is valuable for researchers who are not professional software developers.

Limitations for Research Use

The Mercurial ecosystem is smaller than Git's. Hosting platforms are fewer, and integration with other tools is less extensive. While Bitbucket supports Mercurial, the platform has shifted its focus to Git, and the long-term availability of Mercurial hosting is uncertain.

Mercurial has the same binary file handling limitations as Git. Repositories grow quickly when tracking non-text files, and large data support requires extensions.

The smaller community means fewer tutorials, fewer answers to questions, and less familiarity among potential collaborators. If your collaborators use Git, adopting Mercurial creates friction in collaboration.

When to Choose Mercurial

Choose Mercurial when you want distributed version control with a simpler learning curve, when your team already uses Mercurial, or when you value a stable and predictable tool over ecosystem size.

Subversion: The Centralized Option

Strengths for Research Use

Subversion uses a centralized model, with a single server holding the authoritative repository and working copies on each user's machine. This model is familiar to researchers who have used older version control systems, and it can be easier to understand conceptually.

Subversion handles binary files more gracefully than distributed systems in some respects. The central server stores all versions, and users check out only the files they need. However, the server storage grows with every version, and large binary files can quickly consume disk space.

Subversion's directory-based branching model is intuitive. Branches are simply directories in the repository, and the structure is visible to all users. This can be easier to understand than the branch models of Git and Mercurial.

Subversion has been used in scientific projects for many years. The presentation on using Subversion for Community Earth System Model users and scientists demonstrates that centralized version control has a place in research environments where a central authority manages the codebase.

Limitations for Research Use

Subversion requires a server connection for commits. Researchers cannot commit changes offline, which is a significant limitation for those who work in the field or travel. The centralized model also means that the server is a single point of failure, and the entire project history is lost if the server is not backed up.

Merging in Subversion is more difficult than in distributed systems. Because branches are directories, the system does not track the relationship between branches as carefully as Git or Mercurial. Merging changes from one branch to another can require manual intervention and careful attention.

The Subversion ecosystem is smaller than Git's, and modern hosting options are limited. While Apache provides infrastructure for Subversion, the tool is less integrated with modern collaboration platforms.

When to Choose Subversion

Choose Subversion when your organization already uses it, when you need centralized control over the repository, or when you work in an environment where a central server is available and offline commits are not needed.

Options and Tradeoffs for Research Data

Tracking Code Versus Data

Version control systems are designed for text files, and they work best for code, configuration files, and documents. Binary data files, such as images, spreadsheets, and proprietary formats, are stored as complete copies in every version. This causes repositories to grow quickly and can make cloning and backup slow.

For research projects, a common practice is to track code and small configuration files in version control while storing large data files separately. The Research Data Framework from the National Institute of Standards and Technology provides guidance on managing research data throughout its lifecycle, including storage, backup, and sharing.

Large File Extensions

Git Large File Storage (LFS) replaces large files with text pointers in the repository and stores the actual file content on a separate server. This keeps the repository small while still tracking versions of large files. Mercurial has similar extensions, and Subversion can be configured to exclude large files from version control.

These extensions add complexity and require additional setup. Researchers must decide whether the benefits of tracking large files in version control outweigh the added complexity.

Reproducibility and Data FAIRness

Version control systems can support the FAIR data principles, which emphasize findability, accessibility, interoperability, and reusability. Research on enabling data FAIRness of version control systems explores how these tools can be used to make research data more accessible and reusable.

The practical implication is that version control alone is not sufficient for data management. Researchers need to combine version control with data documentation, metadata, and storage practices to ensure that data are findable and reusable by others.

Observations and Measurements for Tool Selection

Assessing Your Project Needs

Before choosing a version control tool, assess your project needs based on the following factors:

  • Team size and distribution: Are collaborators in the same lab or across institutions?
  • Offline work requirements: Do you need to commit changes without internet access?
  • File types: Are you tracking code, documents, binary data, or a mix?
  • Project size: How large are the files, and how many versions will you create?
  • Collaboration workflow: Do you need branching, code review, or issue tracking?
  • Hosting requirements: Do you need a public repository for sharing code with reviewers?

Measuring Repository Growth

Track the size of your repository over time to understand how your version control choice affects storage. A repository that grows quickly may require a different approach, such as excluding large files or using a large file extension.

Recording Commit Patterns

Monitor how often you and your collaborators commit changes. Regular commits with clear messages indicate a healthy workflow. Infrequent commits with vague messages suggest that the team needs better version control practices.

Evaluating Collaboration Efficiency

Measure how long it takes to integrate changes from multiple collaborators. Frequent merge conflicts indicate that the team needs to coordinate better or that the version control tool is not well suited to the workflow.

Records and Measurements for Version Control Use

Commit Logs

The commit log is the primary record of project history. Each commit should include a timestamp, author, message, and a list of changed files. Review the commit log regularly to ensure that messages are clear and that changes are logically grouped.

Tag and Release Records

Tags mark important points in the project history. Record the tag names and commit identifiers associated with manuscript submissions, published analyses, and other key events. This information supports reproducibility by linking results to specific versions of code and data.

Branch and Merge Records

Document the branching strategy and record when branches are created and merged. This information helps collaborators understand the project structure and avoids confusion about which branch contains which changes.

Backup Records

Maintain backups of the repository, especially for centralized systems like Subversion where the server holds the only copy of the history. For distributed systems, each clone is a backup, but regular backups of the primary repository are still recommended.

Common Failure Patterns in Research Version Control

Committing Large Binary Files

Researchers often commit data files, images, and other binary content without considering the impact on repository size. This leads to slow clones, large backups, and difficulty sharing the repository. The solution is to exclude large files from version control or use a large file extension.

Unclear Commit Messages

Commit messages such as "update" or "fix" provide no context for future readers. Without clear messages, the history is difficult to understand, and researchers cannot determine which version produced a specific result. The solution is to write descriptive messages that explain what changed and why.

Infrequent Commits

Researchers sometimes work for days or weeks without committing changes. This creates large commits that are difficult to review and increases the risk of losing work. The solution is to commit regularly at logical points in the workflow.

Merge Conflicts Ignored

When merge conflicts occur, some researchers resolve them by overwriting files or discarding changes. This can lead to lost work and incorrect results. The solution is to resolve conflicts carefully, understanding both sets of changes before merging.

No Branching Strategy

Some research teams work directly on the main branch without using branches for experiments. This makes it difficult to test new ideas without risking the working code. The solution is to use branches for experiments and merge only when the changes are stable.

Repository as Data Storage

Using the version control repository as the primary storage for large data files leads to repository bloat and performance problems. The solution is to store large data files separately and track only code, small configuration files, and documentation in version control.

Limitations of Version Control for Research

Not a Backup System

Version control tracks changes, but it is not a substitute for regular backups. A repository can be corrupted, a server can fail, and a clone can be lost. Researchers should maintain backups of the repository and the working files.

Not a Data Management System

Version control does not manage data quality, metadata, or data sharing. Researchers still need to document data collection, processing, and analysis steps. The Research Data Framework from the National Institute of Standards and Technology provides guidance on managing research data throughout its lifecycle.

Not a Collaboration Platform

Version control tracks changes, but it does not provide the full range of collaboration features found in platforms like GitHub or GitLab. Researchers may need additional tools for issue tracking, code review, and project management.

Not a Reproducibility Solution

Version control supports reproducibility by tracking code and data versions, but it does not capture the computational environment, software dependencies, or analysis parameters. Researchers need to combine version control with other practices, such as containerization and workflow documentation, to achieve full reproducibility.

Safety and Regulatory Context for Research Data

Data Integrity and Provenance

Version control contributes to data integrity by providing a complete history of changes. This is important for research that may be subject to audit or regulatory review. The ability to demonstrate how data and code evolved over time supports transparency and accountability.

Data Sharing and Open Science

Version control systems, particularly Git with public hosting platforms, support open science by making code and data available to the broader community. The EQUATOR Network provides reporting guidelines for health research, and sharing analysis code through version control supports the transparency that these guidelines promote.

Institutional Policies

Research institutions may have policies regarding data management, code sharing, and version control. Researchers should be aware of these policies and ensure that their version control practices comply with institutional requirements.

Professional Escalation Criteria

Seek professional guidance when:

  • Your repository has grown so large that cloning or backup is impractical
  • You need to track large data files and are unsure how to configure large file extensions
  • Your team is experiencing frequent merge conflicts that cannot be resolved
  • You need to share code with external collaborators and are unsure about hosting options
  • Your institution requires specific data management practices that version control alone cannot meet

Frequently Asked Questions

What is the difference between centralized and distributed version control?

Centralized version control, such as Subversion, uses a single server that holds the authoritative repository. Users check out working copies and commit changes directly to the server. Distributed version control, such as Git and Mercurial, gives every user a complete copy of the repository, including the full history. Users commit changes locally and push them to a shared remote repository when ready.

Which version control system is best for a solo researcher?

For a solo researcher, Git is often the best choice because of its extensive ecosystem and the availability of free hosting on platforms like GitHub. Mercurial is a good alternative if you prefer a simpler command set. Subversion is less suitable for solo research because it requires a central server.

Can I use version control for documents and manuscripts?

Yes, version control works well for text-based documents, including LaTeX manuscripts, Markdown files, and plain text notes. It is less suitable for binary formats like Word documents, where every version is stored as a complete file and the repository grows quickly.

How do I handle large data files in version control?

Large data files should generally be stored outside the version control repository. Track only code, small configuration files, and documentation in version control. If you need to track large files, use a large file extension such as Git LFS, which stores file content on a separate server and keeps the repository small.

What is a merge conflict and how do I resolve it?

A merge conflict occurs when two collaborators modify the same lines of a file and the version control system cannot automatically combine the changes. To resolve a conflict, open the file, review both sets of changes, and decide which changes to keep. After resolving all conflicts, commit the merged result.

How often should I commit changes?

Commit changes at logical points in your workflow, such as after adding a new analysis, fixing a bug, or updating a figure. Each commit should represent a coherent unit of work with a clear message. Regular commits make the history easier to understand and reduce the risk of losing work.

Do I need a hosting platform like GitHub?

A hosting platform is not required for version control, but it provides useful features such as web interfaces, issue tracking, and collaboration tools. For research projects, hosting platforms make it easier to share code with collaborators and reviewers. Free hosting is available for public repositories.

How does version control support research reproducibility?

Version control supports reproducibility by tracking the exact versions of code and data used to produce results. By recording the commit identifier associated with an analysis run, researchers can reconstruct the computational steps at any later time. This practice aligns with broader reproducibility efforts in the research community.

Related Articles

References and Further Reading

This article is educational and does not replace institutional policy, professional advice, or applicable safety and regulatory requirements.