How to De-identify Clinical Data for Sharing: A Practical Guide

By Dr. Zubair Khalid, DVM, MS, PhD · 2026-11-11

By the end of this guide you will be able to take a raw clinical extract, classify every variable by how identifying it is, remove or transform direct and indirect identifiers, run a re-identification check on the result, and deposit the cleaned dataset in a repository that matches its sensitivity. You will need Python 3.10 or newer with the pandas and numpy packages, a dataset you are authorized to work with, your ethics or data governance approval, and an account at the repository you plan to use. Every command below runs in a plain Python script or notebook.

Learning how to de-identify clinical data for sharing starts with a change in how you see the file. A patient table is not a pile of names to delete. It is a lattice of variables, some identifying on their own and some identifying only in combination, and the craft lies in transforming the second kind without destroying the science.

The payoff for that craft is measurable. A decade-long review of bioscience publishing at the University of Edinburgh found that the share of papers releasing all relevant data rose from 7 percent in 2014 to 45 percent in 2023, a shift driven by funder and journal expectations [1]. The pandemic exposed the cost of the opposite pattern, because fragmented and poorly connected health information systems slowed outbreak detection and response [2]. Patient-level health data sits at the sensitive end of this movement and demands more care than sequences or images, since re-identification risk, consent questions, and governance gaps make open-by-default the wrong default for most clinical datasets [3].

What De-identification Actually Means

De-identification is the umbrella term for removing or transforming personal identifiers so a record can no longer be linked to a specific person without disproportionate effort. Two techniques sit underneath the term, and confusing them causes most of the legal trouble in this field.

Pseudonymization replaces direct identifiers, such as a medical record number or a name, with a code. The mapping from codes back to identities is kept in a key stored separately from the data. Under European privacy law, pseudonymized health data still counts as personal data, because whoever holds the key can re-link it [4]. Pseudonymization reduces risk. It does not take a dataset outside privacy regulation.

Anonymization means the transformation is irreversible. Truly anonymized data no longer relates to an identifiable person, and privacy rules generally stop applying to it. For rich patient-level clinical data, genuine anonymization is difficult, because ordinary variables combine in ways that single people out.

That is the role of quasi-identifiers. A quasi-identifier is a variable that looks harmless alone, such as ZIP code, birth year, or sex, but identifies a person when combined with other quasi-identifiers. A 90-year-old woman with a rare diagnosis who lives in a small three-digit ZIP area may be the only such record in the file. k-anonymity is the standard check for this problem. A dataset is k-anonymous when every combination of quasi-identifier values appears at least k times, so no record stands alone. You will compute it in Step 5.

Check the Legal Frame Before You Touch the Data

United States. The federal rule for health data privacy, the HIPAA Privacy Rule, recognizes two de-identification routes. Safe Harbor requires removal of 18 categories of identifiers, including names, street addresses, phone and fax numbers, email addresses, Social Security numbers, medical record numbers, health plan and account numbers, certificate and license numbers, vehicle and device identifiers, web addresses, IP addresses, biometric identifiers, full-face photographs, and any other unique identifying code. Dates more specific than a year must be removed, ages above 89 must be collapsed into a single category, and ZIP codes may keep only their first three digits, and only where that three-digit area contains more than 20,000 people. The second route, expert determination, puts a qualified statistician in charge of measuring re-identification risk and certifying that it is very small. A middle option, the limited data set, keeps some dates and geography but requires a signed data use agreement, a contract that fixes who may use the data and for what.

European Union. The General Data Protection Regulation treats pseudonymized data as personal data, so secondary use of health data needs a defined purpose and a legal basis. Consent from data subjects matters, and narrow exceptions to the general ban on processing sensitive data exist, but member states read the shared rules differently and add national conditions, which complicates any plan to share across borders [4].

Before transforming anything, answer four questions in writing. Does your Institutional Review Board (IRB) approval or ethics protocol cover sharing outside the original study team? What did the consent forms say about future secondary use? Which route applies, Safe Harbor, expert determination, a limited data set, or a European legal basis? Which repository and access model will receive the data? Discovering after publication that the sharing itself was never permitted is the most expensive failure in this workflow, and consent scope is where it usually bites [4].

How to De-identify Clinical Data for Sharing, Step by Step

Step 1: Inventory and classify every variable

Work on a copy of the extract, in a folder that does not sync to cloud storage or backup services. Load the file and look at what you actually have.

import pandas as pd

df = pd.read_csv("clinic_extract_raw.csv")
print(df.shape)
print(df.head(3))

The output looks like this:

(1200, 14)
   mrn   name            zip    dob          admission_date  discharge_date sex age diagnosis      hba1c attending   notes
0  402138  Jane Doe    02139  1958-03-11  2024-01-05      2024-01-09     F   66  T2DM       8.1  Dr. Reyes  transferred from St. Mary's on 3 January
1  402139  John Smith  02140  1949-11-02  2024-01-07      2024-01-12     M   74  T2DM       7.4  Dr. Reyes  follow-up after ED visit
2  402140  Ana Lopez   02139  1972-06-30  2024-01-08      2024-01-08     F   51  prediabetes 6.0  Dr. Chen   new consult

Sort every column into one of five classes.

Free text deserves special suspicion. The first note above reintroduces a place, a month, and a day in a single sentence. Columns like it are the most common reason a technically de-identified file still leaks identities.

Step 2: Confirm the legal basis and consent scope

The legal section above gave you the four questions. Write the answers into your project README now, because the transformation choices in the next steps depend on them. If the route is Safe Harbor, you will delete identifiers outright and reduce dates to years. If the route is European pseudonymization, you will keep a keyed code and treat the file as personal data with a documented purpose and legal basis [4]. If the destination is a controlled-access repository, request the data use agreement template now, since committee review takes weeks.

Step 3: Remove direct identifiers and pseudonymize what must persist

Delete direct identifier columns, then create a study ID for the rows that must stay linkable across visits or files. A plain hash is not enough for this. Anyone can hash a list of guessed record numbers and match the results, so use HMAC, a hash-based message authentication code that mixes a secret key into the hash.

import pandas as pd
import hmac, hashlib, os

df = pd.read_csv("clinic_extract_raw.csv")

# Create the secret key once and store it outside the data folder
key_path = "../keys/study_key.bin"
os.makedirs("../keys", exist_ok=True)
if not os.path.exists(key_path):
    with open(key_path, "wb") as f:
        f.write(os.urandom(32))
with open(key_path, "rb") as f:
    salt = f.read()

def keyed_id(mrn: str) -> str:
    return hmac.new(salt, str(mrn).encode(), hashlib.sha256).hexdigest()[:16]

df["study_id"] = df["mrn"].map(keyed_id)
df = df.drop(columns=["mrn", "name", "attending"])
print(df[["study_id", "zip", "dob", "sex"]].head(3))

Output:

           study_id    zip          dob sex
0  3f9a1c77e2b04d08  02139  1958-03-11   F
1  a41d9e02c7b55f61  02140  1949-11-02   M
2  77b2c4e9d1a03c85  02139  1972-06-30   F

The key file lives on encrypted, access-controlled storage, never in the shared folder and never in the repository upload. Two legal notes follow from the definitions above. Under the Safe Harbor standard, a re-identification code must not be derived from the patient's own information, so a hash of the record number does not qualify and the identifier should simply be dropped. Under the European framework, the keyed code is exactly what pseudonymization means, and the dataset remains personal data that needs a documented purpose and legal basis [4].

Two errors show up here often. Dropping a column that was already removed raises KeyError: "['attending'] not found in axis", which usually means the name carries trailing whitespace, fixed by running df.columns = df.columns.str.strip() first. Filtering a DataFrame without .copy() later triggers A value is trying to be set on a copy of a slice from a DataFrame, so copy explicitly as the code above does.

Step 4: Transform quasi-identifiers

Shift every date for a given patient by the same random offset. Within-patient intervals, such as length of stay and time between lab draws, survive intact, while the link to real calendar dates is cut.

import numpy as np

rng = np.random.default_rng(42)  # log the seed in your README
df["offset_days"] = rng.integers(-365, 366, size=len(df))

for col in ["dob", "admission_date", "discharge_date"]:
    df[col] = pd.to_datetime(df[col]) + pd.to_timedelta(df["offset_days"], unit="D")

df = df.drop(columns=["offset_days"])

Worked example. Patient 0 was admitted on 2024-01-05 and discharged on 2024-01-09, a 4-day stay. With an offset of 187 days, the shifted pair becomes 2024-07-10 and 2024-07-14, still a 4-day stay. If a patient appears in several files, derive the offset from the study ID instead of drawing it at random, so every file shifts that patient by the same amount.

def stable_offset(study_id: str) -> int:
    h = int(hmac.new(salt, study_id.encode(), hashlib.sha256).hexdigest(), 16)
    return (h % 731) - 365  # offsets from -365 to 365 days

Mixed date formats are the common error here. The message ValueError: time data '1/5/24' does not match format '%Y-%m-%d' means one column mixes layouts, fixed by cleaning the column or passing an explicit format argument to pd.to_datetime.

Then band ages and truncate geography.

df["age"] = df["age"].where(df["age"] <= 89, 90)
df["age_band"] = pd.cut(
    df["age"],
    bins=[0, 17, 34, 49, 64, 79, 89, 90],
    labels=["0-17", "18-34", "35-49", "50-64", "65-79", "80-89", "90+"],
    include_lowest=True,
)
df["zip3"] = df["zip"].astype(str).str.zfill(5).str[:3]
print(df[["study_id", "age_band", "zip3", "sex"]].head(3))

Output:

           study_id age_band zip3 sex
0  3f9a1c77e2b04d08    65-79  021   F
1  a41d9e02c7b55f61    65-79  021   M
2  77b2c4e9d1a03c85    35-49  021   F

Everyone older than 89 lands in one 90+ category, which matches the Safe Harbor rule. ZIP codes reduce to three digits, which Safe Harbor permits only where that three-digit area holds more than 20,000 people. For a small catchment area, replace zip3 with a state code or drop it. If you keep shifted dates of birth, recompute age from the shifted date, or band age directly as above, so the file stays internally consistent.

Step 5: Test k-anonymity and suppress small cells

Now measure whether anyone stands alone.

qi = ["age_band", "zip3", "sex"]
sizes = df.groupby(qi).size()
print("smallest cell:", sizes.min())
print("cells of size 1:", (sizes == 1).sum())
print("records in cells under 5:", sizes[sizes < 5].sum())

Output:

smallest cell: 1
cells of size 1: 22
records in cells under 5: 187

Twenty-two records are unique on age band, ZIP3, and sex, which means 22 people could be singled out by an intruder who knows those three facts about them. Fix the problem with suppression, generalization, or both.

cell_size = df.groupby(qi)["study_id"].transform("size")
df_k = df[cell_size >= 5].copy()
print(len(df), "->", len(df_k))

Output:

1200 -> 1013

Suppression deletes records in small cells and costs rows. Generalization widens the bands instead, for example by replacing zip3 with state or by merging adjacent age bands, and keeps rows at the price of precision. Most projects combine the two and rerun the check until the smallest cell clears the threshold. Many teams use a floor of 5 for internal sharing and a larger value for public release, because the right k is a risk decision tied to sensitivity and release model, not a universal constant.

Step 6: Clean free text, images, and file metadata

Notes, images, and file properties are where de-identification quietly fails after the columns already look clean.

Free-text notes. Run a clinical named-entity recognition tool that flags names, dates, locations, and phone numbers, then read a random sample of at least 5 percent of the notes yourself. Tools such as Microsoft Presidio and MITRE Philter are common starting points. No automated pass catches everything, especially nicknames, addresses typed without spaces, and institution names inside clinical phrases.

Medical images. DICOM, the standard file format for medical imaging, stores patient name, ID, birth date, and institution in header fields, and some scans also carry burned-in text on the pixels themselves. Scrub the headers with a standard anonymization profile and inspect a sample of images by eye.

Documents and spreadsheets. Office files store author and organization names, and PDFs store creator metadata. Strip these before sharing. The command exiftool -all= report.pdf removes embedded metadata fields from a PDF.

Finish with a pattern scan over the whole output folder, searching for digit strings shaped like phone or Social Security numbers, month names, and your own institution's name.

Step 7: Measure residual re-identification risk

De-identification lowers risk, it never reaches zero, so quantify what remains before anyone else sees the file. Three checks cover most cases. First, recompute the uniqueness fraction on your final quasi-identifiers and record it. Second, run a motivated intruder test. Give a colleague who knows the cohort, or who holds public records such as obituaries, the shared file plus a real patient's nonclinical facts, and see whether they can find the record. Third, write the residual risk into the README in plain language, including which variables you kept and why.

For high-value or legally delicate datasets, expert determination is the formal alternative. A qualified statistician measures re-identification risk against the capabilities of a realistic adversary and certifies that the risk is very small. That certification, not a checklist, is what regulators accept on the expert determination route.

Step 8: Document, package, and deposit

Three files travel with the dataset. A data dictionary listing every column, its units, and its allowed values. The de-identification script itself, so reviewers can audit exactly what was done. A README stating the residual risk, the transformation choices, and the intended uses.

Write the data availability statement in your manuscript at the same time as the deposit. The Edinburgh review found that papers carrying an explicit data availability statement shared more complete and more reusable data, so draft the statement early rather than treating it as paperwork [1].

A Data Repository Comparison for Clinical Datasets

Match the access model to the residual risk, not the other way around. Across biomedical research, open data works as a spectrum that runs from public aggregate datasets through controlled-access repositories and federated analysis to synthetic data, and durable practice depends on calibrated access rather than maximal openness [3].

Genomic data deserves a specific note. The legal prerequisites for its secondary use are intricate, because European legislation is read differently across member states and national rules add further conditions, so controlled access with a clearly defined purpose and documented legal basis is the standard route [4]. Repository behavior reflects this. The Edinburgh review observed that genomic sequences were shared more often than image data or data on human subjects or samples, a pattern that tracks both funder expectations and the maturity of sequence repositories [1]. Repositories also assign metadata and persistent identifiers that make a dataset FAIR, meaning findable, accessible, interoperable, and reusable, which is the standard funders increasingly apply [1].

Synthetic Data and Privacy-Preserving Computation

Synthetic data. A synthetic dataset is generated by a model trained on the real data, so every record is artificial while the distributions look plausible. It suits teaching, software testing, and early analysis, and it can support research where patient-level release is impossible [5]. The catch is quality assurance. A critical appraisal of synthetic healthcare data found that fidelity and utility scores computed on the whole dataset often miss weaknesses that appear only in specific patient subgroups, illustrated with a lung cancer case study in which aggregate metrics looked acceptable while subgroup behavior did not [5]. Audit synthetic data at the subgroup level, not just in aggregate, before anyone builds on it.

Privacy-preserving computation. When data cannot move at all, computation can move instead. Differential privacy adds mathematically calibrated noise so aggregate results stay accurate while individual contributions are masked, and the parameter epsilon sets the tradeoff between protection and accuracy. One published framework combined homomorphic encryption, secure multi-party computation, and differential privacy for networked devices, injecting calibrated noise at a moderate privacy budget of epsilon = 1.0 and completing aggregation with a relative error of about 2.8 percent [6]. Homomorphic encryption lets a server compute on encrypted values without decrypting them. Secure multi-party computation lets several parties jointly compute a result without revealing their individual inputs. Federated analysis extends the same principle to multi-site research, keeping the data inside each source institution [3]. These architectures cost more engineering effort than a CSV export, and they are the right tool when the data simply cannot leave the building.

Common Mistakes and Limitations

1. Calling pseudonymized data anonymous. A coded dataset with a retained key is still personal data under European law, whatever the file name says [4]. Label the dataset honestly and protect it accordingly.
2. Hashing identifiers without a secret key. An unsalted hash of a medical record number falls to anyone with a list of candidate numbers. Use HMAC with a key stored separately, or drop the identifier.
3. Leaving full dates in place. An admission date, a three-digit ZIP, and sex can isolate a single patient. Shift dates or reduce them to year.
4. Forgetting where identifiers hide. Free-text notes, DICOM headers, burned-in image annotations, spreadsheet author fields, and PDF metadata all carry identifiers after the columns look clean.
5. Ignoring rare values. A rare diagnosis, an extreme age, or a small-town ZIP3 makes one record unique even after textbook transformations. The k-anonymity check exists to catch exactly this.
6. Publishing small cells. A count of 1 in a cross-tab is a person. Suppress or merge cells below your threshold in every table, not only in the main file.
7. Shipping the key with the data. The salt, key file, or codebook linking study IDs to identities belongs on encrypted, access-controlled storage, never in the shared folder or the repository upload.
8. Over-suppressing. Aggressive generalization strips out the signal the science needs. Record what you traded away in the residual risk note, so users can judge the result.
9. Assuming one legal answer covers every site. Member states read European privacy rules differently and add national conditions, so a pipeline that is compliant in one country may need review in another [4].
10. Trusting aggregate metrics for synthetic data. A dataset can look faithful overall and still fail inside clinically meaningful subgroups [5].
11. Assuming open by default. Experience across biomedical data sharing supports calibrated access, clear accountability, and sustained public engagement rather than unrestricted release [3].

De-identification lowers re-identification risk, it does not eliminate it. Determined adversaries holding auxiliary datasets can sometimes re-identify records that pass every standard check, which is why controlled access and data use agreements exist. For decisions about a specific dataset, involve your IRB, data protection officer, or a privacy professional early.

Frequently Asked Questions

What is the difference between de-identification, anonymization, and pseudonymization?

De-identification is the umbrella term for removing or transforming identifiers. Pseudonymization swaps identifiers for codes and keeps a separate key, so the data still counts as personal data under European law. Anonymization is irreversible and takes data outside privacy rules, but it is hard to achieve for rich patient-level records.

Is de-identified patient data still protected by law?

Often yes. Pseudonymized health data remains personal data under the GDPR and stays fully regulated [4]. Data that meets the HIPAA Safe Harbor or expert determination standard is no longer protected health information, though institutional policy and any confidentiality promises still apply.

What does the HIPAA Safe Harbor method require me to remove?

It requires removal of 18 identifier categories, including names, addresses smaller than a state, phone numbers, email addresses, Social Security numbers, medical record numbers, account and license numbers, device and vehicle identifiers, web addresses, IP addresses, biometric identifiers, and full-face photographs. Dates must be reduced to year, ages over 89 must be collapsed into one category, and ZIP codes may keep three digits only where the area holds more than 20,000 people.

Can I just hash medical record numbers instead of deleting them?

No. A plain unsalted hash can be reversed by hashing guesses. A keyed HMAC with a secret salt is the minimum for pseudonymization, and even then the data remains personal data under European law. Under Safe Harbor, a re-identification code must not be derived from the patient's own information, so a hash of the record number does not qualify.

What is k-anonymity and how large should k be?

k-anonymity means every combination of quasi-identifier values, such as age band, ZIP3, and sex, appears at least k times, so no record is unique. Many teams use a floor of 5 for internal sharing and larger values for public release. The right threshold depends on sensitivity and release model.

Is synthetic data a safe substitute for real patient data?

It removes direct exposure to patient records, but it is not automatically accurate or private. Quality scores computed on the whole dataset can hide serious errors inside patient subgroups, so audit synthetic data at the subgroup level before trusting it.

Where should I deposit de-identified clinical data?

Match the repository to the residual risk. Anonymized aggregate data fits open repositories such as Zenodo or Dryad. Patient-level clinical or genomic data usually belongs in controlled-access repositories such as dbGaP or the European Genome-phenome Archive under a data use agreement. Data that cannot move at all may suit federated analysis [3].

Do I still need IRB approval or patient consent to share de-identified data?

Yes, you still need to check. Your ethics approval and consent forms define what secondary use is allowed, and European law requires a defined purpose and legal basis for secondary processing [4]. Confirm the scope with your IRB or data governance office before deposit.

Sources

1. The rise of open data practices among bioscientists at the University of Edinburgh.
2. Strengthening Data Systems and Use for Infectious Outbreak Preparedness in Uganda.
3. Ethical Governance of Open Data Across Biomedical Research, Healthcare, and Public Health: Privacy, Equity, Trust, and Controlled Access.
4. Basic legal prerequisites for secondary use of genomic data for scientific purposes.
5. Quality assessment of synthetic data in healthcare: a critical appraisal with a focus on subgroup bias.
6. An integrated privacy preserving data aggregation framework for IoT networks using homomorphic encryption and secure computation.