Free CSV tool
How to remove duplicate rows from a CSV
Decide what “the same row” means for your file — every column, every column except a timestamp, or just the email address — then keep the most complete copy of each. Columnwise removes the rest and tells you exactly how many went, without the file ever leaving your browser.
Runs entirely in your browser — your file stays on your machine.
How it works
- Load your file. Drag in a CSV, TSV or XLSX file. A Remove duplicates step is added for you and already removes exact duplicate rows.
- Choose what counts as a duplicate. Three options:
- All columns — rows must be identical everywhere.
- All columns except… — tick the columns to ignore, like an import timestamp or a row number that differs on every copy.
- Only these columns… — tick the columns that identify a record, like
emailorpatient_id.
- Choose which copy to keep. The first row, the most complete row (fewest empty cells in the columns you didn't compare on) or the least complete row. Ties go to the row that appears first.
- Check and export. You see how many rows were removed and from how many duplicate groups. The removed rows aren't thrown away: they sit in the step's quarantine and can be downloaded as a separate CSV.
Worked example
Merging a contact list that was exported twice
contacts.csv was built by pasting together two exports. The same people show up more than once, and only one of the copies has their phone number filled in:
| name | phone | city | |
|---|---|---|---|
| ada@example.com | Ada | ||
| grace@example.com | Grace | 555-0102 | York |
| ada@example.com | Ada | 555-0101 | Leeds |
| grace@example.com | Grace | York |
Comparing on all columns finds nothing, because no two rows are identical. Comparing on only email and keeping the most complete row gives you one row per person, each with every detail you had:
| name | phone | city | |
|---|---|---|---|
| grace@example.com | Grace | 555-0102 | York |
| ada@example.com | Ada | 555-0101 | Leeds |
The step reports 2 duplicate rows removed from 2 groups, and the two sparse rows are kept in quarantine in case you want to look at them.
Exactly what counts as a match
Case and stray spaces are ignored by default. John@X.com and john@x.com count as the same email. The row you keep still has its original text; the comparison is the only thing that's relaxed. Untick the option if case matters in your data, for example in case-sensitive product codes.
Blank keys are never merged. When you compare on only a few columns, those columns act as the record's ID. A row with an empty email, or with a name but no date of birth, isn't a duplicate of another row with the same gap, because you can't tell they're the same record. Those rows are kept as they are, and the step tells you how many there were.
Keys can span several columns. Tick name and date_of_birth together, as is common in research data, and two rows only match when both agree.
Nothing disappears unaccounted for. The step shows rows in, rows out and rows removed. The removed rows can be downloaded as their own CSV from the export panel. Saved recipes re-run with the same settings and give the same result, whether the file is a CSV or an Excel workbook.
Just want to know whether a file has duplicate IDs, without changing it? Use the duplicate-key check instead.
Your file never leaves your computer
The file is read with the browser's File API and deduplicated in memory, in your tab. There's no upload step. Open your browser's Network tab while you do this and you won't see a single request carrying the file's contents. The only network activity you might see is a Google Analytics page-view ping, if you've accepted analytics, and it never includes your data. The whole free toolkit works this way; this page isn't a special case.
Related tools
Clean up your file now
No signup, no upload. The workspace opens with a remove-duplicates step ready.
Open the workspace