Duplicate rows get into CSV exports in predictable ways: a form submitted twice, two exports pasted together, a sync that ran twice, a copy-paste that overlapped by a few lines. The result is inflated counts, customers who get the same email twice and totals that are quietly wrong.
Deleting duplicates is easy. Deleting exactly the right ones takes a little more thought. This guide walks through what counts as a duplicate, why some of them are invisible, and a safe order of steps that works whether you use a spreadsheet or a dedicated tool.
What counts as a duplicate
An exact duplicate is a row where every cell matches another row character for character. In this file, the third data row repeats the first:
email,name,plan
ana@example.com,Ana Ruiz,Pro
ben@example.com,Ben Ode,Free
ana@example.com,Ana Ruiz,ProA near duplicate describes the same real-world record but differs in some detail:
email,name,plan
ana@example.com,Ana Ruiz,Pro
Ana@Example.com ,Ana Ruiz,Pro
ana@example.com,Ana Ruiz,FreeThe second row differs only in capitalization and a trailing space, so most people would call it a duplicate. The third row is different: same person, different plan. That may be an upgrade, a downgrade or a data-entry mistake. It is a conflict, and deleting it blindly means picking a winner without looking.
Before you remove anything, decide which of these cases you are dealing with.
Why duplicates hide from you
Two rows can look identical on screen and still be different to a computer. The usual culprits:
- Leading and trailing spaces.
ana@example.comandana@example.comlook the same in a cell, but an exact comparison treats them as different values. - Non-breaking spaces. Text copied from web pages, emails or PDFs often carries non-breaking spaces instead of ordinary ones. In Excel, the
TRIMfunction only removes the ordinary space character, so these survive a quick cleanup. - Capitalization. Most mail systems treat
Ana@Example.comandana@example.comas the same address, but a strict comparison does not. Tools also differ on whether their duplicate check is case-sensitive, so check rather than assume. - Formatting differences.
+1 555 0100and15550100may be the same phone number, but no duplicate remover will know that. Normalizing formats is a separate step.
This is why trimming whitespace comes before removing duplicates. Our guide to finding hidden whitespace in CSV files covers the invisible characters in more detail.
Choose the columns that define a duplicate
Comparing every column finds exact duplicates only. Comparing one or more key columns finds rows that represent the same thing even when other details differ. The right key depends on what one row is supposed to mean.
| Dataset | Typical key | Watch out for |
|---|---|---|
| Newsletter signups | email | Capitalization and trailing spaces |
| Order lines | order_id + sku, or a line ID | One order legitimately spans several rows |
| Card or bank transactions | A transaction ID, if the export has one | Two genuine purchases of the same amount on the same day |
| Survey responses | Response ID | Whether a second submission from the same person should count |
Deduplicating an order export by order_id alone will keep only the first line item of every multi-item order. If a key is not unique by design, add columns until it is.
Which copy to keep
Most tools, including Excel, keep the first occurrence of each duplicate and remove the rest, in file order. That makes row order part of the decision. If your file is sorted oldest first, keeping the first row keeps the oldest record. If you want the newest, sort newest first before you deduplicate.
Keeping one row also means discarding whatever was only in the others. Consider:
email,phone,signup_date
ana@example.com,,2026-03-01
ana@example.com,555-0100,2026-05-12Deduplicating on email and keeping the first row throws away the only phone number. When rows carry different information, merge them by hand (or sort so the most complete row comes first) rather than letting a tool choose.
Step by step: removing duplicates safely
- Keep the original file. Work on a copy so you can always compare or start over.
- Trim whitespace first. Remove leading and trailing spaces, including non-breaking ones, so invisible differences stop hiding duplicates.
- Decide the key. All columns for exact duplicates, or the specific columns that identify one record. Decide whether capitalization matters.
- Look before you delete. Sort by the key column and scan the repeats, or count them with a formula such as
=COUNTIF(A:A,A2)>1in a helper column. Note how many rows you expect to lose. - Sort so the row you want to keep comes first. For example, newest first, or rows with a phone number first.
- Remove the duplicates.
- Check the result. Compare the row count before and after with what you expected, and spot-check a few of the keys that had repeats.
Excel, Google Sheets or a dedicated tool
Excel
Select the data and use Data → Remove Duplicates. You can tick the columns to compare, and Excel reports how many duplicates it removed. The catch is getting the CSV into Excel in the first place: opening a CSV directly can strip leading zeros, turn long IDs into scientific notation and convert some text to dates, which changes what counts as a duplicate. See how to open a CSV in Excel without breaking your data.
Google Sheets
Use Data → Data cleanup → Remove duplicates, choose the columns and say whether the data has a header row. The same menu has a Trim whitespace option, which is worth running first. If you would rather not change the data in place, the UNIQUE function returns the distinct rows of a range into a new area of the sheet. Excel for Microsoft 365 has a UNIQUE function too.
A dedicated tool
A tool that reads the CSV as plain text avoids spreadsheet type conversion entirely, so IDs and ZIP codes stay exactly as they were. It also makes options like ignoring case or surrounding spaces explicit, instead of depending on how a particular app behaves.
Remove duplicates from your file
The free Remove Duplicate Rows tool runs entirely in your browser, so the file is never uploaded. It keeps the first copy of each row and lets you compare only chosen columns, ignore case and ignore leading and trailing spaces. Run the Trim Whitespace tool first if your data came from copy-paste or a web form.
If you clean the same kinds of files regularly, the CleanYourDataset workspace saves datasets privately to your account and removes exact duplicate rows (keeping the first) after showing you how many rows will be removed. You can undo the last change or reset to the original upload. The guide explains the recommended order: trim, remove empty rows, remove duplicates, then fill blanks. Create a free account to try it.