Duplicate Line Remover
Paste a list and keep each line once, in the order it first appeared.
INPUT
CHARS: 0WORDS: 0LINES: 0
OUTPUT
CHARS: 0WORDS: 0LINES: 0
How the Duplicate Line Remover works
Lines are compared after optional trimming, so apple and apple with a trailing space count as the same entry. Case sensitivity is off by default because most lists are meant to be case insensitive.
By default the first copy of each line survives and the rest go. You can keep the last copy instead, which is useful for logs where the newest entry matters. The show only duplicated option flips the tool around and returns just the lines that appeared more than once, which is a quick way to audit a list.
Duplicate Line Remover examples
Deduplicate messy list items
Input
apple orange banana apple orange grape
Result
apple orange banana grape
When it is useful
- •Cleaning an email or keyword list pulled from several sources.
- •Finding which entries in a list were pasted twice.
- •Reducing log output to one line per distinct message.
- •Preparing a unique set of URLs or IDs before an import.
Frequently Asked Questions
Does it change the order of my lines?
No. Lines keep the order of their first appearance. If you also want a sorted list, run the result through Sort Words Alphabetically.
Why are two similar lines not treated as duplicates?
Usually because of invisible differences such as a trailing space, a tab or different capitalization. Turn on trimming and turn off case sensitivity, and check for non-breaking spaces with the Whitespace Remover.
Is there a size limit?
The tool runs in your browser, so the limit is your device memory. Lists of several hundred thousand lines work on a modern computer.
Guides that use this tool
Related tools
- Sort Words AlphabeticallyChoose lines, words or comma separated items and sort them in the order you need.
- Duplicate Word FinderPaste a draft to see which words you lean on too often and where you typed a word twice in a row.
- List ConverterReshape a list for SQL, code or a spreadsheet.
- Whitespace RemoverChoose which kinds of whitespace to clean up and see the tidy result.
- Text Diff CheckerPaste an original and a changed version to see exactly what differs.