Lists pick up duplicates all the time. Merging two keyword exports, combining mailing lists, copying product codes from several orders or pasting log lines from different servers quickly leaves the same entry two, three or ten times. Spreadsheets can remove duplicates, but only after you import the data into columns, and many online tools sort the list alphabetically as a side effect.
Paste your list here, one item per line, and the output shows each line only once, in the order it first appeared. Options let you treat "Apple" and "apple" as the same, ignore stray spaces, and drop blank lines. Three counters tell you how many lines went in, how many came out, and how many duplicates were removed. The work is done in your browser, so it is safe for client lists and internal data.
What "duplicate" means here
By default two lines are duplicates only when they match exactly, character for character. "Apple", "apple" and "Apple " with a trailing space are three different lines. That strict rule is safe for code, passwords and identifiers where capitals and spacing matter.
For human lists, exact matching is often too strict. Ignore case compares lines without regard to capital letters, and Trim spaces removes spaces and tabs at the start and end of each line before comparing, so copied cells with invisible padding still match. Trim spaces also cleans the lines in the output, while Ignore case only affects the comparison: the kept line keeps its own spelling.
Typical de-duplication jobs
Keyword and hashtag research
Combine keyword lists from several tools or hashtag sets from different posts, then remove the repeats with Ignore case on so variations in capitals collapse into one.
Mailing and contact lists
When two exports overlap, paste both lists one after the other. Each address or name appears once, in the order of the first list followed by the new entries from the second.
Logs and error messages
Server logs repeat the same warning hundreds of times. Removing duplicate lines leaves one copy of each distinct message, which makes it easier to see how many different problems there really are.
Cleaning data before import
Product SKUs, IDs, file names and URLs often need to be unique before they go into a database or a spreadsheet. Turn on Trim spaces and Remove empty lines to catch the near-duplicates copy and paste creates.
Keep first or last, and other options
The Keep switch decides which copy of a duplicate survives. First keeps the earliest occurrence and is the usual choice. Last keeps the final occurrence and puts it where that last copy stood, which is useful for logs or exports where a later line is the most up to date.
Remove empty lines drops blank lines entirely. With it off, blank lines are treated like any other line, so a list with many empty lines keeps only the first one, in its original position. The input can use Windows, Mac or Unix line breaks; the output always uses plain line breaks, which every editor and spreadsheet reads.
Lines in counts every line you pasted, Lines out counts the result, and Duplicates removed is the difference, including any empty lines that were dropped. Copy or Download, which saves unique-lines.txt, gives you the cleaned list.
Как пользоваться инструментом
- Paste your list into the input box, with one item on each line.
- Choose Keep First or Last.
- Switch on Ignore case, Trim spaces or Remove empty lines as needed.
- Check Lines in, Lines out and Duplicates removed below the panes.
- Copy the unique lines or download them as a text file.
Limitations
- It compares whole lines. Two lines that differ by one word, or a list where several items share a line, are not merged.
- The output is not sorted. If you also need alphabetical order, sort the result in a spreadsheet or editor afterwards.
- Near-duplicates such as "colour" and "color", or a URL with and without a trailing slash, are treated as different lines.
- Ignore case uses simple Unicode lower-casing, so language-specific rules such as Turkish dotted and dotless i are not applied.