Line Tools: Dedupe, Sort and Shuffle, Explained

After reading this, you will know exactly what happens when you remove duplicate lines, sort text alphabetically or numerically, reverse, shuffle and number it, and you will understand the ordering rules well enough to predict the output before you press a button.

What these operations do

Line tools treat your text as a list of lines split on newline characters. Every operation works on that list: it keeps some lines, reorders them, or rewrites each one. Nothing looks inside a line except when trimming whitespace or comparing lines during dedupe and sort.

Here is the hook. Paste this into the tool:

banana, apple, banana, Apple, cherry (five lines). Turn on deduplication and sort ascending. You get four lines: Apple, apple, banana, cherry. The second banana vanished because it repeated the first. Apple sorts before apple because uppercase A is code point U+0041 (decimal 65) and lowercase a is U+0061 (decimal 97). A plain sort compares those numbers, so 65 comes first.

That single example already shows the two ideas that trip people up: what counts as a duplicate, and what "alphabetical" means to a computer. The rest of this article makes both precise.

When to use line tools, and when not

Reach for line tools when your data is genuinely line-based: a list of email addresses, log lines, exported URLs, one word per line, or rows you plan to paste into a spreadsheet column. These operations are fast, predictable and reversible in the sense that you can always re-run with different options.

Do not use them when the meaning of your text spans multiple lines. A CSV where one record wraps across two lines, a paragraph of prose, or code with indented blocks will all break if you sort or shuffle, because you will scramble units that were never meant to move independently. Sorting a poem destroys the poem.

How ordering actually works

There are two sort modes, and they answer different questions.

Lexicographic (plain) sort

Compare two lines character by character using Unicode code point values. The first position where they differ decides the order. If one line is a prefix of the other, the shorter one comes first.

Natural sort

Split each line into runs of digits and runs of non-digits. Compare non-digit runs as text, but compare digit runs as numbers. This is why item2 comes before item10.

The lexicographic rule can be written as a comparison. For two strings a and b, find the first index i where their code points differ:

Here \text{cp}(a_i) is the Unicode code point of the character at position i in string a, and i is the first position where the two strings differ. A negative result means a sorts first. If no differing position exists before one string ends, the shorter string sorts first.

Concrete check: compare item2 and item10 lexicographically. The first four characters item match. At position 5, both have 1? No: item2 has 2 (code point 50) and item10 has 1 (code point 49). Since 50 > 49, item2 sorts after item10. That is the "wrong" order most people complain about. Natural sort reads the trailing digits as the numbers 2 and 10, and 2 < 10, so item2 comes first.

What deduplication keeps

Deduplication walks the list from top to bottom and keeps the first time it sees each line. Every later copy is dropped. The kept line is always the earliest one, so dedupe never reorders survivors on its own.

Case sensitivity changes what counts as "the same line". With case-sensitive comparison, Apple and apple are two different lines and both survive. With case-insensitive comparison, they are treated as equal, so only the first (Apple, if it appeared first) is kept. The comparison is case-insensitive, but the output preserves the original casing of whichever line was kept.

Order of operations matters here. If you sort first and dedupe second, duplicates become adjacent and the kept copy is whichever the sort put first. If you dedupe first and sort second, the "first occurrence" rule uses your original paste order. When you stack operations in one pass, the tool applies them in a fixed sequence: trim and drop empty lines, then dedupe, then sort or shuffle, then number. Knowing that sequence lets you predict the result.

Reading the line count

The most useful number these operations produce is how many lines survived. A large drop from dedupe tells you something real about your data: it was repetitive. The chart below shows how many unique lines remain as duplicates pile up in a fixed 1000-line file.

Common mistakes

Four traps account for most surprises.

• Expecting numeric order from a plain sort. Plain sort puts 10 before 2 because 1 (49) is less than 2 (50). Use natural sort for anything with numbers in it.
• Uppercase clustering. A lexicographic sort groups all capitalized lines before all lowercase ones, because every uppercase ASCII letter (65 to 90) is below every lowercase letter (97 to 122). If you want dictionary order regardless of case, normalize case first.
• Trailing whitespace hiding duplicates. apple and apple  (with a trailing space) are different lines until you trim. Two visually identical lines can both survive dedupe if one has invisible whitespace.
• Non-ASCII letters sorting oddly. café uses é at code point U+00E9 (decimal 233), far above plain letters, so it can sort after z in a code point sort. Strip accents first if you want café near cafe.

Related tools

Line tools clean structure, but they do not rewrite the content of a line. Pair them with these when you need more.

To fix case before a case-sensitive sort, use the Case Converter. To match and rewrite patterns inside lines, for example stripping a prefix before dedupe, use Find & Replace with Regex. To cut or pad lines to a fixed width after sorting, use Truncate, Pad & Wrap. To make café sort next to cafe, run the Accent & Diacritic Remover first. If you only need to flip line order rather than sort it, the Reverse & Upside-down Text tool handles that directly.

Frequently asked questions

Related tools

Case Converter

camelCase, snake_case, Title Case, UPPER, lower and more.

Accent & Diacritic Remover

Turn café, naïve and Škoda into cafe, naive, Skoda.

Find & Replace with Regex

Search and replace across text, plain or regex, with preview.

Reverse & Upside-down Text

Reverse letters, words or lines; flip text upside down.

Truncate, Pad & Wrap

Cut text to length, pad lines, hard-wrap paragraphs.

Fancy Text Generator

𝓗𝓪𝓷𝓭𝔀𝓻𝓲𝓽𝓲𝓷𝓰, sᴍᴀʟʟ ᴄᴀᴘs, zalgo and other Unicode styles.