• A better script for removing duplicate lines
    By : erdizz ( Wed Oct 10 11:57:34 2007 )

    I suggest this:
    sed 'x; G; /^\(.*\)\n\1$/ d; g; p; d'

    Instead of this:
    sed '$!N; /^\(.*\)\n\1$/!P; D'

    The new variant does not read the whole file into memory, which makes it capable of handling very large files.

    Return to article...

    Suggest a Site