Forensicsdeduplicationfile-processingline-removalhashingduplicates

rling

rling is a faster alternative to the rli utility from hashcat-utils. It removes matching lines or duplicates from files by comparing a single input file against one or more remove files.

Description

rling efficiently processes large files to eliminate duplicates or matching lines, making it ideal for cleaning result files in penetration testing and forensics workflows. Similar to rli but significantly faster, it supports various optimizations for memory usage, threading, and sorting to handle massive datasets encountered in password cracking or log analysis.

Use cases include deduplicating wordlists, removing known lines from capture outputs, or extracting unique entries from hash or password dumps. It integrates well with tools like getpass for password extraction and splitlen for length-based splitting, forming part of a toolkit for result processing.

The tool supports stdin/stdout for piping in Unix pipelines, enabling seamless integration into scripts for automated data cleanup.

How It Works

rling uses hash tables by default for rapid line matching and duplicate removal, with options for binary search (-b), file-based storage (-f), or low-memory rli2 mode (-2) requiring sorted inputs. It processes an input file against remove files (remfil1, remfile2, etc.), optionally writing duplicates to a file (-D), sorting output (-s), or limiting matches by length (-l). Threading (-t), hash table sizing (-p), and temp file directories (-T) optimize performance for large-scale operations.

Installation

bash
sudo apt install rling

Flags

-iIgnore any error/missing files on remove list
-dRemoves duplicate lines from input (on by default)
-D fileWrite duplicates to file
-nDo not remove duplicate lines from input
-cOutput lines common to input and remove files
-sSort output. Default is input order. This will make the -b and -f options substantially faster
-t numberNumber of threads to use
-p primeForce size of hash table
-bUse binary search vs hash (slower, but less memory)
-2Use rli2 mode - all files must be sorted. Low mem usage.
-fUse files instead of memory (slower, but small memory)
-l [len]Limit all matching to a specific length.
-M memsizeMaximum memory to use for -f mode
-T pathDirectory to store temp files in

Examples

Basic usage: remove matching lines from input file, writing unique lines to output, using specified remove files
rling input output [remfil1 remfile2 ...]
Shows version and help (note: --help is unrecognized but displays usage)
rling --help
Remove duplicate lines from input (default behavior)
rling input output -d
Remove matches and write duplicates to specified file
rling input output -D duplicates.txt
Output only lines common to input and remove files
rling input output -c remfile1
Sort output and use 4 threads for faster processing
rling input output -s -t 4
Use binary search instead of hash table for lower memory usage
rling input output -b
Use low-memory rli2 mode (requires sorted files)
rling input output -2
Updated 2026-04-16kali.org ↗