logo

Announcing the fastest rm and cp commands for linux

121 points SUPERCILEX
View on Reddit ↗

Project: https://github.com/SUPERCILEX/fuc

rmz is the first of its kind and there's nothing faster as far as I'm aware. cpz beats everybody else when operating on file trees (i.e. directories with nesting) and remains faster than cp when running through a single directory. Both tools are extremely efficient, doing the minimal work possible. Of course, there's always room for improvement, so in a distant future (there are other exciting projects to move on to!) I'd like to rewrite the threadpool crate to support dynamic thread allocation and investigate a faster MPMC queue.

Benchmarks are available in the comparisons/ folder (and include prior art such as fcp and xcp).

rmz is the most interesting to me because of the constraints on how it can be parallelized: all files (and sub-directories) in a directory must be deleted prior to deleting the directory itself. This leads to an interesting problem where you need to remember which directories to delete after a recursive deletion task has completed. I went through a number of failed approaches before realizing a parent pointer tree backed by Arcs perfectly encodes those constraints. The directory node drop implementation will delete its directory after all children have been dropped (deleted) and the compiler handles the recursion for you. I plan on writing more about this in a few weeks on my blog: https://alexsaveau.dev/blog.

cpz uses the same underlying parallelism algorithm as rmz (perform ops for a specific directory on a single thread, but parallelize across directories), hence why it suffers when there is no directory nesting. At some point, I'd like to look into running all copies in parallel, but I don't want to take the efficiency hit of queuing one task per file. Instead, I'd like to use io_uring if it had support for copy acceleration (which cpz takes advantage of).

Some interesting work that went into putting these tools together: