On File::read_to_end's performance

77 points ducktherapy

Hey, folks!

I came across an unexpected performance degradation on Windows when using File::read_to_end(): the larger the Vec buffer's capacity, the longer the call takes (regardless of actually read bytes). With help from folks at Stackoverflow, we figured out this is due to NtReadFile's behavior.

Now, I've never contributed to Rust. Is this something worth opening an issue for? At least for documentation purposes? Or is this a commonly expected behavior that I failed to identify?

In any case, small boiled-down PSA that might surprise others, like it did me:

  • At least on Windows, do not use File::read_to_end with extraneous capacity as-is. Either shrink the buffer or use read_exact/take with an appropriate length, otherwise the runtime will scale linearly with the unused buffer capacity.

I'd appreciate any pointers. Thanks for reading!