What is the traditional wisdom in processing larger-than-memory datasets?

Hi! I'm using Rust to write my bachelor thesis, which involves Differential SAR Interferometry. In this field, it's often the case that you have a time series of 30+ SLC Images (which are composed of complex numbers), each about 100Mbs to Gbs (in the case of Sentinel-1 bursts, they can be 100s of Mbs, for example, but for a full product in SAFE format it can be way more, up to 8 Gb), which requires some sort of out-of-core algorithms. I'm aware of open-source software such as SNAP, GMTSAR, etc but I'd like to do a few things myself in Rust just for the learning experience (and the fun of RiiR, of course!).

Let me give an example: If you want to compute relevant statistics on each image pixel, e.g. the amplitude dispersion index, which is defiined as the ratio of the standard deviation of the amplitude over the mean amplitude value, that involves taking many pixel samples from many different Images, so the entire dataset would definitely not fit into RAM of a consumer laptop/desktop. Some solutions to this would be:

• Chunked processing
• Memory mapping
• Distributed computing

I'm aware of python libraries like Dask for parallel/distributed computing, but I would like to know more about this in the Rust ecosystem, including best practices, GitHub repos, crates, etc. Any advice is welcome :)