I analyzed 4 billion Reddit messages on a Mac Mini by rewriting my Python pipeline in Rust

Hi everyone,

I recently started a project to analyze the entirety of the 2025 Reddit archive (about 4 billion messages) to find SaaS ideas.

I started with a standard Python stack (Pandas/Postgres), but I hit a wall quickly. I was running this on my home hardware (Mac Mini M1 + older MacBooks), and Python's memory overhead + GC pauses were killing the process. I was getting OOM kills constantly, and throughput was stuck at ~20 messages/sec.

I decided to port the ingestion and classification pipeline to Rust. The difference was night and day.

The Tech Stack:

• Ingestion: Custom Rust ingestor reading .zst streams.
• Queue: Redis (via redis-rs) to decouple producers/consumers.
• Classification: Ported from PyTorch to ONNX Runtime (ORT) in Rust.
• Storage: Polars + Parquet (instead of Postgres rows).

The Results:

• Memory usage dropped from >1GB (swelling) to stable <500MB per worker.
• Throughput went from 20 msg/sec to ~300+ msg/sec on the same hardware.
• The ONNX implementation in Rust was significantly faster and lighter than the Python equivalent.

I wrote a detailed blog post about the architecture, the memory struggles, and the specific Rust implementations I used.

Read the full write-up here: https://teo-miscia.medium.com/i-built-a-saas-idea-generator-by-analyzing-the-entirety-of-reddit-2025-9de42bcddb27

If you're a data engineer struggling with Python at scale, I highly recommend looking into Rust + ONNX. It turned a project that would have taken months into weeks.

Happy to answer questions about the crates I used!