I am using Sea-ORM as an ETL tool and its flawless

Long story short, the Data Engineering team at my company is bad... like really, really bad. They have written themselves a bunch of spaghetti Python code. Due to the chaos of their python tool, they created a CLI python tool to help with their over complicate Data Engineering Python library. On-boarding a new table can take weeks and on-boarding an entire new data source can take months.

This is extremely frustrating when you are working in a fast paced environment and has caused an "underground" analytics world where a bunch of us across different teams and orgs have deployed different ETL tools ourselves. Several of us have tried to help our Data Engineering team, but they are "holier than thou" and refuse to believe anyone could do better than their crap tooling.

I have been learning Rust and had a "hold my beer" moment and challenged their ETL to a competition. I know ORMs are usually for websites, but it has all of the elements to be a good ETL tool, too (and also part of the "hold my beer" challenge was that I could get something up and running in a day). After realizing I would need to implement a trait for every model using SQLx and no wanting to deal with setting up Aysnc on Diesel, I landed on Sea-ORM. Within 2 hours I had 10 tables dumping to an S3 bucket with better logging than the current Data Engineering teams "product". Also, my pipelines run about 60x faster because Rust is fast, and I am utilizing Async and Multithreading, where as their ETL is single threaded Python 3.6 (they spent months trying to upgrade to 3.10 but had to rollback after they found it wasn't possible).

So I essentially use Sea-ORM to read from MySQL/Posgtres then Polars to write to our S3 bucket and then I have lambda and glue take it from there.

So far no errors or failures, it just works. Same can't be said about or Data Engineering team.