python to rust migration

Hey folks,

I currently have a log parsing pipeline project written in python and I'm considering migrating it to rust for true multi treading and speed.

The core of my current implementation includes:

1. Arguments parsing from the command line, each option having an acceptable list of options.

2. An abstract log class that all log types can inherit from and dynamically call their respective abstract method implementation.

3. For every worker the user has indicated in the command line that they want to use to consume a RabbitMQ queue, a dedicated connection and channel to rabbitmq is started (this is because the main py lib for rabbitmq interaction, pika, is not threadsafe).

4. Use RegEx for parsing the data.

5. Validate data using pydantic (I believe the core of the V2 of this is written in rust)

6. Send off clean data to postgres.

7. Use supervisor to run one or multiple instances of the script, and restart a thread if it is unexpectedly terminated.

Right now, I have a producer that will be sending data at the rate of 20,000/sec and my py implementation can barely keep up. At best, I'm getting half of that, with around 50 workers, but I'm skeptical adding more workers will substantially increase throughput.

As someone who has never worked with rust, but really wants to try this, I'm not sure what to expect or how big a learning curve this will be for me. I'm looking at no later than the end of May to make this switch work or just continue with py.

Any suggestion will be appreciated.

P.S. Please let me know if you need other clarifying information.