On using unsafe and startup times

I have a binary that takes about 4-5 seconds on a typical VPS. The startup time involves loading/decoding a 100mb proprietary data file. I believe it's probably as optimized as it can be in rust without unsafe mode (although maybe there are things I am missing).

This leads me to wonder about the general attitudes of people regarding when and if to use unsafe mode. I know that if I forgo utf validation when loading strings, and do unsafe tricks with strings to avoid memcpy (create Strings backed by the data array rather than via copying from the data array) the load time will be significantly decreased.

Thus the question, I wonder if the general sentiment of people is that people are generally always against using unsafe no matter what. Compared to, for example, what portion of people would be for using unsafe if it has real world tangible observable practical benefits for the system you are building.

My very first servers were coded in c many many years ago now, and I am not scared of mucking around with doing unsafe type things, but it does seem to me that there is a general attitude in the rust community that using unsafe even where it is beneficial, is not a good idea. Am I reading the room correctly? or am I wrong?

UPDATE: Thanks to everyone who shared, I’ve learnt some things already. Let’s all assume at a minimum no one should use unsafe unless there is a demonstrable impact on performance that has a flow on real world impact on business operations. (let’s stipulate that premature optimization is silly, and let’s assume using unsafe for minor performance improvements is silly)