superdiff - a way to find similar code blocks in projects (comments appreciated)
I had this idea 3 months back when I was fed up with the copy-pasted code in the codebase I'm working on. So I made a python script that uses Levenshtein distance and a threshold to find similar pieces of code. And it turns out that python is pretty slow, so I rewrote it in rust, which is what I've arrived with now.
It's very good at finding bugs that occur from misspelled variables that you can't catch because you are using PHP.
Are there any other ways to speed up the search process, other than using async and threads?