ngrep: a grep-like tool that extends regexp with word embeddings

Hi everyone!

I got curious about a simple question: regular expressions are purely syntactic, but what happens if you extend them with just a little bit of semantics?

To answer, I ended up building ngrep: a grep-like tool that extends regular expressions with a new operator ~(token) that matches a word by meaning using word2vec style embeddings (FastText, GloVe, Wikipedia2Vec).

A simple demo: ~(big)+ \b~(animal;0.35)+\b ran over the Moby-Dick book text can find different ways used to refer to a large animal. It matches vectors based on cosine similarity, using 0.35 as the similarity threshold for "animal" - surfacing "great whale", "enormous creature", "huge elephant", and so on:

ngrep -o '~(big)+ \b~(animal;0.35)+\b' moby-dick.txt | sort | uniq -c | sort -rn
   7 great whale
   5 great whales
   3 large whale
   3 great monster
   2 great fish
   1 tremendous whale
   1 small fish
   1 small cub
   1 little cannibal
   1 large herd
   1 huge reptile
   1 huge elephant
   1 great hunting
   1 great dromedary
   1 gigantic fish
   1 gigantic creature
   1 enormous creatures
   1 enormous creature
   1 big whale

It is built in Rust on top of the awesome fancy-regex, and ~() composes with all standard operators (negative lookahead, quantifiers, etc.). Currently it is a PoC with many missing optimizations (e.g: no caching, no compilation to standard regex, etc.), obviously without the guarantees of plain regex and subject to the limits of w2v-style embeddings...but thought it was worth sharing!

Repo: https://github.com/0xNaN/ngrep

--
note: I realized after naming it that there is a famous network packet analyzer also called ngrep...this is a completely different tool :)