logo

Konnu - peer to peer search engine

159 points shadowislord
View on Reddit ↗

Here's new peer to peer search engine I've been working on for the past two years. It uses some of the new tech to come out of machine learning and natural language processing, which means there's a bit more capabilities here than the original p2p engines we had back in the 2000's.

It's built around two main ideas:

  • generating dense vectors for text
  • nearest neighbor search in a peer to peer network

Most of the research on peer to peer search engines was done before we had BERT language models or even word embeddings. Those old engines relied on indexing terms which is very difficult due to how quickly they scale with larger document collections.

By using vectors and clustering, it is possible to reduce the # of writes required by a lot, which means it's possible to index larger document collections more efficiently.

I put the code up on GitLab here:

https://gitlab.com/shadowislord/konnu

The architecture is described here (warning: long):

https://gitlab.com/shadowislord/konnu/-/blob/main/ARCHITECTURE.md

The demo program will run a simulated peer to peer network with 1024 nodes. The corpus being indexed is the "20 newsgroups" data set from here: http://qwone.com/\~jason/20Newsgroups/

After the network has been warmed up you can perform search requests.

Example:

    > end of the space age

    14060909 Re: Why not give $1 billion to first year-long moon residents?
    With the continuin talk about the "End of the Space Age" and complaints
    by government over the large cost, why not try something I read about
    that might just work.

    14060842 End of the Space Age
    End of the Space Age
    There is an interesting opinion piece in the business section of today's LA Times (Thursday April 15, 1993, p. D1).